Chapter 9: Monitoring Data Pipelines
You now know how to build data pipelines that are production ready—they are idempotent, use atomic transactions, and are version controlled. They are ready to start running in production, but before you can deploy them, you need to be able to monitor them. There will be errors in the code, in the data, in the network, or in other areas outside your control. You will need to know when errors happen, and when things are running as they should.
In this chapter, we're going to cover the following main topics:
- Monitoring NiFi in the GUI
- Monitoring NiFi using processors
- Monitoring NiFi with Python and the REST API
