Chapter 10: Deploying Data Pipelines
In software engineering, you will usually have development, testing, and production environments. The testing environment may be called quality control or staging or some other name, but the idea is the same. You develop in an environment, then push it to another environment that will be a clone of the production environment and if everything goes well, then it is pushed into the production environment. The same methodology is used in data engineering. So far, you have built data pipelines and run them on a single machine. In this chapter, you will learn methods for building data pipelines that can be deployed to a production environment.
In this chapter, we're going to cover the following main topics:
- Finalizing your data pipelines for production
- Using the NiFi variable registry
- Deploying your data pipelines
