Chapter 9: Deploying and Monitoring Pipelines in Production
The journey of data is fast approaching its destination. In the previous chapter, we learned about data aggregation and how to build the gold layer of a lakehouse. Data in the gold layer is the representation of summary data, which keeps changing every time new data gets ingested. Therefore, it is a cyclical process. Now that we have built and tested all of the necessary pieces of the Electroniz lakehouse, we need to start thinking about deploying the pipeline in production. This is so that it can serve the needs of the end users in the most timely and durable fashion.
In this chapter, we will cover the following topics:
- The deployment strategy
- Developing the master pipeline
- Testing the master pipeline
- Scheduling the master pipeline
- Monitoring pipelines
