The deployment strategy
We have worked extremely hard to get to the point where we have three functional pipelines, as follows:
- The Electroniz ingestion pipeline:
electroniz_batch_ingestion_pipeline - The Electroniz curation pipeline:
electroniz_curation_pipeline - The Electroniz aggregation pipeline:
electroniz_aggregation_pipeline
Just as a recap, in the last few chapters, we followed multiple steps in order to create these pipelines. After their creation, we invoked each one manually to unit test their functionality. Finally, we validated the data that each one produced to make sure it matched the expectation of the Electroniz use cases. That's a lot of work, so we should be proud to have reached this far.
Assuming we are happy with the outcomes of the unit tests performed on the preceding pipelines, it is time to start thinking about the best way to deploy these pipelines in production. As per best practices, the three pipelines should run as one complete...
