Managing Batches and Pipelines

In Chapter 6, Developing a Stream Processing Solution, you grasped the foundational elements of building effective Big Data processing systems with an emphasis on the importance of both batch and streaming solutions in the realm of data engineering. You explored the design principles of streaming systems, utilizing Azure services such as Event Hubs, Azure Stream Analytics (ASA), and Spark Streaming, and handled time series data, including concepts such as windowed aggregates, checkpointing, and schema drifts.

In this chapter, you will be focusing on four broad categories: triggering batch jobs, handling failures in batch jobs, managing pipelines, and configuring version control for pipelines. Once you complete this chapter, you should be able to comfortably set up and manage batch pipelines using Azure Data Factory (ADF) or Synapse pipelines.

Note

This chapter primarily focuses on the Manage batches and pipelines topic of the DP-203: Data Engineering...