Chapter 4: Understanding Data Pipelines
In the previous chapter, we discussed the various services that can be used to build a data lake in Microsoft Azure. We will now focus on how to kick-start the process of building a data lake using data pipelines.
As a data engineer, the process of laying out a data pipeline should be extremely well planned. This chapter will educate data engineers regarding the various phases of a data pipeline creation, with a list of recommended actions and tasks that need to be accounted for during each phase. Better planning results in better execution.
In this chapter, we will cover the following topics:
- Exploring data pipelines
- Process of creating a data pipeline
- Running a data pipeline
- Sample lakehouse project
