Performing data engineering in Microsoft Azure
Data engineering in Microsoft Azure can be performed using the following three options:
- Self-managed data engineering services (IaaS)
- Azure-managed data engineering services (PaaS)
- Data engineering as a service (SaaS):
Figure 3.1 – Data engineering option in Microsoft Azure
Self-managed data engineering services (IaaS)
In the early phases of data engineering, using well-known distributed frameworks such as Hadoop, Spark, and Kafka rose sharply. As a result, many organizations were deploying Hadoop/Spark/Kafka using on-premises infrastructures. Since Hadoop/Spark/Kafka are multi-node frameworks, this meant the installations were performed using physical and virtual machines hosted on either the organization's owned or co-located data centers.
Then came the period when the cloud started to become a reality and organizations started to move their Hadoop/Spark/Kafka clusters to...
