You're reading from Azure Data Engineer Associate Certification Guide A hands-on reference guide to developing your data engineering skills and preparing for the DP-203 exam

Product type Paperback

Published in Feb 2022

Publisher Packt

ISBN-13 9781801816069

Length 574 pages

Edition 1st Edition

Tools

Azure

Concepts

Big Data

Author (1):

Newton Alex

View More author details

Table of Contents (23) Chapters

Preface

1. Part 1: Azure Basics

2. Chapter 1: Introducing Azure Basics FREE CHAPTER

3. Part 2: Data Storage

4. Chapter 2: Designing a Data Storage Structure

5. Chapter 3: Designing a Partition Strategy

6. Chapter 4: Designing the Serving Layer

7. Chapter 5: Implementing Physical Data Storage Structures

8. Chapter 6: Implementing Logical Data Structures

9. Chapter 7: Implementing the Serving Layer

10. Part 3: Design and Develop Data Processing (25-30%)

11. Chapter 8: Ingesting and Transforming Data

12. Chapter 9: Designing and Developing a Batch Processing Solution

13. Chapter 10: Designing and Developing a Stream Processing Solution

14. Chapter 11: Managing Batches and Pipelines

15. Part 4: Design and Implement Data Security (10-15%)

16. Chapter 12: Designing Security for Data Policies and Standards

17. Part 5: Monitor and Optimize Data Storage and Data Processing (10-15%)

18. Chapter 13: Monitoring Data Storage and Data Processing

19. Chapter 14: Optimizing and Troubleshooting Data Storage and Data Processing

20. Part 6: Practice Exercises

21. Chapter 15: Sample Questions with Solutions

22. Other Books You May Enjoy

Transforming data by using ADF

We have already seen a few examples of ADF in the previous chapters. Just to refresh, search for Azure Data Factory from the Azure portal and create a new Azure data factory. Once created, you can launch the Azure data factory, as shown in the following screenshot:

Figure 8.1 – Launching the Azure data factory

This will open up the ADF workspace where you can build your pipelines, as shown in the following screenshot:

Figure 8.2 – ADF Studio

All ADF transformations happen on datasets. So, before we can do any transformation, we will have to create datasets of the source data. You can click on the + symbol to create a new dataset, as shown in the following screenshot:

Figure 8.3 – Creating a new dataset in ADF

ADF provides a wide range of Azure and non-Azure data sources, as shown in the following screenshot.

Figure 8.4 – Dataset...