0

Explore Products

Best Sellers

New Releases

Books

Videos

Audiobooks

Free Learning

Azure Data Engineer Associate Certification Guide

You're reading from Azure Data Engineer Associate Certification Guide Ace the DP-203 exam with advanced data engineering skills

Product type Paperback

Published in May 2024

Publisher Packt

ISBN-13 9781805124689

Length 548 pages

Edition 2nd Edition

Languages

SQL

Tools

Azure

Concepts

Big Data

Authors (3):

Newton Alex

Giacinto Palmieri

Mr. Surendra Mettapalli

View More author details

Table of Contents (17) Chapters

Preface

1. Part 1: Azure Basics FREE CHAPTER

2. Chapter 1: Introducing Azure Basics

3. Part 2: Data Storage

4. Chapter 2: Implementing a Partition Strategy

5. Chapter 3: Designing and Implementing the Data Exploration Layer

6. Part 3:Data Processing

7. Chapter 4: Ingesting and Transforming Data

8. Chapter 5: Developing a Batch Processing Solution

9. Chapter 6: Developing a Stream Processing Solution

10. Chapter 7: Managing Batches and Pipelines

11. Part 4:Secure, Monitor, and Optimize Data Storage and Processing

12. Chapter 8: Implementing Data Security

13. Chapter 9: Monitoring Data Storage and Data Processing

14. Chapter 10: Optimizing and Troubleshooting Data Storage and Data Processing

15. Chapter 11: Accessing the Online Practice Resources

16. Other Books You May Enjoy

Managing Small Files

Processing small files can be a headache for big data systems. Both engines such as Spark, Synapse SQL, and Google BigQuery, and cloud storage platforms such as Blob and ADLS Gen2, thrive on large files. So, how do you optimize your data pipelines? The answer lies in consolidating those pesky small files into more manageable ones.

Note

This section primarily focuses on the Compact small files concept of the DP-203: Data Engineering on Microsoft Azure exam.

In the Azure ecosystem, you can achieve this efficiency boost using ADF and Synapse pipelines. Imagine you have a directory filled with tiny Comma Separated Values (CSV) files, and your goal is to merge them into a single, cohesive large file to pave the way for smoother data processing. The steps for Synapse pipelines will closely mirror those in ADF.

Perform the following steps to streamline your pipelines:

Head over to the ADF portal and select the Copy Data activity, as shown in Figure...

The rest of the chapter is locked

Register for a free Packt account to unlock a world of extra content!

A free Packt account unlocks extra newsletters, articles, discounted offers, and much more. Start advancing your knowledge today.

Unlock this book and the full library FREE for 7 days

Get unlimited access to 7000+ expert-authored eBooks and videos courses covering every tech area you can think of

Start free trial

Renews at $19.99/month. Cancel anytime

Authors (3)

Palmieri

Palmieri

Giacinto Palmieri is a freelance technical trainer and consultant who has been working in the IT secotr for for more than 25 years, He has a double background as a developer and a data engineer and focuses nowadays on Azure development and data services, but also on Power BI. He works for a wide range of training companies and for their clients in different business sectors and geographical areas.He prides himelf in always trying to go to the bottom of things, exploring not only the what and the how but also the why (his educational background is in Philosophy), which then allows him to convey this deep understanding to his course participants and readers

See other products by Palmieri

Alex

Alex

Newton Alex leads several Azure Data Analytics teams in Microsoft, India. His team contributes to technologies including Azure Synapse, Azure Databricks, Azure HDInsight, and many open source technologies, including Apache YARN, Apache Spark, and Apache Hive. He started using Hadoop while at Yahoo, USA, where he helped build the first batch processing pipelines for Yahoo's ad serving team. After Yahoo, he became the leader of the big data team at Pivotal Inc., USA, where he was responsible for the entire open source stack of Pivotal Inc. He later moved to Microsoft and started the Azure Data team in India. He has worked with several Fortune 500 companies to help build their data systems on Azure.

See other products by Alex

Surendra Mettapalli

Surendra Mettapalli

Surendra Mettapalli is leading expert in Azure technologies with proven track record of designing, implementing innovative cloud solutions. As a Lead Data Engineer/Architect, he specializes in pivotal services such as Azure Synapse Analytics, Azure Databricks, Azure Data Factory and Power BI. He has a deep understanding of Cloud Architecture, Data Engineering, AI-driven applications, and has successfully collaborated with the diverse clients across various sectors including technology, finance, retail and government. His expertise has been instrumental in delivering some of the largest and most impactful data projects, enabling organizations to utilize Azure's full potential and enhance their business operations.

See other products by Surendra Mettapalli

Personalised recommendations for you

Based on your interests and search pattern

Modern Computer Vision with PyTorch

Modern Computer Vision with PyTorch

This book provides a hands-on approach to solving over 30 prominent real-world computer vision problems using PyTorch 2.x on actual datasets. Here you'll learn to build a neural network from scratch and optimize hyperparameters, perform image classification, multi-object detection, segmentation, and more. You'll also explore facial expression manipulation and combining CV with NLP and RL techniques, build generative AI applications, and take your model to production on AWS. By the end of this book, you'll master modern NN architectures and confidently solve real-world CV problems.

Jun 2024 24h 52m

Modern Computer Vision with PyTorch

Modern Computer Vision with PyTorch

This book provides a hands-on approach to solving over 30 prominent real-world computer vision problems using PyTorch 2.x on actual datasets. Here you'll learn to build a neural network from scratch and optimize hyperparameters, perform image classification, multi-object detection, segmentation, and more. You'll also explore facial expression manipulation and combining CV with NLP and RL techniques, build generative AI applications, and take your model to production on AWS. By the end of this book, you'll master modern NN architectures and confidently solve real-world CV problems.

Jun 2024 24h 52m

Modern Computer Vision with PyTorch

Modern Computer Vision with PyTorch

This book provides a hands-on approach to solving over 30 prominent real-world computer vision problems using PyTorch 2.x on actual datasets. Here you'll learn to build a neural network from scratch and optimize hyperparameters, perform image classification, multi-object detection, segmentation, and more. You'll also explore facial expression manipulation and combining CV with NLP and RL techniques, build generative AI applications, and take your model to production on AWS. By the end of this book, you'll master modern NN architectures and confidently solve real-world CV problems.

Jun 2024 24h 52m

Modern Computer Vision with PyTorch

Modern Computer Vision with PyTorch

This book provides a hands-on approach to solving over 30 prominent real-world computer vision problems using PyTorch 2.x on actual datasets. Here you'll learn to build a neural network from scratch and optimize hyperparameters, perform image classification, multi-object detection, segmentation, and more. You'll also explore facial expression manipulation and combining CV with NLP and RL techniques, build generative AI applications, and take your model to production on AWS. By the end of this book, you'll master modern NN architectures and confidently solve real-world CV problems.

Jun 2024 24h 52m

Modern Computer Vision with PyTorch

Modern Computer Vision with PyTorch

This book provides a hands-on approach to solving over 30 prominent real-world computer vision problems using PyTorch 2.x on actual datasets. Here you'll learn to build a neural network from scratch and optimize hyperparameters, perform image classification, multi-object detection, segmentation, and more. You'll also explore facial expression manipulation and combining CV with NLP and RL techniques, build generative AI applications, and take your model to production on AWS. By the end of this book, you'll master modern NN architectures and confidently solve real-world CV problems.

Jun 2024 24h 52m

Modern Computer Vision with PyTorch

Modern Computer Vision with PyTorch

This book provides a hands-on approach to solving over 30 prominent real-world computer vision problems using PyTorch 2.x on actual datasets. Here you'll learn to build a neural network from scratch and optimize hyperparameters, perform image classification, multi-object detection, segmentation, and more. You'll also explore facial expression manipulation and combining CV with NLP and RL techniques, build generative AI applications, and take your model to production on AWS. By the end of this book, you'll master modern NN architectures and confidently solve real-world CV problems.

Jun 2024 24h 52m

Modern Computer Vision with PyTorch

Modern Computer Vision with PyTorch

This book provides a hands-on approach to solving over 30 prominent real-world computer vision problems using PyTorch 2.x on actual datasets. Here you'll learn to build a neural network from scratch and optimize hyperparameters, perform image classification, multi-object detection, segmentation, and more. You'll also explore facial expression manipulation and combining CV with NLP and RL techniques, build generative AI applications, and take your model to production on AWS. By the end of this book, you'll master modern NN architectures and confidently solve real-world CV problems.

Jun 2024 24h 52m

Modern Computer Vision with PyTorch

Modern Computer Vision with PyTorch

This book provides a hands-on approach to solving over 30 prominent real-world computer vision problems using PyTorch 2.x on actual datasets. Here you'll learn to build a neural network from scratch and optimize hyperparameters, perform image classification, multi-object detection, segmentation, and more. You'll also explore facial expression manipulation and combining CV with NLP and RL techniques, build generative AI applications, and take your model to production on AWS. By the end of this book, you'll master modern NN architectures and confidently solve real-world CV problems.

Jun 2024 24h 52m

Modern Computer Vision with PyTorch

Modern Computer Vision with PyTorch

This book provides a hands-on approach to solving over 30 prominent real-world computer vision problems using PyTorch 2.x on actual datasets. Here you'll learn to build a neural network from scratch and optimize hyperparameters, perform image classification, multi-object detection, segmentation, and more. You'll also explore facial expression manipulation and combining CV with NLP and RL techniques, build generative AI applications, and take your model to production on AWS. By the end of this book, you'll master modern NN architectures and confidently solve real-world CV problems.

Jun 2024 24h 52m

Modern Computer Vision with PyTorch

Modern Computer Vision with PyTorch

This book provides a hands-on approach to solving over 30 prominent real-world computer vision problems using PyTorch 2.x on actual datasets. Here you'll learn to build a neural network from scratch and optimize hyperparameters, perform image classification, multi-object detection, segmentation, and more. You'll also explore facial expression manipulation and combining CV with NLP and RL techniques, build generative AI applications, and take your model to production on AWS. By the end of this book, you'll master modern NN architectures and confidently solve real-world CV problems.

Jun 2024 24h 52m

Modern Computer Vision with PyTorch

Modern Computer Vision with PyTorch

This book provides a hands-on approach to solving over 30 prominent real-world computer vision problems using PyTorch 2.x on actual datasets. Here you'll learn to build a neural network from scratch and optimize hyperparameters, perform image classification, multi-object detection, segmentation, and more. You'll also explore facial expression manipulation and combining CV with NLP and RL techniques, build generative AI applications, and take your model to production on AWS. By the end of this book, you'll master modern NN architectures and confidently solve real-world CV problems.

Jun 2024 24h 52m

Modern Computer Vision with PyTorch

Modern Computer Vision with PyTorch

This book provides a hands-on approach to solving over 30 prominent real-world computer vision problems using PyTorch 2.x on actual datasets. Here you'll learn to build a neural network from scratch and optimize hyperparameters, perform image classification, multi-object detection, segmentation, and more. You'll also explore facial expression manipulation and combining CV with NLP and RL techniques, build generative AI applications, and take your model to production on AWS. By the end of this book, you'll master modern NN architectures and confidently solve real-world CV problems.

Jun 2024 24h 52m