You're reading from Building ETL Pipelines with Python Create and deploy enterprise-ready ETL pipelines by employing modern methods

Product type Paperback

Published in Sep 2023

Publisher Packt

ISBN-13 9781804615256

Length 246 pages

Edition 1st Edition

Languages

Python

Tools

AWS

Concepts

Data Streaming

Authors (2):

Brij Kishore Pandey

Emily Ro Schoof

View More author details

Table of Contents (22) Chapters

Preface

1. Part 1:Introduction to ETL, Data Pipelines, and Design Principles

2. Chapter 1: A Primer on Python and the Development Environment FREE CHAPTER

3. Chapter 2: Understanding the ETL Process and Data Pipelines

4. Chapter 3: Design Principles for Creating Scalable and Resilient Pipelines

5. Part 2:Designing ETL Pipelines with Python

6. Chapter 4: Sourcing Insightful Data and Data Extraction Strategies

7. Chapter 5: Data Cleansing and Transformation

8. Chapter 6: Loading Transformed Data

9. Chapter 7: Tutorial – Building an End-to-End ETL Pipeline in Python

10. Chapter 8: Powerful ETL Libraries and Tools in Python

11. Part 3:Creating ETL Pipelines in AWS

12. Chapter 9: A Primer on AWS Tools for ETL Processes

13. Chapter 10: Tutorial – Creating an ETL Pipeline in AWS

14. Chapter 11: Building Robust Deployment Pipelines in AWS

15. Part 4:Automating and Scaling ETL Pipelines

16. Chapter 12: Orchestration and Scaling in ETL Pipelines

17. Chapter 13: Testing Strategies for ETL Pipelines

18. Chapter 14: Best Practices for ETL Pipelines

19. Chapter 15: Use Cases and Further Reading

20. Index

Why subscribe?

21. Other Books You May Enjoy

ETL logging in Python

Logging plays a crucial role in ETL pipelines, providing valuable insights into the execution and health of your data processing workflows. In this section, we will explore the importance of logging in Python-based ETL pipelines and discuss best practices for implementing robust logging mechanisms. As mentioned previously, as stewards of data, it is essential to have a clear understanding of what occurs during each stage of the ETL process. Logging provides a detailed record of events, including information about data sources, transformation steps, errors, and system performance, as illustrated in the following diagram:

Figure 14.1: Processes that are logged in an ETL pipeline

In the following subsections, we will discuss some of the most important attributes of logging within your data pipelines.

Debugging and issue resolution

Logging enables you to identify and debug issues by providing insights into the state of both the...

The rest of the chapter is locked

You're reading from Building ETL Pipelines with Python Create and deploy enterprise-ready ETL pipelines by employing modern methods

Table of Contents (22) Chapters

ETL logging in Python

Debugging and issue resolution

Unlock this book and the full library FREE for 7 days

Authors (2)

Personalised recommendations for you