You're reading from Building ETL Pipelines with Python Create and deploy enterprise-ready ETL pipelines by employing modern methods

Product type Paperback

Published in Sep 2023

Publisher Packt

ISBN-13 9781804615256

Length 246 pages

Edition 1st Edition

Languages

Python

Tools

AWS

Concepts

Data Streaming

Authors (2):

Brij Kishore Pandey

Emily Ro Schoof

View More author details

Table of Contents (22) Chapters

Preface

1. Part 1:Introduction to ETL, Data Pipelines, and Design Principles

2. Chapter 1: A Primer on Python and the Development Environment FREE CHAPTER

3. Chapter 2: Understanding the ETL Process and Data Pipelines

4. Chapter 3: Design Principles for Creating Scalable and Resilient Pipelines

5. Part 2:Designing ETL Pipelines with Python

6. Chapter 4: Sourcing Insightful Data and Data Extraction Strategies

7. Chapter 5: Data Cleansing and Transformation

8. Chapter 6: Loading Transformed Data

9. Chapter 7: Tutorial – Building an End-to-End ETL Pipeline in Python

10. Chapter 8: Powerful ETL Libraries and Tools in Python

11. Part 3:Creating ETL Pipelines in AWS

12. Chapter 9: A Primer on AWS Tools for ETL Processes

13. Chapter 10: Tutorial – Creating an ETL Pipeline in AWS

14. Chapter 11: Building Robust Deployment Pipelines in AWS

15. Part 4:Automating and Scaling ETL Pipelines

16. Chapter 12: Orchestration and Scaling in ETL Pipelines

17. Chapter 13: Testing Strategies for ETL Pipelines

18. Chapter 14: Best Practices for ETL Pipelines

19. Chapter 15: Use Cases and Further Reading

20. Index

Why subscribe?

21. Other Books You May Enjoy

AWS big data tools for ETL pipelines

Several AWS tools can be used for creating ETL pipelines in the cloud. In this section, we chose to focus on the most common AWS tools that are best for building cost-effective and scalable ETL workflows.

AWS Data Pipeline

AWS Data Pipeline (https://docs.aws.amazon.com/datapipeline/latest/DeveloperGuide/what-is-datapipeline.html) is a web service for orchestrating data workflows across various AWS services and on-premises systems. It provides a visual pipeline designer that makes it easy to visualize and clearly define pre-built connectors for popular data sources and destinations, scheduling, error handling, and monitoring. Data Pipeline supports a wide range of data formats and protocols, including relational databases, NoSQL databases, and Hadoop clusters.

Amazon Kinesis

Amazon Kinesis (https://aws.amazon.com/kinesis/) is a managed service a big data platform specifically designed for processing large datasets (we’re talking...

The rest of the chapter is locked

You're reading from Building ETL Pipelines with Python Create and deploy enterprise-ready ETL pipelines by employing modern methods

Table of Contents (22) Chapters

AWS big data tools for ETL pipelines

AWS Data Pipeline

Amazon Kinesis

Unlock this book and the full library FREE for 7 days

Authors (2)

Personalised recommendations for you