Data Ingestion with Python Cookbook: A practical guide to ingesting, monitoring, and identifying errors in the data ingestion process

Gláucia Esppenchutz

£20.99 ~~£23.99~~

4.5 (4 Ratings)

eBook May 2023 414 pages 1st Edition

Gláucia Esppenchutz

£20.99 ~~£23.99~~

4.5 (4 Ratings)

eBook May 2023 414 pages 1st Edition

What do you get with eBook?

Instant access to your Digital eBook purchase

Download this book in EPUB and PDF formats

Access this title in our online reader with advanced features

DRM FREE - Read whenever, wherever and however you want

AI Assistant (beta) to help accelerate your learning

View table of contents

Preview Book

Key benefits

Harness best practices to create a Python and PySpark data ingestion pipeline
Seamlessly automate and orchestrate your data pipelines using Apache Airflow
Build a monitoring framework by integrating the concept of data observability into your pipelines

Description

Data Ingestion with Python Cookbook offers a practical approach to designing and implementing data ingestion pipelines. It presents real-world examples with the most widely recognized open source tools on the market to answer commonly asked questions and overcome challenges. You’ll be introduced to designing and working with or without data schemas, as well as creating monitored pipelines with Airflow and data observability principles, all while following industry best practices. The book also addresses challenges associated with reading different data sources and data formats. As you progress through the book, you’ll gain a broader understanding of error logging best practices, troubleshooting techniques, data orchestration, monitoring, and storing logs for further consultation. By the end of the book, you’ll have a fully automated set that enables you to start ingesting and monitoring your data pipeline effortlessly, facilitating seamless integration with subsequent stages of the ETL process.

Who is this book for?

This book is for data engineers and data enthusiasts seeking a comprehensive understanding of the data ingestion process using popular tools in the open source community. For more advanced learners, this book takes on the theoretical pillars of data governance while providing practical examples of real-world scenarios commonly encountered by data engineers.

What you will learn

Implement data observability using monitoring tools
Automate your data ingestion pipeline
Read analytical and partitioned data, whether schema or non-schema based
Debug and prevent data loss through efficient data monitoring and logging
Establish data access policies using a data governance framework
Construct a data orchestration framework to improve data quality

What do you get with eBook?

Instant access to your Digital eBook purchase

Download this book in EPUB and PDF formats

Access this title in our online reader with advanced features

DRM FREE - Read whenever, wherever and however you want

AI Assistant (beta) to help accelerate your learning

Frequently bought together

Exploratory Data Analysis with Python Cookbook

£37.99

£29.99

£26.99

Total £ 94.97

Lincoln Nascimento Jun 26, 2023

The book is amazing, it helped me through daily situations and also gave me new perspectives. The mention to OpenMetadata helped to solve a problem I was facing in my current company. I recommend for new and seasoned Data Engineers.

Amazon Verified review

Om S Jul 10, 2023

"Data Ingestion with Python Cookbook" is a practical and comprehensive guide that equips data engineers and enthusiasts with the knowledge and skills to design and implement efficient data ingestion pipelines. The book seamlessly blends industry best practices with real-world examples, utilizing popular open-source tools to overcome common challenges in the field.From data schema design to creating monitored pipelines with Apache Airflow, the book covers a wide range of topics essential for building robust data ingestion processes. Readers learn to integrate data observability principles into their pipelines, ensuring data quality and facilitating troubleshooting. The book also addresses the complexities of reading various data sources and formats, offering practical solutions and techniques.Throughout the book, readers gain insights into error logging best practices, data orchestration, monitoring, and establishing data access policies through a data governance framework. With a focus on automation and efficiency, the book empowers readers to effortlessly ingest and monitor their data pipelines, laying the foundation for seamless integration with subsequent stages of the ETL process."Data Ingestion with Python Cookbook" is a valuable resource for data engineers and enthusiasts seeking a comprehensive understanding of data ingestion using popular open-source tools. Whether you're a beginner or an advanced learner, this book provides practical examples and theoretical pillars of data governance to address real-world scenarios encountered in data engineering projects.

Vinoth Jun 28, 2023

The book covers life cycle of Data ingestion process (ingesting, monitoring & errors). Author goes in depth about Data Discovery ,Ingesting Data from Structured and Unstructured Databases, PySpark with Defined and Non-Defined Schemas, Designing Monitored Data Workflows, Airflow, Logging and Monitoring Data Ingest in Airflow, Automating Data Ingestion Pipelines, Data Observability for Debugging, Error Handling, and Preventing Downtime. Throughout the book there are lots of examples with code. Highly recommend this book for entry level and experienced data engineers.

Rashi Garg Aug 07, 2023

I just finished reading this book last week. Having worked in data engineering field more than 10 years now , I found this book a great resource for someone who wants to build skills as a hands-on data-engineer . It offers a practical explanation of the data ingestion lifecycle, covering aspects like data discovery, ingestion from various databases, PySpark usage with different schemas, monitored workflows design, Airflow integration, error handling. The book's focus on open-source tools and real-world scenarios equips readers with skills to build efficient pipelines, ensure data quality, and troubleshoot effectively. I really liked the last section of data observability in the book where it covers the statsd , prometheus and grafana setup.Though I would have loved to more details on another popular data engineering stack Dbt,Snowflake, Yet fundamentals of building scalable data pipelines will remain the same regardless of the tech stack. I would highly recommend this book for data engineers seeking practical guidance and understanding of data ingestion challenges and solutions.

Data Ingestion with Python Cookbook: A practical guide to ingesting, monitoring, and identifying errors in the data ingestion process

What do you get with eBook?