You're reading from Data Engineering with Python Work with massive datasets to design data models and automate data pipelines using Python

Product type Paperback

Published in Oct 2020

Publisher Packt

ISBN-13 9781839214189

Length 356 pages

Edition 1st Edition

Languages

Python

Concepts

Data Analysis

Author (1):

Paul Crickard

View More author details

Table of Contents (21) Chapters

Preface

1. Section 1: Building Data Pipelines – Extract Transform, and Load

2. Chapter 1: What is Data Engineering? FREE CHAPTER

3. Chapter 2: Building Our Data Engineering Infrastructure

4. Chapter 3: Reading and Writing Files

5. Chapter 4: Working with Databases

6. Chapter 5: Cleaning, Transforming, and Enriching Data

7. Chapter 6: Building a 311 Data Pipeline

8. Section 2:Deploying Data Pipelines in Production

9. Chapter 7: Features of a Production Pipeline

10. Chapter 8: Version Control with the NiFi Registry

11. Chapter 9: Monitoring Data Pipelines

12. Chapter 10: Deploying Data Pipelines

13. Chapter 11: Building a Production Data Pipeline

14. Section 3:Beyond Batch – Building Real-Time Data Pipelines

15. Chapter 12: Building a Kafka Cluster

16. Chapter 13: Streaming Data with Apache Kafka

17. Chapter 14: Data Processing with Apache Spark

18. Chapter 15: Real-Time Edge Data with MiNiFi, Kafka, and Spark

19. Other Books You May Enjoy

Leave a review - let other readers know what you think

Appendix

Versioning your data pipelines

You can use the NiFi Registry to version your data pipelines inside of a processor group. I have NiFi running and the canvas zoomed in to the SeeClickFix processor group from Chapter 6, Building a 311 Data Pipeline. To start versioning this data pipeline, right-click on the title bar of the processor group and select Version | Start version control, as shown in the following screenshot:

Figure 8.8 – Starting version control on a processor group

Your processor group is now being tracked by version control. You will see a green checkmark on the left of the processor group title box, as shown in the following screenshot:

Figure 8.9 – Processor group using version control

If you browse back to the NiFi Registry, you will see that Scf-DataEngineeringPython is being tracked. You will also see the details by expanding the bar. The details show your description and the version notes (First Commit...