An Introduction to Data Engineering

Data engineering continues to be a fast-growing career path and a role in high demand, as data becomes ever more critical to organizations of all sizes. For those that enjoy the challenge of putting together the “puzzle pieces” that build out complex data pipelines to ingest raw data, and then transform and optimize that data for varied data consumers, it can be a really rewarding career.

In this chapter, we look at the many ways that data has become an important, and increasingly valuable, corporate asset. We also review some of the challenges that organizations face as they deal with increasing volumes of data, and how data engineers can use cloud-based services to help overcome these challenges. We then set the foundations for the hands-on activities in this book by providing step-by-step details on creating a new Amazon Web Services (AWS) account.

Throughout this book, we are going to cover a number of topics that teach the fundamentals of developing data engineering pipelines on AWS, but we’ll get started in this chapter with these topics:

• The rise of big data as a corporate asset
• The challenges of ever-growing datasets
• The role of the data engineer as a big data enabler
• The benefits of the cloud when building big data analytic solutions
• Hands-on – create or access an AWS account for following along with the hands-on activities in this book