You're reading from Data Engineering with AWS Acquire the skills to design and build AWS-based data transformation pipelines like a pro

Product type Paperback

Published in Oct 2023

Publisher Packt

ISBN-13 9781804614426

Length 636 pages

Edition 2nd Edition

Tools

AWS

Concepts

Data Engineering

Author (1):

Gareth Eagar

View More author details

Table of Contents (24) Chapters

Preface

1. Section 1: AWS Data Engineering Concepts and Trends

2. An Introduction to Data Engineering FREE CHAPTER

3. Data Management Architectures for Analytics

4. The AWS Data Engineer’s Toolkit

5. Data Governance, Security, and Cataloging

6. Section 2: Architecting and Implementing Data Engineering Pipelines and Transformations

7. Architecting Data Engineering Pipelines

8. Ingesting Batch and Streaming Data

9. Transforming Data to Optimize for Analytics

10. Identifying and Enabling Data Consumers

11. A Deeper Dive into Data Marts and Amazon Redshift

12. Orchestrating the Data Pipeline

13. Section 3: The Bigger Picture: Data Analytics, Data Visualization, and Machine Learning

14. Ad Hoc Queries with Amazon Athena

15. Visualizing Data with Amazon QuickSight

16. Enabling Artificial Intelligence and Machine Learning

17. Section 4: Modern Strategies: Open Table Formats, Data Mesh, DataOps, and Preparing for the Real World

18. Building Transactional Data Lakes

19. Implementing a Data Mesh Strategy

20. Building a Modern Data Platform on AWS

21. Wrapping Up the First Part of Your Learning Journey

22. Other Books You May Enjoy

23. Index

Hands-on – Working with Apache Iceberg tables in AWS

As discussed in the previous section, Amazon Athena has strong support for the Apache Iceberg format, and as a serverless service, it is the quickest and simplest way to work with Apache Iceberg tables.

For the hands-on section of this chapter, we are going to use the Amazon Athena service to create an Apache Iceberg table, and then explore some of the features of Iceberg as we query and modify the table. To do this, we will create an Iceberg version of one of the tables we created earlier in this book.

Creating an Apache Iceberg table using Amazon Athena

To create our Apache Iceberg table, we will access the Athena console and then run DDL statements to specify the details of the table we want to create. At the time of writing, Amazon Athena supports the creation of Iceberg v2 tables. Remember to refer to the GitHub site for this book for a copy of the SQL statements used in this section (as mentioned at the...