Validating the output using Amazon Athena

The Parquet format data is already available in Amazon S3 with year and month partition, but to make it more consumable for data analysts or data scientists, it would be great if we could enable querying the data through SQL by making it available as a database table.

To make that integration, we can follow a two-step approach:

1. We can run the Glue crawler to create a Glue Data Catalog table on top of the S3 data.
2. We can run a query in Athena to validate the output.

Let's see how you can integrate that.

Defining a virtual Glue Data Catalog table on top of Amazon S3 data

You can follow these steps to create and run the Glue crawler, which will create a Glue Data Catalog table:

1. Navigate to the AWS Glue crawler at https://console.aws.amazon.com/glue/home?region=us-east-1#catalog:tab=crawlers.
2. Then click Add crawler, which will open up the form to configure the crawler.
3. Configure the crawler...