Chapter 7: Metadata Management

Just as with relational databases, AWS Glue relies on the concepts of databases and tables to organize and manage datasets. That said, these concepts are quite different in their execution. In a relational database, the data to be stored and its descriptors (such as the schema and comments, also known as metadata) are stored and managed together: there is no way to store data without describing it first, and there is no way to add metadata to already written data.

In big data environments, the storage and metadata layers are decoupled. There is no centralized storage system because of dataset size limitations, and data is typically dumped without a format onto distributed, large-scale systems such as Apache Hadoop or Amazon S3. This means we as users have to bring the metadata to the data wherever it is stored, cataloging the data and specifying its location, how to read it, and how to understand it (its schema).

In the case of Glue, this centralized...