Managing data pipelines with Glue

Data scientists and data engineers run different jobs to transform, extract, and load data into systems such as S3. For example, we might have a daily job that processes text data and stores a table with the bag-of-words table representation that we saw in Chapter 2, Classifying Twitter Feeds with Naive Bayes. We might want to update the table each day to point to the latest available data. Upstream processes can then only rely on the table name to find and process the latest version of the data. If we do not catalog this data properly, it will be very hard to combine the different data sources or even to know where the data is located, which is where the AWS Glue metastore comes in. Tables in Glue are grouped into databases; however, tables in different databases can be joined and referenced.

...