Designing metastores in Azure Synapse Analytics and Azure Databricks

Metastores store the metadata of data in services such as Spark or Hive. Think of a metastore as a data catalog that can tell you which tables you have, what the table schemas are, what the relationships among the tables are, where they are stored, and so on. Spark supports two metastore options: an in-memory version and an external version.

In-memory metastores are limited in accessibility and scale. They can help jobs running on the same Java virtual machine (JVM) but not much further than this. Also, the metadata is lost once the cluster is shut down.

For all practical purposes, Spark uses an external metastore, and the only supported external metastore at the time of writing this book was Hive Metastore. Hive's metastore is mature and provides generic application programming interfaces (APIs) to access it. Hence, instead of rebuilding a new metastore, Spark just uses the mature and well-designed Hive...