Maintaining metadata
As we have seen in Chapter 4, Designing a Partition Strategy, metastores are like data catalogs that contain information about all the tables you have, the table schemas, the relationships among them, where they are stored, and so on. In that chapter, we learned at a high level about how to access the metadata in Synapse and Databricks. Now, let's learn the details of implementing them.
Metadata using Synapse SQL and Spark pools
Synapse supports a shared metadata model. The databases and tables that use Parquet or CSV storage formats are automatically shared between the compute pools, such as SQL and Spark.
Important Note
Data created from Spark can only be read and queried by SQL pools but cannot be modified at the time of writing this book.
Let's look at an example of creating a database and a table using Spark and accessing it via SQL:
- In the Synapse Spark notebook, create a sample table, as shown in the following screenshot...
