Managing our schemas with code

Our schema app will manage setting up and updating any schema changes we have to Databricks. It’s important to have a mechanism to manage schemas. Data swamps quickly form when the schema is not managed correctly. In this project, we are not referencing the schema app to have a central view of the schema. This might be a good idea for your project but creates the added overhead of dealing with package versioning.

In our configuration folder, we will keep data classes that define how we want our database and tables configured from a high level:

schema-jobs/schema_jobs/jobs/configuration/database_configuration.py
"""
fill in
"""
import abc
from dataclasses import dataclass
class DatabaseConfig(abc.ABC):
    """
    fill in
    """
    database_name = "dev"
@dataclass
class DatabaseConfiguration...