Scheduling notebooks using job clusters

Data processing can be performed using notebooks, but to operationalize it, we need to execute it at a specific scheduled time, depending upon the demands of the use case or problem statement. After a notebook has been created, you can schedule a notebook to be executed at a preferred frequency using job clusters. This recipe will demonstrate how you could schedule a notebook using job clusters.

Getting ready

Create a Databricks workspace, as explained in the Configuring the Azure Databricks environment recipe.

How to do it…

In the following steps, we will import the SampleJob.dbc notebook file into the Databricks workspace and schedule it to be run daily:

1. Log in to portal.azure.com. Go to All resources and find pactadedatabricks, the Databricks workspace created in the Configuring the Azure Databricks environment recipe. Click Launch Workspace to log in to the Databricks portal.
2. Navigate to Workspace | Create...