Learning about Databricks managed resource groups
In this section, we will take a look at the managed resource group of a Databricks workspace. A managed resource group is a resource group that is automatically created by Azure when a Databricks workspace is created.
In the majority of cases, we do not need to do anything in a managed resource group. However, it is helpful to know the components that are created inside the managed resource group of an Azure Databricks workspace. This helps us understand how the Databricks workspace is functioning under the hood.
To start, let's create a new cluster with the following configuration:
- Cluster Mode: Standard
- Databricks Runtime Version:
8.3 (includes Apache Spark 3.1.1 and Scala 2.12) - Autoscaling: Disabled
- Automatic termination: After
30minutes of inactivity - Worker Type:
Standard_DS3_v2 - Number of workers:
1 - Driver Type: Same as the worker
Let's wait for the cluster to spin up and don...
