Learning about Databricks Pools
In this section, we will dive deeper into Azure Databricks Pools. We will start by creating a pool, attaching a cluster to a pool, and then learning about the best practices when using Pools in Azure Databricks.
Creating a pool
To create a pool, head over to the Databricks workspace. Then, click on Compute, select Pools, and click on + Create Pool. This will open a page where we need to define the pool's configuration, as shown in the following screenshot:
Figure 4.7 – Creating a pool in Azure Databricks
Let's discuss the configurations one by one:
- Name: We need to give the pool a suitable name.
- Min Idle: This defines the minimum number of idle instances that will be contained in the pool at any given time. These instances do not terminate and when consumed by a cluster, they will be replaced by another set of idle instances.
- Max Capacity: This defines the maximum number of instances that...
