Setting up your environment

Before we begin this chapter, let’s take some time to set up our working environment.

Python, AWS, and Databricks

As in the previous chapters, this chapter assumes you have a working version of Python 3.6+ installed in your development environment. We will also assume you have set up an AWS account and have set up Databricks with that AWS account.

Databricks CLI

The first step is to install the databricks-cli tool using the pip Python package manager:

pip install databricks-cli

Let’s validate that everything has been installed correctly. If the following command produces the tool version, then everything is working correctly:

Databricks –v

Now, let’s set up authentication. First, go into the Databricks UI and generate a personal access token. The following command will ask for the host that was created for your Databricks instance and the created token:

databricks configure –token

We can determine...