Installing libraries in Azure Databricks
We can make use of third-party or custom code by installing libraries written in Python, Java, Scala, or R. These libraries will be available to notebooks and jobs running on your clusters depending on the level at which the libraries were installed.
In Azure Databricks, installing libraries can be done in different ways, the most important decision being at which level we will be installing these libraries. The options available are at the workspace, cluster, or notebook level:
- Workspace libraries serve as a local repository from which you create cluster-installed libraries. A workspace library might be custom code created by your organization or might be a particular version of an open-source library that your organization has standardized on.
- Cluster libraries are available to be used by all notebooks attached to that cluster. You can install a cluster library from a public repository or create one from a previously installed...
