Enriching and operationalizing

In this section, we will look at loading an already trained ML model from the model registry and generate predictions on new incoming data. Once these predictions have been generated, we will save this data in another Delta table so that we can create a report on it.

Note

The code that will be discussed in this section can be found in the Data Science - Perform Prediction or Scoring notebook. Please make sure you attach the lakehouse (nyctaxilake) you created in the Data and storage – creating a lakehouse and ingesting data using Apache Spark section of this chapter to this notebook.

The steps are as follows:

1. The first step is to import the required libraries into the current Spark session. Next, we must load the trained ML model from the MLflow-based model registry. While specifying the model’s name, we also need to specify the version of the model – in this case, it’s version = 2:

   import mlflow
   from pyspark...