This chapter does not require you to follow along with any hands-on exercises. However, there are some examples used from the BentoML official site: https://docs.bentoml.org/en/latest/tutorial.html.

If you want to try those examples on your local machine, please feel free to install a local version of BentoML following the simple installation instructions here: https://docs.bentoml.org/en/latest/installation.html.

Basically, you need to install BentoML like other Python packages using the following command:

 pip install bentoml

Feel free to follow the quick get-started link to understand the steps involved in model serving, which are highlighted in a later section in this chapter.

Serving is an important step for ensuring the business impact of the applications when we develop the life cycle of application development. The application we have developed needs to be available to the user so that they can use it. For example, let’s say we have developed a game. After the development, if the game just stays on the developer’s machine, then it is not going to be of any use to the users. So, the developer needs to bring the game to the users by serving it through a serving platform such as Apple Store, Google Play Store, and web servers.

So, serving can be seen as a mechanism to distribute our applications/services to end users. The end users can be different based on the applications/services we develop. Serving creates a bridge of communication between the two parties: the developer and the users. This bridge is vital for the business success of our application. If we don’t have people using our service, then we are not gaining any business value or impact from the applications we’ve developed. That’s what we have seen in the past when big corporate companies’ servers go down for some time: they incur a huge amount of loss. Facebook (Meta) lost ~65 million US dollars due to its outage for some hours in October 2021, as per Forbes: https://www.forbes.com/sites/abrambrown/2021/10/05/facebook-outage-lost-revenue/?sh=c1d7d03231ad.

The development-to-serving process usually forms the life cycle of the service or application.

For example, let’s consider the life cycle of web application development in Figure 1.1.

Figure 1.1 – Web development life cycle

The developer develops the website. Then, the website is served using a web server. Only after that are the users able to use the website. The cycle can continue through collecting feedback from users, improving the website, and serving again on the web server.

Now we know what serving is, let’s look at what we are actually serving.

There are a lot of definitions of models from the perspective of various domains. When we define a model or use the term model in this book, we will consider it in the context of ML. A model in ML can be seen as a function that has been fine-tuned through training, using some well-engineered data so that the function can recognize and distinguish patterns in unseen data.

Depending on the problem and business goal, a different model can be used, for example, a linear regression model, a logistic regression model, a naive Bayes model, or a decision tree model. These models’ underlying logical representations are different from each other. However, we use the generic term model and the problem domain and name of the ML algorithm to give us a clear picture of what the model is, how it was trained, and how the model is represented. For example, if we are told that the model is for linear regression, then we know that it was trained by minimizing a cost function iteratively using the training data, and it can be saved by storing the regression parameters’ coefficients and intercepts. Similarly, other models will have different algorithms for training and storing. For a deep learning model, we might have to use forward propagation and backward propagation for training, and for storing we might have to store the weights and biases of all the layers.

The trained model can be stored in different formats to load later for serving and inference. Some popular formats in which to save a model are as follows:

• ONNX
• YAML
• Protobuf
• Pickle
• JSON
• H5
• TFJS
• Joblib

However, model-serving tools usually require the models to be saved in a particular format. So, they provide a function to save the model in its desired format. There are also tools and libraries to convert models from one format to another. For example, in Figure 1.2, we see that an AlexNet model that is pre-trained in PyTorch is loaded and exported to ONNX format in a file named alexnet.onnx.

PyTorch files

It’s worth knowing that PyTorch saves the model using the Python pickle (https://docs.python.org/3/library/pickle.html) library. For further reading on PyTorch strategies for saving and loading models, please check out their official documentation: https://pytorch.org/tutorials/beginner/saving_loading_models.html.

Figure 1.2 – Example code converting a PyTorch pre-trained AlexNet model to ONNX format

Note

Figure 1.2 is an example from the PyTorch official website: https://pytorch.org/docs/stable/onnx.html#example-alexnet-from-pytorch-to-onnx.

Now we should have a good idea about models and how each model is represented and stored. The following section will introduce us to model serving.