Understanding model decomposition and distillation

The third technique we introduce here is called model decomposition and distillation.

At a high level, model decomposition tries to split the giant model into small subnets and thus minimize the communication among those subnets.

For each DNN, we will further reduce its size by performing model pruning, which is also called model distillation.

Now, we will talk about each technique in detail.

Model decomposition

One SOTA approach for model decomposition is sensAI, which can almost eliminate communication among the subnets split from the giant base model. Basically, we assume we have a fully trained model.

For the ease of illustration, we assume the DNN base model here is just a convolutional neural network (CNN) and the fully trained base model is used for image classification between cats and dogs.

The following diagram depicts how we split this model into disconnected subnets:

Figure 8...