Surveying state-of-the-art ML platforms
At a high level, a mature ML system has the components outlined in Figure 6.2. These components are ideally independent and responsible for one particular feature of the system:
Figure 6.2 – Components of an ML platform
Following the lead from SWE modularization, these general components allow us to compare different ML platforms and also specify our PsyStock requirements for each of the components. The components that we choose to use as a reference for architecture comparison are the following:
- Data and feature management: The component of data and feature management is responsible for data acquisition, feature generation, storing, and serving the modules upstream.
- Training infrastructure: The component that handles the process of the training of models, scheduling, consuming features, and producing a final model.
- Deployment and inference: The responsibility of this unit is for the deployment...
