Security, monitoring, and automation
In this section, you will see some common components of the ML platform that apply to all the components and stages we have discussed so far. These components assist you in operationalizing the platform in your organization:
- Data pipeline execution: The outcome of data engineering is a data pipeline that ingests, cleans, and processes data. You have built this pipeline with scaled-down data for development purposes. Now, you need to run this code with production data, or you want a scheduled run with new data available, say, every week. An ML platform allows you to take your code and automate its execution in different environments. This is a big step because the platform not only allows you to run your code but will also manage the packaging of all the dependencies of your code so that it can run anywhere. If the code that you have built is using Apache Spark, the platform should allow you to automate the process of provisioning a Spark...
