Reviewing the standard Pachyderm workflow

As you probably noticed when you were creating a pipeline, there is a certain workflow that you will need to follow when working with Pachyderm. Depending on your automation tools, your team processes, and the software that you use, it might differ, but in general, it boils down to the following common steps:

1. Add your data to Pachyderm.
2. Create a pipeline or pipelines.
3. Add more data.
4. Modify the parameters.
5. Modify your pipeline or pipelines.
6. Output the result to an output repository.
7. Repeat Steps 3–6 as many times as needed or continuously.
8. Serve your model or package your library and make it available for use.

The following diagram demonstrates this process:

Figure 7.1 – Pachyderm workflow

Depending on whether you keep your code in a Docker image, in the pipeline itself, or you use a build pipeline with your Python code, you need to rebuild your Docker image every...