Pipeline specification overview

Typically, when you conduct an ML experiment, it involves multiple sequential steps. In the simplest scenario, your pipeline takes input from an input repository, applies your transformation code, and outputs the result in the output repository. For example, you may have a set of images to apply a monochrome filter to and then output the result in an output repository that goes by the same name as the pipeline. This workflow performs only one operation and can be called a one-step pipeline, or one-step workflow. A diagram for such a pipeline would look like this:

Figure 3.1 – One-step workflow

The specification for this simple pipeline, in YAML format, would look like this:

---
pipeline:
  name: apply-photo-filter
transform:
  cmd:
  - python3
  - "/photo-filter.py"
  image: myregistry/filter
input:
  pfs:
    repo: photos
 ...