Building end-to-end workflows with AWS Step Functions
AWS Step Functions let you define and run workflows based on state machines (https://aws.amazon.com/step-functions/). A state machine is a combination of steps, which can be sequential, parallel, or conditional. Each step receives an input from its predecessor, performs an operation, and passes the output to its successor. Step Functions are integrated with many AWS services, such as Amazon SageMaker, AWS Lambda, container services, Amazon DynamoDB, Amazon EMR, AWS Glue, and more.
State machines can be defined using JSON and the Amazon States Language, and you can visualize them in the service console. State machine execution is fully managed, so you don't need to provision any infrastructure to run.
When it comes to SageMaker, Step Functions has a dedicated Python SDK, oddly named the Data Science SDK (https://github.com/aws/aws-step-functions-data-science-sdk-python).
Let's run an example where we automate training...