Understanding trigger options
In this recipe, we will understand various trigger options that are available in Spark Structured Streaming and learn under which scenarios a specific type of trigger option can be used. The trigger option for a streaming query identifies how quickly streaming data needs to be processed. It defines whether the streaming query needs to be processed in micro-batch mode or continuously. The following are the different types of triggers that are available:
- Default (when unspecified): New data is processed as soon as the current micro-batch completes. No interval is set in this option.
- Fixed Interval – micro-batch: We define a processing time that controls how often the micro-batches are executed. This is preferred in many use cases.
- One Time – micro-batch: This will execute as a micro-batch only once, process all the data that is available, and then stop. It can be used in scenarios where data arrives once every hour or so. ...
