Structured Streaming concepts
To understand Structured Streaming, it’s important for us to understand the different operations that take place in a near-real-time scenario when data arrives. We will understand them in the following section.
Event time and processing time
In Structured Streaming, there are two important notions of time – event time and processing time:
- Event time: Event time refers to the time when an event occurred or was generated. It is typically embedded within the data itself, representing the timestamp or a field indicating when the event occurred in the real world. Event time is crucial for analyzing data based on its temporal order or performing window-based computations.
- Processing time: Processing time, on the other hand, refers to the time when an event is processed by the streaming application. It is determined by the system clock or the time at which the event is ingested by the processing engine. Processing time is useful...
