Amazon Kinesis Data Firehose (KDF)
Amazon KDF was launched in October 2015. It is a fully managed, serverless service for ingesting streaming data and delivering to destinations in AWS, third-party services such as Splunk, or even generic HTTP endpoints. In terms of the five core stages of enabling real-time analytics, Amazon KDF straddles stream storage and real-time stream processing. Some of the core capabilities of Amazon KDF are as follows:
- Ingesting data at high volumes
- Ingesting high-throughput streaming data from myriad data sources
- Buffering and aggregating data
- Transforming and processing data inline
- Sending data to one of a number of destinations
- Handling errors and retries while sending
- Storing ingested data in the service for 24 hours, to enable retries and handle situations when destinations are unavailable
When Amazon KDS first launched, the majority of organizations used the service to ingest streaming data and store it in Amazon...
