Case study on monitoring a Kinesis stream
The data engineering team has been using an AWS EC2 instance with a custom plugin to extract data generated from the Apache server. For some reason, they noticed that it is taking time for the KCL to receive information being sent to it. The instance is restarted and the KPL is restarted, and they notice that the problem persists. What should have been done or configured to ensure that the team is alerted when this sort of incident occurs?
Solution
When the Kinesis stream is created, head on over to the Amazon CloudWatch console and configure an alarm with a specific threshold for the Get records latency metric, to ensure that the team is alerted via an SNS email or SMS notification.