Summary

The concept of big data is continually expanding in conjunction with data engineering. We have spent time explaining what big data is and the different aspects of big data, as well as how these aspects intertwine, from the role of a data engineer who helps to sanitize data and make it more friendly and usable by business intelligence experts, to that of data scientists, machine learning engineers, data analysts, and more. Understanding the process of making this data clean is important and monitoring plays a key role in that. We also explained the concept of ETL, which is one of the fundamental concepts of data engineering. Then, we talked about the AWS tools that are used for ETL work: Kinesis for extraction, Glue for transformation, and S3, where the data is stored or loaded.

Lastly, we introduced one important aspect of monitoring in the AWS ecosystem, which is AWS CloudTrail. We then moved on to explain the importance of CloudTrail, how to use the CloudTrail console...