Looking at data source issues and challenges

Unfortunately, a lot of variabilities are involved in what data sources will be available and the quality of those data sources. We’ll touch on several of the causes of such variability and the challenges they present in the following subsections.

Completeness

The completeness of the data provided by a data source is based on the value of the attributes captured for any given event. We do not want to waste storage resources and bandwidth on data sources that won’t add value to our investigation due to the data they expose. For example, if a system provides logs showing a network connection was established but there are no details on the source/destination of the connection with contextless timestamps and ambiguous time zones, there is likely not much that can be used from that to develop a quality detection. As such, we either ignore or de-prioritize this data source.

As an additional note regarding completeness, some...