How to identify credible cyber threat intelligence

I’m going to give you some guidance on how to identify good CTI versus the questionable threat intelligence I see so often in the industry today. After publishing one of the industry’s best threat intelligence reports for the better part of a decade (OK, I admit I’m biased), I learned a few things along the way that I’ll share with you here. The theme of this guidance is to understand the methodology that your threat intelligence vendors use. If they don’t tell you what their methodology is, then you can’t trust their data, period. Additionally, the only way you’ll be able to truly understand if or how specific threat intelligence can help your organization is to understand its data sources, as well as the methodology used to collect and report the data; without this context, threat intelligence can be distracting and the opposite of helpful.

Data sources

Always understand the sources of CTI data that you are using and how the vendors involved are interpreting the data. If the source of data is unknown or the vendors won’t share the source of the data, then you simply cannot trust it and the interpretations based on it. For example, a vendor claims that 85% of all systems have been successfully infected by a particular family of malware. But when you dig into the source of the data used to make this claim, it turns out that 85% of systems that used the vendor’s online malware cleaner website were infected with the malware referenced. Notice that “85% of all systems” is a dramatic extrapolation from “85% of all systems that used their online tool.”

Additionally, the online tool is only offered in US English, meaning it’s less likely that consumers who don’t speak English will use it, even if they know it exists. Finally, you discover that the vendor’s desktop anti-virus detection tool refers users to the online tool to get disinfected when it finds systems to be infected with the threat. The vendor does this to drive awareness that their super-great online tool is available to their customers. This skews the data as 100% of users referred to the online tool from the desktop anti-virus tool were already known to be infected with that threat. I can’t count how many times I’ve seen stunts like this over the years.

Always dive deep into the data sources to understand what the data actually means to you. The more familiar you are with the data sources, the easier it will be for you to determine the true value of that data to your organization. In Chapter 4, The Evolution of Malware, I spend a lot of time describing the intricacies of the sources of data used in that chapter. This is the only way to understand the picture the data is providing, relative to your organization and the risks it cares about.

For example, if you work at a public sector organization in Japan, how valuable is CTI to you that focuses on a specific industry vertical in the private sector in the United States? The answer is you don’t know until you understand the sources of data and what they might mean to your organization.

Specificity is your friend in this context. Understanding where the data was collected from and how, the limitations of the data sources, and the underlying assumptions and biases present while processing the data are all key to understanding how the resulting CTI might help your organization. CTI is a lot less credible without the context that allows you to understand it. Purveyors of credible CTI are happy to provide this context to you. However, they might not volunteer this information and you might need to request it. Providing such information tends to highlight the limitations of the CTI and the CTI provider’s capabilities. Also, I’ve found that not everyone is a connoisseur of the finer points of CTI; being prepared to ask your own questions is typically the best way to get the context you need to truly understand CTI.

Time periods

When consuming threat intelligence, understanding the time scale and time periods of the data is super important. Are the data and insights provided from a period of days, weeks, months, quarters, or years? The answer to this question will help provide the context required to understand the intelligence. The events of a few days will potentially have a much different meaning to your organization than a long-term trend over a period of years.

Anomalies will typically warrant a different risk treatment than established patterns. Additionally, the conclusions that can be made from CTI data can be dramatically altered based on the time periods the vendor uses in their report.

Let me provide you with an example scenario. Let’s say a vendor is reporting on how many vulnerabilities were exploited in their products for a given period. If the data is reported in regular sequential periods of time, such as quarterly, the trend looks really bad as large increases are evident.

But instead of reporting the trend using sequential quarterly periods, the trend looks much better when comparing the current quarter to the same quarter last year; there could actually be a decrease in the exploitation of vulnerabilities in the current quarter versus the same quarter last year. This puts a positive light on the vendor, despite an increase in the exploitation of vulnerabilities in their products quarter over quarter.

Another potential red flag is when you see a vendor report data that isn’t for a normal period of time, such as monthly, quarterly, or annually. Instead, they use a period of months that seems a little random. If the time period is irregular or the reason it’s used isn’t obvious, the rationale should be documented with the CTI. If it’s not, ask the vendor why they picked the time periods they picked. Sometimes, you’ll find vendors use a specific time period because it makes their story more dramatic, garnering more attention, if that’s their agenda. Alternatively, the period selected might help downplay bad news by minimizing changes in the data.

Understanding why the data is being reported in specific time scales and periods will give you some idea about the credibility of the data, as well as the agenda of the vendor providing it to you.

Recognizing hype

One of the biggest mistakes I’ve seen organizations make when consuming CTI is accepting their vendor’s claims about the scope, applicability, and relevance of their data. For example, a CTI vendor publishes data that claims 100% of attacks in a specific time period involved social engineering or exploited a specific vulnerability. The problem with such claims is that no one in the world can see 100% of all attacks, period.

They’d have to be omniscient to see all attacks occurring everywhere in the world simultaneously, on all operating systems and cloud platforms, in all browsers and applications. Similarly, claims such as 60% of all attacks were perpetrated by a specific APT group are not helpful. Unless they have knowledge of 100% of attacks, they can’t credibly make claims about the characteristics of 60% of them. A claim about the characteristics of all attacks or a subset that requires knowledge of all attacks, even when referencing specific time periods, specific locations, and specific attack vectors, simply isn’t possible or credible. A good litmus test for CTI is to ask yourself, does the vendor have to be omniscient to make this claim? This is where understanding the data sources and the time periods will help you cut through the hype and derive any value the intelligence might have.

Many times, the vendor publishing the data doesn’t make such claims directly in their threat intelligence reports, but the way new intelligence is reported in the headlines is generalized or made more dramatic in order to draw attention to it. Don’t blame CTI vendors for the way the news is reported, as this is typically beyond their control. But if they make such claims directly, recognize them and adjust the context in your mind appropriately. For many years, I made headlines around the world regularly speaking and writing about threats, but we were always very careful not to overstep the mark from conclusions supported by the data. To make bolder claims would have required omniscience and omnipotence.

Predictions about the future

I’m sure you’ve seen some vendors make predictions about what’s going to happen in the threat landscape in the future. One trick that some CTI vendors have used is again related to time periods. Let’s say I’m publishing a threat intelligence report about the last 6-month period covering January through June. By the time the data for this period is collected and the report is written and published, a month or two might have gone by. Now we are in September. If I make a prediction about the future in this report, I have two months of data from July and August that tell me what’s been happening since the end of the reporting period.

If my prediction is based on what the data tells us already happened in July and August, readers of the report will be led to believe that I actually predicted the future accurately, thus reinforcing the idea that we know more about the threat landscape than anyone else. Understanding when the prediction was made relative to the time period it was focused on will help you decide how credible the prediction and results are, and how trustworthy the vendor making the prediction is. Remember, predictions about the future are guesses – what happened in the past does not define what can happen in the future.

Vendors’ motives

Trust is a combination of credibility and character. You can use both to decide how trustworthy your vendors are. Transparency around CTI data sources, time scales, time periods, and predictions about the future can help vendors prove they are credible. Their motives communicate something about their character. Do they want to build a relationship with your organization as a trusted advisor or is their interest limited to a financial transaction? There’s a place for both types of vendors when building a cybersecurity program, but knowing which vendors fall into each category can be helpful, especially during incident response-related activities, when the pressure is on. Knowing who you can rely on for real help when you need it is important.

Those are some of the insights I can offer you from 10 years of publishing threat intelligence reports. Again, the big takeaway here is understanding the methodology and data sources of the CTI you consume - this context is not optional. One final word of advice: do not consume threat intelligence that doesn’t meet this criterion. There is too much fear, uncertainty, doubt, and complexity in the IT industry. You need to be selective about who you take advice from.

I hope you enjoyed this chapter. Over the last few years, the CTI industry has exploded. Finding credible sources of CTI shouldn’t be a challenge for well-funded cybersecurity programs. CTI is being integrated into cybersecurity products and services more and more, which means protecting, detecting, and responding to threats should be easier and faster than ever. However, I have to wonder if this is true, how are attackers being more successful now than ever before? There are many historical examples that teach us that threat intelligence isn’t sufficient by itself to mitigate attacks - defenders need to be willing to act on the intelligence they have and need the capabilities to do so effectively. Despite the proliferation of CTI, organizations still need effective cybersecurity strategies to be successful. In order to develop an effective strategy for your organization, it is helpful to first understand the types of threats you face and how they operate. This is the theme of the next three chapters of this book.