In this chapter, we will be using the following:

• YARA
• pefile
• PyGitHub
• Cuckoo Sandbox
• Natural Language Toolkit (NLTK)
• imbalanced-learn

The code and datasets can be found at https://github.com/PacktPublishing/Machine-Learning-for-Cybersecurity-Cookbook/tree/master/Chapter02.

In static analysis, we examine a sample without executing it. The amount of information that can be obtained this way is large, ranging from something as simple as the name of the file to the more complex, such as specialized YARA signatures. We will be covering a selection of the large variety of features you could obtain by statically analyzing a sample. Despite its power and convenience, static analysis is no silver bullet, mainly because software can be obfuscated. For this reason, we will be employing dynamic analysis and other techniques in later chapters.

Without delving into the intricacies of hashing, a hash is essentially a short and unique string signature....

Unlike static analysis, dynamic analysis is a malware analysis technique in which the expert executes the sample, and then studies the sample's behavior as it is being run. The main advantage of dynamic analysis over static is that it allows you to bypass obfuscation by simply observing how a sample behaves, rather than trying to decipher the sample's contents and behavior. Since malware is intrinsically unsafe, researchers resort to executing samples in a virtual machine (VM). This is called sandboxing.

One of the most prominent tools for automating the analysis of samples in a VM is Cuckoo Sandbox. The initial installation of Cuckoo Sandbox is straightforward; simply run the...

One of the techniques hackers use to sneak their malicious files into security systems is to obfuscate their file types. For example, a (malicious) PowerShell script is expected to have an extension, .ps1. A system administrator can aim to combat the execution of all PowerShell scripts on a system by preventing the execution of all files with the .ps1 extension. However, the mischievous hacker can remove or change the extension, rendering the file's identity a mystery. Only by examining the contents of the file can it then be distinguished from an ordinary text file. For practical reasons, it is not possible for humans to examine all text files on a system. Consequently, it is expedient to resort to automated methods. In this chapter, we will demonstrate how you can use machine learning to detect the file type...

To check whether two files are identical, we utilize standard cryptographic hash functions, such as SHA256 and MD5. However, at times, we would like to also know to what extent two files are similar. For that purpose, we utilize similarity hashing algorithms. The one we will be demonstrating here is ssdeep.

First, let's see how to use ssdeep to compare two strings. This can be useful to detect tampering in a text or script and also plagiarism.

Now, we are going to see how to apply ssdeep to measure the similarity between two binary files. The applications of this concept are many, but one in particular is using the similarity measure as a distance in clustering.

In standard quantitative analysis of text, N-grams are sequences of N tokens (for example, words or characters). For instance, given the text The quick brown fox jumped over the lazy dog, if our tokens are words, then the 1-grams are the, quick, brown, fox, jumped, over, the, lazy, and dog. The 2-grams are the quick, quick brown, brown fox, and so on. The 3-grams are the quick brown, quick brown fox, brown fox jumped, and so on. Just like the local statistics of the text allowed us to build a Markov chain to perform statistical predictions and text generation from a corpus, N-grams allow us to model the local statistical properties of our corpus. Our ultimate goal is to utilize the counts of N-grams to help us predict whether a sample is malicious or benign. In this recipe, we demonstrate how to extract N-gram counts from a sample.

The number of different N-grams grows exponentially in N. Even for a fixed tiny N, such as N=3, there are 256x256x256=16,777,216 possible N-grams. This means that the number of N-grams features is impracticably large. Consequently, we must select a smaller subset of N-grams that will be of most value to our classifiers. In this section, we show three different methods for selecting the topmost informative N-grams.

In this section, we will see how to put together the recipes we discussed in prior sections to build a malware detector. Our malware detector will take in both features extracted from the PE header as well as features derived from N-grams.

Often in applying machine learning to cybersecurity, we are faced with highly imbalanced datasets. For instance, it may be much easier to access a large collection of benign samples than it is to collect malicious samples. Conversely, you may be working at an enterprise that, for legal reasons, is prohibited from saving benign samples. In either case, your dataset will be highly skewed toward one class. As a consequence, naive machine learning aimed at maximizing accuracy will result in a classifier that predicts almost all samples as coming from the overrepresented class. There are several techniques that can be used to tackle the challenge of class imbalance.

In many situations in machine learning, one type of error may be more important than another. For example, in a multilayered defense system, it may make sense to require a layer to have a low false alarm (low false positive) rate, at the cost of some detection rate. In this section, we provide a recipe for ensuring that the FPR does not exceed a desired limit by using thresholding.