Standard encryption algorithms such as symmetric DES and AES or asymmetric RSA are widely used by malware authors. However, the vast majority of samples that include these algorithms never implement these algorithms themselves or copy their code into their malware. They are mainly implemented using core Windows APIs or through a third-party library, such as OpenSSL.
These algorithms are mathematically more complicated than simple encryption algorithms or RC4. You don't need to understand their mathematical background to understand how they are implemented—you only need to understand how to identify the way any of these algorithms can be used and how to figure out the exact algorithm used, the encryption/decryption key(s), and the data.
