Responding to security incidents (Advanced)

It will happen. Large market share along with insufficient security practices and attack persistency will eventually result in a security breach, especially if attackers have incentives. If you operate a low-profile CMS, most likely you will be defaced. If you operate something for profit in nature, attackers might go for passwords or sensitive data. In other cases, your server may be compromised and used as a platform for other attacks, such as DDoS botnets.

You will notice defacements, strange files lying around your filesystem, processes that you know nothing about running, and so on. Those are clear signs that something is going on; although, there might be others, and for a while, there might be none making operations even more important.

Getting ready

There are two schools for responding to a security incident: shutting the power off or not. Among other things, it depends on whether you or your company prosecute violators. Although not everyone wants to go through a legal process, learning can be a very important driver of your very own internal forensic effort.

Shutting the power off, and then cloning the disk for a forensic analysis is a common approach in those cases; although, not powering it off might help gather important live information. Also, shutting it off for forensic analysis means your time-to-recovery will be significantly higher.

How to do it…

The following is a non-authoritative, non-specific list of suggestions to respond to a security incident:

1. Don't unplug the network cable. Advanced malware, especially those installed as a result of Advanced Persistent Threats (APTs, not to be confused with the packaging systems), can inflict damage when the network goes down.

2. If you decide to shut it off, pull the energy cord. This is not a clean shutdown, and you will want to take care of any critical pending operations first. Take the disk(s) out and replicate them using read-only access, ideally using a forensics duplicator. If you don't have the hardware or the knowledge, and you want to prosecute, take it to forensic professionals.

3. Recover your server as specified in Restore your environment; however, after that, you need to find and patch whichever vulnerability led to the intrusion in the first place!

4. Examination of system logs is essential, although they might have been cleared. On a web application, you'd like to take a look at the web server logs (/var/log/apache2/*, /var/log/nginx/*), especially for any error lines that might be indicative of an application vulnerability leveraged to inject code or provoke an intrusion. Also, you want to take a look at any seemingly random garbage or erroneous requests that can evidence someone is trying to exploit an error and /var/log/auth.log containing authentication activity (SSH logins, and so on). In addition to this, you should also look at the /var/log/wtmp, /var/log/lastlog and /var/log/btmp files that can be read with the last and lastlog commands and might contain important authentication information. Database logs should also be reviewed and users (in some cases, regulatory organizations) notified of privacy breaches. Also, logs of products such as RKHunter might be helpful.

5. Finally, for critically modified information such as defaced files, you'd want to check the MAC times. When you use –l, you get the modification time of a file but by adding –c, you can check the creation time and with –u, the access (use) time. Specialized forensic toolkits, such as The Coroner's Toolkit, will help you detect mismatches and timelines if necessary.

6. In most cases, if you're using a CMS or some other prepackaged web software, you can upgrade your software, themes, and modules (especially third party modules) and check on specialized mailing lists (such as full disclosure) if there are any known security vulnerabilities. Upgrading your operating system to patch any potential vulnerability is also advised.