By Kurt Seifried
We've all been there; in fact, many of us have been there recently. You wake up to find email from Bugtraq or Full-Disclosure, or perhaps a notification from your vendor about a security update - a security update that affects almost all your systems, and for which exploit code has been publicly available for several days [1]. Of course, I easily could launch into the power of open source and talk about how quickly the Linux kernel team fixed the vulnerability, allowing vendors to push out updates in a timely manner. Or I could talk about the importance of inventorying your systems and having a patch-management system in place so that they can be patched promptly. But the point is this: All systems are vulnerable, all the time. Let me repeat that just to make sure you don't think I made a mistake: All systems are vulnerable, all the time.
Security flaws are a lot like Schrödinger's cat: neither dead nor alive until you look [2] (Figure 1). Do security vulnerabilities appear out of the ether as if by magic when someone observes them, in effect collapsing the quantum probability state of the vulnerability in question to a known state? That is to say, as long as no entity recognizes the security vulnerability (and more importantly, as long as no one has exploit code for it), the vulnerability doesn't really exist. This also is known as the "tree falling in a forest" theory of information security.
Alternatively, one could argue that even though you don't know for sure when the isotope will decay, triggering the poison gas vial to release it, it will decay eventually, and probably sooner than you'd like! The reality is that a modern system has so much code installed and running that it is likely that some exploitable vulnerability will be found and used to compromise the system. So, what can you do?
Much like the old saying "plan for the worst but hope for the best," you should plan for failures in your security systems. All the standard advice about hardening systems, layering security, and limiting access and privileges where possible is still good advice, but it largely fails to answer the question: What do you do when your security fails?
Because it is almost inevitable that your systems will be vulnerable at some point - most likely for a period of time during which there is public access to exploit code - your system probably will be compromised. A perfect example of this is seifried.org. I had SELinux enabled, which prevents all manner of security vulnerabilities from being exploited but also prevents a lot of third-party, web-based software from working properly. After fighting SELinux for some time, I gave up and disabled it.
This was my first mistake for two reasons. First, by disabling SELinux, I made my system much easier to break into and also reduced the amount of logging that would occur if someone did break in. If you must turn SELinux off, at least use permissive mode, which logs violations.
My second mistake was classic administrative laziness. To ease system administration, I had established SSH trust relationships between my hosts and also enabled sudo access without having it prompt for a password on my machines.
In effect, once an attacker broke into a system and gained access to the administrative account, they would have access to all of my systems. Thus, when one of my systems was compromised, I ended up having to rebuild all of them because I was pretty sure they all had been modified. Often, this design flaw also occurs with firewalls and other security mechanisms: Organizations implement a hard shell of exterior security but fail to implement any internal security, leaving a soft chewy center for attackers to feast upon.
| Break-in Tip: MySQL |
|
One commonly used method to reset a lost MySQL root password is to use the --skip-grant-tables option, which loads the database with no protection at all. If you must use this method, use --skip_networking as well, to allow only local connections. Local users would still be able to connect, so make sure you boot all of them off, too. Perhaps a better method is to use the --init-file: "Read SQL statements from this file at startup. Each statement must be on a single line and should not include comments." So all you need to do is create a text (reset-password.txt) file with an SQL statement that will change root's password, SET PASSWORD FOR
'root'@'localhost' = PASSWORD('MyNewPassword');
and then start mysqld with the --init-file [filename] option and, once loaded, simply restart MySQL normally and log in with your new password. |
As I mentioned, I had disabled SELinux instead of leaving it in permissive mode. With hard-disk space relatively cheap now, it rarely makes sense to forego audit logs to save disk space on most systems.
Additionally, with the relative ease of setting up network logging for Syslog, there is no excuse not to centrally log events (thus making it much harder for an attacker to erase any trail they leave). However, note that many attacks will leave little if any meaningful messages in the logs.
Often, a message saying that a server program has crashed will be all that is left, which hardly warrants a full-scale investigation each time it occurs.
Having a trustworthy audit trail can mean the difference between needing to rebuild an entire group of machines and only needing to clean off one. Logging also is important in providing the information needed to fix a system. Simply restoring a system to its original state won't prevent the attacker from breaking in again.
Often backups are a neglected side of information security. Backups are crucial for ensuring that systems can be rebuilt properly and data restored if necessary. If you back up nothing else, at least back up your data and make sure to store it offsite. Also, having restoration systems that work is just as important.
Too often, I have seen backups work perfectly until they were needed, bare-metal recovery images that were incompatible with replacement hardware, or exports of data that were corrupted or missing data. For example, I have a backup script that was quietly failing for three months because I forgot the "overwrite older files" option, so all the changed files had not been backed up in some time.
One of the most powerful practices I've seen is conducting a "pre-mortem." Essentially, you sit down and assume the system has already failed. Having an idea of what to do in this scenario will help prevent confusion if it ever does happen and can offer insight into preemptive action to take, such as loading host-based firewalls on every computer, regardless of whether the machine is attached to a public network.
Much like a compromised system, it doesn't really matter if the cat is dead or alive. At some point, the poison gas will be released, you will end up with a dead cat, and eventually it will start stinking the place up.
The trick is to find effective ways to deal with failures so that you can be back up and running quickly and securely!
| INFO |
|
[1] SecurityFocus copy_from_user_mmap_sem() vulnerability: http://www.securityfocus.com/bid/27796
[2] Schrödinger's cat: http://en.wikipedia.org/wiki/Schr%C3%B6dinger's_cat |
| THE AUTHOR |
|
Kurt Seifried is an Information Security Consultant specializing in Linux and networks since 1996. He is married and has four cats but no fish (because the cats are more hungry than afraid of water). He often wonders how it is that technology works on a large scale but often fails on a small scale. |
