In a previous column, “Chain of Custody”, I wrote about some of the ways a piece of software could be compromised between the developer and the consumer. Here, I describe how developers can improve their practices in administering their development workstations to prevent compromise at the source.
I have a favorite saying: “If you are a systems administrator, you have the keys to the kingdom. If you are an open-source programmer, you don't know which or how many kingdoms you have the keys to.” We send our programs out into the world to be run by anyone for any purpose. Think about that: by anyone, for any purpose. Your code might be running in a nuclear reactor right now, or on a missile system or on a medical device, and no one told you. This is not conjecture; this is everyday reality. Case in point: the US Army installed gpsd on all armor (tanks, armored personnel carriers and up-armored Humvees) without telling its developers (esr.ibiblio.org/?p=3818).
This two-part series focuses on the needs of infrastructure software developers—that is, developers of anything that runs as root, has a security function, keeps the Internet as a whole working or is life-critical. Of course, one never knows where one's software will be run or under what circumstances, so feel free to follow this advice even if all you maintain is a toddler login manager.
Part I covers basic security concepts and hygiene: how to think about security needs and how to keep your development system in good shape to reduce the risk of major computing security mishaps. Part II, in a future issue of Linux Journal, will take things a step further and discuss what it takes to improve the security culture and practices of the projects you develop for.
This guide isn't going to teach you everything about security. It will give you an idea of what to do, but in many cases, you'll need to rely on man pages and other documentation to get the “how”. I did that both for brevity and to ensure that this article covers various Linux distributions equally and without becoming out of date in a matter of weeks.
I chose the controls here carefully. It is the set of controls that is consistently available across Linux distributions, realistic for developers to maintain even if they are developing open-source software as a side project and can't put many hours into it. It's maintainable without extensive training and has highest impact for the security of the software being developed. All of those things are judgment calls, and I welcome debate about them. The goal of this guide is not “ultimate security” or the fabled “uncrackable system”. It is to raise the bar for security hygiene among open-source infrastructure software developers significantly from where it is right now.
I'd love to find that, in a year from now, we're all much more secure and can iterate on our standards again. In my perfect world, I write this article every spring, we all up our game a notch, and the following spring, we are prepared to make the jobs of ransomware developers, spammers, oppressive governments, corporate spies and so on even harder than before.
I know programmers—being one myself—and programmers don't just want to be told that something works, we want to know why and how it works. Thus, before I introduce a checklist for the open-source developer, I'm beginning with some underlying security concepts.
CIA and Software Development:
No, not that CIA. Confidentiality, Integrity and Availability: these are the three goals of security. In the Open Source world, we usually are most concerned with integrity: is this the software the developer I trust made, and can I be sure it hasn't been tampered with? Availability typically comes second: can I get a copy of this software, and its documentation, when I need it? Confidentiality usually comes in last, as it applies only to a few parts of our practice: private keys and other credentials, vulnerabilities we still are working to patch and some sensitive intra-project communications. Even so, it is rare that those things need remain confidential indefinitely.
Risk:
“A ship is safe in harbor, but that's not what ships are for.”—William G.T. Shedd
I can make my laptop perfectly secure. I can do this by removing its battery, filling its ports with epoxy, tying it to some bricks, and then dropping it to the bottom of the Marianas Trench. There, in the deepest ocean chasm, it will be awfully hard to get to, and anyone who tries will find a hunk of pulverized metal and plastic that couldn't take the pressure or the salt water.
Of course, at that point, what good does the laptop do me or anyone else?
Computing involves risk. It always has involved risk, but never more so than now, when we are constantly connected and running systems so complex as to be virtually un-auditable. It is rare for me to work on a machine so small in scope that I could audit its every line of code in a lifetime, let alone before the next kernel patch comes out. When I do see such a machine, it is invariably a single-purpose, life-critical component maintained at great expense.
Still, the world doesn't seem to be ending (yet). This is because we do have the power to mitigate risk to manageable levels:
We can render a risk irrelevant (if I don't store credit-card numbers, I'm not at risk of a credit-card database breach).
We can transfer a risk to someone else (if I insure my laptop, the insurance company pays if it is stolen, not me).
We can lower the likelihood of a risk (if I never transmit or store passwords in the clear, it is less likely that they will be compromised).
We can lower the impact of a risk (if I use two-factor authentication, compromising a password alone does nothing).
A control is something that mitigates risk in information security.
Full-Disk Encryption:
“Full-disk encryption”, often abbreviated FDE, is something of a misnomer. It really means full partition encryption in most cases. Many distributions, such as Red Hat and Ubuntu, offer full-disk encryption as a check-box option at install time. Some others, such as Slackware and Gentoo, require some manual intervention in preparing the disks, but there is reasonably good documentation available.
Now that you know how easy it is, let's talk about why you do it: encrypting all of your storage (including swap!) protects you against theft of your powered-down or hibernated computer. If you are not using FDE, and your machine is lost or stolen, not only is your personal information compromised, but so are your code-signing keys, the SSH keys you use to check in code and access servers, and any information you may have on not-yet-patched vulnerabilities. An attacker could release patches that look as if they came from you, and most likely no one would be the wiser.
Off-line attacks on the passphrases of private keys can and do happen. Full-disk encryption makes it unlikely that the keys can be recovered and almost certain that you will have had time to revoke them in the meantime. It also gives your team plenty of time to patch and publish vulnerabilities before a thief can make use of any information that may have been on your machine.
Operating System:
Quite simply: if you need Windows, run it in a VM or on another machine. Do not dual-boot on your development machine. Windows is generally prone to collecting malware, and in a scenario where the machine boots directly to Windows (as opposed to running it in a virtual machine), Windows may have the opportunity to overwrite your motherboard's firmware with malicious software that would then impact your Linux system.
This isn't to say that Windows cannot be reasonably secured, but it's a large, hard-to-manage attack surface, especially for those of us who specialize in Linux and may keep a Windows system around only for something like gaming purposes.
Password Management:
Pick a decent password manager, and use it. Recycling passwords is not okay. Using weak passwords is not okay. No one can remember a large assortment of strong passwords. Save your brain by memorizing only what you have to.
Many Linux users ask me if a password manager is really safe. What if my password is held in RAM? What if my laptop is stolen? Full-disk encryption will protect your machine's contents, including your password management database, and most password management software has a layer of encryption of its own. I'm not talking about perfect security (and you may want to keep your 2–5 most valuable passwords in your head). I'm talking about making decent security manageable. A weak password out in the wild is far more open to attack than a password management database behind a password on an encrypted hard drive on your laptop.
I could write an article entirely about the pros and cons of various password managers, but the short version is this: any password manager that doesn't upload all your credentials to the cloud is better than not using a password manager.
Key Management:
How you handle your private encryption keys is paramount to their usefulness. If attackers get a copy of one of your keys—especially if two-factor authentication is not in play—they easily can brute-force its password off-line and use it to impersonate you. The NSA's or China's or EvilCorp's latest back door could go out under your signature. So, do the following:
Never store private keys on a system that does not have full-disk encryption.
Avoid creating passwordless private keys unless you understand the implications of doing so, and have another protection in place to prevent their abuse, such as encrypting the key with a key stored on a separate hardware token.
Keep a log of which keys you have in use where, so that if a key is compromised, you can ensure that it is revoked and replaced with a new key.
Use a different key on each of your machine. If one of your keys is compromised, once it is used, you know from which key it was which of your machines is in question. If all machines use the same key, you likely will have no idea where the compromise came from. Having multiple keys also is helpful when you are having someone else revoke a key for you. If I find that my laptop's key may be compromised, but I'm away at a conference, I can make a couple phone calls to get that key revoked in the two or three places where it could cause the most damage. Then, when I get home, I can log in via my desktop (which has its own key) to place a new public key for my laptop in the appropriate places.
For GPG keys and other systems that allow it, create a revocation certificate for each key and have a friend store those certificates in case of emergency. This way, even if you have a major compromise or lose access to your private key, your friend can mark that key as compromised and no longer valid. This does not give your friend the ability to impersonate you, only to revoke your keys.
Backups:
Your backups should be protected as well as you protect your primary systems, otherwise they are as much a liability as they are an asset. Backups always should be encrypted, and it is especially important that if you back up to a cloud service, you have configured your backup system such that the relevant encryption keys reside with you locally and never are shared with the service storing the encrypted data. Be sure to keep a backup of your keys or passwords for your backup data store somewhere safe and separate from that data store, such as in a fireproof safe, if they are not memorized.
Multifactor Authentication:
Quite simply, multifactor authentication is your friend. Use it wherever it is available. In most cases, this is either a password or an SSH key combined with a second authentication factor, such as a hardware token, a soft token on your phone that provides a time-sensitive shortcode or the service's ability to send you an SMS or other out-of-band confirmation.
Passwords are fairly easy to compromise. SSH keys are less so, but at this point, two-factor (or more) authentication has become so easy and accessible, there is no excuse not to use it.
GitHub, Google and many of the other services we use regularly offer two-factor authentication to help protect our accounts, and several different options exist for employing it on our own infrastructure as well.
Configuration Management:
If you have a complex enough system that the overhead is worth it, consider a configuration management system, such as Ansible or Puppet. For most of us, with respect to our individual development machines, this will not be the case. A much simpler and lighter solution is to install and enable etckeeper, which will keep a revision history of your system configuration in a git or hg repository, automatically updating it on package manager events. You can trigger updates manually when editing configuration files yourself.
Although etckeeper doesn't give you the centralized management features of a true configuration management system, when combined with good backups, it provides the feature most important to the security of a single-machine setup: auditability of configuration. That audit trail can be invaluable when things go wrong.
Updates:
Keeping up with security updates should be a no-brainer, but many developers simply become lazy or avoid updates because they are afraid they'll have to resolve some new conflict or failure as a result.
However, doing critical development work on a machine that is a week or more behind on security patches is simply taking a massive risk on behalf of every user of your software for the sake of some convenience. You have a package manager; use it.
Firewall:
If your machine is a desktop that will live behind a hardware firewall, you may not need to run a firewall on the machine itself. However, if you do not have a hardware firewall protecting your machine, or if it is mobile (a laptop you travel with, or a desktop if you ever bring it to hackfests or LAN parties), it needs to be firewalled.
Almost every Linux firewall is a wrapper around iptables, which is good, because iptables is fast, powerful and reliable. Which wrapper you use doesn't matter terribly much. I have been using ipkungfu for a while now to avoid having to hand-compose iptables rules on my laptop, but other options are just as good.
SSH:
You may want to ssh between your development machines for a variety of reasons: to transfer files, to check just one setting or to use a build environment home from a laptop abroad. There are many good guides to running SSH servers, but here are some general tips for your first-pass check over sshd configuration:
Never run sshd on your development machine unless you really need to and don't have it start on boot. A development machine is a machine you are usually sitting at, so running sshd when not in use is a needless risk, and it's easy to start it when it is needed.
Never allow ssh in as root. If you need root access, you can ssh in as a regular user and then use su or sudo to gain root privileges. This helps to ensure that a single compromised key or password will not give an attacker root privileges.
Never allow password-only SSH. Either require key-based authorization or a password plus a second factor, such as a one-time password from a soft token application.
Check logs or run a monitoring script to stay informed about attempts to brute-force your sshd.
Isolating Users and Services:
One simple way to increase your system's security is to isolate various services from one another using system accounts. Some distributions do this by default most of the time, but you should check that your machine is doing it wherever practical:
Do not use root, or a user that can exercise root privileges without a password (for example, via passwordless sudo) for everyday tasks. Always use the least system access necessary for any given task.
Do not run every service on your machine as root. In general, never run anything as root unless you must. Developers frequently run local instances of a number of services for testing and development purposes. These should be isolated to their own runs-as users to help contain any exposure that one service causes. Apache may run as “apache” or “www-data”; a git server may run as “git” or “gitolite”; a mail service may run as “mail”. Whatever the names are, it is important that services are separated.
Never allow two users to share the same system account. System accounts are free; make more of them as needed.
What Not to Use on a Development Machine:
If you must run Adobe Flash (try not to run it at all), enable it in only one browser, and do everything other than the one thing you tolerate Flash for in a different browser. Better yet: run Flash only within a VM dedicated to that purpose, if you can. Flash is closed-source, riddled with holes and simply can't be secured.
Do not run an FTP service. FTP is incredibly insecure, as is FTPS. This is not to be confused with SFTP, which serves a similar function but operates over the more trustworthy SSH protocol.
Disable all FireWire and Lightning ports on your machine, in the BIOS or UEFI firmware if possible, otherwise by physically disconnecting them or filling them with epoxy. FireWire and Lightning use direct memory access, meaning that an attacker connecting to one of these ports while you are at a conference and looking away for a moment (even with your computer's screen locked) can dump the contents of your RAM (or change it) as long as your machine is powered up.
