Let's Automate Let's Encrypt

HTTPS is a small island of security in this insecure world, and in this day and age, there is absolutely no reason not to have it on every Web site you host. Up until last year, there was just a single last excuse: purchasing certificates was kind of pricey. That probably was not a big deal for enterprises; however, if you routinely host a dozen Web sites, each with multiple subdomains, and have to pay for each certificate out of your own dear pocket—well, that quickly could become a burden.

Now you have no more excuses. Enter Let's Encrypt (https://letsencrypt.org), a free Certificate Authority that officially left Beta status in April 2016.

Aside from being totally free, there is another special thing about Let's Encrypt certificates: they don't last long. Currently all certificates issued by Let's Encrypt are valid for only 90 days, and you should expect that someday this term will become even shorter (https://letsencrypt.org/2015/11/09/why-90-days.html). Although this short lifespan definitely creates a much higher level of security, many people consider it as an inconvenience, and I've seen people going back from using Let's Encrypt to buying certificates from commercial certificate authorities for this very reason.

Of course, if you are running multiple Web sites, having to renew several certificates manually every three months quickly could become annoying to say the least. Some day you even may forget (and you will regret that forgetfulness). Let's leave routines to computers, right?

If you are using Apache under a Debian-based distribution, Let's Encrypt already has you covered with the libaugeas0 package, and it is capable of both issuing and renewing certificates. If, like me, you prefer nginx and want to have zero-downtime automatic certificate updates with industrial-grade encryption, keep reading. I'm going to show you how to get there.

First things first—some assumptions and requirements:

Now, check whether your nginx supports TLS:

sudo nginx -V

Usually it is supported by default and should yield the following:

TLS SNI support enabled

You also need a place to store certificates:

sudo mkdir -m 755 /etc/letsencrypt

Don't sweat the permissions for this directory; the certificates themselves will not be publicly accessible. Now you need to make a small change in your nginx configuration. Create a new file /etc/nginx/letsencrypt.inc with the following contents:

location ^~ /.well-known/acme-challenge/ {
    root /tmp/letsencrypt/www;
    break;
}

Then find your “server” section in the nginx configuration, and add the following line to each Web site you host:

include /etc/nginx/letsencrypt.inc;

So the final result will look like this:

server {
    listen 80;
    server_name example.com www.example.com;
    ...
        include /etc/nginx/letsencrypt.inc;
        ...
}

After saving both files, ask nginx to reload the configuration:

sudo /usr/sbin/nginx -t && sudo service nginx reload

Notice that you are only reloading the nginx configuration—and nginx knows very well how to do it without dropping connections.

Now, let's go get some certificates! Needless to say, all domain names for which you are going to issue certificates should resolve to your server IP address; otherwise, it would be possible to issue certificates for somebody else's domain and use those certificates for man-in-the-middle attacks.

The following will pull and start a new Docker image with the official Let's Encrypt client:

mkdir -p /tmp/letsencrypt/www

# make sure you have the latest version of this image,
# and not some pre-beta - those used to be notoriously buggy
docker pull quay.io/letsencrypt/letsencrypt:latest

docker run --rm -it --name letsencrypt \
-v /etc/letsencrypt:/etc/letsencrypt \                                                                                                                   
-v /tmp/letsencrypt/www:/var/www \
    quay.io/letsencrypt/letsencrypt:latest \
    auth --authenticator webroot \
    --webroot-path /var/www \
    --domain=example.com --domain=www.example.com \
    --email=admin@example.com

As you can see, you share two data volumes between the host and the container:

The webroot plugin that runs inside the container will create a temporary challenge file for each of your domains, then Let's Encrypt validation servers will send an HTTP request to ensure that you are really controlling this domain and this server. These files are temporary and needed only during issuing or renewing a certificate.

You will need to agree on TOS by pressing a button, and after several seconds, your certificate is ready. If you have several subdomains, as in this example, you can enumerate all of them, which will result in one shared certificate issued for all of these subdomains. However, if you have several domains, it would be much more convenient to have a separate certificate for each of them—just repeat this last docker run ... command for each domain you have (and thank me later if someday you decide to move one of your domains to a different server).

As you can see, the procedure for obtaining certificates is painless and safe. Almost all the heavy work is done for you behind the scenes, and if you've ever had to deal with certificates using some other traditional certification authority, you will know exactly what I mean. Whatever runs inside the container can access only two directories on the server, and only while it runs.

After you get all the certificates, it's safe to remove the temporary directory:

rm -rf /tmp/letsencrypt

Let's go back to the nginx configuration. Getting an A+ grade from SSLLabs requires some additional effort. Create a new Ephemeral Diffie-Hellman prime (if this is the first time you've see this term, see https://wiki.openssl.org/index.php/Diffie_Hellman for more information):

sudo openssl dhparam -out /etc/pki/tls/private/dhparam.pem 4096

Caution: if you absolutely need to support ancient versions of client software, for example, Java 6 clients, you need to skip this step and comment the ssl_dhparam line in the following step. These old clients do not support Diffie-Hellman parameters longer than 1024 bytes, so you need to make a choice between supporting those clients and security.

Now, have a hot beverage; it will take some time to generate. Add these lines to the “http” section of /etc/nginx/nginx.conf:

http {
        ...
        ssl_dhparam /etc/pki/tls/private/dhparam.pem;
        ssl_session_cache shared:SSL:10m;
        ssl_session_timeout 60m;
        ...
}

Create a new file /etc/nginx/ssl_options.inc:

ssl on;
ssl_prefer_server_ciphers on;
ssl_protocols TLSv1 TLSv1.1 TLSv1.2;
    ssl_ciphers "ECDH+AESGCM DH+AESGCM ECDH+AES256 DH+AES256
     ↪ECDH+AES128 DH+AES ECDH+3DES DH+3DES RSA+AESGCM 
     ↪RSA+AES RSA+3DES !aNULL !MD5 !DSS";
# Enable HSTS (HTTP Strict Transport Security) for half a year
add_header Strict-Transport-Security 
 ↪"max-age=15768000;includeSubDomains";

And create a new “server” section:

server {
    listen 443;
    server_name example.com www.example.com;

    include /etc/nginx/letsencrypt.inc;
include /etc/nginx/ssl_options.inc;

ssl_certificate /etc/letsencrypt/live/example.com/fullchain.pem;
ssl_certificate_key /etc/letsencrypt/live/example.com/privkey.pem;

        # enable OCSP stapling to speed up first connect
    ssl_stapling on;
    ssl_stapling_verify on;
    ssl_trusted_certificate 
     ↪/etc/letsencrypt/live/example.com/chain.pem;

...
}

Warning: the Strict-Transport-Security header will tell each visitor that you promise always to use HTTPS in the future. It's a one-way street, and once you set it, there is no way back—your visitor's browser will remember your promise and insist on having HTTPS. Also note: for more information on OCSP stapling, see https://en.wikipedia.org/wiki/OCSP_stapling.

After making all of these changes, reload the nginx configuration again:

sudo /usr/sbin/nginx -t && sudo service nginx reload

At this point, your Web site should have HTTPS up and running. Try to open https://www.example.com/ in a browser and enjoy the green lock sign in the address line. To verify the quality of encryption, go to https://www.ssllabs.com/ssltest, and submit your hostname for a check (usually it takes several minutes).

So, now that you have HTTPS, how about disabling HTTP? Go back to the HTTP “server” section and make the following improvement:

server {
    listen 80;
    server_name example.com www.example.com;
        include /etc/nginx/letsencrypt.inc;
        ...
if ($scheme = "http") {
        rewrite ^/(.*)$ https://$host/$1 permanent;
}
        ...
}

This will redirect all traffic from HTTP to HTTPS, automatically bringing all clients to the secure version of your Web site. Reload the nginx configuration to activate the changes.

Now it's time to automate certificate renewals. Let's Encrypt's current policy allows you to request five certificate renewals for a domain within seven days. That means it wouldn't be wise (and wouldn't make much sense either) to try to renew certificates every day. On the other hand, leaving it for the last moment before expiration also is quite dangerous. Luckily, there is an easy way to renew these certificates only when they have less than 30 days before expiration. To me, 30 days sounds just right. That means my certificates will be reissued every 60 days on average, and if something fails afterward, I will have a whole month to fix whatever is broken.

Create a script for renewal (I placed it in/root/update_keys.sh) with these contents:

#!/bin/bash

mkdir -p /tmp/letsencrypt/www

ADMIN_EMAIL=admin@example.com
HOSTNAME=$(hostname)

OUTPUT="$((docker run --rm -i --name letsencrypt \
    -v /etc/letsencrypt:/etc/letsencrypt \
    -v /tmp/letsencrypt/www:/var/www \
    quay.io/letsencrypt/letsencrypt:latest renew) 2>&1)"

if [[ $? -eq 0 ]]; then
    echo "${OUTPUT}" | grep -q "No renewals were attempted"
    if [[ $? -eq 0 ]]; then
        # all certificates have more than 30 days left - 
        # nothing to do
        exit 0
    fi
    echo "${OUTPUT}" | tr -Cd '[:print:]\n' \
        | mail -s "${HOSTNAME}: Let's Encrypt keys renewal -
         ↪success" "${ADMIN_EMAIL}"
    else
        echo "${OUTPUT}" | tr -Cd '[:print:]\n' \
            | mail -s "${HOSTNAME}: Let's Encrypt keys renewal -
             ↪failed, exit code $?!" "${ADMIN_EMAIL}"
        exit 1
    fi

    # test config, reload if successful
    /usr/sbin/nginx -t &> /dev/null 
    if [[ $? -ne 0 ]]; then
        echo 'please fix configfile problem' \
            | mail -s "${HOSTNAME}: nginx unable to reload"
             ↪"${ADMIN_EMAIL}"
        logger "nginx has errors - not reloaded"
    else
        service nginx reload
        logger "nginx reloaded"
    fi

    rm -rf /tmp/letsencrypt

Remember to assign proper access rights:

sudo chmod u+x /root/update_keys.sh

And create a crontab entry:

sudo crontab -e

with a line like this:

17 2 * * * /root/update_keys.sh

That will trigger execution of this update script at 2:17 every day. The update script will check whether your certificates have more than 30 days left, and if they don't, it will attempt to renew all expiring certificates. Are you wondering why I used 2:17 am? Well, there is a simple explanation for that: almost everybody else did not. Most people, when creating cron jobs, use some simple value like 1:00 am, 2:00 am, 3:30 am, 4:15 pm and so on, and that is a really, really bad choice if your cron job is supposed to talk to an external service, because that means the service will experience maximum loads every once in a while. It is bad for the service, and it is not good for you; the chance of getting a timeout is significantly higher if you send a request during these peak loads.

So, for this job, please, please do not use an even value, and do not use my value; use some random value instead, and everything will be fine.

As you can see, Let's Encrypt managed to make the full automation of certificate maintenance possible. If you are using it right, it just works—and it's free.