Strong TLS for Zotonic using nginx and Let's Encrypt

This gets more interesting. I'm running on SmartOS with no front-end web server. In this article I will show the steps I took to introduce TLS to my blog.

The first thing I did was research Zotonic's support for TLS. It isn't great and it's complicated to get right, and that's more Erlang's fault than Zotonic's. I immediately pivoted to using a front-end web server. I have historically used Apache https, but I know it isn't the best for resource and performance reasons. I chose nginx based on the successful experience of Allele Dev in moving her https://queertypes.com/ blog to HTTPS using Let's Encrypt!.

The next thing I did was set up nginx on my laptop as a proxy to my real blog and altered my /etc/hosts file to point alainodea.com and blog.alainodea.com to 127.0.0.1. This allowed me to locally experiment with basic nginx configuration.

Once I had the basics of HTTP configuration down in nginx I looked into TLS configuration and set up a self-signed certificate. I referred to https://cipherli.st/ to get a set of vetted strong TLS configuration settings for nginx. This includes restricting the cipher suites to only those that can support perfect forward secrecy, disabling dangerous backward compatibility (like SSL and TLS 1.0 support), OCSP stapling, and HSTS, and requiring 128 bit or better symmetric keys.

Then I validated the TLS configuration (aside from CA certification) using Chrome. Once that was done I created a new SmartMachine base infrastructure container to run it properly. I configured nginx exactly how it was configured on my laptop (literally copied nginx.conf). I changed /etc/hosts to point alainodea.com and blog.alainodea.com to the SmartMachine and tested in Chrome again. All good. Then I switched nginx.conf to port 80 and 443 for HTTP and HTTPS to make it one step closer to reality. I tested again. Still good. Time to get a CA signed cert to give commodity a way to trust my site's integrity.

The EFF Certbot tool can be used to easily obtain certificates from Let's Encrypt! on most common server operating systems. Unfortunately it does not have complete support for Joyent SmartOS. I am running my personal deployments on SmartOS as a deliberate trade-off for operational simplicity and enhanced security. SmartOS is objectively more secure than Linux at this point (that may change). The downside is reduced software support particularly in community and documentation outside SmartOS itself (for which the community is outstandingly generous, knowledgeable, and very helpful).

How did I address that? I ran certbot in manual mode on my laptop instead. Manual mode echoes commands to run on the server that establish the ACME challenges used to do domain validation. Let's Encrypt! requires domain validation to ensure that you have control of the domain and the private keys you are signing. For commercial environments this practically means that you need to allow access to the challenge URLs from anywhere on the Internet. I would prefer if this were different (authentication of some kind at least), but it's understandable (wise really) given the goal of simplifying TLS enrolment and making it ubiquitous.

Certbot has some very good docs so I'll just share the essential nugget needed to understand the domain validation piece: Let's Encrypt! expects you to have a file at /.well-known/acme-challenge/$base64nonce with a challenge response. Certbot automates that completely normally, but on SmartOS I needed to use the manual mode to generate commands for me to run on SmartOS.

So I ran:

certbot certonly --manual

It gave me a set of commands to create the acme-challenge response files on a temporary web server. Looking at the commands that create the temp folders I created similar folders on my static file path on my nginx SmartMachine. I then ran the command to create the challenge response files in .well-known/acme-challenge/ for each SAN (alainodea.com and blog.alainodea.com). I messed this up the first time around because I didn't realize I needed an ACME challenge per SAN.

I then copied the fullchain.pem and privkey.pem from /etc/letsencrypt/live/alainodea.com/ to the /opt/local/etc/nginx/ folder on the nginx SmartMachine.

I checked the site in Chrome again. All good. Then I altered the firewall configuration to allow TCP/443 to go to the nginx SmartMachine (left TCP/80 going to the Zotonic SmartMachine).