[#915] Automatically request/renew Let's Encrypt SSL/TLS certificates for all vhosts



Detailed description

Wald - current situation

We're running a FusionForge 6.1.x instance named "Wald", reachable via http://wald.intevation.org/ and https://wald.intevation.org/

Currently the certificate used for https is from our own CA (root certificate downloadable at https://ssl.intevation.de/), which isn't known by default by any web browsers, so we would like to switch to certificates from Let's Encrypt.

Let's Encrypt client

On other servers we are currently using the program "acmetool" from https://github.com/hlandau/acmetool to request/renew Let's Encrypt certificates. To support the new ACMEv2 protocol, at least version 0.2.1 (currently only available as a beta release) is needed. A simple alternative client could be https://dehydrated.io/

Implementation proposal

Originally I thought we would need a certificate including SAN entries for *.wald.intevation.org, wald.intevation.org and every alternative host name pointing to Wald, e.g. mpuls.org and http://www.mpuls.org.

But wildcard certificates and Let's Encrypt are a little bit complicated to handle automatically, especially in combination with additional SAN entries for different domains, so I was thinking about the following way:

A script on Wald provides a list of all used host names, i.e.: - http://www.wald.intevation.org - wald.intevation.org - scm.wald.intevation.org - hg.wald.intevation.org - lists.wald.intevation.org - what other host names *.wald.intevation.org did I miss? - all project pages, e.g. adminton.wald.intevation.org - the manually configured vhosts that are listed in httpd-ssl.vhosts (e.g. mpuls.org and http://www.mpuls.org), but EXCLUDING those that have their DNS entry no longer pointing to Wald, e.g openvas.org.

Wald could then request a single SSL certificate containing SAN entries for all hostnames using e.g.

acmetool want http://www.wald.intevation.org wald.intevation.org ...more...

and use this single certificate for everything (assuming Let's Encrypt allows that many SAN entries, I haven't verified this).

Before using "acmetool want", the old list of vhosts should be cleared using "acmetool unwant ...old list of hostnames...", so no further verification requests are triggered for hostnames that are no longer in use and may now point to a different webserver.

The web server of Wald would need to direct all requests to /.well-known/acme-challenge/ for all configured vhosts to the challenge directory created by acmetool.

I know that projects on a FusionForge server can be configured to use their own SSL certificates (I assume using SNI), but I don't want to manually do this, especially since the verification challenges by Let's Encrypt have to be done anyway and the certificates have a short life span (I think 3 months).


The steps to implement this would be:

  1. A script that lists all used host names that have a DNS entry pointing to the server
  2. A script that uses the output of 1. determines if the list has changed. If yes calls "acmetool unwant (all previous hostnames)" followed by "acmetool want (all current hostnames)". (this script will regularly run, e.g. via cron or triggered by vhost changes)
  3. A configuration change to Apache to offer /.well-known/acme-challenge/ on all configured vhosts.
General Information
Submitted by:
Thomas Arendsen Hein
Date Submitted: 2020-01-28 12:12
Last Modified by:
Thomas Arendsen Hein
Last Modified: 2020-03-10 11:16
Permalink: https://fusionforge.org/tracker/a_follow.php/915
Internal Fields
Data Type: Support requests
Assigned to: Nobody (None)
State: Open
Priority: 3
Default ExtraField 2-columns Widget
Follow-up tabs
Message  ↓
Date: 2020-03-10 11:16
Sender: Thomas Arendsen Hein

Of course you will need to add a cronjob entry to regularly run: dehydrated --cron && systemctl reload apache2 && dehydrated --cleanup

Certificates are valid for 3 months and no renew will happen within the first 30 days (unless the list of hostnames has changed), you can either run every 7-30 days (and run it manually after changing the list of hostnames) or just run it every night.

You could optionally use a dehydrated hook to execute "systemctl reload apache2" only if the certificate has changed.

And if you put above call in a script, you can use chronic (from moreutils) to only output the result (and thus generate a mail with cron) when something bad happened, e.g. the verification failed, which can happen with so many hostnames.

Date: 2020-03-10 10:51
Sender: Thomas Arendsen Hein

Update: I have a working solution that is not fully automated (I have to manually prepare a list of hosts), but otherwise works well. I decided to use dehydrated instead of acmetool.

First the staging setup:

To fix the hg/svn challenge URLs (send redirects to the main host to avoid using the hg ScriptAlias or svn webdav:

  • /etc/apache2/conf-available/dehydrated-redirects.conf: &lt;IfModule alias_module&gt; &lt;If &quot;%{HTTP_HOST} in { 'hg.wald.intevation.org', 'svn.wald.intevation.org' }&quot;&gt; RedirectMatch &quot;^/.well-known/acme-challenge/(.*)$&quot; &quot;<a href="http://wald.intevation.org/.well-known/acme-challenge/$1&quot" target="_blank">http://wald.intevation.org/.well-known/acme-challenge/$1&quot</a>; &lt;/If&gt; &lt;/IfModule&gt;
  • a2enconf dehydrated-redirects
  • systemctl reload apache2
  • /etc/dehydrated/domains.txt (see /usr/share/doc/dehydrated/docs/domains_txt.md): wald.intevation.org ...(many hostnames) &gt; wald-test
  • dehydrated --cron
  • dehydrated --cleanup

Running dehydrated with about 75 hostnames (hosts like foo.wald.intevation.org and project homepages like http://www.mpuls.org -- the limit for let's encrypt is 100 hostnames per certificate!) worked fine with the staging CA, so I prepared everything for production:

  • mv /etc/dehydrated/conf.d/staging.sh{,.off}
  • mv /var/lib/dehydrated{,.sik1}
  • mkdir -m755 -p /var/lib/dehydrated/acme-challenges
  • echo dehydrated > /var/lib/dehydrated/acme-challenges/test
  • /etc/dehydrated/domains.txt (see /usr/share/doc/dehydrated/docs/domains_txt.md): wald.intevation.org lists.wald.intevation.org scm.wald.intevation.org hg.wald.intevation.org svn.wald.intevation.org sitetest.wald.intevation.org adminton.wald.intevation.org ...many more... <a href="http://www.mpuls.org" target="_blank">http://www.mpuls.org</a> &gt; wald
  • dehydrated --register
  • dehydrated --register --accept-terms
  • dehydrated --cron
  • # Error message "JWS has no anti-replay nonce", worked on the second try:
  • dehydrated --cron
  • dehydrated --cleanup

Now use the new certificate:

  • cd /etc/fusionforge
  • mkdir -m700 old
  • mv ssl-cert.key ssl-cert.pem old/
  • ln -sf /var/lib/dehydrated/certs/wald/privkey.pem ssl-cert.key
  • ln -sf /var/lib/dehydrated/certs/wald/fullchain.pem ssl-cert.pem
  • systemctl reload apache2

No attached documents

No related commits.

No changes have been made to this item

No relations found.