Letsencrypt only successfully create/renew a few subdomains

General
1

HI,

I am since quite some time using the linuxserver/letsencrypt container running on a Synology DS716+II.
In my router (asus wrt merlin) i am forwarding 80/443/tcp to my NAS where the container is running on -p 80:80 and
-p 443:443 (i have changed the ports for the Synology nginx server to other ports).

I have a domain hosted with cloudns.net with a number of a records updated using ddns
double checked that all points to the actual external IP of my router.

I have had the setup running with 3 subdomains nextcloud, bitwarden, gogs and the container
when started with docker-compose up -d successfully create certificate for all 3 without problems.

Now comes the problem, i want to add a 4th subdomain, but now all of a sudden the verification
is failing randomly for each of the subdomains.
Some times 2 are created and for 2 verification fails, sometimes 3 are created and 1 verification fail.

As you can see the port mapping setup is correct and working (if not then 3 subdomains would not work either!) but for some reason it seems that multiple simultaneous requests for verification above
3 seems to fail.

Is this a known problem? Or is there any other way to configure this so that i can use +3 certificates?
Ex. wildcard cert or similar?

br

Mikael

2

What validation method are you using? With http, I have done as many as 10 subdomains and not had any issues. It tries them one by one.

Post a full log and we’ll take a look (feel free to redact part of the info)

3

Just to add to this, i currently have 28 subdomains (don’t judge me!) using http validation with 0 issues.

4

Hi again!

I am unfortunately unable to post the log as i am only allowed to use two links in one post
and it seems that the startup log of the container contains things which are considered to be links.

Any ideas on how to successfully upload the log file?
I even tried renaming my .txt to .jpg but no fooling the system said that possibly the image is broken. :slight_smile:
/mikael

5

you only posted one link in the previous post and it is a link to cloudns.net

And the limit is per post I believe, not thread

6

Here at least is the last part of the validation which then fails when i have +3 a records.

Generating new certificate
Saving debug log to /var/log/letsencrypt/letsencrypt.log
Plugins selected: Authenticator standalone, Installer None
Obtaining a new certificate
Performing the following challenges:
http-01 challenge for bw
http-01 challenge for ha
http-01 challenge for nc
http-01 challenge for rc
Waiting for verification…
Challenge failed for domain rc
http-01 challenge for rc
Cleaning up challenges
Some challenges have failed.
IMPORTANT NOTES:

  • The following errors were reported by the server:

    Domain: rc
    Type: connection
    Detail: Fetching
    http://rc/.well-known/acme-challenge/WiTshUnqjL2cTsMl4EGSLtmtcD9mhqoTGEjZhEPS1I0:
    Timeout during connect (likely firewall problem)

    To fix these errors, please make sure that your domain name was
    entered correctly and the DNS A/AAAA record(s) for that domain
    contain(s) the right IP address. Additionally, please check that
    your computer has a publicly routable IP address and that no
    firewalls are preventing the server from communicating with the
    client. If you’re using the webroot plugin, you should also verify
    that you are serving files from the webroot path you provided.

  • Your account credentials have been saved in your Certbot
    configuration directory at /etc/letsencrypt. You should make a
    secure backup of this folder now. This configuration directory will
    also contain certificates and private keys obtained by Certbot so
    making regular backups of this folder is ideal.
    ERROR: Cert does not exist! Please see the validation error above. The issue may be due to incorrect dns or port forwarding settings. Please fix your settings and recreate the container

7

And here is what comes before (sorry for posting in wrong order!)

Variables set:
PUID=1026
PGID=100
TZ=Europe/Berlin
URL=example.com
SUBDOMAINS=ha,bw,nc,rc
EXTRA_DOMAINS=
ONLY_SUBDOMAINS=true
DHLEVEL=2048
VALIDATION=http
DNSPLUGIN=
EMAIL=
STAGING=false

2048 bit DH parameters present
SUBDOMAINS entered, processing
SUBDOMAINS entered, processing
Only subdomains, no URL in cert
Sub-domains processed are: -d ha -d bw -d nc -d rc
E-mail address entered: mikael@example.com
http validation is selected
Different validation parameters entered than what was used before. Revoking and deleting existing certificate, and an updated one will be created
Saving debug log to /var/log/letsencrypt/letsencrypt.log
No match found for cert-path /config/etc/letsencrypt/live/ha/fullchain.pem!

8

It’s only erroring on rc. Did you create a CNAME for it?

9

I created it as an a name record and i used it to verify when i installed nginx (as the port forwarding guide suggested).

It even works when trying to access rocket.chat (rc=rocketchat) but with invalid certificate since it
is not in the list of domains…

10

I see this a couple of months old now, but did you ever resolve the issue? I’m experiencing the exact same thing; everything’s fine with 2 subdomains, with 3 I get occasional failures but it works with a couple of restarts, with 4 subdomains I get no successes at all. I’d need 4-5. My config is very similar to yours (attaching it below)

Just like you described, I’m also certain my DNS records and port forwards lead to the container ok as any one of the subdomains is fine alone or in smaller groups. The actual applications also work fine if I leave one subdomain out - that one just won’t have a valid cert.

At some point I suspected the fail2ban, but I see no signs of it kicking in. Besides, I understand that it only works on the nginx logs and the cert-phase is done before even starting the nginx - that’s why the apps stay down if any of the certs fail.

Basically, I’m testing it somewhat like this:

docker run \
--cap-add=NET_ADMIN \
--name=my-proxy \
--network my-network \
--restart unless-stopped \
-v /webapp/proxy/config:/config \
-e PGID=989 \
-e PUID=993 \
-e EMAIL=my.email@mydomain.org \
-e URL=mydomain.org \
-e SUBDOMAINS=cloud,gogs,rocket,wiki \
-e ONLY_SUBDOMAINS=true \
-e VALIDATION=http \
-p 80:80 \
-p 443:443 \
-e TZ=Europe/Helsinki \
linuxserver/letsencrypt
11

I’ve tested dozens of subdomains with http validation with no issues. All it does is the letsencrypt server sends an http request to each subdomain address one by one and checks to see if it’s getting the response with the correct key (different for each) for each one (from the letsencrypt container). That’s all.

And you are correct that it is not dependent on nginx as that process uses a basic web server that certbot temporarily puts up

12

Right, no doubt it must be something in my setup, I just can’t figure out what. The OPs problem seems identical, so I was hoping for a clue on where to look. I guess I’ll try to set up a second minimal proxy with some more subdomains to try and narrow it down.

13

Just to let you know; I did a fair bit of really convoluted attempts to fix this to no avail… until I finally noticed a convenient little feature in my router’s built-in firewall: DDOS protection. Apparently the validations were just one too many near-simultaneous requests. Disabling that, everything is now working smoothly.

Thanks for the quality images, I will now disappear back into the shadows whence I came.

14

Thanks so much for reporting back. That would definitely do it. Like I said, the letsencrypt server attempts to connect to all the domains and subdomains in a row and that could be misinterpreted by the router, which I had not thought of.