Let's Encrypt SSL certificates could NOT be updated  [SOLVED]

Have you discovered a bug? Tell us about it.
Post Reply
User avatar
george
Posts: 87
Joined: Fri 3. Jan 2020, 05:53
Location: AUSTRALIA

Let's Encrypt SSL certificates could NOT be updated

Post by george »

Over the last several days, I have received error notifications (by email), that the Let's Encrypt SSL certificates could NOT be updated. Here is excerpt of an email (hostname and IP changed to protect the innocent):

Code: Select all

During the routine check of the SSL/TLS certificates, the following problems occurred:

------------------------------------
Certificate name: zzz.com (Let's Encrypt)

Verification ended with an error.
Details: 101.1.101.1: Fetching https://zzz.com/.well-known/acme-challenge/uzGGNG6sqw-PeQ9mhbSGPspu3BMcret8SBOoraY9w70: Timeout during connect (likely firewall problem)
Type: urn:ietf:params:acme:error:connection
Full response: {"type":"http-01","status":"invalid","error":{"type":"urn:ietf:params:acme:error:connection","detail":"101.1.101.1: Fetching https:\/\/zzz.com\/.well-known\/acme-challenge\/uzGGNG6sqw-PeQ9mhbSGPspu3BMcret8SBOoraY9w70: Timeout during connect (likely firewall problem)","status":400},"url":"https:\/\/acme-v02.api.letsencrypt.org\/acme\/chall-v3\/192206579837\/JxWLlA","token":"uzGGNG6sqw-PeQ9mhbSGPspu3BMcret8SBOoraY9w70","validationRecord":[{"url":"http:\/\/zzz.com\/.well-known\/acme-challenge\/uzGGNG6sqw-PeQ9mhbSGPspu3BMcret8SBOoraY9w70","hostname":"zzz.com","port":"80","addressesResolved":["101.1.101.1","2600:a100:1111:11:ded:beeb:baab:beeb"],"addressUsed":"2600:a100:1111:11:ded:beeb:baab:beeb"},{"url":"http:\/\/zzz.com\/.well-known\/acme-challenge\/uzGGNG6sqw-PeQ9mhbSGPspu3BMcret8SBOoraY9w70","hostname":"zzz.com","port":"80","addressesResolved":["101.1.101.1","2600:a100:1111:11:ded:beeb:baab:beeb"],"addressUsed":"101.1.101.1"},{"url":"https:\/\/zzz.com\/.well-known\/acme-challenge\/uzGGNG6sqw-PeQ9mhbSGPspu3BMcret8SBOoraY9w70","hostname":"zzz.com","port":"443","addressesResolved":["101.1.101.1","2600:a100:1111:11:ded:beeb:baab:beeb"],"addressUsed":"2600:a100:1111:11:ded:beeb:baab:beeb"}],"validated":"2023-01-03T13:00:27Z"}
Valid until: 2023-01-22 23:00:30 (18 day(s) left)
After investigation, it appears that something changed regarding the redirection to HTTP instead of HTTPS, for the verification on /.well-known/acme-challenge/ addresses.
I "unchecked" the following setting in Domains > Security settings for each domain, to allow HTTP access for the ACME challenge.

Image

I noticed from the following post Alexander is doing an update for openssl:
viewtopic.php?t=12015

I am on Debian 11, and hope that this covers the hostname domain, as I have not been able to override the HTTP > HTTPS redirect found in the file:
/etc/apache2/keyhelp/keyhelp.conf

Code: Select all

# Redirect HTTP -> HTTPS
<VirtualHost *:80>
    ServerName host.domain.com

    <IfModule mod_rewrite.c>
        RewriteEngine On
        RewriteCond %{HTTP_HOST} ^(25[0-5]|2[0-4][0-9]|[0-1]?[0-9]?[0-9])\.(25[0-5]|2[0-4][0-9]|[0-1]?[0-9]?[0-9])\.(25[0-5]|2[0-4][0-9]|[0-1]?[0-9]?[0-9])\.(25[0-5]|2[0-4][0-9]|[0-1]?[0-9]?[0-9]).*$
        RewriteRule ^(.*)$ https://%{HTTP_HOST}/
    </IfModule>

    Redirect / https://host.domain.com/
</VirtualHost>
So all my domains can update SSL cerfificates now, but NOT the main host!
Eoler
Posts: 17
Joined: Tue 2. Jul 2019, 01:20

Re: Let's Encrypt SSL certificates could NOT be updated

Post by Eoler »

george wrote: Wed 4. Jan 2023, 08:56 Over the last several days, I have received error notifications (by email), that the Let's Encrypt SSL certificates could NOT be updated.
Me too, on Ubuntu 20.04.
User avatar
Alexander
Keyweb AG
Posts: 3809
Joined: Wed 20. Jan 2016, 02:23

Re: Let's Encrypt SSL certificates could NOT be updated

Post by Alexander »

Hello,

I would assume that you would have received the Let's Encrypt certificates without changing anything. But it might rather have something to do with the timing of the "Maintenance of SSL/TLS certificates" - maybe the server is busy at the given times or something else is causing the problem at that time.

Just give it a try: Call "keyhelp-toolbox" -> 6) Start maintenance tasks -> 8) Maintenance of SSL/TLS certificates

There have been no changes recently and Let's Encrypt has not announced any changes that would affect the maintenance of the certificates.
I noticed from the following post Alexander is doing an update for openssl:
viewtopic.php?t=12015
This has nothing to do with Let's Encrypt and has not been released yet.
Mit freundlichen Grüßen / Best regards
Alexander Mahr

**************************************************************
Keyweb AG - Die Hosting Marke
Neuwerkstr. 45/46, 99084 Erfurt / Germany
http://www.keyweb.de - http://www.keyhelp.de
**************************************************************
User avatar
george
Posts: 87
Joined: Fri 3. Jan 2020, 05:53
Location: AUSTRALIA

Re: Let's Encrypt SSL certificates could NOT be updated

Post by george »

Alexander, it is the same with keyhelp-toolbox. I also tried changing the task time as you suggested (Settings > Maintenance Intervals), and have run the task manually from there too, without effect.
It looks like the HTTP redirect is the issue, as I corrected the problem on ALL the domains, I just can't force it on the hostname domain.
User avatar
Alexander
Keyweb AG
Posts: 3809
Joined: Wed 20. Jan 2016, 02:23

Re: Let's Encrypt SSL certificates could NOT be updated

Post by Alexander »

Have you not configured some proxy settings in your Apache? Maybe they are interfering.

I can't reproduce the problem with a standard KeyHelp system on Debian 11.
Mit freundlichen Grüßen / Best regards
Alexander Mahr

**************************************************************
Keyweb AG - Die Hosting Marke
Neuwerkstr. 45/46, 99084 Erfurt / Germany
http://www.keyweb.de - http://www.keyhelp.de
**************************************************************
User avatar
george
Posts: 87
Joined: Fri 3. Jan 2020, 05:53
Location: AUSTRALIA

Re: Let's Encrypt SSL certificates could NOT be updated

Post by george »

Alexander wrote: Wed 4. Jan 2023, 10:59 Have you not configured some proxy settings in your Apache?
I'm not sure what you mean "some proxy settings in your Apache". No settings have been changed for a while. It is a HTTP redirect problem - using the following 2 websites I can see if the verification on /.well-known/acme-challenge/ is possible (HTTP), and on the hostname domain it is NOT:
https://geekflare.com/tools/url-redirection-checker
https://letsdebug.net/

A quick fix for now, would be a way to implement Apache directives that would FORCE HTTP on /.well-known/acme-challenge/, for the hostname domain.
User avatar
george
Posts: 87
Joined: Fri 3. Jan 2020, 05:53
Location: AUSTRALIA

Re: Let's Encrypt SSL certificates could NOT be updated

Post by george »

I currently use this for all domains:

Code: Select all

# SSL Redirect
RewriteEngine On
RewriteCond %{REQUEST_URI} !^/\.well\-known/acme\-challenge.* [NC]
RewriteCond %{HTTPS} off
RewriteRule ^(.*)$ https://%{HTTP_HOST}%{REQUEST_URI} [L,R=301]
User avatar
Alexander
Keyweb AG
Posts: 3809
Joined: Wed 20. Jan 2016, 02:23

Re: Let's Encrypt SSL certificates could NOT be updated

Post by Alexander »

I thought you had made some changes to your apache settings and domain settings.
(I haven't checked your earlier posts yet, I'm just remembering from memory).

Anyway, what modifications have been done to the Apache settings?

How does he complete log of the ssl-maintenance.log (System Stauts -> Logs) look like, regarding the panel domain.
Mit freundlichen Grüßen / Best regards
Alexander Mahr

**************************************************************
Keyweb AG - Die Hosting Marke
Neuwerkstr. 45/46, 99084 Erfurt / Germany
http://www.keyweb.de - http://www.keyhelp.de
**************************************************************
User avatar
george
Posts: 87
Joined: Fri 3. Jan 2020, 05:53
Location: AUSTRALIA

Re: Let's Encrypt SSL certificates could NOT be updated

Post by george »

No major changes to apache, I use .htaccess mainly for added security, but I have tested without these too.
Here is what the log looks like:

Code: Select all

[04-Jan-2023 20:35:12] INFO  --> starting ssl certification maintenance
[04-Jan-2023 20:35:12] INFO  --> checking (normal) SSL/TLS certificates
[04-Jan-2023 20:35:12] INFO  --> check certificate "[ID 1]"
[04-Jan-2023 20:35:12] INFO  --> certificate name is "default"
[04-Jan-2023 20:35:12] INFO  --> certificate is valid until 2032-04-23 17:00:01 (3396 days left)
[04-Jan-2023 20:35:12] INFO  --> checking lets encrypt certificates
[04-Jan-2023 20:35:12] INFO  --> remove unused accounts / certificates
[04-Jan-2023 20:35:12] INFO  --> check domain "domain.com'
[04-Jan-2023 20:35:12] INFO  --> certificate is valid until 2023-04-04 11:17:21 (89 days left)
[04-Jan-2023 20:35:12] INFO  --> check domain "www.domain.com'
[04-Jan-2023 20:35:12] INFO  --> certificate is valid until 2023-04-04 11:47:21 (89 days left)

... (more domains checked here) ...

[04-Jan-2023 20:35:12] INFO  --> check domain "host.domain.com'
[04-Jan-2023 20:35:12] INFO  --> certificate is valid until 2023-01-22 23:01:06 (18 days left)
[04-Jan-2023 20:35:12] INFO  --> certificate is in renewal period
[04-Jan-2023 20:35:12] INFO  --> renew cert
[04-Jan-2023 20:35:12] INFO  --> Using certificate authority: "https://acme-v02.api.letsencrypt.org/" (PRODUCTION).
[04-Jan-2023 20:35:12] INFO  --> Getting endpoint URLs.
[04-Jan-2023 20:35:13] INFO  --> Account "keyhelp" already registered. Continue.
[04-Jan-2023 20:35:13] INFO  --> Requesting Key ID.
[04-Jan-2023 20:35:13] INFO  --> Sending signed request to "https://acme-v02.api.letsencrypt.org/acme/new-acct".
[04-Jan-2023 20:35:14] INFO  --> Start certificate generation.
[04-Jan-2023 20:35:14] INFO  --> Token stored at: /home/keyhelp/www/.well-known/acme-challenge/local-check-63b5485296bb74.92287446
[04-Jan-2023 20:35:14] INFO  --> Local resolving checks of domains successfully completed.
[04-Jan-2023 20:35:14] INFO  --> Requesting challenges for domain "host.domain.com".
[04-Jan-2023 20:35:14] INFO  --> Sending signed request to "https://acme-v02.api.letsencrypt.org/acme/new-order".
[04-Jan-2023 20:35:15] INFO  --> Sending signed request to "https://acme-v02.api.letsencrypt.org/acme/authz-v3/192441466327".
[04-Jan-2023 20:35:15] INFO  --> Start authorization process for "host.domain.com".
[04-Jan-2023 20:35:15] INFO  --> Deploy challenge.
[04-Jan-2023 20:35:15] INFO  --> Token stored at: /home/keyhelp/www/.well-known/acme-challenge/Pd1Lv2BA2mWt3DFJ6XdI5PW76T0ZdgSkC95kJhwqULs
[04-Jan-2023 20:35:16] INFO  --> Notify CA that the challenge is ready.
[04-Jan-2023 20:35:16] INFO  --> Sending signed request to "https://acme-v02.api.letsencrypt.org/acme/chall-v3/192441466327/nyUqiA".
[04-Jan-2023 20:35:16] INFO  --> Sending signed request to "https://acme-v02.api.letsencrypt.org/acme/chall-v3/192441466327/nyUqiA".
[04-Jan-2023 20:35:17] INFO  --> Waiting for verification...
[04-Jan-2023 20:35:19] INFO  --> Sending signed request to "https://acme-v02.api.letsencrypt.org/acme/chall-v3/192441466327/nyUqiA".
[04-Jan-2023 20:35:20] INFO  --> Waiting for verification...
[04-Jan-2023 20:35:22] INFO  --> Sending signed request to "https://acme-v02.api.letsencrypt.org/acme/chall-v3/192441466327/nyUqiA".
[04-Jan-2023 20:35:22] INFO  --> Waiting for verification...
[04-Jan-2023 20:35:24] INFO  --> Sending signed request to "https://acme-v02.api.letsencrypt.org/acme/chall-v3/192441466327/nyUqiA".
[04-Jan-2023 20:35:25] INFO  --> Waiting for verification...
[04-Jan-2023 20:35:27] INFO  --> Sending signed request to "https://acme-v02.api.letsencrypt.org/acme/chall-v3/192441466327/nyUqiA".
[04-Jan-2023 20:35:28] INFO  --> Waiting for verification...
[04-Jan-2023 20:35:30] INFO  --> Sending signed request to "https://acme-v02.api.letsencrypt.org/acme/chall-v3/192441466327/nyUqiA".
[04-Jan-2023 20:35:30] INFO  --> Waiting for verification...
[04-Jan-2023 20:35:32] INFO  --> Sending signed request to "https://acme-v02.api.letsencrypt.org/acme/chall-v3/192441466327/nyUqiA".
[04-Jan-2023 20:35:33] INFO  --> Waiting for verification...
[04-Jan-2023 20:35:35] INFO  --> Sending signed request to "https://acme-v02.api.letsencrypt.org/acme/chall-v3/192441466327/nyUqiA".
[04-Jan-2023 20:35:36] INFO  --> Waiting for verification...
[04-Jan-2023 20:35:38] INFO  --> Sending signed request to "https://acme-v02.api.letsencrypt.org/acme/chall-v3/192441466327/nyUqiA".
[04-Jan-2023 20:35:38] INFO  --> Waiting for verification...
[04-Jan-2023 20:35:40] INFO  --> Sending signed request to "https://acme-v02.api.letsencrypt.org/acme/chall-v3/192441466327/nyUqiA".
[04-Jan-2023 20:35:41] ERROR --> a Let's Encrypt error occurred: Verification ended with an error.
Details: 101.1.101.1: Fetching https://host.domain.com/.well-known/acme-challenge/Pd1Lv2BA2mWt3DFJ6XdI5PW76T0ZdgSkC95kJhwqULs: Timeout during connect (likely firewall problem)
Type: urn:ietf:params:acme:error:connection
Full response: {"type":"http-01","status":"invalid","error":{"type":"urn:ietf:params:acme:error:connection","detail":"101.1.101.1: Fetching https:\/\/host.domain.com\/.well-known\/acme-challenge\/Pd1Lv2BA2mWt3DFJ6XdI5PW76T0ZdgSkC95kJhwqULs: Timeout during connect (likely firewall problem)","status":400},"url":"https:\/\/acme-v02.api.letsencrypt.org\/acme\/chall-v3\/192441466327\/nyUqiA","token":"Pd1Lv2BA2mWt3DFJ6XdI5PW76T0ZdgSkC95kJhwqULs","validationRecord":[{"url":"http:\/\/host.domain.com\/.well-known\/acme-challenge\/Pd1Lv2BA2mWt3DFJ6XdI5PW76T0ZdgSkC95kJhwqULs","hostname":"host.domain.com","port":"80","addressesResolved":["101.1.101.1","2600:a100:1111:11:ded:beeb:baab:beeb"],"addressUsed":"2600:a100:1111:11:ded:beeb:baab:beeb"},{"url":"http:\/\/host.domain.com\/.well-known\/acme-challenge\/Pd1Lv2BA2mWt3DFJ6XdI5PW76T0ZdgSkC95kJhwqULs","hostname":"host.domain.com","port":"80","addressesResolved":["101.1.101.1","2600:a100:1111:11:ded:beeb:baab:beeb"],"addressUsed":"101.1.101.1"},{"url":"https:\/\/host.domain.com\/.well-known\/acme-challenge\/Pd1Lv2BA2mWt3DFJ6XdI5PW76T0ZdgSkC95kJhwqULs","hostname":"host.domain.com","port":"443","addressesResolved":["101.1.101.1","2600:a100:1111:11:ded:beeb:baab:beeb"],"addressUsed":"2600:a100:1111:11:ded:beeb:baab:beeb"}],"validated":"2023-01-04T09:35:16Z"}
[04-Jan-2023 20:35:41] INFO  --> finished
====
User avatar
george
Posts: 87
Joined: Fri 3. Jan 2020, 05:53
Location: AUSTRALIA

Re: Let's Encrypt SSL certificates could NOT be updated  [SOLVED]

Post by george »

I found the best way to test Let's Encrypt "acme-challenge" to work on port 80, is to put a file named "TEST" (containing some text for output), in directory /home/keyhelp/www/.well-known/acme-challenge then check it by curl:

Code: Select all

curl http://host.domain.com/.well-known/acme-challenge/TEST
I took another look at the HTTPS redirects in /etc/apache2/keyhelp/keyhelp.conf. It was the last line of VirtualHost *:80 directives, which stopped the acme-challenge, as it kept getting redirected to https. After commenting out that line (and apache restart), it worked fine, and updated the host certificate.

Code: Select all

# Redirect HTTP -> HTTPS
<VirtualHost *:80>
    ServerName host.domain.com

    <IfModule mod_rewrite.c>
        RewriteEngine On
        RewriteCond %{HTTP_HOST} ^(25[0-5]|2[0-4][0-9]|[0-1]?[0-9]?[0-9])\.(25[0-5]|2[0-4][0-9]|[0-1]?[0-9]?[0-9])\.(25[0-5]|2[0-4][0-9]|[0-1]?[0-9]?[0-9])\.(25[0-5]|2[0-4][0-9]|[0-1]?[0-9]?[0-9]).*$
        RewriteRule ^(.*)$ https://%{HTTP_HOST}/
    </IfModule>

#    Redirect / https://host.domain.com/
</VirtualHost>
I'm not sure what exactly changed with ssl redirection recently, but this fixed it for now. If the file gets changed by future updates, I will make a little script to check and update.
Post Reply