apache stops working continuously

[nobody]

I'm sure the cause of the problem is within KeyHelp

No idea

Server operating system + version
Ubuntu 24.04.4


KeyHelp version + build number
Last build


Problem description / error messages
AH10159: server is within MinSpareThreads of MaxRequestWorkers, consider raising the MaxRequestWorkers setting
AH00484: server reached MaxRequestWorkers setting, consider raising the MaxRequestWorkers setting

Expected result
Apache shouldn't going down

Actual result
Apache stops working

Hi there,

since the last update of keyhelp apache stops working continuously, websites, traffic and so on didn't change.
I have 8GB of ram and only 30-50% are in use, so ram isn't a problem.
Where can i find which processes are cause of the above errors find out which processes are causing the errors above?
Thanks in advance for helping!



.

[nobody]

Apparently it's a website that's causing the issue.
Until i don't find the cause i've raised the pm.children to 24 (instead of 12).
Any idea on how to find the leak, ecc..?
I've also looked into the error log of the website and it's full of requests by bots.
They rotate the IP continuously so how to configure a rule on fail to ban or similar?
Also, the error log is full of errors like:
AH02604: Unable to configure certificate domain.com:443:0 for stapling
But this isn't a issue (i think).

Sorry, i read the wrong php fpm log file from 2 years ago, lol.
So nothing has been solved until now.
Keyhelp version : 25.3 (Build 3569)

[Alexander]

Hello,

like the message said:

AH10159: server is within MinSpareThreads of MaxRequestWorkers, consider raising the MaxRequestWorkers setting
AH00484: server reached MaxRequestWorkers setting, consider raising the MaxRequestWorkers setting

Have you considered this ?

[nobody]

Have you considered this ?

lol, sure, but first i would like to know what process(es)/ script, etc. are causing the errors, could also be a website on the server, no idea.
I've looked everywhere, didn't find anything.

The mpm_event.conf file in /etc/apache2/mods-available (if it's the right file) is the default one, and the configuration was ok until the last update.

StartServers 2
MinSpareThreads 25
MaxSpareThreads 75
ThreadLimit 64
ThreadsPerChild 25
MaxRequestWorkers 150
MaxConnectionsPerChild 0

[Tiresias]

Hi!

To check what Apache is doing in real time, you can use the built-in server status page. The correspondend module should be enabled by default. But on Ubuntu 24 you can do this with

Code: Select all

a2enmod status

and then reload Apache using

Code: Select all

systemctl reload apache2

. The default configuration usually allows access only from localhost via /server-status.

To view the status in a readable way on the command line, it is recommended to install a text-based browser like lynx. You can install it with

Code: Select all

apt install lynx

. After that, simply run

Code: Select all

apachectl fullstatus

, which will automatically use lynx to display the status page.

When looking at the output, the most important things are the number of busy and idle workers, the scoreboard, and the list of current requests. If you see many busy workers and almost no idle ones, the server is under load. Repeated requests to the same URL or from the same IP often indicate bots or a problematic endpoint. Many workers in “W” state usually means the backend (for example PHP or the database) is slow. Many “K” states point to keep-alive connections holding workers open. High values in request time or duration indicate slow or stuck requests.

You can also use this view to identify which virtual host is causing trouble. In the VHost column you will see the domain handling each request. If you notice that most active or slow requests belong to the same domain, that virtual host is very likely the source of the problem. This view is a snapshot of the current moment, so it is best to check it while the problem is actually happening.

For bots you can enable "kh-bad-bots"-Jail in Fail2Ban.

[nobody]

For bots you can enable "kh-bad-bots"-Jail in Fail2Ban.

Since the default "keyhelp.conf" file gets overwritten on each update, i've made my own long time ago, but that didn't help me out, as fail2ban gives a lot of false positives by monitoring the https port.

I also read the apache fullstatus but that didn't give me any further clue.
Would be good if there was a log file to see past activities as it's impossible to monitor the server 24/7, lol

There are a couple of server reached max_children setting log entries from 2 years ago with php8.1, but nothing from today.

Somestimes like this morning apache went down regular, trying to find out why.
At first i tought it was the network so i reconfigured it already twice but it didn't help.
Changing the mpm children isn't a solution because until a couple of months ago it worked fine with these settings.
It's only since i last updated keyhelp that i have these issues.
Could also be a coincidence since i don't monitor actively all users on the websites.
Keyhelp devs are awesome so i don't think that there are bugs that causes the server to go down

[Alexander]

Hello,

To check what Apache is doing in real time, you can use the built-in server status page. The correspondend module should be enabled by default. But on Ubuntu 24 you can do this with

Code: Select all

a2enmod status

and then reload Apache using

Code: Select all

systemctl reload apache2

. The default configuration usually allows access only from localhost via /server-status.

No need to do this all manually. KeyHelp has this feature build in below "System Status -> Apache Server Status".

Since the default "keyhelp.conf" file gets overwritten on each update...

Just a note: Each File with "keyhelp" in file name or which is located in a folder called something with "keyhelp" might get overridden by KeyHelp at some point.

Somestimes like this morning apache went down regular, trying to find out why.

What does the syslog, apache and php-fpm and access/error logs of the domains say at the particular time?

It's only since i last updated keyhelp that i have these issues.

There have not been any changes within the web server configuration in the 25.3 update.

[nobody]

Just a note: Each File with "keyhelp" in file name or which is located in a folder called something with "keyhelp" might get overridden by KeyHelp at some point.

Exactly. For this reason i've made my own so they don't get overwritten.

What does the syslog, apache and php-fpm and access/error logs of the domains say at the particular time?

There's nothing related to the problem; from the php-fpm log there are a couple of entries on 29th and 30th of march when the max children has been reached of a particolar domain where the server didn't go down and the next entry is when i restarted the server yesterday.
php8.3-fpm:

Code: Select all

[29-Mar-2026 10:52:07] WARNING:  server reached max_children setting (3), consider raising it
[30-Mar-2026 11:52:41] WARNING: server reached max_children setting (3), consider raising it
[01-Apr-2026 12:30:59] NOTICE: Terminating ...
[01-Apr-2026 12:30:59] NOTICE: exiting, bye-bye!

What i've noticed is that some of the domains are still using php8.1 but the last php8.1 log entry is from 2024.
Weird?

apache error log:

Code: Select all

[Wed Apr 01 11:45:10.989730 2026] [core:error] [pid 1963116:tid 128917771892416] [client 51.77.199.82:47854] AH10244: invalid URI path (/cgi-bin/.%2e/.%2e/.%2e/.%2e/.%2e/.%2e/.%2e/.%2e/.%2e/.%2e/bin/sh)
[Wed Apr 01 11:45:11.045503 2026] [core:error] [pid 1963116:tid 128918325548736] [client 51.77.199.82:47868] AH10244: invalid URI path (/cgi-bin/%%32%65%%32%65/%%32%65%%32%65/%%32%65%%32%65/%%32%65%%32%65/%%32%65%%32%65/%%32%65%%32%65/%%32%65%%32%65/bin/sh)
[Wed Apr 01 12:30:59.732584 2026] [mpm_event:notice] [pid 805601:tid 128918956791680] AH00492: caught SIGWINCH, shutting down gracefully
[Wed Apr 01 12:31:03.268050 2026] [core:warn] [pid 805601:tid 128918956791680] AH00045: child process 1962876 still did not exit, sending a SIGTERM
[Wed Apr 01 12:31:03.268156 2026] [core:warn] [pid 805601:tid 128918956791680] AH00045: child process 1963116 still did not exit, sending a SIGTERM
ecc....

syslog just shows a lot of

Code: Select all

2026-04-01T12:21:35.389353+01:00 vps named[1613932]: client @0x71d48535bb28 141.98.83.48#22469 (vcor.com): query (cache) 'vcor.com/ANY/IN' denied (allow-query-cache did not match)

entries before restarting

I've already looked other times in the log files but i couldn't find anything.

There have not been any changes within the web server configuration in the 25.3 update.

oh ok, thanks for the information.
And in the 25.1/ 25.2 update, no changes at all in the software, ecc. ?

In the meantime i think i should change the mpm_event.conf file in /etc/apache2/mods-available, right?
Question: Is this the right file in the correct folder location?

Which is just a temporary solution, because i need the find the cause of the casual hangings of apache
But if the log files don't show anything, how to find it?

[Alexander]

Code: Select all

[29-Mar-2026 10:52:07] WARNING:  server reached max_children setting (3), consider raising it
[30-Mar-2026 11:52:41] WARNING: server reached max_children setting (3), consider raising it

FYI In case, you want to raise this too, you can do so via user management of the particullar user -> Tab: PHP-FPM

What i've noticed is that some of the domains are still using php8.1 but the last php8.1 log entry is from 2024.
Weird?

You have checked this log, right? /opt/keyhelp/php/8.1/var/log/php-fpm.log

Normally, KeyHelp should restart Apache, if it detects apache is not running anymore.
This would be logged in /var/log/keyhelp/cronjob/master.log: " "apache2" is not running, restarting... "
On your system: Is apache dead until you restart it?

In the meantime i think i should change the mpm_event.conf file in /etc/apache2/mods-available, right?
Question: Is this the right file in the correct folder location?

Yes.

Which is just a temporary solution, because i need the find the cause of the casual hangings of apache
But if the log files don't show anything, how to find it?

Sure. Seems you are also not alone with this issue (non-KeyHelp-related):
https://www.google.com/search?q=mpm_eve ... gracefully
Some have switched to mpm_worker, you might want to give it a shot: KeyHelp -> Configuration -> Webserver -> Multi-Processing-Modul (MPM)
I will have a deeper look into the search results when i find the time.

[nobody]

FYI In case, you want to raise this too, you can do so via user management of the particullar user -> Tab: PHP-FPM

done

You have checked this log, right? /opt/keyhelp/php/8.1/var/log/php-fpm.log

Now that you have mentioned it, yes. Didn't know there was a separate log file. There was an entry, but it's from the 28th of march, nothing related. Because on 28th the server was running

Code: Select all

[28-Mar-2026 10:53:26] WARNING: [pool domain_php81] server reached max_children setting (12), consider raising it

------------------------------------ formatting posting line --------------------------------------

This would be logged in /var/log/keyhelp/cronjob/master.log: " "apache2" is not running, restarting... "

I've checked all these logs now back to master.log5. There wasn't any entry of keyhelp restarting apache.
Could you please do in a test environment assign a min. amount of children (2,3) to a heavy server load to see if keyhelp restart apache?

On your system: Is apache dead until you restart it?

exactly. I know it because my own domain is running there and if roundcube is hanging i know apache went down.
Or when a customer calls me telling me his website is down.
I have other 3 servers with keyhelp running and there are also casual entries of max_children reached but they never went down.
It's only this server that hangs. Initially i reconfigured the network twice thinking it was the cause but apache kept hanging from time to time.
But if the apache guys don't do properly logging, there's not much to do expect guessing.

Some have switched to mpm_worker, you might want to give it a shot:

What are the pros and cons switching to mpm_worker? On the keyhelp UI i read:

The mpm_event and mpm_worker both work reliably, with mpm_event providing better performance.

------------------------------------ formatting posting line --------------------------------------

I will have a deeper look into the search results when i find the time.

Thanks for the link, need to read it too, but needs a lot of time to extract the needed information