apache stops working continuously

[nobody]

I'm sure the cause of the problem is within KeyHelp

No idea

Server operating system + version
Ubuntu 24.04.4


KeyHelp version + build number
Last build


Problem description / error messages
AH10159: server is within MinSpareThreads of MaxRequestWorkers, consider raising the MaxRequestWorkers setting
AH00484: server reached MaxRequestWorkers setting, consider raising the MaxRequestWorkers setting

Expected result
Apache shouldn't going down

Actual result
Apache stops working

Hi there,

since the last update of keyhelp apache stops working continuously, websites, traffic and so on didn't change.
I have 8GB of ram and only 30-50% are in use, so ram isn't a problem.
Where can i find which processes are cause of the above errors find out which processes are causing the errors above?
Thanks in advance for helping!



.

[nobody]

Apparently it's a website that's causing the issue.
Until i don't find the cause i've raised the pm.children to 24 (instead of 12).
Any idea on how to find the leak, ecc..?
I've also looked into the error log of the website and it's full of requests by bots.
They rotate the IP continuously so how to configure a rule on fail to ban or similar?
Also, the error log is full of errors like:
AH02604: Unable to configure certificate domain.com:443:0 for stapling
But this isn't a issue (i think).

Sorry, i read the wrong php fpm log file from 2 years ago, lol.
So nothing has been solved until now.
Keyhelp version : 25.3 (Build 3569)

[Alexander]

Hello,

like the message said:

AH10159: server is within MinSpareThreads of MaxRequestWorkers, consider raising the MaxRequestWorkers setting
AH00484: server reached MaxRequestWorkers setting, consider raising the MaxRequestWorkers setting

Have you considered this ?

[nobody]

Have you considered this ?

lol, sure, but first i would like to know what process(es)/ script, etc. are causing the errors, could also be a website on the server, no idea.
I've looked everywhere, didn't find anything.

The mpm_event.conf file in /etc/apache2/mods-available (if it's the right file) is the default one, and the configuration was ok until the last update.

StartServers 2
MinSpareThreads 25
MaxSpareThreads 75
ThreadLimit 64
ThreadsPerChild 25
MaxRequestWorkers 150
MaxConnectionsPerChild 0

[Tiresias]

Hi!

To check what Apache is doing in real time, you can use the built-in server status page. The correspondend module should be enabled by default. But on Ubuntu 24 you can do this with

Code: Select all

a2enmod status

and then reload Apache using

Code: Select all

systemctl reload apache2

. The default configuration usually allows access only from localhost via /server-status.

To view the status in a readable way on the command line, it is recommended to install a text-based browser like lynx. You can install it with

Code: Select all

apt install lynx

. After that, simply run

Code: Select all

apachectl fullstatus

, which will automatically use lynx to display the status page.

When looking at the output, the most important things are the number of busy and idle workers, the scoreboard, and the list of current requests. If you see many busy workers and almost no idle ones, the server is under load. Repeated requests to the same URL or from the same IP often indicate bots or a problematic endpoint. Many workers in “W” state usually means the backend (for example PHP or the database) is slow. Many “K” states point to keep-alive connections holding workers open. High values in request time or duration indicate slow or stuck requests.

You can also use this view to identify which virtual host is causing trouble. In the VHost column you will see the domain handling each request. If you notice that most active or slow requests belong to the same domain, that virtual host is very likely the source of the problem. This view is a snapshot of the current moment, so it is best to check it while the problem is actually happening.

For bots you can enable "kh-bad-bots"-Jail in Fail2Ban.

[nobody]

For bots you can enable "kh-bad-bots"-Jail in Fail2Ban.

Since the default "keyhelp.conf" file gets overwritten on each update, i've made my own long time ago, but that didn't help me out, as fail2ban gives a lot of false positives by monitoring the https port.

I also read the apache fullstatus but that didn't give me any further clue.
Would be good if there was a log file to see past activities as it's impossible to monitor the server 24/7, lol

There are a couple of server reached max_children setting log entries from 2 years ago with php8.1, but nothing from today.

Somestimes like this morning apache went down regular, trying to find out why.
At first i tought it was the network so i reconfigured it already twice but it didn't help.
Changing the mpm children isn't a solution because until a couple of months ago it worked fine with these settings.
It's only since i last updated keyhelp that i have these issues.
Could also be a coincidence since i don't monitor actively all users on the websites.
Keyhelp devs are awesome so i don't think that there are bugs that causes the server to go down