Serious Bugs on Usage & Logs [SOLVED]

[hexacube]

I have installed keyhelp, configured properly with 30 domains. Everything went fine. After 5 days, no websites were loading. When i did SSH failed. Tried serial port access and found out of disk space. Resized and logged in and found few log files sizes were 6GB / 12GB etc., error.log, errol.log1 etc.., Deleted the same and try to run instance but no luck. All ports blocked. No access to http, https, sftp, ssh etc., Restored daily snapshots one by one and finally able to run snapshot from today-2 days. lost all progress i did for past 48 hours. There is a serious issue with keyhelp and needs bug fixes and improvements in following areas:

1) Firewall - Completely confusing and resets during migration. Option to disable firewall is needed, since now all cloud providers have network level firewalls which is sufficient enough.
2) Log Rotation - How could log files (error.log) under each domain can occupy 12GB etc., This has to be fixed.
3) Quota and Disk size calculation - Even after resize, new disk size is not reflecting in dashboard. After installing WordPress in a domain, file size remains at 1.4 MB which is wrong. Tried manually update through crons listed in dashboard. still no luck.
4) Keyhelp is not catching public IP. It always displays and locks private IP. Incase of migration, the firewalls not allowing newly assigned ip and the instance becomes completely inaccesible except serial ports, which is a big headache. (Solution for this has been suggested here for GCP VM) viewtopic.php?f=12&t=9840

I like the way keyhelp was designed and functioning. If theses things are fixed keyhelp is the best CPANEL alternative. Sadly moving back to serverpilot panel. Thank You keyhelp team. Looking forward.

[Alexander]

Hello,

Which operating system does the server use?

Firewall - Completely confusing and resets during migration. Option to disable firewall is needed, since now all cloud providers have network level firewalls which is sufficient enough.

The answer depends on which OS you are running.
However I plan to release a new firewall system for the upcoming KeyHelp release. The current system will therefor not receive much love anymore.

Log Rotation - How could log files (error.log) under each domain can occupy 12GB etc., This has to be fixed.

The file normally not grow that big, they are rotated after beeing above 500KB in size each day. So there must be something strange happen on your server to let them grow that big. Can have a look which messages are listened and send me a few lines?

Quota and Disk size calculation - Even after resize, new disk size is not reflecting in dashboard. After installing WordPress in a domain, file size remains at 1.4 MB which is wrong. Tried manually update through crons listed in dashboard. still no luck.

Have you installed Wordpress as the user itself or with a root user - asking because it is a common mistake to install or copy files while being root. KeyHelp uses quotas, so every file have to belong to the corresponding user to get calculated.

Disk space is calculated once per hour (by default), you may want to have a look into the corresponding log (System status -> Logs -> diskspace.log)

[hexacube]

1) The Operating system used was Ubuntu 18.04 LTS on Google Cloud VM instance.
2) I created new instance with fresh Keyhelp. The server is up and running for past 2 weeks.
3) Faced the same out of space error. Disk was suddenly full. Unable to access instance via SSH too. The issue happened around 9.00 to 10.00 am ETC.
4) Since I create snapshots of instance, I was fortunate to restore latest good working instance and now running smoothly.
5) I am waiting for the disk full problem to occur again. Unable to identify the cause.

Now I am using Keyhelp for my production server with lot of snapshot backups.

The panel is fantastic by the way. Hope this unknown disk full issue will be fixed soon. (I have doubled disk size for safety purpose).

[george]

Hi hexacube, I strongly suggest reading logs, as the problem will appear there. As Alexander also suggested, it would be helpful to see some log entries, especially error.log/s and syslog here. For a log to grow that big, it is getting written to constantly, probably with obvious errors.

You could also check for what is consuming the disk. This command will output the top 10:

Code: Select all

du -ahx / | sort -rh | head -10

[hexacube]

The issue has been spotted. One of the domain is undergoing brute force attack. The log file size is growing enormous. Now the size of error.log file is 19.7GB. It is a WordPress website with word fence enabled.

Even though I have enabled under attack mode in cloud flare dns, the site is still receiving bruteforce attack. Now I need to find someway to protect the website from bruteforce attacks. Screenshot attached for reference.

Is there any way to limit log file size? Help needed. Thank You.

[hexacube]

Output of

Code: Select all

du -ahx / | sort -rh | head -10 

suggested by george is giving the same output. Screenshot attached.

[Tobi]

You could write a little root cronjob an reset the error.log every X minutes.

[Daniel]

Hello,

have you checked the content of the "error.log". It should not even raise that large and errors appearing there should become resolved.
If the error.log does increase due the effect of the incomming bruteforce attack, you could temporary disable the error logging at all for this vhost.

Code: Select all

ErrorLog /dev/null

[hexacube]

Thank You @Tobi. I will look for options to clean logs folders using cron, but the same has to be done for multiple domains. Need to explore ways to automate this during addition / removal of users and domains.

[hexacube]

Thank you @Daniel. I was unable to check contents of error.log. The size was too large, i dont want to download it. Simply deleted it. As suggested by you I will disable error logging.

I have a question. Is there any way to disable error logging for all vhosts / domains? I never used this error logs for any purposes. Help needed.

[Daniel]

Hello,

you do not need to download the whole file. You can even display it at the terminal through ssh with a short simple command.

Code: Select all

tail -n30 /home/users/tasteofsouthindia/logs/error.log

There are possibilties to disable logging in general. But its not recommended to go for it. People usually forget about it and troubleshooting is nearly impossible then. Further reasons that I am not specifying. Basically every vhost-entry overrides log settings of the global configuration.

To answer your question correct: No, there is no possibility available to disable logging in Keyhelp.

[hexacube]

[Tue Sep 29 09:01:43.525320 2020] [proxy_fcgi:error] [pid 32364:tid 140449298831104] [client 173.245.54.59:63102] AH01071: Got error 'PHP message: PHP Fatal error: Uncaught Error: Call to un
defined function is_shop() in /home/users/tasteofsouthindia/www/www.tasteofsouthindia.com/wp-content/th ... 112\nStack trace:\n#0 /home/users/tasteofsouthindia/www/www.tas
teofsouthindia.com/wp-includes/template.php(730): require_once()\n#1 /home/users/tasteofsouthindia/www/www.tasteofsouthindia.com/wp-includes/template.php(676): load_template('/home/users/tas.
..', true, Array)\n#2 /home/users/tasteofsouthindia/www/www.tasteofsouthindia.com/wp-includes/g ... te.php(48): locate_template(Array, true, true, Array)\n#3 /home/users/tasteofsouthi
ndia/www/www.tasteofsouthindia.com/wp-content/th ... ive.php(8): get_header()\n#4 /home/users/tasteofsouthindia/www/www.tasteofsouthindia.com/wp-includes/t ... r.php(106)
: include('/home/users/tas...')\n#5 /home/users/tasteofsouthindia/www/www.tasteofsouthindia.com/wp-blog-header.php(19): require_once('/home/users/tas...')\n#6 /home/users/tasteofsouthindia/ww
w/www.tasteofsou...\n'
[Tue Sep 29 09:31:46.518830 2020] [proxy_fcgi:error] [pid 6573:tid 140449155102464] [client 162.158.79.189:10664] AH01071: Got error 'PHP message: PHP Fatal error: Uncaught Error: Call to un
defined function is_shop() in /home/users/tasteofsouthindia/www/www.tasteofsouthindia.com/wp-content/th ... 112\nStack trace:\n#0 /home/users/tasteofsouthindia/www/www.tas
teofsouthindia.com/wp-includes/template.php(730): require_once()\n#1 /home/users/tasteofsouthindia/www/www.tasteofsouthindia.com/wp-includes/template.php(676): load_template('/home/users/tas.
..', true, Array)\n#2 /home/users/tasteofsouthindia/www/www.tasteofsouthindia.com/wp-includes/g ... te.php(48): locate_template(Array, true, true, Array)\n#3 /home/users/tasteofsouthi
ndia/www/www.tasteofsouthindia.com/wp-content/th ... ive.php(8): get_header()\n#4 /home/users/tasteofsouthindia/www/www.tasteofsouthindia.com/wp-includes/t ... r.php(106)
: include('/home/users/tas...')\n#5 /home/users/tasteofsouthindia/www/www.tasteofsouthindia.com/wp-blog-header.php(19): require_once('/home/users/tas...')\n#6 /home/users/tasteofsouthindia/ww
w/www.tasteofsou...\n'



This is the error.log. Again the error.log file has shootup to 34.7 GB. Help needed to solve this issue.

[george]

Doesn't look like a KeyHelp issue (maybe). You might need to visit the right forum for this one, the one for your php application.
Good luck!

[Tobi]

Looks like your script http://www.tasteofsouthindia.com/wp-con ... header.php is using an unknown php function called "is_shop()".

If you don´t expirience any issues on your website you might add this little snipplet in your functions.php which is located in the theme-directory.

Code: Select all

function is_shop(){
	return;
}

[hexacube]

@Tobi. Implemeted the change u mentioned in fuctions.php. Website is working fine. Let me update the status after a week. Tq for the timely help.