Apache2 - PHP-Fpm not working properly with nextcloud.

[Victor]

I'm sure the cause of the problem is within KeyHelp
(Problems not related to KeyHelp belong in the Offtopic forum)
I am sure, it's something in the KeyHelp configs.

Server operating system + version
Debian 12.2 Bookworm


Server virtualization technology used
None, physical server.


KeyHelp version + build number
23.2.1 (Build 3125)


Problem description / error messages
Ok, so this nightmare started about 2 weeks ago, I rebooted the machine for a kernel upgrade and mysql/mariadb wouldn't restart, it would just give out a very weird error about options. In any case, I had to manually remove all mariadb packages and configs, reinstalled the mysql server, copied back the databases and inno files and mysql started, but some websites started to experience problems, mainly the ones hosting nextcloud installations.
Revelant portion of apache2 config for the vhost in question :

Code: Select all

 # PHP support
  <FilesMatch "\.php$">
    <If "-f %{REQUEST_FILENAME}">
      SetHandler "proxy:unix:/run/php/keyhelp_sysadmins_php82.socket|fcgi://keyhelp_sysadmins_php82.socket"
    </If>
  </FilesMatch>

  # Set proxy properties (the string "unique-domain-name-string" should match
  # the one set in the FilesMatch directive.
  <Proxy fcgi://keyhelp_sysadmins_php82.socket>
    ProxySet connectiontimeout=5 timeout=600
  </Proxy>

Apache2 error log :

Code: Select all

[Fri Dec 08 14:53:49.715651 2023] [proxy:error] [pid 1385417:tid 140482944431808] (2)No such file or directory: AH02454: FCGI: attempt to connect to Unix domain socket /run/php/keyhelp_sysadmins_php82.socket (keyhelp_sysadmins_php82.socket:8000) failed
[Fri Dec 08 14:53:49.715781 2023] [proxy_fcgi:error] [pid 1385417:tid 140482944431808] [client 86.x.x.x:50719] AH01079: failed to make connection to backend: httpd-UDS

I would like to mention that the socket file is properly created and in the path, with proper permissions, however I don't understand why it tries to connect to port :8000, there is no :8000 mention in apache config.

Code: Select all

srw-rw-rw- 1 sysadmins    sysadmins     0 Nov 27 17:18 /run/php/keyhelp_sysadmins_php74.socket
srw-rw-rw- 1 sysadmins    sysadmins     0 Nov 27 17:17 /run/php/keyhelp_sysadmins_php81.socket
srw-rw-rw- 1 sysadmins    sysadmins     0 Dec  8 14:53 /run/php/keyhelp_sysadmins_php82.socket
lrwxrwxrwx 1 root         root         39 Nov 30 16:19 /run/php/keyhelp_sysadmins.socket -> /run/php/keyhelp_sysadmins_php82.socket

Relevant section of php-fpm :

Code: Select all

;
[sysadmins_php82]

; Unix user/group of processes
user = sysadmins
group = sysadmins

; Define how to accept requests
listen = /run/php/keyhelp_sysadmins_php82.socket
listen.owner = sysadmins
listen.group = sysadmins
listen.mode = 0666

Php-fpm errro log :

Code: Select all

[08-Dec-2023 15:22:45 Europe/Bucharest] PHP Warning:  Zend OPcache can't be temporary enabled (it may be only disabled till the end of request) in Unknown on line 0

Modules are loaded correctly :

Code: Select all

root@key:/var/log/php-fpm# /opt/keyhelp/php/8.2/bin/php -m|grep Zend
Zend OPcache
[Zend Modules]
Zend OPcache

Total modules installed :

Code: Select all

root@key:/var/log/php-fpm# dpkg -l|grep php8.2
ii  libapache2-mod-php8.2              8.2.7-1~deb12u1                         amd64        server-side, HTML-embedded scripting language (Apache 2 module)
ii  php8.2                             8.2.7-1~deb12u1                         all          server-side, HTML-embedded scripting language (metapackage)
ii  php8.2-apcu                        5.1.22+4.0.11-2                         amd64        APC User Cache for PHP
ii  php8.2-bcmath                      8.2.7-1~deb12u1                         amd64        Bcmath module for PHP
ii  php8.2-bz2                         8.2.7-1~deb12u1                         amd64        bzip2 module for PHP
ii  php8.2-cli                         8.2.7-1~deb12u1                         amd64        command-line interpreter for the PHP scripting language
ii  php8.2-common                      8.2.7-1~deb12u1                         amd64        documentation, examples and common module for PHP
ii  php8.2-curl                        8.2.7-1~deb12u1                         amd64        CURL module for PHP
ii  php8.2-fpm                         8.2.7-1~deb12u1                         amd64        server-side, HTML-embedded scripting language (FPM-CGI binary)
ii  php8.2-gd                          8.2.7-1~deb12u1                         amd64        GD module for PHP
ii  php8.2-gmp                         8.2.7-1~deb12u1                         amd64        GMP module for PHP
ii  php8.2-igbinary                    3.2.13-1                                amd64        igbinary PHP serializer
ii  php8.2-imagick                     3.7.0-4                                 amd64        Provides a wrapper to the ImageMagick library
ii  php8.2-intl                        8.2.7-1~deb12u1                         amd64        Internationalisation module for PHP
ii  php8.2-ldap                        8.2.7-1~deb12u1                         amd64        LDAP module for PHP
ii  php8.2-mbstring                    8.2.7-1~deb12u1                         amd64        MBSTRING module for PHP
ii  php8.2-mysql                       8.2.7-1~deb12u1                         amd64        MySQL module for PHP
ii  php8.2-opcache                     8.2.7-1~deb12u1                         amd64        Zend OpCache module for PHP
ii  php8.2-phpdbg                      8.2.7-1~deb12u1                         amd64        server-side, HTML-embedded scripting language (PHPDBG binary)
ii  php8.2-pspell                      8.2.7-1~deb12u1                         amd64        pspell module for PHP
ii  php8.2-readline                    8.2.7-1~deb12u1                         amd64        readline module for PHP
ii  php8.2-redis                       5.3.7+4.3.0-3                           amd64        PHP extension for interfacing with Redis
ii  php8.2-soap                        8.2.7-1~deb12u1                         amd64        SOAP module for PHP
ii  php8.2-xml                         8.2.7-1~deb12u1                         amd64        DOM, SimpleXML, XML, and XSL module for PHP
ii  php8.2-zip                         8.2.7-1~deb12u1                         amd64        Zip module for PHP
root@key:/var/log/php-fpm#

Interesting enough, I tried switching to multiple php versions, including default, and I get the same errors.

An error from the nextcloud log :

Code: Select all

{"reqId":"fSNbZNsMoL0A0UMh6GdI","level":3,"time":"2023-11-27T13:22:50+00:00","remoteAddr":"86.x.x.x","user":"Victor","app":"PHP","method":"PROPFIND","url":"/remote.php/dav/files/Victor/Kituri","message":"Zend OPcache can't be temporary enabled (it may be only disabled till the end of request) at Unknown#0","userAgent":"Mozilla/5.0 (Windows) mirall/3.10.1stable-Win64 (build 20231025) (Nextcloud, windows-10.0.22631 ClientArchitecture: x86_64 OsArchitecture: x86_64)","version":"27.1.3.2","data":{"app":"PHP"}}

Expected result
To work properly just like before.

Actual result
Doesn't work.

Steps to reproduce
No idea, I just rebooted the server and all hell broke loose.

Additional information
(e.g. recent changes to the server, excerpts from log files (/var/log/*, /var/log/keyhelp/php-error.log, etc.))
I've put an phpinfo file trying to debug issues and it's available on https://cloud.sysadmins.ro/blah.php if anyone wants to see it.
Other problems I'm experiencing are with dovecot.
I have a user with about 80gb of mails and I'm unable to raise the vsz_limit from the Dovecot configs (I raise it but Dovecot ignores it)

Code: Select all

2023-12-08T14:40:23.534501+02:00 key dovecot: imap(lorena@xxx.com)<1384136><uGzy4v4LL8RWfgeL>: Disconnected: Logged out in=212 out=1163
2023-12-08T14:40:23.592587+02:00 key dovecot: imap(lorena@xxx.com)<1384133><mwAi4/4LMsRWfgeL>: Fatal: block_alloc(4194304): Out of memory
2023-12-08T14:40:23.598097+02:00 key dovecot: imap(lorena@xxx.vom)<1384133><mwAi4/4LMsRWfgeL>: Error: Raw backtrace: #0 test_subprocess_fork[0x7f1749b95a20] -> #1 backtrace_append[0x7f1749b95c90] -> #2 backtrace_get[0x7f1749b95e20] -> #3 execvp_const[0x7f1749ba2f90] -> #4 i_syslog_fatal_handler[0x7f1749ba3900] -> #5 i_fatal_status[0x7f1749af9758] -> #6 i_fatal_status[0x7f1749afda58] -> #7 pool_alloconly_create_clean[0x7f1749bc4f30] -> #8 hash_table_create_direct[0x7f1749ba8a10] -> #9 maildir_uidlist_sync_next_uid[0x7f1749d091e0] -> #10 maildir_uidlist_sync_next[0x7f1749d096d0] -> #11 maildir_list_index_update_sync[0x7f1749d02750] -> #12 maildir_list_index_update_sync[0x7f1749d031f0] -> #13 maildir_list_index_update_sync[0x7f1749d036a0] -> #14 maildir_storage_sync_force[0x7f1749d037a0] -> #15 maildir_file_do[0x7f1749d09a70] -> #16 maildir_sync_is_synced[0x7f1749cfe280] -> #17 mail_get_stream_because[0x7f1749cccdd0] -> #18 index_mail_prefetch[0x7f1749d43010] -> #19 mail_prefetch[0x7f1749cc82a0] -> #20 index_search_get_mail[0x7f1749d4b2a0] -> #21 index_storage_search_next_nonblock[0x7f1749d4b800] -> #22 mailbox_search_next_nonblock[0x7f1749cd97d0] -> #23 mailbox_search_next[0x7f1749cd9820] -> #24 imap_fetch_begin[0x55f6e1100170] -> #25 imap_fetch_more[0x55f6e1100550] -> #26 cmd_fetch[0x55f6e10f4c00] -> #27 command_exec[0x55f6e10fd880] -> #28 cmd_x_cancel[0x55f6e1103510] -> #29 cmd_x_cancel[0x55f6e1103510] -> #30 cmd_x_cancel[0x55f6e1103510] -> #31 client_handle_input[0x55f6e1103880] -> #32 client_input[0x55f6e1103d80] -> #33 io_loop_call_io[0x7f1749bbac70] -> #34 io_loop_handler_run_internal[0x7f1749bbc970] -> #35 io_loop_handler_run[0x7f1749bbcb00] -> #36 io_loop_run[0x7f1749bbccd0] -> #37 master_service_run[0x7f1749b2d180] -> #38 main[0x55f6e10efa00] -> #39 __libc_init_first[0x7f17498e2150] -> #40 __libc_start_main[0x7f17498e2200] -> #41 _start[0x55f6e10f0000]
2023-12-08T14:40:23.600070+02:00 key dovecot: imap(lorena@xxx.com)<1384133><mwAi4/4LMsRWfgeL>: Fatal: master: service(imap): child 1384133 returned error 83 (Out of memory (service imap { vsz_limit=256 MB }, you may need to increase it) - set CORE_OUTOFMEM=1 environment to get core dump)

If anyone has any idea how to also fix this, I would appreciate it very much
But this is is a side issue, my main concern is nextcloud not running.
Forgot to mention, I used keyhelp-toolbox to re-write all the config files of apache/php etc multiple times, for panel configs and user configs, nothing changed.
Thank you!
Victor

[Florian]

Hallo,

since you can call phpinfo() under https://cloud.sysadmins.ro/blah.php the PHP is running on thsi subdomain normally.

The problems of your Nextcloud have got other causes.

Internal Server Error on Nextcloud can be caused by problems with the database or with activating caching tools like Redis

[Victor]

Hello Florian and thank you for the reply!
You were right, seems that after the mysql crash the nextcloud database got corrupted, I managed to re-install and everything is working fine, the last thing I thought to check was the damn mysql database.
About the Dovecot error do you know any solutions ? I'm guessing it's because of the huge inbox size, but I should be able to raise the vsz_limit to 3-4 GB, since the server has 32GB ram, but it seems the settings don't take effect.
I tried :

Code: Select all

# Default VSZ (virtual memory size) limit for service processes. This is mainly
# intended to catch and kill processes that leak memory before they eat up
# everything.
default_vsz_limit = 4096M

But I still get the 256mb error limit when the user receives a mail for example, also the user complains of constant disconnects from the imap server.
Thank you again for the support!

[Florian]

Hello,

create file /etc/doevcot/local.conf

Fill in your configuraton, e.g.

Code: Select all

service imap {
vsz_limit = 1024MB
}

Restart Dovecot.