I’ve just updated a nextcloud container (offical image) and its associated mariadb (linuxserver.io image) and redis (official image) containers for the first time since 23rd August 2022 and I’ve run into a problem…
I peg the nextcloud container to a particular version (the update was to change from 24.0.4 to 24.0.5) but let mariadb and redis run the latest version that docker can pull.
Nextcloud now won’t start and the web interface gives the following error:
Internal Server Error
The server encountered an internal error and was unable to complete your request.
Please contact the server administrator if this error reappears multiple times, please include the technical details below in your report.
More details can be found in the server log.
The nextcloud container log didn’t reveal anything useful at all; it just showed the web request that displayed the error above.
Trying nextcloud’s associated mariadb container (this is the linuxserver.io version of mariadb) reveals a log containing the following sequence that repeats every second:
Caught SIGTERM signal!
cat: /var/run/mysqld/mysqld.pid: No such file or directory
221023 12:10:36 mysqld_safe Logging to '/config/databases/nextcloud-mariadb.err'.
221023 12:10:36 mysqld_safe Starting mariadbd daemon with databases from /config/databases
I’ve tried exec’ing into the mariadb container to touch /var/run/mysqld/mysqld.pid but the file is never created despite the parent directory having 777 permissions. Can’t create it as either the root, abc, or mysql users. Can create files find in /var/run, just not in /var/run/mysqld.
Updating the host OS (Ubuntu 20.04 LTS) and rebooting has not made a difference. And I’m unsure where to go from here as the error affects an ephemeral directory, not one mapped to a docker volume/
Other output:
root@nextcloud-mariadb:/# ls -l /var/run
lrwxrwxrwx 1 root root 4 Oct 5 18:33 /var/run -> /run
root@nextcloud-mariadb:/# ls -l /run
total 4
drwxrwxrwx 1 abc users 0 Oct 23 13:40 mysqld
drwxr-xr-x 1 root root 90 Oct 23 13:38 s6
drwxr-xr-x 1 root root 16 Oct 23 13:38 s6-linux-init-container-results
lrwxrwxrwx 1 root root 23 Oct 23 13:38 s6-rc -> s6-rc:s6-rc-init:LHjKnb
drwxr-xr-x 1 root root 82 Oct 23 13:38 s6-rc:s6-rc-init:LHjKnb
drwxr-xr-x 1 root root 152 Oct 23 13:38 service
root@nextcloud-mariadb:/# ls -l /var/run
lrwxrwxrwx 1 root root 4 Oct 5 18:33 /var/run -> /run
root@nextcloud-mariadb:/# ls -l /var/run/mysqld/
total 0
looking elsewhere for usefil logs, there’s also a repeating loop of the following from /config/databases/nextcloud-mariadb.err
221023 13:54:43 mysqld_safe mysqld from pid file /var/run/mysqld/mysqld.pid ended
221023 13:54:44 mysqld_safe Starting mariadbd daemon with databases from /config/databases
2022-10-23 13:54:44 0 [Note] /usr/bin/mariadbd (server 10.6.10-MariaDB-log) starting as process 221180 ...
2022-10-23 13:54:44 0 [ERROR] /usr/bin/mariadbd: Error while setting value 'none' to 'innodb-checksum-algorithm'
2022-10-23 13:54:44 0 [ERROR] Parsing options for plugin 'InnoDB' failed.
2022-10-23 13:54:44 0 [Note] Plugin 'FEEDBACK' is disabled.
2022-10-23 13:54:44 0 [ERROR] /usr/bin/mariadbd: unknown variable 'innodb_buffer_pool_size=1G'
2022-10-23 13:54:44 0 [ERROR] Aborting
221023 13:54:44 mysqld_safe mysqld from pid file /var/run/mysqld/mysqld.pid ended
find all maridb config files mentioning innodb_buffer_pool_size:
root@nextcloud-mariadb:/# find / -name "*cnf" -exec grep -H innodb_buffer_pool_size {} \;
/etc/my.cnf.d/custom.cnf:#innodb_buffer_pool_size= 256M
/etc/my.cnf.d/custom.cnf:innodb_buffer_pool_size = 1G
/config/custom.cnf:#innodb_buffer_pool_size= 256M
/config/custom.cnf:innodb_buffer_pool_size = 1G
/defaults/my.cnf:innodb_buffer_pool_size = 256M
find all mariadb config files mentioning innodb_checksum_algorithm:
root@nextcloud-mariadb:/# find / -name "*cnf" -exec grep -H innodb_checksum_algorithm {} \;
/etc/my.cnf.d/custom.cnf:innodb_checksum_algorithm = none
/config/custom.cnf:innodb_checksum_algorithm = none
also noting that:
root@nextcloud-mariadb:/# ls -l /etc/my.cnf.d/custom.cnf
lrwxrwxrwx 1 root root 18 Oct 23 13:38 /etc/my.cnf.d/custom.cnf -> /config/custom.cnf
I’ve had a quick search for innodb_buffer_pool_size being unknown, but the few results I’ve looked at so far relate to having that variable in the wrong section of the config file, instead of in [mysqld] - and it’s in the right section of my config file (along with a bunch of other innodb options that don’t seem to cause trouble.
Much as I dislike leaving a problem unsolved like this, I have to go out for the rest of the day now, so does anyone have any tips for the next troubleshooting steps I could take when I get back?
thanks, Richard