Thelounge container fails to launch with file watcher limit reached errors

LSIO Discussion Container Support
1

Hey all,

I’m trying to run amd64-latest version of this container assisted by Portainer on an OpenMediaVault install (which is debian 10). This host is a haswell core i5 with 16gb ram, my config directories all live on a ZFS RAIDz1 pool of 4x 2tb enterprise drives. This host also runs about 20 other docker containers including many from LinuxServer.io and a handful of other services. It’s not heavily used, just my small family and a couple family access it.

Launching the container populates my config directory and I can make the tweaks mentioned in the instructions at but regardless of what I’ve changed so far, the container fails to start. Logs show this:

2020-04-30 20:35:10 [INFO] The Lounge v4.1.0 (Node.js 12.15.0 on linux x64)
2020-04-30 20:35:10 [INFO] Configuration file: /config/config.js
2020-04-30 20:35:10 [INFO] Available at http://[::]:9000/ in private mode
internal/fs/watchers.js:173
    throw error;
    ^
Error: ENOSPC: System limit for number of file watchers reached, watch '/config/packages/package.json'
    at FSWatcher.start (internal/fs/watchers.js:165:26)
    at Object.watch (fs.js:1329:11)
    at watchPackages (/usr/lib/node_modules/thelounge/src/plugins/packages/index.js:144:5)
    at Object.loadPackages (/usr/lib/node_modules/thelounge/src/plugins/packages/index.js:140:2)
    at Server.<anonymous> (/usr/lib/node_modules/thelounge/src/server.js:187:12)
    at Object.onceWrapper (events.js:312:28)
    at Server.emit (events.js:223:5)
    at emitListeningNT (net.js:1340:10)
    at processTicksAndRejections (internal/process/task_queues.js:80:21) {
  errno: -28,
  syscall: 'watch',
  code: 'ENOSPC',
  path: '/config/packages/package.json',
  filename: '/config/packages/package.json'
}
[cont-finish.d] executing container finish scripts...
[cont-finish.d] done.
[s6-finish] waiting for services.
[s6-finish] sending all processes the TERM signal.

I realize this post is similar to Thelounge v. 4.0.0 "System limit for number of file watchers reached" but I hope I can provide more reproducible and useful information here.

Thanks!

2

in #jupiterbroadcasting on geekshed irc:

│07:32:40 akik | ndroftheline: is that about fs.inotify.max_user_watches ?
│07:32:58 akik | if so you can change it with sysctl config
│07:33:22 akik | the change on the host gets propagated to the container

this apperas to have been the culprit.

here’s some more detail: inotify(7) - Linux manual page

a nice simple article on how to change it in debian: How to increase the max number of inotify watches on Debian/Ubuntu – Alex Dumitru

this worked for me, hope it helps someone else too

3

i find it slightly strange that other, seemingly much more file intensive apps like jellyfin don’t have a problem with the default filesystem watches of 8192 on debian 10, but thelounge does - which, at least to my lazy eye, shouldn’t do a whole lot with the filesystem.

more references:

docker inherits the proc filesystem from the host, you can’t set it per container. you can set it per user in theory, i didn’t try that. any case: https://github.com/cdr/code-server/issues/628

this has a good answer that discusses the implications of just slamming fs.inotify.max_user_watches to 524288 (in short, more or less legligible amount of increased RAM usage, up to 32MB): https://askubuntu.com/questions/154255/how-can-i-tell-if-i-am-out-of-inotify-watches

so, with a debian host:
on the docker host (not in the container), with root priv:

sysctl fs.inotify.max_user_watches=524288

test. if it works, you can make it survive reboots by adding that string to /etc/sysctl.conf:
echo fs.inotify.max_user_watches=524288 | sudo tee -a /etc/sysctl.conf && sudo sysctl -p

1 Like