Linuxserver/jellyfin container won't start while jellyfin/jellyfin does

Brian · 4 November 2023 16:51

I am unable to start the lscr.io/linuxserver/jellyfin:latest lately. On a long-running server I performed no update, just added some media files (relatively small library). But, trying with the jellyfin/jellyfin image, all goes well.

linuxserver/jellyfin details

docker-compose

My docker-compose is very close to the suggested:

version: "2.1"
services:
  jellyfin:
    image: lscr.io/linuxserver/jellyfin:latest
    container_name: jellyfin
    environment:
      - PUID=1000
      - PGID=1000
      - TZ=Europe/London
      # - JELLYFIN_PublishedServerUrl=192.168.1.5 #optional
    volumes:
      - /mnt/DC67-0ADB/jellyfin/config:/config
      - /mnt/DC67-0ADB/TV:/data/tvshows
      - /mnt/DC67-0ADB/Movies:/data/movies
    ports:
      - 8096:8096
      - 8920:8920 #optional
      - 7359:7359/udp #optional
      - 1900:1900/udp #optional
    restart: unless-stopped

It had previously been running without issue for over 6 months.

Device and software details

Device: Jetson Nano (arm64)
Docker version: 20.10.7
Portainer version: 20.15.1
linuxserver/jellyfin version: "build_version": "Linuxserver.io version:- 10.8.11-1-ls231 Build-date:- 2023-10-31T12:43:55+00:00"

Jellyfin Container Logs

Repeat 174562 times:
--------------------------------
   at System.IO.FileSystemWatcher+RunningInstance+WatchedDirectory.Write(System.Text.StringBuilder, Boolean)
--------------------------------
   at System.IO.FileSystemWatcher+RunningInstance+WatchedDirectory.GetPath(Boolean, System.String)
   at System.IO.FileSystemWatcher+RunningInstance.AddDirectoryWatchUnlocked(WatchedDirectory, System.String)
   at System.IO.FileSystemWatcher+RunningInstance.AddDirectoryWatchUnlocked(WatchedDirectory, System.String)
   at System.IO.FileSystemWatcher+RunningInstance.AddDirectoryWatchUnlocked(WatchedDirectory, System.String)
   at System.IO.FileSystemWatcher+RunningInstance.AddDirectoryWatchUnlocked(WatchedDirectory, System.String)
   at System.IO.FileSystemWatcher+RunningInstance.AddDirectoryWatchUnlocked(WatchedDirectory, System.String)
   at System.IO.FileSystemWatcher+RunningInstance..ctor(System.IO.FileSystemWatcher, Microsoft.Win32.SafeHandles.SafeFileHandle, System.String, Boolean, System.IO.NotifyFilters, System.Threading.CancellationToken)
   at System.IO.FileSystemWatcher.StartRaisingEvents()
   at System.IO.FileSystemWatcher.StartRaisingEventsIfNotDisposed()
   at System.IO.FileSystemWatcher.set_EnableRaisingEvents(Boolean)
   at Microsoft.Extensions.FileProviders.Physical.PhysicalFilesWatcher.TryEnableFileSystemWatcher()
   at Microsoft.Extensions.FileProviders.Physical.PhysicalFilesWatcher.CreateFileChangeToken(System.String)
   at Microsoft.Extensions.FileProviders.PhysicalFileProvider.Watch(System.String)
   at Microsoft.Extensions.Configuration.FileConfigurationProvider.<.ctor>b__1_0()
   at Microsoft.Extensions.Primitives.ChangeToken.OnChange(System.Func`1<Microsoft.Extensions.Primitives.IChangeToken>, System.Action)
   at Microsoft.Extensions.Configuration.FileConfigurationProvider..ctor(Microsoft.Extensions.Configuration.FileConfigurationSource)
   at Microsoft.Extensions.Configuration.Json.JsonConfigurationSource.Build(Microsoft.Extensions.Configuration.IConfigurationBuilder)
   at Microsoft.Extensions.Configuration.ConfigurationBuilder.Build()
   at Jellyfin.Server.Program.CreateAppConfiguration(Jellyfin.Server.StartupOptions, MediaBrowser.Common.Configuration.IApplicationPaths)
   at Jellyfin.Server.Program+<StartApp>d__9.MoveNext()
   at System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[[Jellyfin.Server.Program+<StartApp>d__9, jellyfin, Version=10.8.11.0, Culture=neutral, PublicKeyToken=null]](<StartApp>d__9 ByRef)
   at System.Runtime.CompilerServices.AsyncTaskMethodBuilder.Start[[Jellyfin.Server.Program+<StartApp>d__9, jellyfin, Version=10.8.11.0, Culture=neutral, PublicKeyToken=null]](<StartApp>d__9 ByRef)
   at Jellyfin.Server.Program.StartApp(Jellyfin.Server.StartupOptions)
   at CommandLine.ParserResultExtensions.MapResult[[System.__Canon, System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e],[System.__Canon, System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]](CommandLine.ParserResult`1<System.__Canon>, System.Func`2<System.__Canon,System.__Canon>, System.Func`2<System.Collections.Generic.IEnumerable`1<CommandLine.Error>,System.__Canon>)
   at Jellyfin.Server.Program.Main(System.String[])
   at Jellyfin.Server.Program.<Main>(System.String[])
Stack overflow.
Repeat 174511 times:

... <abbreviated for space. many repeats>... 

Repeat 174510 times:
--------------------------------
   at System.IO.FileSystemWatcher+RunningInstance+WatchedDirectory.Write(System.Text.StringBuilder, Boolean)
--------------------------------
   at System.IO.FileSystemWatcher+RunningInstance+WatchedDirectory.GetPath(Boolean, System.String)
   at System.IO.FileSystemWatcher+RunningInstance.AddDirectoryWatchUnlocked(WatchedDirectory, System.String)
   at System.IO.FileSystemWatcher+RunningInstance.AddDirectoryWatchUnlocked(WatchedDirectory, System.String)
   at System.IO.FileSystemWatcher+RunningInstance..ctor(System.IO.FileSystemWatcher, Microsoft.Win32.SafeHandles.SafeFileHandle, System.String, Boolean, System.IO.NotifyFilters, System.Threading.CancellationToken)
   at System.IO.FileSystemWatcher.StartRaisingEvents()
   at System.IO.FileSystemWatcher.StartRaisingEventsIfNotDisposed()
   at System.IO.FileSystemWatcher.set_EnableRaisingEvents(Boolean)
   at Microsoft.Extensions.FileProviders.Physical.PhysicalFilesWatcher.TryEnableFileSystemWatcher()
   at Microsoft.Extensions.FileProviders.Physical.PhysicalFilesWatcher.CreateFileChangeToken(System.String)
   at Microsoft.Extensions.FileProviders.PhysicalFileProvider.Watch(System.String)
   at Microsoft.Extensions.Configuration.FileConfigurationProvider.<.ctor>b__1_0()
   at Microsoft.Extensions.Primitives.ChangeToken.OnChange(System.Func`1<Microsoft.Extensions.Primitives.IChangeToken>, System.Action)
   at Microsoft.Extensions.Configuration.FileConfigurationProvider..ctor(Microsoft.Extensions.Configuration.FileConfigurationSource)
   at Microsoft.Extensions.Configuration.Json.JsonConfigurationSource.Build(Microsoft.Extensions.Configuration.IConfigurationBuilder)
   at Microsoft.Extensions.Configuration.ConfigurationBuilder.Build()
   at Jellyfin.Server.Program.CreateAppConfiguration(Jellyfin.Server.StartupOptions, MediaBrowser.Common.Configuration.IApplicationPaths)
   at Jellyfin.Server.Program+<StartApp>d__9.MoveNext()
   at System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[[Jellyfin.Server.Program+<StartApp>d__9, jellyfin, Version=10.8.11.0, Culture=neutral, PublicKeyToken=null]](<StartApp>d__9 ByRef)
   at System.Runtime.CompilerServices.AsyncTaskMethodBuilder.Start[[Jellyfin.Server.Program+<StartApp>d__9, jellyfin, Version=10.8.11.0, Culture=neutral, PublicKeyToken=null]](<StartApp>d__9 ByRef)
   at Jellyfin.Server.Program.StartApp(Jellyfin.Server.StartupOptions)
   at CommandLine.ParserResultExtensions.MapResult[[System.__Canon, System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e],[System.__Canon, System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]](CommandLine.ParserResult`1<System.__Canon>, System.Func`2<System.__Canon,System.__Canon>, System.Func`2<System.Collections.Generic.IEnumerable`1<CommandLine.Error>,System.__Canon>)
   at Jellyfin.Server.Program.Main(System.String[])
   at Jellyfin.Server.Program.<Main>(System.String[])

What I’ve tried

searched discourse “jellyfin”. no relevant topics found (as far as I can tell)
googled the container logs
Phind queried the container logs
increased inotify limit: https://jellyfin.org/docs/general/administration/troubleshooting/#real-time-monitoring. No success, no change to container error logs.
Tried running Plex, on the same media volume. works just fine.
checked uid:gid on the host id $whoami → # uid=1000 gid=1000
checked uid:gid inside the container id $whoami → # uid=0(root) gid=0(root) groups=0(root)
… thus I tried swapping 0:0 in the docker-compose. no bueno. per the troubleshooting above for user: 0:0 and/or for the environment: PUID=0 GUID=0. no bueno.
checked all files in the /mnt/ location

(host): ls -l /path/to/media/
    drwxrwxrwx 1 root root       1048576 Jan 15  2023 ...
    -rwxrwxrwx 1 root root    1441080880 May  5  2018 ...

And from inside the docker container:

(container) ls -l /data/movies #
   drwxrwxrwx 1 root root    1048576 Jan 15  2023 ...
   -rwxrwxrwx 1 root root 1441080880 May  5  2018 ...

Setup and run the jellyfin/jellyfin image (per https://jellyfin.org/docs/general/installation/container/#using-docker-compose). With user: 1000:1000

Conclusion

I’m pretty stumped, folks. Any help would be much appreciated. I think I’ve exhausted all the googling I can do on this one.

driz · 6 November 2023 12:57

It will probably be best for you to pop into discord for support. I will note that we do not support or recommend deploying our containers with portainer. It also looks like you are using a remote mount for /config, we do not support this either. Finally, your docker version is CRAZY OLD and there have been a ton of critical updates since your version was released in June of 2021.

This is because you checked the uid/gid incorrectly inside the container, you went in as root, so you saw root. docker exec -it -u abc jellyfin id $whoami

Brian · 6 November 2023 15:14

Thank you for the response!

TLDR; Seems Solved

I changed the config volume from the mounted HDD to local storage:

volumes:
  - /home/my_username/Documents/jellyfin-lscr/config:/config

All seems to be running correctly now. I’ll update back if anything goes wrong after metadata are re-downloaded et c.

Further Debugging Steps, for Posterity

I wanted to post to Discord before here, but the phone number verification is too restrictive
re Portainer: I get ya. In this case, I don’t think it’s related however? If I run the same docker-compose directly ( (host)$ docker compose up) The error is identical.
re checking uid/gid. whoops, goof on my part. Issue is similar: (host)$ docker exec -it -u abc jellyfin-lscr id $whoami uid=1000(abc) gid=1000(abc) groups=1000(abc),100(users) (note that I changed the container name to jellyfin-lscr just for distinction from jellyfin/jellyfin)
re Docker version, for all the above, I first upgraded docker: ((host)$ docker --version Docker version 24.0.2, build cb74dfc) . Same error.
re Remote mount: I’m linking from a physical HDD directly connected via USB to the host (jetson nano). /mnt/DC67-0ADB is the mount location, setup in fstab sometime ago (and didn’t exactly know what I was doing. like here…). Changing the mount to on-board storage:

volumes:
  - /home/my_username/Documents/jellyfin-lscr/config:/config

seems to have solved the issue.

Remaining Question

I wonder why the config location became a problem after all this time. Especially since it works on the jellyfin/jellyfin build (unless that was a ticking clock as well).

Close

Thanks for the help! Hopefully if anyone else comes across this, it helps solve their issue as well.

driz · 6 November 2023 15:24

it uses a database and file based remote mounts (NFS/CIFS/SMB/etc) can cause locking issues with databases because they are not intended to be used on this kind of storage. So it can work for long times and then just magically break or worse, become corrupted for months until your backups age out the pre-corruption and you’re stuck it’s a bad time. If you MUST use remote storage for any volume with a database (with our containers, we keep this in /config) then you need to used a block based remote storage type, FC, FCoE, iSCSI, etc

system · 11 November 2023 15:25

This topic was automatically closed 5 days after the last reply. New replies are no longer allowed.