ModuleNotFoundError: No module named 'python_on_whales' when using INSTALL_PIP_PACKAGES

[jk2018]

I am running Duplicati to back up my Nextcloud container instance. I need to run a python script using the run-script-before-required Advanced option in the GUI. Here is my docker-compose.yml:

---
version: "2.1"
services:
  duplicati:
    image: lscr.io/linuxserver/duplicati:latest
    container_name: duplicati
    environment:
      - PUID=1000
      - PGID=1000
      - TZ=America/Chicago
      - CLI_ARGS= #optional
      - DOCKER_MODS=linuxserver/mods:universal-package-install
      - INSTALL_PACKAGES=python3
      - INSTALL_PIP_PACKAGES=python-on-whales
    volumes:
      - ./config:/config
      - ./scripts:/scripts #<--Location of python scripts
      - /backup:/backups #<--Location where backups are placed
      - /data:/source #<--Location of source files
    ports:
      - 8200:8200
    restart: unless-stopped

I get no errors when bringing the Duplicati container up, and pip loads the python-on-whales module. I can docker exec into the container and confirm this with pip list:

root@2249d4e70ed8:/# pip list
WARNING: The directory '/config/.cache/pip' or its parent directory is not owned or is not writable by the current user. The cache has been disabled. Check the permissions and owner of that directory. If executing pip with sudo, you should use sudo's -H flag.
Package            Version
------------------ --------
certifi            2023.5.7
charset-normalizer 3.1.0
click              8.1.3
idna               3.4
pip                23.1.2
pydantic           1.10.9
python-on-whales   0.62.0
requests           2.31.0
setuptools         68.0.0
tqdm               4.65.0
typer              0.9.0
typing_extensions  4.6.3
urllib3            2.0.3
wheel              0.40.0

But, if I try to manually run a script from the command line inside the container, I get this error:

root@2249d4e70ed8:/# /usr/bin/python3 /scripts/backup_nextcloud.py
Traceback (most recent call last):
  File "/scripts/backup_nextcloud.py", line 6, in <module>
    from python_on_whales import docker
ModuleNotFoundError: No module named 'python_on_whales'

Here is the beginning of my python script (backup_nextcloud.py) that is relevent:

#! /usr/bin/python3

from subprocess import Popen
from datetime import datetime, date
from pathlib import Path
from python_on_whales import docker
from python_on_whales import DockerClient

It seems to be able to find the other modules, just not the one I installed through the docker-compose.yml file using the INSTALL_PIP_PACKAGES.

How do I point python to the python-on-whales module? Or, am I doing something incorrectly?

[aptalca]

Do docker exec duplicati which python3 and you’ll see why

[jk2018]

>>> $ docker exec duplicati which python3
/usr/bin/python3

I had already added this path in my command, which is why this is so confusing.

[aptalca]

@jk2018

$ docker exec duplicati which python3
/usr/bin/python3

That’s not the right PATH

Are you using portainer or synology gui to manage the container?

Here’s my output:

$ docker run -d --rm --name test -e DOCKER_MODS=linuxserver/mods:universal-package-install -e INSTALL_PIP_PACKAGES=python-on-whales lscr.io/linuxserver/duplicati
81530be80488f83668fec2d0f382b65bf4325df58a5eaac83eca1af91b5be679

$ docker exec test which python3
/lsiopy/bin/python3

$ docker exec test bash -c 'echo $PATH'
/lsiopy/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin

tl;dr, our newer images use a venv that is located inside the container at /lsiopy and PATH inside the container is fixed to include that location first. So when you run python3 or pip, it should run /lsiopy/bin/python3. If yours doesn’t do that, likely your PATH inside the container is messed up and portainer and synology gui are known to do that.

See here: Known Issue: Portainer & Synology Docker UI | Info :: LinuxServer.io

[jk2018]

I do not use synology or portainer. I like the control of spinning up my stacks manually from the command line using compose files.

After verifying I was getting a different python path than you, but that /lsiopy/bin was in my path, I brought down my container, and ran

docker system prune -a

I spun up my container again, and ran the commands you gave me:

>>> $ docker exec duplicati which python3
/lsiopy/bin/python3
>>> $ docker exec duplicati bash -c 'echo $PATH'
/lsiopy/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin

Rebuilding my container seems to have fixed the problem. I am getting a new error, I believe is related to needing the docker.sock configured, but that is not related to this topic.

Thank you for your help, @aptalca ! You gave me some troubleshooting steps I can use in the future.

When in doubt… docker system prune -a