Swag network gateway returned as Client/Real IP in access logs

I have SWAG running on three different servers, 1 self-hosted, the other 2 are Hetzner Cloud VPS. Everything works, except for seemingly the Client/Real IP. Looking at the access logs shows the swag docker network’s gateway as the client for nearly every request, but strangely not all (on some of the servers). SWAG is configured near exactly the same on all three servers. Self-hosted server is Solus 4.5. One Hetzner server is Debian 12, the other is NixOS 23.11

This URL queries for the accessing client’s IP address using my Moa (SearXNG fork) instance: https://find.ports.exposed/searx/search?q=ip
The IP it returns is 172.19.0.1, which is the swag network’s Gateway.

Solus Nginx access log snippet:

192.168.1.1 - - [21/May/2024:23:27:39 -0500] "GET /ScheduledTasks?IsEnabled=true HTTP/2.0" 200 2234 "-" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/124.0.0.0 Safari/537.36"
192.168.1.1 - - [21/May/2024:23:27:39 -0500] "GET /ScheduledTasks?IsEnabled=true HTTP/2.0" 200 2234 "-" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/124.0.0.0 Safari/537.36"
192.168.1.1 - - [21/May/2024:23:27:39 -0500] "GET /ScheduledTasks?IsEnabled=true HTTP/2.0" 200 2234 "-" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/124.0.0.0 Safari/537.36"
46.142.105.8 - - [21/May/2024:23:27:39 -0500] "PUT /_matrix/federation/v1/send/1716331983509 HTTP/1.1" 499 0 "-" "Synapse/1.90.0"
192.168.1.1 - - [21/May/2024:23:27:40 -0500] "GET /ScheduledTasks?IsEnabled=true HTTP/2.0" 200 2234 "-" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/124.0.0.0 Safari/537.36"
192.168.1.1 - - [21/May/2024:23:27:40 -0500] "GET /ScheduledTasks?IsEnabled=true HTTP/2.0" 200 2234 "-" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/124.0.0.0 Safari/537.36"
192.168.1.1 - - [21/May/2024:23:27:41 -0500] "GET /ScheduledTasks?IsEnabled=true HTTP/2.0" 200 2234 "-" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/124.0.0.0 Safari/537.36"
192.168.1.1 - - [21/May/2024:23:27:43 -0500] "GET /ScheduledTasks?IsEnabled=true HTTP/2.0" 200 2234 "-" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/124.0.0.0 Safari/537.36"
192.168.1.1 - - [21/May/2024:23:27:43 -0500] "GET /ScheduledTasks?IsEnabled=true HTTP/2.0" 200 2234 "-" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/124.0.0.0 Safari/537.36"
192.168.1.1 - - [21/May/2024:23:27:43 -0500] "GET /ScheduledTasks?IsEnabled=true HTTP/2.0" 200 2234 "-" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/124.0.0.0 Safari/537.36"
192.168.1.1 - - [21/May/2024:23:27:43 -0500] "GET /ScheduledTasks?IsEnabled=true HTTP/2.0" 200 2234 "-" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/124.0.0.0 Safari/537.36"
192.168.1.1 - - [21/May/2024:23:27:44 -0500] "GET /ScheduledTasks?IsEnabled=true HTTP/2.0" 200 2234 "-" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/124.0.0.0 Safari/537.36"
192.168.1.1 - - [21/May/2024:23:27:44 -0500] "GET /ScheduledTasks?IsEnabled=true HTTP/2.0" 200 2234 "-" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/124.0.0.0 Safari/537.36"
192.168.1.1 - - [21/May/2024:23:27:44 -0500] "GET /ScheduledTasks?IsEnabled=true HTTP/2.0" 200 2234 "-" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/124.0.0.0 Safari/537.36"
192.168.1.1 - - [21/May/2024:23:27:45 -0500] "GET /ScheduledTasks?IsEnabled=true HTTP/2.0" 200 2234 "-" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/124.0.0.0 Safari/537.36"
192.168.1.1 - - [21/May/2024:23:27:45 -0500] "GET /ScheduledTasks?IsEnabled=true HTTP/2.0" 200 2234 "-" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/124.0.0.0 Safari/537.36"

Debian:

172.28.0.1 - - [22/May/2024:07:27:24 +0300] "GET /api/v1/polls/5802599 HTTP/2.0" 200 249 "-" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/124.0.0.0 Safari/537.36"
159.69.152.150 - - [22/May/2024:07:27:25 +0300] "POST /inbox HTTP/1.1" 400 24 "-" "http.rb/5.2.0 (Mastodon/4.3.0-alpha.3; +https://101010.pl/)"
172.28.0.1 - - [22/May/2024:07:27:33 +0300] "GET /api/v1/polls/5802599 HTTP/2.0" 200 249 "-" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/124.0.0.0 Safari/537.36"
172.28.0.1 - - [22/May/2024:07:27:38 +0300] "GET /api/v1/pleroma/admin/announcements HTTP/2.0" 200 2 "-" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/124.0.0.0 Safari/537.36"
172.28.0.1 - - [22/May/2024:07:27:38 +0300] "GET /api/v1/announcements HTTP/2.0" 200 2 "-" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/124.0.0.0 Safari/537.36"
172.28.0.1 - - [22/May/2024:07:27:55 +0300] "GET /api/v1/polls/5802599 HTTP/2.0" 200 249 "-" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/124.0.0.0 Safari/537.36"
172.28.0.1 - - [22/May/2024:07:27:58 +0300] "GET /api/v1/pleroma/admin/reports HTTP/2.0" 200 208657 "-" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/124.0.0.0 Safari/537.36"
172.28.0.1 - - [22/May/2024:07:28:04 +0300] "GET /api/v1/polls/5802599 HTTP/2.0" 200 249 "-" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/124.0.0.0 Safari/537.36"
172.28.0.1 - - [22/May/2024:07:28:08 +0300] "GET /api/v1/announcements HTTP/2.0" 200 2 "-" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/124.0.0.0 Safari/537.36"
52.195.120.189 - - [22/May/2024:07:28:10 +0300] "POST /inbox HTTP/1.1" 400 24 "-" "http.rb/5.1.1 (Mastodon/4.1.15; +https://pawoo.net/)"
172.28.0.1 - - [22/May/2024:07:28:26 +0300] "GET /api/v1/polls/5802599 HTTP/2.0" 200 249 "-" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/124.0.0.0 Safari/537.36"
172.28.0.1 - - [22/May/2024:07:28:35 +0300] "GET /api/v1/polls/5802599 HTTP/2.0" 200 249 "-" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/124.0.0.0 Safari/537.36"
162.158.183.185 - - [22/May/2024:07:28:35 +0300] "POST /api/notes/global-timeline HTTP/2.0" 200 27448 "-" "axios/1.6.0"
172.69.58.167 - - [22/May/2024:07:28:36 +0300] "GET /api/v1/timelines/public?only_media=false&since_id=0 HTTP/2.0" 200 46628 "-" "Apache-HttpClient/4.5.13 (Java/11.0.20)"
23.88.105.145 - - [22/May/2024:07:28:38 +0300] "POST /inbox HTTP/1.1" 200 4 "-" "http.rb/5.1.1 (Mastodon/4.2.8; +https://bgme.me/)"
172.68.192.135 - - [22/May/2024:07:28:42 +0300] "POST /inbox HTTP/2.0" 202 0 "-" "http.rb/5.1.1 (Mastodon/4.2.8; +https://gametoots.de/)"
35.133.163.115 - - [22/May/2024:07:28:46 +0300] "POST /inbox HTTP/1.1" 200 4 "-" "Misskey/2024.3.1 (https://misskey.bubbletea.dev/)"
35.133.163.115 - - [22/May/2024:07:28:46 +0300] "GET /.well-known/nodeinfo HTTP/1.1" 200 227 "-" "Misskey/2024.3.1 (https://misskey.bubbletea.dev/)"
35.133.163.115 - - [22/May/2024:07:28:46 +0300] "GET /nodeinfo/2.1.json HTTP/1.1" 200 28044 "-" "Misskey/2024.3.1 (https://misskey.bubbletea.dev/)"
35.133.163.115 - - [22/May/2024:07:28:47 +0300] "GET /manifest.json HTTP/1.1" 200 471 "-" "Misskey/2024.3.1 (https://misskey.bubbletea.dev/)"
35.133.163.115 - - [22/May/2024:07:28:47 +0300] "GET / HTTP/1.1" 200 76625 "-" "Misskey/2024.3.1 (https://misskey.bubbletea.dev/)"
192.99.44.144 - - [22/May/2024:07:28:54 +0300] "POST /inbox HTTP/2.0" 200 4 "-" "AodeRelay (ap-relay/v0.3.82-HEAD-d644e837; +https://relay.gay)"
172.28.0.1 - - [22/May/2024:07:28:57 +0300] "GET /api/v1/polls/5802599 HTTP/2.0" 200 249 "-" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/124.0.0.0 Safari/537.36"
172.28.0.1 - - [22/May/2024:07:28:59 +0300] "GET /api/v1/pleroma/admin/reports HTTP/2.0" 200 208657 "-" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/124.0.0.0 Safari/537.36"
172.28.0.1 - - [22/May/2024:07:29:06 +0300] "GET /api/v1/polls/5802599 HTTP/2.0" 200 249 "-" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/124.0.0.0 Safari/537.36"
172.28.0.1 - - [22/May/2024:07:29:09 +0300] "GET /api/v1/announcements HTTP/2.0" 200 2 "-" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/124.0.0.0 Safari/537.36"
202.61.242.89 - - [22/May/2024:07:29:10 +0300] "POST /inbox HTTP/1.1" 200 4 "-" "http.rb/5.1.1 (Mastodon/4.2.8; +https://gametoots.de/)"
35.133.163.115 - - [22/May/2024:07:29:10 +0300] "POST /inbox HTTP/1.1" 200 4 "-" "Misskey/2024.3.1 (https://misskey.bubbletea.dev/)"
172.28.0.1 - - [22/May/2024:07:29:28 +0300] "GET /api/v1/polls/5802599 HTTP/2.0" 200 249 "-" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/124.0.0.0 Safari/537.36"
172.28.0.1 - - [22/May/2024:07:29:37 +0300] "GET /api/v1/polls/5802599 HTTP/2.0" 200 249 "-" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/124.0.0.0 Safari/537.36"

NixOS:

172.19.0.1 - - [22/May/2024:00:34:07 -0400] "POST /api/v4/users/ids?since=1716342050217 HTTP/2.0" 200 2 "-" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.6099.276 Electron/28.2.2 Safari/537.36 Mattermost/5.7.0"
172.19.0.1 - - [22/May/2024:00:34:07 -0400] "GET /plugins/com.mattermost.calls/config HTTP/2.0" 200 503 "-" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.6099.276 Electron/28.2.2 Safari/537.36 Mattermost/5.7.0"
172.19.0.1 - - [22/May/2024:00:34:07 -0400] "GET /plugins/github/api/v1/connected?reminder=false HTTP/2.0" 200 271 "-" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.6099.276 Electron/28.2.2 Safari/537.36 Mattermost/5.7.0"
172.19.0.1 - - [22/May/2024:00:34:08 -0400] "GET /plugins/com.mattermost.calls/channels HTTP/2.0" 200 46 "-" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.6099.276 Electron/28.2.2 Safari/537.36 Mattermost/5.7.0"
172.19.0.1 - - [22/May/2024:00:34:08 -0400] "GET /plugins/com.mattermost.calls/ybkwthwu33b13c5bue69ihxomh HTTP/2.0" 200 44 "-" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.6099.276 Electron/28.2.2 Safari/537.36 Mattermost/5.7.0"
172.19.0.1 - - [22/May/2024:00:34:18 -0400] "GET /plugins/github/api/v1/lhs-content HTTP/2.0" 200 703 "-" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.6099.276 Electron/28.2.2 Safari/537.36 Mattermost/5.7.0"
172.19.0.1 - - [22/May/2024:00:35:07 -0400] "POST /api/v4/users/status/ids HTTP/2.0" 200 851 "-" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.6099.276 Electron/28.2.2 Safari/537.36 Mattermost/5.7.0"
172.19.0.1 - - [22/May/2024:00:36:07 -0400] "POST /api/v4/users/status/ids HTTP/2.0" 200 851 "-" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.6099.276 Electron/28.2.2 Safari/537.36 Mattermost/5.7.0"
172.19.0.1 - - [22/May/2024:00:37:07 -0400] "POST /api/v4/users/status/ids HTTP/2.0" 200 851 "-" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.6099.276 Electron/28.2.2 Safari/537.36 Mattermost/5.7.0"
172.19.0.1 - - [22/May/2024:00:38:07 -0400] "POST /api/v4/users/status/ids HTTP/2.0" 200 851 "-" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.6099.276 Electron/28.2.2 Safari/537.36 Mattermost/5.7.0"
172.19.0.1 - - [22/May/2024:00:38:41 -0400] "GET / HTTP/1.1" 200 6630 "-" "Mozilla/5.0 (X11; Linux x86_64; rv:45.0) Gecko/20100101 Firefox/45.0"
172.19.0.1 - - [22/May/2024:00:38:43 -0400] "GET /?SSL_Labs_Renegotiation_Test=User_Agent_May_Not_Show HTTP/1.0" 400 0 "-" "SSL Labs (https://www.ssllabs.com/about/assessment.html)"
172.19.0.1 - - [22/May/2024:00:39:07 -0400] "POST /api/v4/users/status/ids HTTP/2.0" 200 851 "-" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.6099.276 Electron/28.2.2 Safari/537.36 Mattermost/5.7.0"
64.41.200.114 - - [22/May/2024:00:39:37 -0400] "GET / HTTP/1.1" 200 6630 "-" "Mozilla/5.0 (X11; Linux x86_64; rv:45.0) Gecko/20100101 Firefox/45.0"
64.41.200.114 - - [22/May/2024:00:39:39 -0400] "GET /?SSL_Labs_Renegotiation_Test=User_Agent_May_Not_Show HTTP/1.0" 400 0 "-" "SSL Labs (https://www.ssllabs.com/about/assessment.html)"
172.19.0.1 - - [22/May/2024:00:40:07 -0400] "POST /api/v4/users/status/ids HTTP/2.0" 200 851 "-" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.6099.276 Electron/28.2.2 Safari/537.36 Mattermost/5.7.0"

NixOS docker network inspect swag:

[
    {
        "Name": "swag",
        "Id": "9ae7ac8501fa8286be07f4edb3ae6cbe48f6cfe832d7c7ccea4a3f1c826cfe50",
        "Created": "2024-05-07T15:41:39.167375928-05:00",
        "Scope": "local",
        "Driver": "bridge",
        "EnableIPv6": false,
        "IPAM": {
            "Driver": "default",
            "Options": {},
            "Config": [
                {
                    "Subnet": "172.19.0.0/16",
                    "Gateway": "172.19.0.1"
                }
            ]
        },
        "Internal": false,
        "Attachable": false,
        "Ingress": false,
        "ConfigFrom": {
            "Network": ""
        },
        "ConfigOnly": false,
        "Containers": {
            "14d55b11d508dc983700a53abe770a68231d90b851778303897a5bb0a4b6a41d": {
                "Name": "moa",
                "EndpointID": "0b36f2e9672839f8f95a79838fddee68df2369963bd267ca7fe3f90f281a797c",
                "MacAddress": "02:42:ac:13:00:05",
                "IPv4Address": "172.19.0.5/16",
                "IPv6Address": ""
            },
            "33e63a75da372b6a0cc476d589e11b683f280fa2ebe0b76a7cac41153f2120be": {
                "Name": "morty",
                "EndpointID": "d3be164ae292d7009640437fd749f2ae0f711795af994b66a3a6dbb40cfd8d38",
                "MacAddress": "02:42:ac:13:00:06",
                "IPv4Address": "172.19.0.6/16",
                "IPv6Address": ""
            },
            "6510af7fa6a7de9e5ea7ad3fe550ead741a1fdaccfb4e630d4ceef51380de124": {
                "Name": "focalboard",
                "EndpointID": "dabc454a053315b53f3f20a8ac498df40bd378fba89c1041fc2e3dae9f9176e7",
                "MacAddress": "02:42:ac:13:00:04",
                "IPv4Address": "172.19.0.4/16",
                "IPv6Address": ""
            },
            "80c08e4cd241aa1a527d2a036371c4cd35260668af00d5c1a33b7abcbff5a3db": {
                "Name": "portainer",
                "EndpointID": "6600d2cfbfd83c82c0f6d96fcfb9151006c601b969964e81c8d4d687e94c4e7d",
                "MacAddress": "02:42:ac:13:00:07",
                "IPv4Address": "172.19.0.7/16",
                "IPv6Address": ""
            },
            "900c2384c93db8a8f55a43322a98da75cf8a2a9928b3282425ecb21d2b00c7e1": {
                "Name": "swag",
                "EndpointID": "789b4260d111f8e0f55dbfe7b3f62e2e4f303ec899bf9b24253a21987887c288",
                "MacAddress": "02:42:ac:13:00:02",
                "IPv4Address": "172.19.0.2/16",
                "IPv6Address": ""
            },
            "fe73f2ceee1813e65df1bf177504d19e82616e0fd8d116e04650814146aecb94": {
                "Name": "mattermost",
                "EndpointID": "45ebe5aad1d93b76f86878e0474b1b2dd8b83794405691573192a0b5e1be0647",
                "MacAddress": "02:42:ac:13:00:03",
                "IPv4Address": "172.19.0.3/16",
                "IPv6Address": ""
            }
        },
        "Options": {},
        "Labels": {}
    }
]

NixOS SWAG docker-compose.yml:

---
version: "2.1"
services:
  swag:
    image: lscr.io/linuxserver/swag:latest
# ghcr.io/linuxserver/swag:latest
    container_name: swag
    networks:
      - proxy-tier
      - crowdsec
    cap_add:
      - NET_ADMIN
    environment:
      - PUID=1000
      - PGID=1000
      - TZ=America/New_York
      - URL=****.**
      - SUBDOMAINS=****,****
      - VALIDATION=http
      - CERTPROVIDER=#zerossl
      - EMAIL=admin@****.**
      - ONLY_SUBDOMAINS=true
      - EXTRA_DOMAINS=find.ports.exposed,morty.ports.exposed,****
      - STAGING=false
      - DOCKER_MODS=linuxserver/mods:swag-crowdsec
#linuxserver/mods:universal-docker|linuxserver/mods:swag-crowdsec
#|linuxserver/mods:universal-apprise
#|linuxserver/mods:swag-cloudflare-real-ip
      - CROWDSEC_API_KEY=${CROWDSEC_API_KEY}
      - CROWDSEC_LAPI_URL=http://crowdsec:8080
    volumes:
      - ./config:/config
      - /var/run/docker.sock:/var/run/docker.sock:ro
    ports:
      - 443:443
      - 80:80
      - 8448:8448
      - 8008:8008
#      - 8443:8443/udp
#      - 8443:8443/tcp
    restart: unless-stopped
  crowdsec:
    image: docker.io/crowdsecurity/crowdsec:latest
    container_name: crowdsec
    environment:
      - GID=1000
      - BOUNCER_KEY_SWAG=${CROWDSEC_API_KEY}
      - COLLECTIONS=crowdsecurity/nginx crowdsecurity/http-cve crowdsecurity/whitelist-good-actors
    volumes:
      - crowdsec-db:/var/lib/crowdsec/data/
      - /opt/appdata/swag/log/nginx:/var/log/swag:ro
    networks:
      - crowdsec
    restart: unless-stopped
    security_opt:
      - no-new-privileges=true

volumes:
  crowdsec-db:

networks:
  crowdsec:
    driver: bridge
  proxy-tier:
    name: swag
    external: true

Moa / SearXNG proxy-conf:

server {
    listen 443 ssl http2;
    listen [::]:443 ssl http2;

    server_name find.ports.exposed;

    include /config/nginx/ssl.conf;

    client_max_body_size 0;

    # enable for ldap auth, fill in ldap details in ldap.conf
    #include /config/nginx/ldap.conf;

    # enable for Authelia
    #include /config/nginx/authelia-server.conf;

    location / {
        # enable the next two lines for http auth
        #auth_basic "Restricted";
        #auth_basic_user_file /config/nginx/.htpasswd;

        # enable the next two lines for ldap auth
        #auth_request /auth;
        #error_page 401 =200 /ldaplogin;

        # enable for Authelia
        #include /config/nginx/authelia-location.conf;

        include /config/nginx/proxy.conf;
        include /config/nginx/resolver.conf;
        set $upstream_app moa;
        set $upstream_port 8088;
        set $upstream_proto http;
        proxy_pass $upstream_proto://$upstream_app:$upstream_port;

# I know the headers below are mostly set in proxy.conf, this was just an attempt to get it working:
        proxy_set_header   Host             $host;
        proxy_set_header   Connection       $http_connection;
        proxy_set_header   X-Real-IP        $remote_addr;
        proxy_set_header   X-Forwarded-For  $proxy_add_x_forwarded_for;
        proxy_set_header   X-Scheme         $scheme;
        proxy_set_header   X-Script-Name    /searx;
    }

    location /static/ {
        alias /usr/local/moa/moa-src/moa/static/;
    }
}

server {
    listen 443 ssl http2;
    listen [::]:443 ssl http2;

    server_name morty.ports.exposed;

    include /config/nginx/ssl.conf;

    client_max_body_size 0;

    # enable for ldap auth, fill in ldap details in ldap.conf
    #include /config/nginx/ldap.conf;

    # enable for Authelia
    #include /config/nginx/authelia-server.conf;

    location / {
        # enable the next two lines for http auth
        #auth_basic "Restricted";
        #auth_basic_user_file /config/nginx/.htpasswd;

        # enable the next two lines for ldap auth
        #auth_request /auth;
        #error_page 401 =200 /ldaplogin;

        # enable for Authelia
        #include /config/nginx/authelia-location.conf;

        include /config/nginx/proxy.conf;
        include /config/nginx/resolver.conf;
        set $upstream_app morty;
        set $upstream_port 3000;
        set $upstream_proto http;
        proxy_pass $upstream_proto://$upstream_app:$upstream_port;

        proxy_set_header   Host             $host;
        proxy_set_header   Connection       $http_connection;
        proxy_set_header   X-Real-IP        $remote_addr;
        proxy_set_header   X-Forwarded-For  $proxy_add_x_forwarded_for;
        proxy_set_header   X-Scheme         $scheme;
    }
}

The NixOS server is the main one I need reporting the correct client IP at the moment so that I can get Mattermost calls working and the IP SearXNG feature working so that it can block IPs, along with getting Fail2Ban & CrowdSec working.

I’ve been troubleshooting this issue literally all day and I feel like I’m even more confused than when I started. Any help would be greatly appreciated!

EDIT: Forgot to mention, the NixOS firewall has been completely disabled, along with the firewall on Solus. Those two rely on the Hetzner firewall & Unifi firewall respectively.
2nd EDIT: find.ports.exposed is using Cloudflare for DNS only, unsure if that’s important since the issue is occuring for domains not behind Cloudflare as well.

Still experiencing this issue, haven’t figured anything out.

Only time I’ve seen something similiar is on retail nas boxes like synology, you have to adjust iptables for client ip’s to pass through: Exposing the client IPs to Docker containers in Synology NAS

Unfortunatly I don’t use NixOS so unable to help specifically with that.