Swag vs. docker DNS vs. slow docker startup

Hi,

I’ve been running swag for quite some time now on my minimalist server running on a Raspberry Pi 4. Recently, I’ve started running into issues of 502 errors after:
A- an automatic update of a container by WatchTower – let’s not focus on that, as there might be solutions via WatchTower depending on what the root explanation turns out to be
B- a full system reboot

In both cases, it seems that swag fails to find some container (usually my grafana one) and gives me an upstream error: http://172.0.0.14 : connection refused. The configuration files do not use IPs, but Docker DNS names (i.e. ‘grafana’, so it normally connects to http://grafana). However, I now see IP addresses in the error logs. That it errors makes sense: when I check, the IP it’s trying to connect to is not (or no longer) the Docker-assigned IP corresponding to the container that should be reached. My hypothesis is that swag gets a DNS translation from Docker that matches the previous version of that container, because the new container still has to load. Then when the new container is loaded, it has a different IP than before. Indeed, the one thing that has changed recently on my system is that I’ve added quite a few new containers, so Docker is now much slower to start everything. This, I think, could explain scenario B above.

For reference my containers are created via Ansible, and I do not use docker-compose. My swag container and the container it needs are on a dedicated Docker bridge network. I also use Portainer for quick experiments, log-checking, and restarts.

My questions are:
1- is my hypothesis correct? or is there another reason why I’m seeing IP addresses in nginx logs when my config is not using IP addresses?
2- is there a clean solution to this problem?
3- lacking a better solution, would it work to set up my containers with a fixed IP chosen arbitrarily by me for each of them, so that that IP doesn’t change between container or system restarts?

Thanks in advance for your help!
Pierric.

Post your nginx config for grafana.

Sure, here you go!

server {
   listen       80;
   server_name  graf.*;
   rewrite ^ https://$http_host$request_uri? permanent;    # force redirect to https
   server_tokens off;
}

server {
    listen 443 ssl;
    listen [::]:443 ssl;

    server_name  graf.*;

    include /config/nginx/ssl.conf;
    include /config/nginx/authelia-server.conf;

    client_max_body_size 0;

    location / {
        include /config/nginx/authelia-location.conf;
        include /config/nginx/proxy.conf;
        include /config/nginx/resolver.conf;
        proxy_pass         http://grafana:3000;

        access_log      /config/log/nginx/grafana.access.log;
        error_log       /config/log/nginx/grafana.error.log;
    }
}     

Also, my bad, the error isn’t connection refused, but host unreachable. Here’s an example:

2022/11/08 21:23:32 [error] 296#296: *13203 connect() failed (113: Host is unreachable) while connecting to upstream, client: 192.168.1.34, server: graf.*, request: "GET /favicon.ico HTTP/2.0", upstream: "http://172.22.0.14:3000/favicon.ico", host: "graf.edited.net", referrer: "https://graf.edited.net/d/moMbc5igz/pi-monitoring?orgId=1"

From memory, when I checked the IP of the grafana container after that error, it was something like 172.22.0.4 (not 14)

Pierric.

Yes your assumption is correct. When using the container name in the proxy_pass line, nginx only resolves the IP at startup.

This can, and is solved with the stock configs in swag, by using nginx variables, hence the upstream_app. Simply adopt the upstream_app paradigm and you should be set to go.

1 Like

Thank you @Roxedus ! Based on your response I could find the related section of the docs, and it all makes sense. I guess I must have started with that format early on and then decided to “simplify”, not knowing the consequences! I will revert to using variables in all my configs when I get a chance.

This topic was automatically closed 5 days after the last reply. New replies are no longer allowed.