Newbie SWAG help - trying to use in unRAID to reverse proxy to a VM not a container

NLS · 31 October 2021 16:15

I installed SWAG in my unRAID server, using SpaceInvader One’s guides, although there is no “pure” guide for SWAG.

Anyway I tried to follow it as much as I could, by my MAIN usage is not to use it to reverse proxy to other containers, but also use it to reach other servers in VMs I have on unRAID.

Some info about my setup:

unRAID on a server (duh!).
NethServer on a VM.
I have my own domain (the host uses cpanel)… let’s call it my.domain.
Technitium DNS on a container, but using “br0” network type (seems to work ok like that), I use that as DNS for my internal network. My internal domain is in the form local.my.domain.
My router supports only certain ddns providers, but I have it set properly.
I have properly set in cpanel to resolve (cname) to ddns (this works fine, way before I installed SWAG).
I defined a custom container network for SWAG, although I haven’t used it on another container yet.

So…

First, the SECONDARY thing I want to point out (not the actual reason I post):
When I changed the container to validate using dns, cpanel, I came across two issues:

I couldn’t edit /appdata/swag/dns-conf/cpanel.ini (to put my proper cpanel credentials) from anything else than open a terminal to my unRAID server and edit it from there. Are the security settings correct?
(whereas proxy-confs is editable from a computer in my LAN)
When SWAG booted, it warned me “Plugin legacy name certbot-dns-cpanel:cpanel may be removed in a future version. Please use cpanel instead.”
…what is that and how do I fix it?

Anyway, even with the above minor issues, the above works, here comes my main issue:

My first use as I said, is to use SWAG to reach WebTop mail on my NethServer.
As is set by NethServer, WebTop is reachable internally (and was also externally before using SWAG), with:
https:// remote.my.domain/webtop and https:// mynethserver.local.my.domain/webtop

To move to SWAG, I added a cname entry for this in my cpanel (external) DNS, let say “mymail.my.domain”.
I edited webtop.subdomain.conf and the two entries I touched were:
server_name mymail.*;
and
proxy_pass https:// mynethserver.local.my.domain/webtop
(the above normally takes me to WebTop properly.

When I did that and tried to access WebTop from an outside device, it failed and said it couldn’t find /webtopwebtop!
So, first thing I realized was that “something” added webtop in the URL anyway!

First (actually #3 in this post) question is WHAT adds “webtop” word on the URL?

After I saw this, I eliminated from webtop.subdomain.conf the “/webtop” from proxy_pass.
This time, it reached WebTop login screen properly!
(and I saw it translated it to https:// mymail.my.domain/webtop)
And here is the main issue
When I tried to login though… it failed because it switched to URL “https:// 127.0.0.1/webtop”!

Why the heck did it switch to loopback???
So, resolving this issue is the main reason I post this.
Bonus question: Since I have an internal DNS, is there any point using it in SWAG?

Thank you for your time!

(all spaces in URL is to get by the limitation of new users in this server)

stetho · 31 October 2021 19:42

(1) doesn’t make sense. How are you accessing the server on the LAN? I’m guessing that you’re connecting to a share rather than SSHing in to the server. If so, stop doing that. Terrible idea. Privileges set on shares can affect the was Docker runs. Just SSH to your UnRAID server. If that isn’t what you meant, can you clarify how you’re connecting?

(2) It’s the naming scheme for the (third party) authentication system. I believe (someone correct me) the as long as you’re using the newest version of SWAG the error will fix itself when the cert is renewed. You can safely ignore it.

(3)

WHAT adds “webtop” word on the URL

You are. Proxy_pass literally takes the string xxx and replaces the domain name domain .com. So if you request domian .com/webmail and your proxy_pass is internaldomain .com/webmail NGinX will try and pass your request to domain. com/webmailwebmail. As you’ve seen. To expand on this so you understand what is happening - when you go to a URL www. example. com/some/path/to/something - DNS will ask for the record that directs it to the server hosting www. example. com. It will contact the server and say "I’m looking for www. example. com. The server will respond with yes, that’s here. Then your browser will say I want “/some/path/to/something”. So when you’re contacting mynethserver .local .my .domain it responds and then NGinX asks for /webtopwebtop because that’s how the rest of the URL has been written.
TL;DR - proxy server to server. www. myexternaldomain .com to www. myinternaldomain. com - let the web server, not the proxy, handle the bit after the /

This is usually because webtop (or anything really) needs to be told that it’s being contacted by a proxy, not the originating host. When you proxy, you talk to the proxy. the proxy talks to the host. You’re no longer in direct contact with the host. I think this is what you want but I don’t use Nethserver

https://docs.nethserver.org/en/v7/webtop5.html#change-the-public-url

Depends on what you mean. You can’t use it to prove ownership of the domain.

NLS · 31 October 2021 22:13

Thanks for your reply!

Indeed I was trying to edit the file through a (hidden and restricted) share. And indeed I managed to do it using ssh.
Not so sure. You see the cert was fresh, I just installed SWAG. Also I renewed a few times because of the edits and tests I did, but the message kept popping when starting SWAG.
I am not sure I follow.
BEFORE SWAG (using just port forwarding): I could access my WebTop like this:
https:// remote.my.domain/webtop
AFTER SWAG (going through SWAG instead of directly to my WebTop), I typed this on an external client:
https:// mymail.my.domain
…so not sure where the double /webtop got added.
When I had it included in proxy_pass parameter, then I got it double.
When I removed in from that parameter, I got it just once (which was correct, although I would prefer if it didn’t show at all). I don’t see it anywhere else. It must be somewhere preconfigured, unless something else I don’t understand happens.
I will try touching this and get back to this. Not sure how this results in 127.0.0.1 after authentication.
Correct. I thought SWAG uses some kind of DNS for internal resolution (of dockers etc.) too.
I see resolver being “127.0.0.11” which I am not sure were it points.

The important thing right now is to solve #4, so keep ideas coming and preferably tell me exactly what to do.

NLS · 31 October 2021 22:35

#4… I found the setting “Public URL” in NethServer’s WebTop config. Also found in there a parameter called “Virtual host name” that I also set. WebTop services, were restarted automatically.

Didn’t change a thing with what happens after WebTop login page.
(again: Login page appears fine and when I supply credentials and click ok, it drops to https://127.0.0.1/webtop which of course goes nowhere).

stetho · 31 October 2021 23:01

LetsEncrypt have made a change to the requirements for the way third party plugins are named. I believe there’s a bug already to fix this in swag (which is what I meant by the “someone correct me” in case I’m wrong) that means with the next release of Swag and the first cert renewal after the update, the error will go away.
You said in your first post that you’re using

proxy_pass https:// mynethserver. local. my. domain/webtop

The /webtop at the end is where the problem comes from. Your proxy should be server to server - so the host is myserver .mydomain .com and the internal server is server .internaldomain .com - Anything after the / should be handled by the receiving server, not the proxy.

If, however, what you’re trying to do is go to mymail.domain.com and be taken to nethserver. my. server/webtop, you’ll need to refer to the help document I referred to in my previous reply. Essentially, you need to tell Webtop (not SWAG) that it now responds to “mymai l.domain. com”. As it’s something I’ve never used I’ll be no more use to you than the help document BUT you might find it easier if you use the IP address of your nethserver as the proxy address, e.g.

proxy_pass https:// 192. 168. 1. 50/

To explain - you type mymail. domain. com in to a browser SWAG will pass that URL to the server at 192. 168. 1. 50. It’s then up to the server at 192. 168. 1. 50 to know what to serve for requests to mymail. domain. com which is where the help document comes in.

It goes to 127 .0 0. 1 because it doesn’t know what its base URL is. It doesn’t know it’s mymail.domain.com so it serves 127 .0 .0 .1.

1. 1. 11 is Docker’s internal DNS (not SWAG’s). Every container can refer to another container by its name. SWAG uses it in nginx so you can proxy other containers on the same host. For this particular problem it’s not relevant and can be ignored (even deleted). But if you look at the SWAG NextCloud example you’ll see it sets up a container called ‘mariadb’ and then refers to it as ‘mariadb’ instead of 10.4. 32. 19 or some URL.

Here’s an example from my own setup

server {
    listen 443 ssl;
    listen [::]:443 ssl;
    server_name house. domain. com;
    include /config/nginx/ssl.conf;
    client_max_body_size 0;
    location / {
        include /config/nginx/proxy.conf;
        proxy_pass http://10. 244. 101. 178:8123;
    }
}

As you can see I’ve deleted the ‘upstream’ lines because they’re not needed and I proxy_pass to the host, not a DNS name.

The real problem here is that I don’t use NethServer or WebTop. I can give you lots of pointers but I have no idea what the answer is!

NLS · 31 October 2021 23:14

I will try using the internal IP instead of an FQDN.
This is why I asked if SWAG would need “the help” of my existing DNS (and how).

Back to the /webtop thing (which is not an issue, just curiosity):
Remember that https ://mynetheserver.my.domain also hosts other things.
Referring to that FQDN without a port or a subdir, just goes to NethServer’s own “portal” page (a landing page).
To reach WebTop, you actually need to use /webtop.
This is why I initially added it in my webtop SWAG .conf.
BUT it was not needed!
Something else actually added /webtop, so when I included it in my proxy_pass, it double-added it!
So I removed it from proxy_path so that “something else” added a single “/webtop” in the URL.
I am trying to find that “something else” (to understand how things work).

EDIT: Changing proxy_pass to use my WebTop LAN IP, still takes it to 127.0.0.1 after (again: after) I go to WebTop authentication page and click to proceed. So main issue remains.

NLS · 31 October 2021 23:24

OK I tried an idea I had I HAVE DIFFERENT RESULT FINALLY! (but not a solution…)

I created an “resolver-2.conf” where the resolver was my own DNS server (Technitium) and pointed webtop .conf to use that instead.

Now I reach WebTop login screen (like before) and when I enter credentials and click to go in… It goes BACK to login screen!

No 127.0.0.1 any more, but no actual entry to WebTop either.

EDIT: I think it is solved… I needed to use IP and disable include of the resolver. Just one of the two was not enough.

Strange but ok.
I still don’t get what adds /webtop to the path.
This might be a problem if I try to use other web services provided by the same IP.
We’ll see when I try.

stetho · 31 October 2021 23:41

That’s because the server doesn’t know how to handle requests to mymail.mydomain.com. It is up to the server, not SWAG to know that. As an example, my SWAG instance hosts 31 different services, a mixture of local containers and remote servers. The example I gave in my previous post is my Home Assistant server. SWAG doesn’t know what Home Assistant is. All it knows is that when it receives a request for house.my.domain it has to forward it to the server at 10. 244. 101. 178, port 8123 using http.

I am trying to find that “something else” (to understand how things work).

I explained this really badly in my first response. I’ll try again.

proxy_pass only cares about server.domain.com and nethserver.internaldomain.com. Anything else is not relevant to SWAG. Let’s see if I can explain it better with some examples:

Simple
server = www.domain.com
proxypass = http://192.168.1.50/
I request www.domain.com
The server at 192.168.1.50 serves www.domain.com

Next step
server = www.domain.com
proxypass = http://192.168.1.50/
I request www.domain.com/folder
The server at 192.168.1.50 serves www.domain.com/folder

What you did
server = www.domain.com
proxypass = http://192.168.1.50/webtop
I request www.domain.com/webtop
The server at 192.168.1.50 (tries to) serve www.domain.com/webtopwebtop

Given a proxy directive of (server)/(other_stuff) Nginx doesn’t care about (other_stuff) only the host - so it will tack on (other_stuff) to the end of the URL. If you’ve already requested (other_stuff) and your proxy says that, you get double (other_stuff).

NLS · 1 November 2021 00:03

No no… What I initially did was that:

server = www.domain.com
proxypass = http://192.168.1.50/webtop
I request www.domain.com
The server at 192.168.1.50 (tries to) serve www.domain.com/webtopwebtop

Then I did this:

server = www.domain.com
proxypass = http://192.168.1.50
I request www.domain.com (yes I never in any case typed /webtop in my browser)
The server at 192.168.1.50 (tries to) serve www.domain.com/webtop …which works.

In the meantime, I also made a proxy-conf entry for mattermost.
This one uses a virtual host team.domain.com.
So I made a copy of webtop .conf file and I just edited
server_name team.*;
AND NOTHING ELSE (as it uses the same IP and port as WebTop)… Well guess what, it successfully lands on mattermost page.

Then I added a .conf for another subdomain I have in my certificate, remote.domain.com.
This uses the SAME IP, SAMED PORT, but it just lands directly on that server’s root URL.
I just edited
server_name remote.*;
…and guess what. It does land on NethServer landing page as it should.
So I have three .conf files where proxy_pass is EXACTLY the same and only server_name differs.
And still they go to different pages properly.

I am confused on how it works.

stetho · 1 November 2021 00:23

Have you set the virtual host name on your nethserver to team.domain.com?

That’s how it works. Web servers by default serve everything on port 80 for http and 443 for https. If you haven’t changed that then you don’t need to specify a port. When your request for team.domain.com hits your nethserver its webserver will say “If I receive a request for team.domain.com I have a virtual host for that which tells me to server up MatterMost”.

Think of a proxy as a switchboard. You dial 555-1234 it will make the phone at 555-1234 ring. When they answer you still have to say “Is Mr. Mattermost there, please?” - the switchboard doesn’t know which occupant of the house you’re calling.

Another aspect that might help you understand how it works - If you’re hosting everything on nethserver and nowhere else - this doesn’t work if you have more than one backend source - you can have just one SWAG proxy config that has the server set to *.domain.com and the proxy_pass to http://192.168.1.50. Whether you request https://team.domain.com or https://mail.domain.com or even https://calendar.domain.com, all of those requests will be forwarded to 192.168.1.50 and 192.168.1.50 will find the corresponding virtual host.

With the /webtop thing - i’m still guessing but I assume there’s a URL rewrite going on that makes webtop the default site for some URL. That is, if you go to doesn’t_exist.domain.com the server serves up domain.com/webtop instead of a 404 error. But that really is a guess.

NLS · 1 November 2021 00:41

Yes I understand all that.
What I don’t understand yet is how /webtop was added.
Especially before I even defined within NethServer any mention about mymail.mydomain.com belonging to webtop service.

Anyway. Since you insist nothing in SWAG adds it and probably something in NethServer does, I believe you…