I installed the docker-letsencrypt container and was able to configure it to reverse proxy non-docker services by creating server blocks. But how do I reverse proxy container services? Is that also done via a server block, or do I have to do something else when I start the container I want to be reverse proxied?
Hmm I wonder if the readme says anything about that 
https://github.com/linuxserver/docker-letsencrypt/blob/master/README.md#site-config-and-reverse-proxy
No, it doesn’t mention non-container services vs containerised services.
Not explicitly, but you gotta keep reading.
The other readme referenced there tells you all about how to reverse proxy other containers via preset proxy confs.
There is really no difference between proxying a container vs native service. Nginx just has to be able to reach the url you put into the proxy_pass statement and for other containers it’s either host ip and mapped port, or container’s own ip if using macvlan, or simply the container name as hostname if they are both in the same user defined bridge network. That last one is all explained in there.