Preferred way to set up a reverse proxy web server with Podman containers

[gaufde]

Hello,

I am a beginner looking for advice on the most secure way / best practices to set up a simple web server with Podman containers.

I would like to run a few websites/apps in containers and have Caddy running in a container for my server. I've been struggling a bit with Podman networking, and posted a question about preserving IP addresses on the Caddy forum as well as in the Podman discord. Here is a link to the Caddy forum where I fully document that question: https://caddy.community/t/preserving-source-ip-in-rootless-podman-network/25461.

However, as I was doing more research about networking, I ran across this discussion here about how using rootfull Podman might be more secure: #13728

So, now I think I should take a step back to learn if what I was trying to do in my forum post above is actually the right move.

The goal is to have a simple VPS running Debian 12, some IP-aware security measures like fail2ban, a Caddy webserver, some web apps running PHP (Wordpress, Kirby, etc.), and maybe an app running Node (Plausible/Umami for analytics).

What would be the best practice for this setup? The route I was originally going was to use rootless Podman to have a Caddy container in its own Pod, and then have other pods for each of the apps. I was originally thinking that something like fail2ban would be run directly on the server, but I also saw some mentions that made me think people were running that in a container or maybe they were running all of their containers in a container. I'm not exactly sure.

Thanks!

[gaufde]

I just watched this video by @Luap99, and one of the key things mentioned that stuck with me is:

...the problem with both slirp4netns and pasta is like they isolate each container individually, and they don't allow the communication between containers....

Earlier in the video he demonstrated how a bridge/custom network, created via podman network create, can have an internal dns system that makes it ideal for reverse proxy systems like I was imagining using with Caddy. However, it seems like a bridge/custom network in rootless mode has an issue which is that the source IP address is not preserved—which I believe I saw and documented on the Caddy forum before I even understood what type of network I was using.

The proposed solution is to use pasta, which does preserve source IP, to connect to a rootless bridge/custom network; however, it sounds like that currently isn't possible but is being worked on (see timestamp 21:23).

Also, I re-read the discussion about rootful vs rootless security and realized I missed something quite important in a thread where @rhatdan explains:

I have a blog in progress, the problem is it is arguable which is more secure. $ sudo podman run --userns=auto ... Runs each containers in a separate user namespace where each container is running with different UIDs and almost assuradly does not match any other UIDs on the host. BUT the podman command runs as root which means it pulls images as root. Pulling images as root can be risky, because flaws in the pulling could (have in the past) cause root to write anywhere. $ podman run ... Runs the container in a single user namespace where it could attack the users homedir and potentially get a users data. Also all containers run in the same user namespace.

So, now it seems like I basically have two options:

• Use rootful Podman and --userns=auto. I think I can use a bridge/custom network with DNS which will make networking pretty easy and because it it rootful I will have the source IP address as well. I don't know if using --userns=auto will add any difficulties or not yet, so I will need to do some reading and testing on that.
• Use rootless Podman with separate Pasta networks and tie them together by opening ports on the host: https://www.reddit.com/r/podman/comments/1c46q54/comment/kzndcgq/.

There is a possible third option which involves using socket activation, but that seems a bit more complicated to me and it isn't actually clear if would solve all the issues: https://www.reddit.com/r/podman/comments/1c46q54/comment/kzpt6v9/.

So, I think I can now revise my question to be a bit more specific. Basically, am I right in thinking that the 3 options above are currently the best ways to run a production web server that proxies traffic to a few containerized apps? Am I missing any other options or security concerns?

[eriksjolund]

There is a feature request for adding socket activation support to Caddy:

• Support passing FDs (socket activation) caddyserver/caddy#6296

See also this PR that implements socket activation

• Implement issue #6296 passing FDs / socket activation caddyserver/caddy#6543

If this get released in Caddy, you could then use Caddy as an HTTP reverse proxy for a custom network in Podman.

If you don't want to wait for socket activation support in Caddy, you could also use
nginx
https://github.com/eriksjolund/podman-nginx-socket-activation
or
traefik
https://github.com/eriksjolund/podman-traefik-socket-activation/

Side note: I should probably add another example to
https://github.com/eriksjolund/podman-nginx-socket-activation
that only uses systemd user services.

The examples that use systemd system services combined with User= are a bit too experimental. I just tried it out quickly and noticed it seems to work.

[gaufde]

@Luap99 Thanks for the detailed reply! I really appreciate you adding your opinions as well since that helps me understand what is considered good or bad practice.

However, it seems like a bridge/custom network in rootless mode has an issue which is that the source IP address is not preserved—which I believe I saw and documented on the Caddy forum before I even understood what type of network I was using.

Correct the tracking issue is #8193 and while I would like to get this fixed sooner than later the work is not so trivial and I work on other priorities so I cannot promise any timeline.

Of course! I'm mostly interested in learning what the best practices are now so I can get started.

Note sure if debian 12 is new enough for that.

Is it not possible to install any version of Podman on Debian? I'm new to Linux in general, so I hadn't considered that. But I thought there was something called a flatpak that would allow one to install newer versions of software than what the distro's built-in package manager had.

This solves the source ip issue for the web sever as the socket was opened on the host, the benefit is that there is no pasta/slirp4netns overhead so the network speed is the same as on the host.
But then this only works if the application has socket activation support but for a reverse proxy it is a decent choice IMO
@eriksjolund wrote some great docs about it: https://github.com/eriksjolund/podman-networking-docs?tab=readme-ov-file#socket-activation-systemd-user-service

This is great. So it sounds like the main two contenders are actually using rootful Podman with --userns=auto or rootless Podman with socket activation. @eriksjolund Thank you for the information on socket activation with Caddy! That looks very promising and I'll be sure to stay away from the experimental User= combination.

If I tried to use socket activation with rootless Podman, would it still be advisable to use --userns=auto? I was reading a bit more about UIDs in Podman containers, and this article by @rhatdan mentions:

With Podman, you want to allow users to run any container image on any container registry as non-root if the user chooses. And I believe that running containers as non-root should always be your top priority for security reasons.

This makes me think that using --userns=auto is important no matter if you use rootless or rootful Podman.

Also, thing brings me to another source of confusion. On places like reddit, I have seen a sentiment repeated that effectively says, "in production it is best to use rootless Docker or rootless Podman for security." However, now we are talking about using rootful Podman with --userns=auto as acceptably secure.

 Would it be correct to say that using rootful Podman with --userns=auto is somewhat equivalent to using rootless Docker? That even though Podman is run with root privileges when the containers are started, once the containers are running they are running rootless?

 People talk about Docker running with root privileges by default in a way that is considered bad; is it possible to accidentally run Podman fully rootful in a similar way (I guess by forgetting userns=auto)?

Edit: I think my source of confusion for the above questions is a lack of understanding about rootless vs rootfull Podman and rootless vs rootfull containers. Are they the same thing, because it seems like there might be a distinction I don’t know yet?

[kraftreich]

So, now it seems like I basically have two options:

• Use rootful Podman and --userns=auto. I think I can use a bridge/custom network with DNS which will make networking pretty easy and because it it rootful I will have the source IP address as well. I don't know if using --userns=auto will add any difficulties or not yet, so I will need to do some reading and testing on that.
• Use rootless Podman with separate Pasta networks and tie them together by opening ports on the host: https://www.reddit.com/r/podman/comments/1c46q54/comment/kzndcgq/.

There is a possible third option which involves using socket activation, but that seems a bit more complicated to me and it isn't actually clear if would solve all the issues: https://www.reddit.com/r/podman/comments/1c46q54/comment/kzpt6v9/.

So, I think I can now revise my question to be a bit more specific. Basically, am I right in thinking that the 3 options above are currently the best ways to run a production web server that proxies traffic to a few containerized apps? Am I missing any other options or security concerns?

Give Podman + HaProxy a try. I started with docker-compose files I didn't understand with Caddy and ended with Podman + HaProxy + simple readable quadlet files.
I run a layer 4 proxy (ha-proxy) rootless in a pod and all my web services in different rootless pods and get the source IP in the webservices pods (over ha proxy protocol). This is a very simple and straightforward and for my understanding secure setup. You are not limited to layer 7 HTTP, so you are able to proxy ssh, smtp, vpn....
All traffic is tls secured from the client to the webservice, no need to terminate it @ the proxy pod.

[saolof]

Would it be possible to have something like the keep-id mode, except that as root you get to pick an unprivileged user that gets mapped in? To ensure that the application runs unprivileged and that this unprivileged user owns all files created by the container process, but without crippling networking?

[rhatdan]

keep-id allows you to specify the UID and GID you want to map your UID to, is this what you are after?