Yann39/self-hosted-n100

Yann39/self-hosted-n100 has 36 GitHub stars and 1 forks. GitHot measured 0 stars in 24 hours and 0 in 7 days. Last checked 2026-11-09 16:03 UTC. Last push 2026-10-18.

How these counts are measured

Personal self-hosted infrastructure setup for N100-based mini PC

36 stars 1 forks

Measured star growth

in 24h: 0
in 7 days: 0
in 15 days: Update delayed
in 1 month: Update delayed

Net star changes over overlapping periods, not daily totals. Missing measurements are left blank. Stars measure attention, not software quality.

README.md

Original

Personal self-hosting guide

This project describes my personal self-hosted infrastructure setup, running on a mini PC (N100 based).

This was meant to be just a reminder for me, but I wrote it as a guide, in case it might help someone.

It uses only free and open source software.

[!NOTE] This project is based on my previous home lab setup running on a Banana Pi board (ARM).
This one contains more up-to-date instructions and is closer to my current setup.
The original project can be found at https://github.com/Yann39/self-hosted.

[!IMPORTANT] The content of this repository is provided "as is", with no guarantee that the information is complete or error-free. The techniques and tools discussed here come with inherent risks. The author takes absolutely no responsibility for possible consequences due to the use of the related software.

Table of Content

Overview

Plan

The goal remains the same as with my first home lab: hosting a few useful applications, in an environment :

100% self-hosted (privacy preserving, full control over data and software)
Secure (authentication, SSL/TLS, reverse proxy, firewall, ad blocking, DDOS protection, rate limiting, custom DNS resolver, ...)
Lightweight (runs smoothly with minimal hardware and software requirements)
Container-ready (isolated, portable, scalable applications)
Accessible (some services accessible only locally, some only through VPN, some publicly)
Supervised (monitoring, alerting, tracking, backup tools)

These are the tools we are going to run :

Name	Repository	Description
Docker	https://github.com/docker	Help to build, share, and run container applications
Docker Compose	https://github.com/docker/compose	Run multi-container applications with Docker
Arcane	https://github.com/getarcaneapp/arcane	Management platform for containerized applications
Traefik	https://github.com/traefik/traefik	Modern HTTP reverse proxy and load balancer
Sablier	https://github.com/sablierapp/sablier	Workload scaling on demand
PocketID	https://github.com/pocket-id/pocket-id	Simple OIDC provider for passkey authentication
CrowdSec	https://github.com/crowdsecurity/crowdsec	Collaborative intrusion prevention, bans attackers
CrowdSec Web UI	https://github.com/TheDuffman85/crowdsec-web-ui	Web dashboard for CrowdSec alerts and decisions
Wireguard	https://github.com/WireGuard	Simple yet fast and modern VPN
WGDashboard	https://github.com/WGDashboard/WGDashboard	Web interface to manage WireGuard peers
Pi-hole	https://github.com/pi-hole/pi-hole	Network-wide ad blocking
Unbound	https://github.com/NLnetLabs/unbound	Validating, recursive, and caching DNS resolver
Homer	https://github.com/bastienwirtz/homer	Static application dashboard
Homebox	https://github.com/sysadminsmedia/homebox	Inventory and organisation system for the home
Dashdot	https://github.com/MauriceNino/dashdot	Minimal server dashboard and monitoring
Prometheus	https://github.com/prometheus/prometheus	Metrics collection and time series database
Grafana	https://github.com/grafana/grafana	Dashboards and visualization for metrics
Gatus	https://github.com/TwiN/gatus	Uptime monitoring and alerting, status page
Ghostfolio	https://github.com/ghostfolio/ghostfolio	Wealth management and portfolio tracking
Homelable	https://github.com/Pouzor/homelable	Homelab infrastructure map with live status
Medusa	https://github.com/medusajs/medusa	Headless e-commerce platform, with a storefront
Backrest	https://github.com/garethgeorge/backrest	Web UI for restic backups (snapshots, encryption)
GoatCounter	https://github.com/arp242/goatcounter	Privacy-friendly web analytics, no cookies
Lychee	https://github.com/LycheeOrg/Lychee	Free photo-management tool
PhpMyAdmin	https://github.com/phpmyadmin/phpmyadmin	Web user interface to manage MySQL databases

And also some personal applications :

My first PHP / MySQL website from the early 2000's ! : https://github.com/Yann39/defrag-life
A GraphQL API (Java / Spring Boot) for one of my Flutter mobile applications : https://github.com/Yann39/ccteam-graphql

All of this runs on a Trigkey G4 mini PC ! With the following specifications :

Intel Alder Lake N100 12th gen (4Core, up to 3.4 GHz)
Integrated Intel UHD GPU handling 4K@60Hz
16GB DDR4 3200MHz
500GB M.2 NVME SSD
1 GbE ethernet & Wi-Fi 6
4 x USB 3.2 Gen2

[!NOTE] This hardware is not designed for high loads, I only have a few users on my public applications, of course if you need to handle more load you might consider a better machine.

It should also work on many other x86 based computers.

Target architecture

Here is a diagram representing the global network "architecture" we are going to set up, simplified with only the most relevant services. See Network flow to get a view of the request flow.

This architecture allows exposing applications to the internet while restricting access to some of them only through VPN or from the local network. It's up to you to choose the accessibility level you need for each service, you may want some to be accessible only from your local network, some only via VPN, and others to anyone from the internet.

%%{init: {"look": "classic", "flowchart": {"nodeSpacing": 15, "rankSpacing": 50, "padding": 8, "diagramPadding": 10, "curve": "basis", "wrappingWidth": 220, "inheritDir": false}}}%%
flowchart TB
    style HOSTING_PROVIDER fill: #bbfd795e
    style DDNS_PROVIDER fill: #e2b6ff6e
    style INTERNET_SERVICE_PROVIDER fill: #45d8ff57
    style SERVER_DEVICE fill: #febfb45e
    style CONTAINER_ENGINE fill: #6701022d
    style TRAEFIK_CONTAINER fill: #66010136
    style PIHOLE_CONTAINER fill: #66010136
    style UNBOUND_CONTAINER fill: #66010136
    style POCKETID_CONTAINER fill: #66010136
    style PUBLIC_APP_CONTAINER fill: #66010136
    style PRIVATE_APP_CONTAINER fill: #66010136
    style CROWDSEC_CONTAINER fill: #66010136
    style SABLIER_CONTAINER fill: #66010136
    style WIREGUARD_HOST fill: #66010136
    style TRAEFIK_ROUTER fill: #defc1e37
    style TRAEFIK_MIDDLEWARE fill: #defc1e37
    style VPN_CLIENT fill: #18fdb144
    style PIHOLE_DNS_RECORDS fill: #defc1e37
    style CROWDSEC_COMMUNITY fill: #bbfd795e
    style CLIENT padding:40
    DOMAIN(example.com)
    SUBDOMAIN_WIREGUARD(wireguard.example.com)
    SUBDOMAIN_MYAPP(public-app.example.com)
    DDNS(myddns.ddns.net)
    ROUTER[public IP]
    ROUTER_PORT80{{80/tcp}}
    ROUTER_PORT443{{443/tcp}}
    ROUTER_PORT51820{{51820/udp}}
    DOCKER_WIREGUARD_PORT51820{{51820/udp}}
    DOCKER_MYAPP_PORT5000{{5000/tcp}}
    DOCKER_PRIVATE_APP_PORT8080{{8080/tcp}}
    DOCKER_PIHOLE_PORT53{{53/udp}}
    DOCKER_SABLIER_PORT10000{{10000/tcp}}
    DOCKER_TRAEFIK_PORT443{{443/tcp}}
    DOCKER_TRAEFIK_PORT80{{80/tcp}}
    DOCKER_UNBOUND_PORT53{{53/udp}}
    DOCKER_POCKETID_APP_PORT1411{{1411/tcp}}
    TRAEFIK_ROUTER_MYAPP(public-app)
    TRAEFIK_ROUTER_TRAEFIK(private-app)
    ROOT_DNS_SERVERS[Root DNS servers]
    DNS_ISP[DNS 1 & 2]
    DOCKER_PIHOLE_DNS[DNS 1 & 2]
    PIHOLE_DNS_TRAEFIK[private-app.example.com]
    PIHOLE_DNS_MYAPP[public-app.example.com]
    CROWDSEC_BOUNCER(CrowdSec bouncer)
    CROWDSEC_ENGINE[Security engine\n+ local API]
    ACCESS_LOG[(access log)]
    CROWDSEC_COMMUNITY[CrowdSec\ncommunity blocklist]

    subgraph VPN_CLIENT[VPN CLIENT]
        WIREGUARD_CLIENT_ENDPOINT[Endpoint]
        WIREGUARD_CLIENT_DNS[DNS]
    end

    subgraph HOSTING_PROVIDER[DOMAIN NAME REGISTRAR]
        DOMAIN -->|subdomain| SUBDOMAIN_MYAPP
        DOMAIN -->|subdomain| SUBDOMAIN_WIREGUARD
    end

    subgraph DDNS_PROVIDER[DYNAMIC DNS PROVIDER]
        SUBDOMAIN_MYAPP --->|CNAME| DDNS
        SUBDOMAIN_WIREGUARD --->|CNAME| DDNS
    end

    subgraph INTERNET_SERVICE_PROVIDER[INTERNET SERVICE PROVIDER]
        DDNS --->|DynDNS| ROUTER
        ROUTER --> ROUTER_PORT443
        ROUTER --> ROUTER_PORT80
        ROUTER --> ROUTER_PORT51820
        DNS_ISP
    end

    subgraph SERVER_DEVICE[MINI PC]

        subgraph CONTAINER_ENGINE[DOCKER]

            subgraph WIREGUARD_HOST[WIREGUARD ON THE HOST]
                DOCKER_WIREGUARD_PORT51820
            end
            
            subgraph TRAEFIK_CONTAINER[TRAEFIK CONTAINER]
                subgraph TRAEFIK_ROUTER[TRAEFIK HTTP ROUTER]
                    TRAEFIK_ROUTER_TRAEFIK
                    TRAEFIK_ROUTER_MYAPP
                end

                subgraph TRAEFIK_MIDDLEWARE[TRAEFIK MIDDLEWARE]
                    IP_WHITELISTING(IP whitelist)
                    SABLIER(Sablier dynamic)
                end

                CROWDSEC_BOUNCER
                ACCESS_LOG
                REDIRECT(HTTPS redirect)
                DOCKER_TRAEFIK_PORT80
                DOCKER_TRAEFIK_PORT443
            end

            subgraph SABLIER_CONTAINER[SABLIER CONTAINER]
                DOCKER_SABLIER_PORT10000
                WAITING_PAGE(Waiting page)
            end

            subgraph PIHOLE_CONTAINER[PIHOLE CONTAINER]
                subgraph PIHOLE_DNS_RECORDS[LOCAL DNS RECORDS]
                    PIHOLE_DNS_TRAEFIK ~~~ PIHOLE_DNS_MYAPP
                end

                DOCKER_PIHOLE_PORT53 ~~~ DOCKER_PIHOLE_DNS
            end

            subgraph CROWDSEC_CONTAINER[CROWDSEC CONTAINER]
                CROWDSEC_ENGINE
            end

            subgraph PUBLIC_APP_CONTAINER[PUBLIC APP CONTAINER]
                DOCKER_MYAPP_PORT5000
            end

            subgraph PRIVATE_APP_CONTAINER[PRIVATE APP CONTAINER]
                DOCKER_PRIVATE_APP_PORT8080
            end

            subgraph POCKETID_CONTAINER[POCKETID CONTAINER]
                DOCKER_POCKETID_APP_PORT1411
            end

            subgraph UNBOUND_CONTAINER[UNBOUND CONTAINER]
                DOCKER_UNBOUND_PORT53
            end
        end
    end
    
    CLIENT((User)) -->|with VPN| VPN_CLIENT
    BROWSER@{ shape: docs, label: "Browser" } --> HOSTING_PROVIDER
    CLIENT -->|without VPN| BROWSER
    VPN_CLIENT --> BROWSER
    WIREGUARD_CLIENT_ENDPOINT -.->|" Server static IP\n192.168.0.16 "| SERVER_DEVICE
    WIREGUARD_CLIENT_DNS -->|" Server tunnel address\n10.0.0.1 "| SERVER_DEVICE
    ROUTER_PORT51820 -->|port forward| DOCKER_WIREGUARD_PORT51820
    ROUTER_PORT443 ------>|port forward| DOCKER_TRAEFIK_PORT443
    ROUTER_PORT80 ------>|port forward| DOCKER_TRAEFIK_PORT80
    DNS_ISP ------>|Server static IP| DOCKER_PIHOLE_PORT53
    PIHOLE_DNS_MYAPP --->|Server internal IP\n192.168.0.16| DOCKER_TRAEFIK_PORT443
    PIHOLE_DNS_TRAEFIK ------>|Server internal IP\n192.168.0.16| DOCKER_TRAEFIK_PORT443
    DOCKER_PIHOLE_PORT53 --> PIHOLE_DNS_RECORDS
    DOCKER_TRAEFIK_PORT443 ---> CROWDSEC_BOUNCER
    REDIRECT -.....->|301 to https| BROWSER
    CROWDSEC_BOUNCER ----->|IP not banned| TRAEFIK_ROUTER
    DOCKER_TRAEFIK_PORT80 --> REDIRECT
    CROWDSEC_BOUNCER -..->|every request logged| ACCESS_LOG
    ACCESS_LOG -...->|reads, detects attacks| CROWDSEC_ENGINE
    CROWDSEC_ENGINE -........->|decisions| CROWDSEC_BOUNCER
    CROWDSEC_ENGINE <-..->|signals / community blocklist| CROWDSEC_COMMUNITY
    TRAEFIK_ROUTER_MYAPP ----> SABLIER
    TRAEFIK_ROUTER_TRAEFIK --> IP_WHITELISTING
    SABLIER <-.->|return status| DOCKER_SABLIER_PORT10000
    SABLIER -->|not ready| WAITING_PAGE
    SABLIER -->|ready| DOCKER_MYAPP_PORT5000
    DOCKER_SABLIER_PORT10000 <-.->|check status| DOCKER_MYAPP_PORT5000
    IP_WHITELISTING --> DOCKER_PRIVATE_APP_PORT8080
    PRIVATE_APP_CONTAINER <-->|OIDC| DOCKER_POCKETID_APP_PORT1411
    DOCKER_PIHOLE_DNS --------> DOCKER_UNBOUND_PORT53
    UNBOUND_CONTAINER <-----------> ROOT_DNS_SERVERS

    linkStyle 0 stroke-width: 3px, stroke: red
    linkStyle 1 stroke-width: 3px, stroke: #f08c00
    linkStyle 2 stroke-width: 3px, stroke: red
    linkStyle 3 stroke-width: 3px, stroke: #f08c00
    linkStyle 4 stroke-width: 3px, stroke: red
    linkStyle 5 stroke-width: 3px, stroke: red
    linkStyle 6 stroke-width: 3px, stroke: red, stroke-dasharray: 8
    linkStyle 7 stroke-width: 3px, stroke: #f08c00
    linkStyle 8 stroke-width: 3px, stroke: #f08c00
    linkStyle 9 stroke-width: 3px, stroke: #3b82f6, stroke-dasharray: 8
    linkStyle 10 stroke-width: 3px, stroke: #f08c00
    linkStyle 11 stroke-width: 3px, stroke: red
    linkStyle 12 stroke-width: 3px, stroke: red
    linkStyle 13 stroke-width: 3px, stroke: #f08c00
    linkStyle 14 stroke-width: 3px, stroke: #f08c00, stroke-dasharray: 8
    linkStyle 15 stroke-width: 3px, stroke: #3b82f6, stroke-dasharray: 8
    linkStyle 16 stroke-width: 3px, stroke: #f08c00
    linkStyle 17 stroke-width: 3px, stroke: red
    linkStyle 18 stroke-width: 3px, stroke: red, stroke-dasharray: 8
    linkStyle 19 stroke-width: 3px, stroke: #3b82f6, stroke-dasharray: 8
    linkStyle 20 stroke-width: 3px, stroke: #f08c00
    linkStyle 21 stroke-width: 3px, stroke: #f08c00
    linkStyle 22 stroke-width: 3px, stroke: #f08c00
    linkStyle 23 stroke-width: 3px, stroke: red
    linkStyle 24 stroke-width: 3px, stroke: red, stroke-dasharray: 8
    linkStyle 25 stroke-width: 3px, stroke: red
    linkStyle 26 stroke-width: 3px, stroke: red, stroke-dasharray: 8
    linkStyle 27 stroke-width: 3px, stroke: red, stroke-dasharray: 8
    linkStyle 28 stroke-width: 3px, stroke: red, stroke-dasharray: 8
    linkStyle 29 stroke-width: 3px, stroke: red, stroke-dasharray: 8
    linkStyle 30 stroke-width: 3px, stroke: red, stroke-dasharray: 8
    linkStyle 31 stroke-width: 3px, stroke: red
    linkStyle 32 stroke-width: 3px, stroke: #f08c00
    linkStyle 33 stroke-width: 3px, stroke: red, stroke-dasharray: 8
    linkStyle 34 stroke-width: 3px, stroke: red, stroke-dasharray: 8
    linkStyle 35 stroke-width: 3px, stroke: red
    linkStyle 36 stroke-width: 3px, stroke: red, stroke-dasharray: 8
    linkStyle 37 stroke-width: 3px, stroke: #f08c00
    linkStyle 38 stroke-width: 3px, stroke: #f08c00, stroke-dasharray: 8
    linkStyle 39 stroke-width: 3px, stroke: #3b82f6, stroke-dasharray: 8
    linkStyle 40 stroke-width: 3px, stroke: #3b82f6, stroke-dasharray: 8

Basically all services will be accessible via dedicated subdomains which will point to our local network, either through dynamic DNS or through local DNS records, then a reverse proxy will be responsible for routing the requests to the right application running in Docker containers.

We make the ISP upstream DNS (from router configuration or on each device) point to the server IP address, so that we reroute the entire traffic through Pi-hole and thus take advantage of its benefits, see Network configuration.

In this example private-app (private-app.example.com) is only accessible through VPN and from the local network thanks to local DNS records and IP whitelisting, while public-app (public-app.example.com) is also accessible from the internet publicly, behind Sablier which start/stop the container on demand, see Scale to zero with Sablier. In addition, private-app is behind OIDC authentication through PocketID, see PocketID.

On top of that, CrowdSec watches the Traefik access log and its bouncer, plugged on the HTTPS entrypoint, rejects the IP addresses flagged as malicious (by our own scenarios or by the community blocklist) before they reach any service, see CrowdSec.

You will find more details on how all this has been implemented later in this guide.

Install and prepare system

By default, the Mini PC came with Windows 11, I simply installed Debian 12 instead (then followed version up to 13.4, which is the version I use at the time of writing this guide). Backup the Windows key before, just in case.

Download latest Debian image for amd64 :
- debian-12.5.0-amd64-netinst.iso
Download Rufus or equivalent software to be able to write the image to a USB key
Simply select the image in Rufus and write it to the USB key with the default proposed options
Insert the USB key into the mini PC and start it, you may need to access the bios to change the boot device priority, to boot on the USB key
Then follow the Debian installation instructions, I personally installed the basic system without GUI (no desktop environment)

System user

When installing Debian, you should have been asked to create a regular user account. We will simply use that user for the whole guide.

For security reasons, do not use the root user directly. If you run a program as root and a security flaw is exploited, the attacker has access to the whole system without restriction. Using a regular user, even with sudo enabled, will require running sudo and will still prompt for the account password as an additional security step. It is also safer in case you unintentionally issue a command that could hurt the system (like deleting system files, etc.).

[!NOTE] We may also create specific users inside Docker containers for some applications, specially when creating our own Dockerfile, but we'll clarify then whether additional permissions need to be added in case they need access to the local filesystem through a bind mount.

sudo is not installed on Debian by default. You have to install it. So, become root and install sudo :

su -
apt install sudo

Then add user in the sudoers :

/sbin/adduser username sudo

SSH access

Generally, you'll want to leave your machine in a cool, quiet corner, rather than letting it land around in your feet and having to connect a keyboard/mouse/screen every time you want to access it.

A solution is simply to access it as a remote computer via SSH, from your main computer.

In the normal Debian images, SSH is not enabled by default, so you need to install openssh server to allow SSH connections :

apt update
apt install openssh-server

Then simply use the ssh command from the client machine to establish a secure and authenticated SSH connection to the mini PC (here named n100) :

ssh username@n100

Enter your password then you are ready to go !

You can also use your preferred SSH client.

[!NOTE] Unless you want to be able to do some operations from outside your local network, there is no need to open the SSH port to the internet. If you do so consider using it behind a VPN (even if SSH itself is very secure).

Basic tools

We need to install some basic tools which will be useful for the next steps.

Install curl (for transferring data through URLs) :

sudo apt install curl

Install netstat (to check network connections) :

sudo apt install net-tools

Optionally install vim (improved vi) :

sudo apt install vim

Directory structure

We will place every application configuration into the /opt/apps directory, as follows :

/
|- opt
    |- apps
        |- traefik
        |- arcane
        |- phpmyadmin
        |- dashdot
        |- ...

Usually this directory (/opt) is reserved for any software and packages that are not part of the default installation, but feel free to choose another location.

You can already create the directory :

sudo mkdir /opt/apps

We will create the subdirectories associated with each application when we install them.

Docker & Docker Compose

We will use Docker to containerize and run our different applications.

Docker enables to separate applications from the infrastructure, it provides the ability to package and run an application in an isolated environment called a container. Containers contain everything needed to run the application, so you don't need to rely on what's installed on the host.

We will also install Docker Compose, so we can define and run multi-container Docker applications.

Docker provides an installation script, but in Debian 13, we can install the Docker engine through the package manager and a more modern version of Docker Compose is available as a plugin.

Complete Docker setup on Debian 13 :

# 1. Remove any conflicting packages
sudo dpkg --remove --force-depends docker-buildx-plugin docker-compose-plugin docker-compose
sudo apt --fix-broken install
sudo apt autoremove

# 2. Install Docker Engine + plugins
sudo apt update
sudo apt install docker.io docker-compose-plugin docker-buildx-plugin

# 3. Enable and start Docker
sudo systemctl enable --now docker
sudo systemctl status docker

# 4. Add user to docker group
sudo usermod -aG docker $USER
newgrp docker  # or log out/in

# 5. Check Docker installation
sudo docker info

[!IMPORTANT] In this guide I systematically use latest images (:latesttag), but usually you better want to avoid using :latest tags in production. Anyway if you use latest tags and want to update an image in the future, simply pull it again and rerun your container / compose file, i.e. :
sudo docker-compose pull
sudo docker-compose up -d
Then remove any old images.

Network configuration

Before installing our services, we need to configure the network, so we can reach our applications from different locations.

The idea is to have :

A main domain name
A subdomain name for each application that must be reachable
A dynamic DNS name to avoid having to use a static public IP address
A reverse proxy to handle HTTP request that will be port forwarded to the applications

We will also use Pi-Hole's ability to manage local DNS records, each pointing to the server's internal IP address. They serve two distinct purposes, and are needed for every service, public or not :

services that are not exposed to the internet have no public DNS record at all, so a local record is the only way to reach them by name
services that are exposed still need one, otherwise a device at home resolves the public name, leaves to the router and comes back through its NAT loopback : that costs about half of the throughput, makes Traefik see your own public IP instead of the device's address (so CrowdSec and the logs can no longer tell your devices apart), and depends on a router feature that not all of them provide

In short, the public record is for your visitors, the local one is for you.

Here is an overview of the route for each case, when a client requests myapp.example.com. Note that myapp is a public service here, and that a client on the local network still reaches it through the local DNS record rather than through the internet :

:small_blue_diamond: Internet access (external) :

flowchart LR
    style HOSTING_PROVIDER fill: #bbfd795e
    style DDNS_PROVIDER fill: #e2b6ff6e
    style INTERNET_SERVICE_PROVIDER fill: #45d8ff57
    style TRAEFIK_CONTAINER fill: #66010159
    style APPLICATION fill: #66010159
    style SERVER_DEVICE fill: #febfb45e
    CLIENT((Client))
    SUBDOMAIN_MYAPP(myapp\n.example.com)
    DDNS(myddns\n.ddns.net)
    ROUTER[public IP]
    ROUTER_PORT{{port}}
    DOCKER_TRAEFIK_PORT{{port}}
    APPLICATION_PORT{{port}}

    subgraph HOSTING_PROVIDER[DOMAIN NAME REGISTRAR]
        SUBDOMAIN_MYAPP
    end

    subgraph DDNS_PROVIDER[DYNAMIC DNS PROVIDER]
        DDNS
    end

    subgraph INTERNET_SERVICE_PROVIDER[INTERNET SERVICE PROVIDER]
        ROUTER
        ROUTER_PORT
    end

    subgraph SERVER_DEVICE[MINI PC]
        subgraph TRAEFIK_CONTAINER[REVERSE PROXY]
            DOCKER_TRAEFIK_PORT
        end

        subgraph APPLICATION[APPLICATION]
            APPLICATION_PORT
        end
    end

    CLIENT --> SUBDOMAIN_MYAPP
    SUBDOMAIN_MYAPP -->|CNAME| DDNS
    DDNS -->|DynDNS| ROUTER
    ROUTER --> ROUTER_PORT
    ROUTER_PORT -->|port forward| DOCKER_TRAEFIK_PORT
    DOCKER_TRAEFIK_PORT -->|HTTP router| APPLICATION_PORT

:small_blue_diamond: VPN access :

flowchart LR
    style VPN fill: #bbfd795e
    style TRAEFIK_CONTAINER fill: #66010159
    style PI_HOLE fill: #66010159
    style APPLICATION fill: #66010159
    style WIREGUARD fill: #66010159
    style SERVER_DEVICE fill: #febfb45e
    CLIENT((Client))
    VPN_CLIENT(DNS)
    VPN_ENDPOINT(Endpoint)
    PIHOLE_DNS_MYAPP(myapp\n.example.com)
    DOCKER_TRAEFIK_PORT{{port}}
    APPLICATION_PORT{{port}}
    WIREGUARD_PORT{{port}}
    PIHOLE_DNS{{port}}

    subgraph VPN[VPN CLIENT]
        VPN_CLIENT
        VPN_ENDPOINT
    end

    subgraph SERVER_DEVICE[MINI PC]
        subgraph PI_HOLE[PI-HOLE]
            PIHOLE_DNS
            PIHOLE_DNS_MYAPP
        end

        subgraph TRAEFIK_CONTAINER[REVERSE PROXY]
            DOCKER_TRAEFIK_PORT
        end

        subgraph APPLICATION[APPLICATION]
            APPLICATION_PORT
        end

        subgraph WIREGUARD[VPN SERVER]
            WIREGUARD_PORT
        end
    end

    VPN_ENDPOINT --> WIREGUARD_PORT
    CLIENT --> VPN_CLIENT
    VPN_CLIENT --> PIHOLE_DNS
    PIHOLE_DNS_MYAPP --->|A| TRAEFIK_CONTAINER
    DOCKER_TRAEFIK_PORT -->|HTTP router| APPLICATION_PORT

:small_blue_diamond: Local network access :

flowchart LR
    style INTERNET_SERVICE_PROVIDER fill: #45d8ff57
    style TRAEFIK_CONTAINER fill: #66010159
    style PI_HOLE fill: #66010159
    style APPLICATION fill: #66010159
    style SERVER_DEVICE fill: #febfb45e
    CLIENT((Client))
    ISP_DNS(DNS)
    PIHOLE_DNS_MYAPP(myapp\n.example.com)
    DOCKER_TRAEFIK_PORT{{port}}
    APPLICATION_PORT{{port}}
    PIHOLE_DNS{{port}}

    subgraph INTERNET_SERVICE_PROVIDER[INTERNET SERVICE PROVIDER]
        ISP_DNS
    end

    subgraph SERVER_DEVICE[MINI PC]
        subgraph PI_HOLE[PI-HOLE]
            PIHOLE_DNS
            PIHOLE_DNS_MYAPP
        end

        subgraph TRAEFIK_CONTAINER[REVERSE PROXY]
            DOCKER_TRAEFIK_PORT
        end

        subgraph APPLICATION[APPLICATION]
            APPLICATION_PORT
        end

    end

    CLIENT ---->|DNS server set to the mini PC| PIHOLE_DNS
    CLIENT -..->|only if the router really relays DNS| ISP_DNS
    ISP_DNS -..-> PIHOLE_DNS
    PIHOLE_DNS_MYAPP --->|A| TRAEFIK_CONTAINER
    DOCKER_TRAEFIK_PORT -->|HTTP router| APPLICATION_PORT

[!NOTE] My router offers all the required features (DHCP server, DNS server, port forwarding, dynDNS, etc.) for the steps described below. Most of the routers also have those features (they rarely purely route packets), but if this is not your case, you may have to perform double NAT to allow more advanced configurations. I obviously cannot go through the configuration specific to each router.

IP settings

The following changes to the IP settings are required if you want the DNS requests of your whole local network to go through Pi-Hole and the custom DNS resolver (Unbound) (only the DNS requests : the ad blocking is done at DNS level, the traffic itself does not need to go through the mini PC) :

Assign a static IP address to the mini PC, for example 192.168.0.16 (I have local DHCP enabled)
Make the devices use the mini PC as DNS server (192.168.0.16), either through the router (the DNS server it hands out with DHCP), or manually on each device

Of course Pi-Hole container have to expose port 53 to receive incoming DNS requests. Refer to Pi-hole setup for more details.

[!WARNING] Setting the mini PC as "DNS server" in the router configuration is not always enough : many ISP boxes keep answering the DNS queries of the LAN devices themselves with the ISP resolvers, and the devices silently bypass Pi-Hole. Always verify from a device which server actually answers :
nslookup doubleclick.net
The answering server must be the mini PC (192.168.0.16), and a domain from the block lists must resolve to 0.0.0.0. If the router does not hand out the mini PC address, set the DNS manually on each device (on Windows : Settings -> Network -> Ethernet -> DNS server assignment -> Manual). In that case :

leave the alternate DNS empty : Windows does not strictly respect the primary/secondary order, a public secondary DNS ends up bypassing Pi-Hole

leave "DNS over HTTPS" off : Pi-Hole only speaks plain DNS on port 53, and this leg never leaves your LAN anyway (the privacy part is Unbound resolving directly from the root servers)

disable "secure DNS" / DNS-over-HTTPS in the browsers too, else they use their own resolver and bypass Pi-Hole

If you don't want the whole network to use Pi-Hole, skip the second point, then only the VPN clients (and the devices you configure manually) will use it.

Dynamic DNS

When connecting from outside our network (from the internet), we need to know the public IP address of our router to connect to. But unless we have a static public IP (not necessarily the safest option), we are getting dynamically-assigned public IP addresses (via DHCP), so we would need to update the configuration everytime the IP changes, which is very uncomfortable.

Fortunately we can register a dynamic host record (DynDNS), and configure it in our router configuration so that when the public IP address changes, a call is made to the DynDNS service provider to update the record. That way our network will always be reachable from the internet via the DynDNS record no matter the IP address.

Well, simply register a dynamic DNS hostname from a provider (there are free ones), for example No-IP, DuckDNS, etc. :

hostname : myddns.ddns.net
IP / target : internet box external IP (public IP)
type : A

Then activate DynDNS on the router :

Service provider : No-IP (adapt to your provider)
Hostname : myddns.ddns.net
Username : xxxxxxxx
Password : xxxxxxxx

The IP will be updated automatically when a change will be detected.

[!NOTE] Your ISP may only support some dynamic DNS provider that can be configured in the router, so you may want to pick one that is supported natively, else you will have to set up an update client that will be responsible to regularly check for IP change.

Domain and subdomains

You will need to buy a domain from you preferred domain provider, for this guide I will use example.com.

[!IMPORTANT] I advise you to also subscribe to a domain privacy option in order to hide you personal data. Domain Privacy protects the contact information of the owner of a domain name in the WHOIS directory. Normally, this public database is used to verify the availability of a domain name and who it belongs to, but marketing companies and scammers can also exploit it for other purposes, like sending spam or identity theft.

You can check the information that are available publicly about your domain using the whois command :

whois example.com

Right, we will then use subdomains to locate each service as a separate website to avoid having to buy a new domain name for each. A subdomain is simply a prefix added to the original domain name, it functions as a separate website from its domain.

So, let's create subdomains from the domain name registrar settings, for every service to be exposed on the internet :

wireguard.example.com : To access the WireGuard server
quake.example.com : To access the Defrag-life website
lychee.example.com : To access the Lychee website
ccteam.example.com : To access the CCTeam APIs
goatcounter.example.com : So that GoatCounter can track the traffic on the exposed websites
shop.example.com : To access the Medusa shop
medusa.example.com : For the store API of Medusa, called by the browsers of the shop visitors

Then add corresponding CNAME records to point to the dynamic DNS myddns.ddns.net :

CNAME wireguard myddns.ddns.net
CNAME quake myddns.ddns.net
CNAME lychee myddns.ddns.net
CNAME ccteam myddns.ddns.net
CNAME goatcounter myddns.ddns.net
CNAME shop myddns.ddns.net
CNAME medusa myddns.ddns.net

A CNAME record is just a records which points a name to another name instead of pointing to an IP address (like A records).

[!NOTE] Services that will only be accessible from the local network or through VPN do not need to have a subdomain defined at this level. We will use Pi-Hole's local DNS records for that. See Pi-Hole configuration.

However, while the VPN stuff is fully functional and to be able to do the configuration easily from your client machine, you may want to temporarily create subdomains and add CNAME records for the following subdomains (also remove the IP whitelisting middleware in the corresponding service configuration, else you will be blocked by IP whitelisting) :

arcane.example.com : To manage Docker containers (start/stop, check logs, etc.)

pihole.example.com : To configure the local DNS records

This is optional, you can of course manage containers and do the Pi-Hole configuration from command line.

Port forwarding

For our services to be reachable from the internet, we need to forward incoming requests to our mini PC so that they will be handled by the reverse proxy. This can be done through port forwarding.

Port forwarding directs the router to send any incoming data from the internet to a specified device on the network. It is safe to forward ports on your router as long as you have a reverse proxy or a firewall running in between.

Allow access without VPN

If you decide that at least one of the applications must be reachable from the outside directly through HTTP or HTTPS without requiring a VPN, then simply port forward the related TCP ports to the mini PC.

Go to your router configuration and add a port forward rule for the TCP port 80 :

Name : Traefik
Input port : 80
Target port : 80
Device : n100
Protocol : TCP

and 443 :

Name : Traefik SSL
Input port : 443
Target port : 443
Device : n100
Protocol : TCP

We will configure Traefik later to redirect HTTP requests to HTTPS. But if you prefer you can only open the HTTPS port (if you are going to use Let's encrypt' HTTP challenge, it's enough for the TLS certificates to be generated, see the warning box a little further below though).

Allow access through VPN

If you want some applications to be available from the outside through VPN, then open the VPN port :

Go to your router configuration and add a port forward rule for the UDP port 51820 :

Name : VPN
Input port : 51820
Target port : 51820
Device : n100
Protocol : UDP

Of course if you want the applications to be available only through VPN, then only open the VPN port, remove any opened HTTP/HTTPS port.

[!WARNING] Note that if you use Let's Encrypt' HTTP challenge to issue and renew SSL/TLS certificates, target websites must be reachable from the internet. That mean you will have to open the HTTP (S) port at least when issuing/renewing certificates, you could also keep them open and restrict access to the necessary IP ranges, if your router supports that. If you really don't want to open HTTP (S) ports (better for security), then you will have to configure DNS challenge instead of HTTP challenge, if your DNS provider support it. See HTTP challenge and DNS challenge below when configuring Traefik.

Reverse proxy

Traefik is an open source HTTP reverse proxy and load balancer that can integrate easily with our Docker infrastructure. We will use it to intercept and route every incoming request to the corresponding backend services.

It will listen to our services and instantly generates the routes, so that they are connected to the outside world. We will also use it to automatically generate and renew SSL/TLS certificates through Let's Encrypt.

Here is an overview of the network flow on our setup :

flowchart LR
    style INCOMING_REQUEST fill: #45d8ff57
    style TRAEFIK_CONTAINER fill: #6601013c
    style MYAPP1_CONTAINER fill: #6601013c
    style MYAPP2_CONTAINER fill: #6601013c
    style TRAEFIK_ROUTER fill: #defc1e37
    style TRAEFIK_MIDDLEWARE fill: #defc1e37
    style SERVER_DEVICE fill: #fec9bf5e
    style CONTAINER_ENGINE fill: #67010030
    INCOMING_REQUEST((INCOMING\nREQUEST))
    DOCKER_TRAEFIK_PORT443{{443/tcp}}
    DOCKER_TRAEFIK_PORT80{{80/tcp}}
    DOCKER_TRAEFIK_PORT8080{{8080/tcp}}
    DOCKER_MYAPP1_PORT{{exposed port}}
    DOCKER_MYAPP2_PORT{{exposed port}}
    TRAEFIK_ROUTER_MYAPP1(myapp1.example.com)
    TRAEFIK_ROUTER_MYAPP2(myapp2.example.com)
    TRAEFIK_ROUTER_TRAEFIK(traefik.example.com)

    subgraph SERVER_DEVICE[MINI PC]
        subgraph CONTAINER_ENGINE[DOCKER]
            subgraph MYAPP1_CONTAINER[MYAPP1 CONTAINER]
                DOCKER_MYAPP1_PORT
            end
            subgraph MYAPP2_CONTAINER[MYAPP2 CONTAINER]
                DOCKER_MYAPP2_PORT
            end

            subgraph TRAEFIK_CONTAINER[TRAEFIK CONTAINER]
                subgraph TRAEFIK_ROUTER[TRAEFIK HTTP ROUTER]
                    TRAEFIK_ROUTER_TRAEFIK
                    TRAEFIK_ROUTER_MYAPP1
                    TRAEFIK_ROUTER_MYAPP2
                end
                subgraph TRAEFIK_MIDDLEWARE[TRAEFIK MIDDLEWARE]
                    IP_WHITELISTING(IP whitelist)
                    AUTH(PocketID auth)
                end
                REDIRECT(HTTPS redirect\non the web entrypoint)
                DOCKER_TRAEFIK_PORT80
                DOCKER_TRAEFIK_PORT443
                DOCKER_TRAEFIK_PORT8080
            end

        end
    end

    INCOMING_REQUEST --> DOCKER_TRAEFIK_PORT80
    INCOMING_REQUEST --> DOCKER_TRAEFIK_PORT443
    DOCKER_TRAEFIK_PORT80 --> REDIRECT
    REDIRECT -.->|301 to https| INCOMING_REQUEST
    DOCKER_TRAEFIK_PORT443 --> TRAEFIK_ROUTER
    TRAEFIK_ROUTER_TRAEFIK --> IP_WHITELISTING
    TRAEFIK_ROUTER_MYAPP2 --> IP_WHITELISTING
    TRAEFIK_ROUTER_MYAPP1 --> DOCKER_MYAPP1_PORT
    IP_WHITELISTING --> AUTH
    IP_WHITELISTING ---> DOCKER_MYAPP2_PORT
    AUTH --> DOCKER_TRAEFIK_PORT8080

It redirects HTTP to HTTPS on its web entrypoint, and handles IP whitelisting and authentication (through PocketID, or basic authentication) through custom middlewares. In this example myapp1 is accessible from the internet, myapp2 is accessible only through VPN, and Traefik (dashboard and APIs) is accessible only through VPN after OIDC authentication.

I've deliberately left out Sablier for the moment, to keep things simple, but basically this would simply add a middleware that checks the state of the application, in order to temporarily display a waiting page while not ready, refer to Scale to zero with Sablier for more information.

Installation

First, create a folder to hold data and configuration :

sudo mkdir /opt/apps/traefik

Then copy the files from this project's traefik directory into the /opt/apps/traefik directory :

docker-compose.yml : The Traefik service definition
traefik.yml : The Traefik static configuration
.env : The secrets read by the service (DNS provider token, CrowdSec bouncer key), to fill in
credentials.txt : A file that will hold users credentials to access the Traefik dashboard (only if you want it to be restricted with basic authentication), see Generate basic authentication credentials

Files should be ready to use, simply replace the e-mail address ([email protected]) in the traefik.yaml file with your e-mail address, and example.com with your hostname.

You will also need to create the JSON file to hold the certificates, see TLS certificates.

Anyway you will find below more details about each file (see Configuration files details) and some further configuration.

Generate basic authentication credentials

This is optional, use this only if you want the Traefik dashboard to be protected with basic authentication rather than via PocketID, allowed users have to be added to the credentials.txt file.

You can generate a user/password using htpasswd :

Install the needed package if not present :
```
sudo apt install apache2-util
```
Generate the credentials (we use bcrypt with a computing time of 10) :
```
htpasswd -nbBC 10 admin xxxxxxxx
```

Then copy the output to the credentials.txt file.

[!NOTE] Actually as Traefik will be accessible only from local network and through VPN, we don't really need to set up authentication, it's more for demonstration, and it's always better to have 2 layers of security than one.

TLS certificates

To enable HTTPS on our websites, we need to get TLS certificates from a certificate authority. A TLS certificate certifies, in a way, the authenticity of a website (actually it proves that we have the ownership of the public key used for TLS encryption), preventing hackers from intercepting any data transmitted between a device and the site.

We will use Let's Encrypt, a nonprofit certificate authority which provide free TLS certificates.

Let's Encrypt can automatically generate certificates via Traefik, for that we need to create a acme.json file that will hold the generated certificates (file is mapped to a volume in the Compose file), so that the certificates are persisted between container restarts (not generated each time which could raise Let's Encrypt rate limits), we also need to change the permissions so that Traefik can access and edit this file :

cd /opt/apps/traefik
touch /opt/apps/traefik/acme.json
chmod 600 /opt/apps/traefik/acme.json

HTTP challenge

If you use HTTP challenge, Let's Encrypt will validate that you control the domain names by trying to reach the web server through HTTP or HTTPS. So you must open and port forward ports 80 or 443 for the TLS certificate to be issued correctly.

The corresponding certificate resolver configuration would be :

tlsChallenge: { }

[!WARNING] Note that Let’s Encrypt will not let you use this challenge to issue wildcard certificates.

DNS challenge

When using DNS challenge, Let's Encrypt will validate that you control the domain names by querying the DNS system for a TXT record under the target domain name. So you don't need to open HTTP or HTTPS port on your router.

First, check that your DNS provider is supported by Traefik to automate the DNS verification, a list can be found here : https://doc.traefik.io/traefik/https/acme/.

Then :

Create an access token / API key from your provider interface
Add the necessary environment variables required by your provider to the .env file next to the Compose file (loaded with env_file), i.e. :
```
MYPROVIDER_ACCESS_TOKEN=<access_token_here>
```

The corresponding certificate resolver configuration would be :

dnsChallenge:
  provider: <your_provider_here>

IP whitelisting

We will set up IP whitelisting so that we can allow only traffic from the local network or from the VPN for some of our services. Indeed, even if we do not have defined public subdomains for these services, they can still be reached via the IP address (actually in that case Traefik will not route the request, but it is still better to have this additional security).

Basically it involves creating a Traefik middleware for defining the IP whitelist and apply it to the needed services. It is declared once, in the dynamic configuration directory :

:page_facing_up: traefik/dynamic/vpn-whitelist.yml :

http:
  middlewares:
    vpn-whitelist:
      ipAllowList:
        sourceRange:
          - "192.168.0.0/24" # your LAN
          - "10.0.0.0/24" # Wireguard subnet

So we allow exactly 2 IP ranges :

the local IP range : IPs assigned to the devices on your local network (computers, mobile devices, ...)
the WireGuard subnet : the VPN peers keep their tunnel address when they reach Traefik, as WireGuard runs on the host and the peers' traffic is not NATed towards the containers, see WireGuard for more details.

That way :

Requests coming from the local network come with a local address assigned by the router DHCP, and are accepted.
Requests coming from the internet through VPN come with a 10.0.0.x address, and are accepted.
Requests coming from the internet without VPN come with a public IP address and are rejected, as it does not match any whitelisted address.

[!IMPORTANT] A request from your own network to a name that resolves to your public IP goes through the NAT loopback of the router and reaches Traefik with the public IP as source : rejected as well. So the private services must resolve to the LAN address of the mini PC for the devices that use them (Pi-Hole's local DNS records, see Pi-hole), and a container that has to call another one (the Traefik plugin fetching a token from PocketID for instance) must use the internal name (i.e.http://pocketid:1411), or a public name that Traefik carries as a network alias on the private network (see PocketID), never a public URL resolving to the public IP.

[!WARNING] Never whitelist a Docker network range. A container is not a trusted client, and with the network segmentation below, a whitelisted Docker range would let a compromised public container walk straight into the private services.

Then the middleware just needs to be referenced in the middlewares section of every router that must stay private (vpn-whitelist@file), as you will see in the services definitions. Keep in mind that it only protects the requests that go through Traefik : what a container can reach directly on the Docker networks is the job of the network segmentation.

Network segmentation

Every service behind the reverse proxy must share a Docker network with Traefik to be reachable by name, but containers on the same network can also talk to each other directly, without going through Traefik and its middlewares. With a single shared network, a vulnerability in one of the applications exposed to the internet (an old PHP website, a photo gallery, an API) gives an attacker a foothold from which every other container is one HTTP request away : Pi-Hole's admin interface, Arcane (and through it the Docker socket, i.e. root on the host), the Traefik dashboard, ... The IP whitelist does not help there, it never sees this traffic.

So Traefik sits on two networks, and nothing else is allowed to be on both :

Network	Who	Reachable from
`traefik-private-net`	Traefik and the private services : Pi-Hole, Arcane, Dashdot, Homer, PhpMyAdmin, PocketID, Sablier, ...	local network and VPN only (`vpn-whitelist`)
`traefik-public-net`	Traefik and the services exposed to the internet : Lychee, Defrag-life, Medusa, ...	anyone

A compromised public container can then only see Traefik and the other public applications, never the private ones. A few rules go with it :

a public application never joins traefik-private-net, a private one never joins traefik-public-net, and no application joins both
the databases stay on the private network of their own stack (lychee-net, defrag-life-net, ...), never on a Traefik network
containers holding the Docker socket (Arcane, Sablier) are private by construction
PocketID stays private : a public application that would authenticate through it does so with the browser, through the public URL and Traefik, it does not need a shared network
a public application monitored by Prometheus shares a dedicated network with Prometheus only (prometheus-<app>-net), never prometheus-net nor its own database network, see Prometheus

Configuration files details

Static configuration file :

:page_facing_up: traefik.yaml :

api:
  dashboard: true

# Health check endpoint (/ping) for Gatus, served on the internal "traefik" entrypoint (8080), not published on the host
ping: { }

entryPoints:
  web:
    address: ':80'
    # Every HTTP request is redirected to HTTPS, before any router : nothing is ever served on port 80
    http:
      redirections:
        entryPoint:
          to: websecure
          scheme: https
          permanent: true

  websecure:
    address: ':443'
    http:
      middlewares:
        # Every request on 443 is checked against the CrowdSec decisions first (see the CrowdSec section)
        - crowdsec@file

providers:
  docker:
    watch: true
    exposedByDefault: false
  file:
    directory: /etc/traefik/dynamic
    watch: true

certificatesResolvers:
  default:
    acme:
      email: [email protected]
      storage: acme.json
      caServer: 'https://acme-v02.api.letsencrypt.org/directory'
      dnsChallenge:
        provider: <your_provider_here>

experimental:
  plugins:
    sablier:
      moduleName: "github.com/sablierapp/sablier-traefik-plugin"
      version: "v1.1.0"
    traefik-oidc-auth:
      moduleName: "github.com/sevensolutions/traefik-oidc-auth"
      version: "v0.18.0"
    crowdsec-bouncer-traefik-plugin:
      moduleName: "github.com/maxlerebourg/crowdsec-bouncer-traefik-plugin"
      version: "v1.7.1"
log:
  level: info

accessLog:
  # One JSON line per request, written to a file shared (read-only) with the CrowdSec container
  filePath: /var/log/traefik/access.log
  format: json
  fields:
    headers:
      names:
        # Request headers are dropped from the log by default, the User-Agent is needed by the CrowdSec scenarios
        User-Agent: keep

This config file :

enables the Traefik dashboard (UI that provides a detailed overview of the current configuration)
enables the ping endpoint (/ping), used by Gatus to check that Traefik is alive. It is served on the internal traefik entrypoint (port 8080), created automatically and not published on the host
defines 2 entrypoints, named web (for port 80) and websecure (for port 443) so that we can receive requests on these ports
redirects every request of the web entrypoint to HTTPS (301, or 308 for the methods other than GET and HEAD), before any router is even considered : no router, no middleware and no service ever handles plain HTTP
defines a docker provider so that we can use container labels for retrieving routing configuration. We have configured it to not expose containers by default, so that containers that do not have a traefik.enable=true label are ignored from the resulting routing configuration. You can omit this if you use only dynamic configuration files
defines a default certificate resolver for Let's Encrypt to automatically generate certificates
set log level to info (you can set it to debug when you need more information on what's going on)
writes the access log as JSON lines in /var/log/traefik/access.log (a folder bound in the Compose file), one line per request with the client IP, the router and the status code : the fastest way to understand why a request is rejected, and the input of CrowdSec. Request headers are dropped from the log by default, the User-Agent is kept for the CrowdSec scenarios
declares the Traefik plugins used by the middlewares (Sablier, OIDC authentication, CrowdSec bouncer), downloaded when Traefik starts
sets the crowdsec middleware on the websecure entrypoint, so that every HTTPS request is checked against the CrowdSec decisions before reaching any router

Service definition :

:page_facing_up: docker-compose.yml :

services:

  traefik:
    image: traefik:latest
    container_name: traefik
    restart: unless-stopped
    ports:
      - "80:80"
      - "443:443"
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock:ro    # So that Traefik can listen to the Docker events
      - ./traefik.yml:/etc/traefik/traefik.yml:ro       # Traefik static configuration
      - ./dynamic:/etc/traefik/dynamic:ro               # Traefik dynamic configuration
      - ./acme.json:/acme.json                          # For Let's Encrypt certificate storage
      - ./credentials.txt:/credentials.txt:ro           # For Traefik dashboard credentials
      - ./logs:/var/log/traefik                         # Access log, shared (read-only) with CrowdSec
    networks:
      # private : services reachable from the local network and the VPN only
      traefik-private-net:
        aliases:
          # Lets the containers of the private network resolve the public name of the OIDC provider to Traefik itself,
          # so that applications authenticating natively against PocketID can use its public issuer URL (see PocketID)
          - pocketid.example.com
          # Same for the public services, so that Gatus checks them through Traefik with their real name
          # (routing, TLS certificate, CrowdSec), without joining the public network nor depending on Pi-hole
          - lychee.example.com
          - quake.example.com
          - goatcounter.example.com
          - ccteam.example.com
          - shop.example.com
          - medusa.example.com
      # public : services exposed to the internet
      traefik-public-net:
        aliases:
          # Lets the Medusa storefront call the store API by its public name from its server-side code,
          # through Traefik but without going out through the box (see Medusa)
          - medusa.example.com
    env_file: .env    # DNS provider token for the DNS challenge, CrowdSec bouncer key
    labels:
      - "traefik.enable=true"

      # The HTTP to HTTPS redirection is done on the "web" entrypoint, see traefik.yml

      # Configure dashboard with HTTPS
      - "traefik.http.routers.dashboard.rule=Host(`traefik.example.com`)"
      - "traefik.http.routers.dashboard.entrypoints=websecure"
      - "traefik.http.routers.dashboard.service=dashboard@internal"
      - "traefik.http.routers.dashboard.tls=true"
      - "traefik.http.routers.dashboard.tls.certresolver=default"

      # Configure API with HTTPS
      - "traefik.http.routers.api.rule=Host(`traefik.example.com`) && PathPrefix(`/api`)"
      - "traefik.http.routers.api.entrypoints=websecure"
      - "traefik.http.routers.api.service=api@internal"
      - "traefik.http.routers.api.tls=true"
      - "traefik.http.routers.api.tls.certresolver=default"

      # Secure dashboard/API behind VPN and PocketID authentication (or basic authentication)
      - "traefik.http.routers.dashboard.middlewares=vpn-whitelist@file,traefik-auth@file"
      - "traefik.http.routers.api.middlewares=vpn-whitelist@file,traefik-auth@file"
      # - "traefik.http.middlewares.auth.basicauth.usersfile=/credentials.txt" # only if you use basic auth

networks:

  traefik-private-net:
    name: traefik-private-net

  traefik-public-net:
    name: traefik-public-net

This Compose file mainly :

exposes ports 80 and 443 to receive incoming HTTP/HTTPS requests
binds the logs folder where the access log is written, shared read-only with the CrowdSec container
defines two networks : traefik-private-net for the services that must stay private (reachable from the local network and the VPN only) and traefik-public-net for the services exposed to the internet, see Network segmentation
gives Traefik network aliases on the private network : the containers of that network resolve these public names to Traefik itself. pocketid.example.com is used by the applications authenticating natively against PocketID (see PocketID), the public services by Gatus to check them through Traefik
gives Traefik a network alias on the public network as well, medusa.example.com : the Medusa storefront calls the store API by its public name from its server side, through Traefik, without looping through the router
loads its secrets from the .env file (see Environment variables) : the DNS provider access token used to issue Let's Encrypt certificates through DNS challenge, and the CrowdSec bouncer key
defines an HTTP router that will match traefik.example.com URL on our websecure entrypoint to point to our service
configures dashboard and api routers to use secure HTTPS endpoint with our certificate resolver to generate related Let's Encrypt certificates
secures dashboard and API endpoints with the vpn-whitelist middleware (requests from the local network and the VPN only) and the traefik-auth middleware (authentication through PocketID, basic authentication being the alternative)

[!CAUTION] The order in which the middlewares are defined in relation to a router is important, they will be applied in the same order as their declaration.

Environment variables :

:page_facing_up: .env :

# Access token / API key of your DNS provider, used by the Let's Encrypt DNS challenge (variable name depends on the provider, see Traefik documentation)
MYPROVIDER_ACCESS_TOKEN=<access_token_here>
# Key of the CrowdSec bouncer (same value as BOUNCER_KEY_traefik in crowdsec/.env), read by traefik/dynamic/crowdsec.yml
CROWDSEC_BOUNCER_KEY=<bouncer_key>

MYPROVIDER_ACCESS_TOKEN is the token of your DNS provider, its name depends on the provider (see DNS challenge)
CROWDSEC_BOUNCER_KEY is read by the crowdsec middleware in dynamic/crowdsec.yml through a template (dynamic configuration files are Go templates, {{ env "..." }} reads a variable of the Traefik container), so that no secret sits in a configuration file. Same value as BOUNCER_KEY_traefik in crowdsec/.env (see CrowdSec)

Run

Finally, run the Compose file :

sudo docker-compose -f /opt/apps/traefik/docker-compose.yml up -d
# You may need to force recreate if you changed a config from an already running configuration
sudo docker-compose -f /opt/apps/traefik/docker-compose.yml up -d --force-recreate

You should end-up with a running traefik container.

It should also have generated the needed Let's Encrypt certificates in the acme.json file.

So you can reach the dashboard at https://traefik.example.com.

VPN and ad-blocking

We will install WireGuard, Pi-hole and Unbound to create a virtual private network (VPN) with ad-blocking and DNS privacy/caching capabilities.

WireGuard is a free and open-source modern VPN that utilizes state-of-the-art cryptography to securely encapsulates IP packets over UDP, in order to lower the environment attack surface. As a VPN it establishes a secure connection between a computer and the internet by making all the traffic going through an encrypted tunnel. The point of self-hosting our own VPN server is to ensure a private and secure connection to our services from the internet, without having to trust third-party VPN providers, and to keep complete freedom and control over the browsing data.

Pi-hole is a network-level ad blocking and internet tracker blocking application. It has the ability to block traditional website advertisements as well as advertisements in unconventional places such as mobile apps ads. It can also be used as a DNS server and has a built-in DHCP server.

Unbound is a validating, recursive, caching DNS resolver, that has the ability to contact DNS authority servers directly in order to validate and cache the queries on your network and serve them to you directly, so you don’t have to rely on your ISP or third-party DNS resolvers (like Cloudflare or Google).

So the idea is that every client in any network can use the VPN to reach our applications while taking advantage of Pi-Hole and Unbound :

flowchart TB
    style WINDOWS11 fill: #01fcbf29
    style LAPTOP fill: #01fcbf29
    style MOBILE fill: #01fcbf29
    style MACOS fill: #01fcbf29
    style WIREGUARD_SERVER fill: #fd500c38
    style PIHOLE fill: #8a200161
    style UNBOUND fill: #5a0f0191
    style INTERNET fill: #bbfd795e
    style HOME_NETWORK fill: #6996fc45
    style 5G_NETWORK fill: #6996fc45
    style WORK_NETWORK fill: #6996fc45
    style MINI_PC fill: #fec1f647
    WINDOWS11(Peer 1 \n Home PC - Windows 11)
    LAPTOP(Peer 2 \n Home laptop - Ubuntu 22)
    MOBILE(Peer 3 \n Phone - Android 14)
    MACOS(Peer 4 \n Work PC - MacOS 13)
    WIREGUARD_SERVER(WireGuard server - Secure VPN)
    PIHOLE(Pi-Hole - Firewall & ad-blocking)
    UNBOUND(Unbound - Custom DNS resolver)
    INTERNET((Internet))

    subgraph HOME_NETWORK[Home network]
        WINDOWS11
        LAPTOP
    end

    subgraph 5G_NETWORK[Mobile network]
        MOBILE
    end

    subgraph WORK_NETWORK[Work network]
        MACOS
    end

    subgraph MINI_PC[Mini PC]
        WIREGUARD_SERVER
        PIHOLE
        UNBOUND
    end

    WINDOWS11 -- WireGuard tunnel --> WIREGUARD_SERVER
    LAPTOP -- WireGuard tunnel --> WIREGUARD_SERVER
    MOBILE -- WireGuard tunnel --> WIREGUARD_SERVER
    MACOS -- WireGuard tunnel --> WIREGUARD_SERVER
    WIREGUARD_SERVER -- DNS queries --> PIHOLE
    PIHOLE -- Filtered DNS queries --> UNBOUND
    UNBOUND -- DNS resolution --> INTERNET

WireGuard runs directly on the host (kernel module, managed by wg-quick), Pi-Hole and Unbound run as two small Compose stacks. Everything about performance is in VPN connection speed.

Installation

First, create the folders that will hold data and configuration :

sudo mkdir -p /opt/apps/pihole /opt/apps/unbound /opt/apps/wgdashboard/data

Then from this project's pihole, unbound and wgdashboard directories, copy the docker-compose.yml files into the matching /opt/apps folders. For more details about these files, see Configuration files details below.

WireGuard itself is a Debian package :

sudo apt install wireguard

Now let's take a look at the configuration for each service.

Configuration

WireGuard

WireGuard runs directly on the host : the kernel module is part of Debian, wg-quick manages the interface, and the peers are managed in the configuration file (or with the wg command). Compared to running it in a container, this removes a few hops for every packet (Docker bridge, veth pair, a second NAT layer and the userland proxy) and makes the network stack much easier to observe and tune.

Generate the keys (wg genkey | tee private.key | wg pubkey > public.key, on the server and on each peer) and create the configuration :

sudo nano /etc/wireguard/wg0.conf

:page_facing_up: /etc/wireguard/wg0.conf (enp1s0 is the LAN interface of the mini PC, %i is replaced by the interface name) :

[Interface]
Address = 10.0.0.1/24
ListenPort = 51820
MTU = 1420
PrivateKey = <server private key>
PostUp = iptables -N DOCKER-USER 2>/dev/null || true; iptables -C DOCKER-USER -i %i -j ACCEPT 2>/dev/null || iptables -I DOCKER-USER 1 -i %i -j ACCEPT; iptables -C DOCKER-USER -o %i -j ACCEPT 2>/dev/null || iptables -I DOCKER-USER 2 -o %i -j ACCEPT; iptables -A FORWARD -i %i -j ACCEPT; iptables -A FORWARD -o %i -j ACCEPT; iptables -t nat -C POSTROUTING -s 10.0.0.0/24 -o enp1s0 -j MASQUERADE 2>/dev/null || iptables -t nat -A POSTROUTING -s 10.0.0.0/24 -o enp1s0 -j MASQUERADE; iptables -t mangle -C FORWARD -p tcp --tcp-flags SYN,RST SYN -j TCPMSS --clamp-mss-to-pmtu 2>/dev/null || iptables -t mangle -A FORWARD -p tcp --tcp-flags SYN,RST SYN -j TCPMSS --clamp-mss-to-pmtu; iptables -t raw -C PREROUTING -i enp1s0 -p udp --dport 51820 -j NOTRACK 2>/dev/null || iptables -t raw -A PREROUTING -i enp1s0 -p udp --dport 51820 -j NOTRACK; iptables -t raw -C OUTPUT -o enp1s0 -p udp --sport 51820 -j NOTRACK 2>/dev/null || iptables -t raw -A OUTPUT -o enp1s0 -p udp --sport 51820 -j NOTRACK; tc qdisc replace dev %i root cake bandwidth 860mbit besteffort || true
PostDown = iptables -D DOCKER-USER -i %i -j ACCEPT || true; iptables -D DOCKER-USER -o %i -j ACCEPT || true; iptables -D FORWARD -i %i -j ACCEPT || true; iptables -D FORWARD -o %i -j ACCEPT || true; iptables -t nat -D POSTROUTING -s 10.0.0.0/24 -o enp1s0 -j MASQUERADE || true; iptables -t mangle -D FORWARD -p tcp --tcp-flags SYN,RST SYN -j TCPMSS --clamp-mss-to-pmtu || true; iptables -t raw -D PREROUTING -i enp1s0 -p udp --dport 51820 -j NOTRACK || true; iptables -t raw -D OUTPUT -o enp1s0 -p udp --sport 51820 -j NOTRACK || true

[Peer]
# desktop-home
PublicKey = <peer public key>
AllowedIPs = 10.0.0.2/32

[Peer]
# phone
PublicKey = <peer public key>
AllowedIPs = 10.0.0.3/32

Then protect and enable it :

sudo chmod 600 /etc/wireguard/wg0.conf
sudo systemctl enable --now wg-quick@wg0

The PostUp line looks scary, but each piece has a reason (and wg-quick runs the hooks with set -e, so anything that may legitimately fail has to be guarded with || true or a -C check, else the interface is torn down) :

DOCKER-USER fast path : Docker sets the FORWARD policy to DROP and inserts about a hundred rules (four per bridge network) that every relayed packet walks through. The DOCKER-USER chain is evaluated first and is never flushed by Docker, so accepting the tunnel traffic there short-circuits the whole chain. The plain FORWARD rules are a fallback in case wg0 comes up before Docker at boot.
NAT : the peers' traffic leaves with the mini PC address (on my machine the rule was already set globally, keeping it here makes the file self-contained).
TCPMSS clamp : TCP inside the tunnel can carry 1380 bytes per segment at most, clamping the MSS on the SYN packets prevents fragmentation and black holes for the relayed connections.
NOTRACK : connection tracking is useless for the encrypted UDP flow (WireGuard authenticates every packet itself), this saves a lookup per packet.
cake : gives every flow inside the tunnel its own queue and keeps the latency low. Without it, the fq_codel queue of the physical interface sees the whole tunnel as a single flow, so a big download can starve a video stream or a call. 860mbit is what a gigabit link carries once the tunnel overhead is added, it costs nothing measurable.

The DNS pushed to the peers is the tunnel address of the server, 10.0.0.1 : Docker publishes Pi-Hole's port 53 on every address of the host, including this one, so the peers reach Pi-Hole (then Unbound) without any extra route, and it also works in split tunnel mode since the address is inside the tunnel subnet.

Peers configuration

On each device, the client configuration looks like this :

[Interface]
PrivateKey = <peer private key>
Address = 10.0.0.2/32
DNS = 10.0.0.1
MTU = 1420

[Peer]
PublicKey = <server public key>
Endpoint = 192.168.0.16:51820
# full tunnel : 0.0.0.0/1, 128.0.0.0/1 — split tunnel : 10.0.0.0/24
AllowedIPs = 0.0.0.0/1, 128.0.0.0/1
PersistentKeepalive = 25

[!TIP] A few things I learned the hard way about the peers configuration :

At home, use the LAN IP address of the server as endpoint (192.168.0.16:51820), not the public hostname : going through the public IP from inside the LAN makes the router do NAT loopback (hairpin) in software, which cost me about half of the throughput (350/440 Mbit/s instead of 570/860). Easiest is to keep two tunnels on the device : a "home" one with the LAN endpoint and an "away" one with the public hostname.

At home, a full tunnel brings nothing : the traffic leaves through the same router anyway, it only adds encryption and relaying work for the server (and costs about 40 % of the download speed, see VPN connection speed). Use a split tunnel (AllowedIPs limited to the VPN subnet, here 10.0.0.0/24, which contains the DNS address so that the DNS still goes through the tunnel), or simply no tunnel at all with the device DNS pointing to the mini PC : the ad blocking is done at DNS level, it is identical in all cases.

0.0.0.0/1, 128.0.0.0/1 also disables the kill switch and the DNS leak protection of the Windows client (only a 0.0.0.0/0 route enables them), so Windows silently falls back to the router DNS if Pi-Hole does not answer within about a second.

WGDashboard

Managing the peers in wg0.conf with an editor works, but a web interface is more comfortable : WGDashboard shows the interfaces, the peers, their last handshake and their traffic, creates a peer with its keys and its QR code, and serves the configuration file to download.

[!IMPORTANT] The container must share the host's network namespace (network_mode: host). WireGuard runs on the host, so the wg0 interface lives in the host's namespace : a container with its own namespace can read wg0.conf through the bind mount, and will happily list your peers, but it cannot query the interface itself. The symptom is unmistakable : the peers are displayed but always as disconnected, with no handshake and no traffic, whatever their real state.

Two consequences follow from the host namespace, and they are the whole difficulty of this service :

Traefik can no longer reach it by container name, since the container is on no Docker network. The service points at the mini PC address instead, http://192.168.0.16:10086
the dashboard is reachable directly at that same address, hence bypassing Traefik, the IP whitelist and any authentication middleware. Its own login therefore remains the barrier that covers every path, and it is the reason we do not put PocketID in front of it : that would only protect the nice URL while leaving the direct one open

To close that direct path, app_ip in the [Server] section of wg-dashboard.ini can bind the dashboard to the gateway address of the private Traefik network (docker network inspect traefik-private-net --format '{{range .IPAM.Config}}{{.Gateway}}{{end}}') so that only Traefik and the containers of that network can reach it. Keep in mind that this address depends on the subnet Docker assigns, and would change if the network were recreated.

[!NOTE] OIDC is not available for the admin dashboard, only for the client side app. The [OIDC] admin_enable setting and the /api/oidc/toggle endpoint exist, and the Admin section of wg-dashboard-oidc-providers.json can be filled, but nothing consumes them : dashboard.py never instantiates the DashboardOIDC module, so no provider is registered, no request is made to the provider, nothing is logged, and no button appears. Don't spend an evening looking for a configuration mistake.

The single sign-on documented by the project applies to the client side app (/client), a self-service portal where people sign in to download the peers assigned to them : fill the Client section instead, set client_enable = true, and register the portal URL itself as the callback. The (empty) Clients tab of the admin dashboard lists those portal accounts, not your peers, an empty tab is normal when you are the only user.

The extra_hosts entry of the Compose file is there for that case : in the host namespace the container resolves names through the host resolver, which knows nothing of the private names, so the provider would not even be resolved. See PocketID for the general problem and its other solutions.

Setting up

Create the folder, then copy the docker-compose.yml file from this project's wgdashboard directory into /opt/apps/wgdashboard, and the wgdashboard.yml file from traefik/dynamic into /opt/apps/traefik/dynamic :

sudo mkdir -p /opt/apps/wgdashboard/data

Then add a local DNS record wgdashboard.example.com pointing to the mini PC (see Pi-hole), start the container (see Run) and open https://wgdashboard.example.com. The default credentials are admin / admin, change them immediately in the settings, where TOTP can also be enabled.

Your existing peers appear on their own : the dashboard reads the very wg0.conf the interface uses, so nothing has to be imported and nothing is duplicated.

[!WARNING] The dashboard can start and stop the interface, but wg0 is managed by wg-quick@wg0 through systemd. Avoid switching it from both sides, otherwise systemd and the dashboard end up with diverging views of what is running.

Peers created from the dashboard are written directly into wg0.conf. Keep a copy of that file with your backups : it holds the server's private key, and losing it means every client has to be reconfigured.

Pi-hole

The Compose file will run a Pi-Hole instance which need to be configured.

First, Pi-Hole must accept the queries coming from other interfaces than its own Docker network (the VPN peers, the LAN) : by default it only answers "local" requests, and "local" for Pi-Hole is the Docker bridge network. The Compose file sets this once and for all with theFTLCONF_dns_listeningMode: 'all' environment variable (the equivalent of Settings -> DNS -> Interface settings -> "Permit all origins" in the web UI).

The web UI is reachable at https://pihole.example.com through Traefik, the router is declared in a file of Traefik's dynamic configuration directory (see Traefik routing below), restricted to the local network and the VPN peers.

[!IMPORTANT] Chicken and egg : the private services have no public DNS record (see Domain and subdomains), so pihole.example.com can only be resolved by Pi-Hole itself through a local DNS record... which is created in the web UI you cannot reach yet. Until it exists the browser gets NXDOMAIN (or, if a public record for the name still exists, reaches Traefik through the NAT loopback of the router with the public IP as source and gets a 403, see IP whitelisting). Create the first record from the command line, it is applied immediately :
sudo docker exec pihole pihole-FTL --config dns.hosts '[ "192.168.0.16 pihole.example.com" ]'
sudo docker exec pihole nslookup pihole.example.com 127.0.0.1
Then make sure the device you use has Pi-Hole as DNS server (192.168.0.16, see IP settings), flush its cache (ipconfig /flushdns on Windows) and restart the browser. --config dns.hosts replaces the whole list : to add entries later from the command line, repeat the complete list, or simply use the web UI once it is reachable.

I don't set a Pi-Hole password : authentication is handled in front of it by the reverse proxy, with an OIDC middleware backed by PocketID (see PocketID), and the network segmentation keeps the container out of reach of the applications exposed to the internet. The image generates a random password at first start, remove it (or set yours) with :

sudo docker exec -it pihole pihole setpassword

In Settings -> DNS, untick every public upstream and add Unbound as custom upstream DNS server : 10.2.0.200#53 (its static address in the pihole-net Docker network, see Services definition).

Then we need to add local DNS records so that the domain names can be resolved from VPN or local network (remember the DNS requests of the VPN peers and of the configured devices go through Pi-Hole). We simply need to associate domain names with the internal IP address of the mini PC, so they can be handled by the reverse proxy.

Go to Settings -> Local DNS Records (or repeat the pihole-FTL --config dns.hosts command above with the complete list) and add a DNS record entry for every subdomain that must only be reachable from the local network or through VPN :

arcane.example.com                  192.168.0.16
ccteam.example.com                  192.168.0.16
crowdsec.example.com                192.168.0.16
dashboard.example.com               192.168.0.16
dashdot.example.com                 192.168.0.16
gatus.example.com                   192.168.0.16
ghostfolio.example.com              192.168.0.16
goatcounter.example.com             192.168.0.16
grafana.example.com                 192.168.0.16
homebox.example.com                 192.168.0.16
homelable.example.com               192.168.0.16
lychee.example.com                  192.168.0.16
medusa.example.com                  192.168.0.16
phpmyadmin.example.com              192.168.0.16
pihole.example.com                  192.168.0.16
pocketid.example.com                192.168.0.16
prometheus.example.com              192.168.0.16
quake.example.com                   192.168.0.16
shop.example.com                    192.168.0.16
speedtest-tracker.example.com       192.168.0.16
traefik.example.com                 192.168.0.16
wgdashboard.example.com             192.168.0.16

Add the public services as well (Lychee, Defrag-life, ...), even though they have a public DNS record. Without a local record, a device at home resolves them to the public IP and the traffic loops through the NAT loopback of the router : it costs about half of the throughput (measured in VPN connection speed), and Traefik sees the requests coming from your public IP address instead of the device's one, so they are treated like internet traffic by the IP whitelist and by CrowdSec (a misbehaving device at home could get your whole household banned from your own sites). With a local record, everything stays on the LAN.

[!NOTE] Consequence for the VPN peers away from home : they use Pi-Hole through the tunnel, so these names resolve to 192.168.0.16 for them too, which is only reachable with a full tunnel or with 192.168.0.0/24 added to AllowedIPs. Do that on the away profile only : on the home profile, routing the LAN subnet through the tunnel would send the traffic to your printer or TV through the mini PC.

You can also configure rate limiting (default to 1000 queries per minute), domain whitelisting, DNS settings, etc. but I will not go through all Pi-Hole configuration, the default should work just fine.

If it is working you should be able to see activity in the dashboard.

Unbound

The first time you will run Unbound, it may fail because a few files included in the default configuration will be missing (at least in the image version I'm using), indeed the following files are included in the default unbound.conf file (which should have been created correctly in /etc/unbound/unbound.conf) :

/opt/unbound/etc/unbound/a-records.conf
/opt/unbound/etc/unbound/srv-records.conf
/opt/unbound/etc/unbound/forward-records.conf

You could manually create these files (you can find default ones from the Unbound GitHub repository), and then mount them into the Unbound container, before running again the Compose file.

But that way it would run Unbound in forwarder mode, meaning that the DNS server will forward all the queries to Cloudflare. This was my first try and a DNS leak test confirmed that it uses Cloudflare, indeed the default forward-records.conf file includes the following forwarding rules :

forward-addr: 1.1.1.1@853#cloudflare-dns.com
forward-addr: 1.0.0.1@853#cloudflare-dns.com

So, if you want to run Unbound without forwarding, just remove or comment the lines that includes the above files from the unbound.conf file. Do not create any of these files at all and do not bind them in the container, just remove the includes from the unbound.conf file.

A DNS leak test should now show your IP address as DNS server.

[!IMPORTANT] If you use the default forward-records.conf file, Unbound will run in forwarder mode, meaning that it will forward all queries to Cloudflare. To remove the default forwarding to Cloudflare and make your unbound container a recursive-only server, edit the unbound.conf file and remove include of the forward-records.conf file.

Then there are a few settings in unbound.conf that are essential when Unbound runs in a container. I ran for months with a resolver that returned SERVFAIL for most names that were not already in cache (login.live.com, www.apple.com, the Twitch video servers, ...), cached names being served fine, which made streams randomly fail to start and Windows painfully slow at boot when the tunnel was up :

server:
    # the container has no IPv6 connectivity : without this, Unbound keeps trying the IPv6 addresses of the authoritative
    # servers, burns its retry budget and ends up with SERVFAIL ("exceeded the maximum number of sends")
    do-ip6: no
    # 0x20 case randomization breaks with load balanced domains (Microsoft, Akamai, Twitch, ...) that answer differently
    # on each query, Unbound then cannot validate its fallback ("0x20 failed, then got different replies in fallback")
    use-caps-for-id: no
    # 1 is plenty, 5 (debug) formats a huge amount of text for every single query, even when it ends up in /dev/null
    verbosity: 1
    # log the reason of each SERVFAIL to the container output (sudo docker logs unbound)
    log-servfail: yes
    logfile: ""
    use-syslog: no

A quick way to validate such changes without touching the running resolver is to start a throwaway Unbound with the modified file on the same Docker network, and to compare both on names that are not cached :

sudo docker run -d --name unbound-test --network wireguard_net -v /tmp/unbound-test.conf:/opt/unbound/etc/unbound/unbound.conf:ro mvance/unbound:latest
dig @$(sudo docker inspect unbound-test --format '{{range .NetworkSettings.Networks}}{{.IPAddress}}{{end}}') login.live.com
sudo docker logs unbound-test | grep SERVFAIL
sudo docker rm -f unbound-test

With my original file 9 names out of 16 failed, with do-ip6: no alone all 16 succeeded, and use-caps-for-id: no on top made them faster.

Do not enable log-queries for daily use, Unbound logs a lot.

Configuration files details

Services definition

:page_facing_up: pihole/docker-compose.yml :

services:

  pihole:
    container_name: pihole
    image: pihole/pihole:latest
    restart: unless-stopped
    ports:
      - "53:53/tcp"
      - "53:53/udp"
    environment:
      TZ: "Europe/Zurich"
      FTLCONF_dns_listeningMode: 'all'
    networks:
      pihole-net:
        ipv4_address: 10.2.0.100
      traefik-private-net:
    volumes:
      - "./etc-pihole/:/etc/pihole/"
    cap_add:
      - NET_ADMIN
      - SYS_TIME
      - SYS_NICE

networks:

  pihole-net:
    name: pihole-net
    ipam:
      driver: default
      config:
        - subnet: 10.2.0.0/24

  traefik-private-net:
    name: traefik-private-net
    external: true

:page_facing_up: traefik/dynamic/pihole.yml :

http:
  services:
    pihole:
      loadBalancer:
        servers:
          - url: http://pihole:80

  routers:
    pihole:
      rule: 'Host(`pihole.example.com`)'
      entryPoints:
        - websecure
      tls:
        certResolver: default
      service: pihole
      middlewares:
        - vpn-whitelist@file
        - pihole-auth@file

This Compose file :

defines the pihole-net network with the subnet 10.2.0.0/24 (shared with Unbound)
references the traefik-private-net Traefik network so that the web UI can be reached through the reverse proxy (the router itself is declared on the Traefik side, see below)
defines the pihole service :
- publishes port 53 (TCP and UDP) on every address of the host, which is what makes Pi-Hole reachable from the LAN (192.168.0.16) and from the VPN peers (10.0.0.1) without any extra rule
- sets the timezone and FTLCONF_dns_listeningMode: 'all' (see Pi-hole)
- assigns the static IP address 10.2.0.100
- binds the /etc/pihole folder to keep the configuration and the databases
- adds the NET_ADMIN, SYS_TIME and SYS_NICE capabilities recommended by the Pi-Hole image (DHCP server, time synchronisation, scheduling priority)
it uses Traefik dynamic config file to :
- define the pihole service pointing to the container on port 80 (reachable by name thanks to the shared traefik-private-net network)
- define the router matching pihole.example.com on the websecure entrypoint with a Let's Encrypt certificate
- restrict the web UI to the local network and the VPN peers with the vpn-whitelist middleware
- add a forward-auth middleware pihole-auth in front of it (I use PocketID) to require authentication (see PocketID)

[!NOTE] Do not lower the MTU of the Docker networks "to fit the tunnel" (I had com.docker.network.driver.mtu: "1280" on all of them for a long time) : the containers don't need it, MSS clamping and PMTU discovery take care of TCP through the tunnel and DNS answers fit anyway. A small bridge MTU only means more packets for the same data and a dependency on ICMP for the inbound traffic, and back when WireGuard itself ran in a container it forced the kernel to fragment every single encrypted packet.

:page_facing_up: unbound/docker-compose.yml :

services:

  unbound:
    image: "mvance/unbound:latest"
    container_name: unbound
    restart: unless-stopped
    hostname: "unbound"
    volumes:
      - "./unbound:/opt/unbound/etc/unbound/"
    networks:
      pihole-net:
        ipv4_address: 10.2.0.200

networks:

  pihole-net:
    name: pihole-net
    external: true

This Compose file only defines the unbound service, on the same (external) pihole-net network with the static IP address 10.2.0.200, and binds the configuration folder so that unbound.conf can be edited (see Unbound). Unbound is not exposed at all, only Pi-Hole talks to it.

:page_facing_up: wgdashboard/docker-compose.yml :

services:

  wgdashboard:
    image: ghcr.io/wgdashboard/wgdashboard:latest
    restart: unless-stopped
    container_name: wgdashboard
    volumes:
      - /etc/wireguard:/etc/wireguard
      - ./data:/data
    network_mode: host
    cap_add:
      - NET_ADMIN
    extra_hosts:
      - "pocketid.example.com:192.168.0.16"

This one is the odd one out : no network and no published port, because network_mode: host puts it in the host's network namespace, the only way for it to see the live state of wg0 (see WGDashboard). NET_ADMIN lets it act on the interface, /etc/wireguard is shared with the host so that it edits the very file wg-quick uses, and data holds its own database and settings. extra_hosts is only useful if you enable the single sign-on of the client side app.

Traefik routing

:page_facing_up: traefik/dynamic/pihole.yml (to copy into /opt/apps/traefik/dynamic/, the directory watched by the file provider of traefik.yml) :

http:
  services:
    pihole:
      loadBalancer:
        servers:
          - url: http://pihole:80

  routers:
    pihole:
      rule: 'Host(`pihole.example.com`)'
      entryPoints:
        - websecure
      tls:
        certResolver: default
      service: pihole
      middlewares:
        - vpn-whitelist@docker
        - pihole-auth@file

It declares the pihole service pointing to the container on port 80 (reachable by name thanks to the shared traefik-private-net network) and the router matching pihole.example.com on the websecure entrypoint with a Let's Encrypt certificate, exactly what the Traefik labels used to do, but Traefik picks up the file without restarting anything. The vpn-whitelist middleware keeps the web UI private (local network and VPN peers only). The pihole-auth middleware is a forward-auth middleware (I use PocketID) to require authentication (see PocketID).

:page_facing_up: traefik/dynamic/wgdashboard.yml :

http:
  services:
    wgdashboard:
      loadBalancer:
        servers:
          - url: http://192.168.0.16:10086

  routers:
    wgdashboard:
      rule: 'Host(`wgdashboard.example.com`)'
      entryPoints:
        - websecure
      tls:
        certResolver: default
      service: wgdashboard
      middlewares:
        - vpn-whitelist@file

The WGDashboard router is the same, with one difference : its service points at the mini PC address rather than at a container name, since the container has no Docker network of its own. Only the IP whitelist is applied, for the reason explained in WGDashboard.

Run

Once the tunnel is up (see WireGuard), run the three Compose files :

sudo docker-compose -f /opt/apps/pihole/docker-compose.yml up -d
sudo docker-compose -f /opt/apps/unbound/docker-compose.yml up -d
sudo docker-compose -f /opt/apps/wgdashboard/docker-compose.yml up -d

You should end up with the wg0 interface (sudo wg show) and 3 running containers, pihole, unbound and wgdashboard. Unbound is not exposed, Pi-Hole is reachable at https://pihole.example.com and WGDashboard at https://wgdashboard.example.com.

Test the network

DNS resolution

Each service should be resolvable through its subdomain name.

When a user enters the URL in the browser, the browser need to know the IP address corresponding to the domain name, so it can send the queries to. For that it :

Checks the browser cache (most browsers cache DNS data by default), and use the address corresponding to the provided name if found
Checks the OS cache and return the address to the browser if found
Checks the local host table file (usually /etc/hosts on Linux/Mac systems and C: \Windows\System32\Drivers\etc\hosts on Windows) to see if an entry matches the specified name, if so, it will directly return it to the browser
Invokes the local resolver, on Windows, it is defined at the network adapter level, usually Control Panel > Network and Internet > Network Connections then in the advanced properties of the desired connection (Higher Priority Connection). On Linux/Mac it is usually /etc/resovl.conf. In my Windows system it is automatically configured to point to my router local IP Address. The resolver checks its cache to see if it already has the address for this name. If it does, it returns it immediately to the browser
Checks the router cache and return the address to the browser if found
Checks the ISP cache and return the address to the browser if found
Checks the ISP resolving name server which will call the root DNS servers (root server <--> TLD server <--> Authoritative Name Server) to find the IP address from the DNS server responsible for the domain name

In our case the requests from the local network should reach Pi-hole (directly, or through the router if it really forwards them, see IP settings), so the IP resolving goes through Pi-Hole and Unbound. See Network flow later below for a more graphical representation of the network flow.

You can first test that each service is resolvable using nslookup command, i.e. :

C:\Users\Yann39>nslookup myapp.example.com
Server :   pi.hole
Address:  10.2.0.100

Name :     myapp.example.com
Address:  192.168.0.16

If the answering server is the router instead of Pi-Hole, the router does not forward the queries (see IP settings). If names resolve fine once cached but fail (SERVFAIL) or take a second the first time, the problem is on Unbound's side, see the settings in Unbound.

Then you can look for DNS leak using any online checker, to determine which DNS servers the browser is using to resolve domain names, it should end up showing your public IP address, not Cloudflare or Google, etc. as we use Unbound (see Unbound for configuration).

You could also use tools like Wireshark to look closely at DNS resolution or to confirm that the traffic is effectively going through the VPN when connected (in that case the "Protocol" column should be WireGuard for all queries). I will not go through a Wireshark tutorial, but it is a very useful and interesting tool for viewing what going on in your network.

Reachability

To verify that the network is set up correctly, we can simply try to access some services and see if we can reach them or not, from different device and connection type.

[!NOTE] I simply temporarily added a CNAME record in my domain name registrar for the services to be checked, to point to my DDNS for testing the IP whitelisting, Traefik will not route request if you try to access a service via the public IP address.

For example if we try to access a service that must be accessible only through VPN (and local network), here are the results :

Device	Connection	VPN status	Public IP	Remote address (request header)	Traefik	Response
PC	cable	:red_circle: off	144.12.117.3	192.168.0.16	192.168.0.11	:heavy_check_mark: 200 OK
PC	cable	:green_circle: on	144.12.117.3	192.168.0.16	192.168.0.11	:heavy_check_mark: 200 OK
Mobile	wifi	:red_circle: off	144.12.117.3	192.168.0.16	192.168.0.12	:heavy_check_mark: 200 OK
Mobile	wifi	:green_circle: on	144.12.117.3	192.168.0.16	172.21.0.1	:heavy_check_mark: 200 OK
Mobile	4G	:red_circle: off	81.165.84.189	144.12.117.3	81.165.84.189	:x: 403 Forbidden
Mobile	4G	:green_circle: on	144.12.117.3	192.168.0.16	172.21.0.1	:heavy_check_mark: 200 OK

192.168.0.16 is the mini PC's private IP address
144.12.117.3 is the router's public IP address
192.168.0.11 is the desktop PC's local IP address
192.168.0.12 is the mobile phone's local IP address
172.21.0.1 is the Traefik Bridge network IP address
81.165.84.189 is the public IP address on the mobile 4G network

These are expected results, we can see that the service is reachable from the local network and from anywhere when using the VPN, and it is not accessible outside the local network if we don't use the VPN.

We can also confirm this by looking at the Traefik logs (you have to set level to debug in traefik.yml file to see the debug logs) which shows that the vpn-whitelist middleware blocks any IP address that is not whitelisted :

level=debug msg="Authentication succeeded" middlewareType=BasicAuth middlewareName=auth@docker
level=debug msg="Accepting IP 192.168.0.16" middlewareName=vpn-whitelist@docker middlewareType=IPWhiteLister
level=debug msg="Accepting IP 172.21.0.1" middlewareName=vpn-whitelist@docker middlewareType=IPWhiteLister
level=debug msg="Rejecting IP 81.165.84.189: \"81.165.84.189\" matched none of the trusted IPs" middlewareName=vpn-whitelist@docker middlewareType=IPWhiteLister

VPN connection speed

To verify that the VPN is not killing the connection speed, first run an online speed test with and without the tunnel, from a wired device (Wi-Fi adds its own variability). These are my results with a symmetric gigabit fiber line, from the home PC :

Test (home PC, Ethernet)	Download / upload (Mbit/s)
No VPN	920 / 920
Split tunnel (only the VPN subnet routed)	910 / 920
Full tunnel, endpoint = LAN IP of the server	570 / 860
Full tunnel, endpoint = public hostname (router hairpin)	350 / 440

The upload is fine, the hairpin case is explained in Peers configuration, and the download ceiling took me an evening of measurements to understand. Here is what I learned, so you don't have to.

Configure MTU

Most Ethernet connections have an MTU of 1500. You can confirm this on your network by running the ping command with the right parameters :

ping www.google.com -f -l 1472
ping www.google.com -f -l 1473

1472 will work and 1473 will warn that the packet needs to be fragmented, because the IPv4 header is 20 bytes and the ICMP header is 8 bytes (1472 + 20 + 8 = 1500).

WireGuard adds its own headers, 60 bytes on IPv4 and 80 bytes on IPv6, so the tunnel MTU must be 1500 - 80 = 1420 (the wg-quick default). Beware of tools defaulting to 1450 (WireGuard UI did) : that produces 1510 bytes packets that get fragmented, and a fragmented tunnel is dramatically slow (a few percent of the line rate). Set 1420 on the server and on every peer, and don't go lower : a smaller MTU only means more packets for the same data.

Measure where the limit is

Speed tests only give the end result. To know which part of the path limits, use iPerf 3 between the peer and the server, in both directions, in UDP and in TCP. Install iperf3 on the server (sudo apt install iperf3) and on the client (Windows builds are available on iperf.fr), run iperf3 -s on the client (allow it in the Windows firewall), then from the server, with the tunnel up (10.0.0.2 being the tunnel address of the peer and 192.168.0.12 its LAN address) :

# reference : LAN, no tunnel, both directions
iperf3 -c 192.168.0.12 -t 10 -P 4
iperf3 -c 192.168.0.12 -t 10 -P 4 -R
# through the tunnel, UDP at a fixed rate : does the path carry the packets at all ?
iperf3 -c 10.0.0.2 -u -b 900M -l 1350 -t 10
# through the tunnel, TCP : what does a real transfer get ?
iperf3 -c 10.0.0.2 -t 10 -P 4
iperf3 -c 10.0.0.2 -t 10 -P 4 -R

And while a test runs, watch the receive drops of the network card and the state of the TCP connections on the server :

ethtool -S enp1s0 | grep rx_missed      # before / after : frames the card dropped because its receive ring was full
ss -ti dst 10.0.0.2                     # rtt, cwnd and retrans of the running connections

My results on the N100 :

LAN without tunnel : 940 Mbit/s both ways, zero retransmission. Card, cable, router and PC are fine.
Tunnel, UDP : 900+ Mbit/s both ways with 0.00 % loss at line rate. The whole path, encryption on the N100 and decryption on the PC included, carries the full gigabit.
Tunnel, TCP, upload (peer to internet) : 860 to 930 Mbit/s.
Tunnel, TCP, download (internet to peer) : 550 to 600 Mbit/s, whatever I tried, with rx_missed climbing on the server (50 to 1000 per second) while the CPU never went above 70 % on the busiest core.

So the limit is neither the CPU nor WireGuard, it is the network card. The Realtek RTL8168H of this mini PC (r8169 driver) has a single queue, a single interrupt handled by a single core, and a receive ring of 256 descriptors (hardware maximum), which holds about 3 ms of gigabit traffic. When the card receives and transmits at ~600 Mbit/s at the same time, which is exactly what relaying a download through the tunnel does, the ring overflows during the small scheduling gaps of the receive path, and the dropped frames make the TCP senders on the internet back off. "Polite" senders (speed test servers) settle around 560 Mbit/s, aggressive ones (public iPerf servers with 10 Gbit/s uplinks) push ~850 Mbit/s through at the price of tens of thousands of retransmissions. The upload is not affected because the plaintext sent out benefits from segmentation offload (far fewer packets to handle), and UDP is not affected because it does not react to drops.

For the record, here is what does not move that ceiling (I measured each one) : pinning the card interrupt to a dedicated core and steering the rest with RPS, interrupt coalescing (receive coalescing even multiplied the drops by 30), threaded NAPI with real-time priority, real-time ksoftirqd, a bigger NAPI budget, disabling Ethernet flow control, TSO/GSO, cake on the tunnel interface, an ingress shaper, a fast path in iptables. Some of them lower the CPU usage, none of them changes the size of the receive ring.

What does help :

Don't use a full tunnel at home, see Peers configuration : a split tunnel, or no tunnel at all with the DNS pointing to Pi-Hole, gives the same ad blocking at 920 Mbit/s. Away from home, the remote connection is the limit anyway.
If you really want line rate through the tunnel, the fix is hardware : a multi-queue network card (for example an Intel i226 on an M.2 A+E adapter, brings 4 queues and receive rings up to 4096 descriptors).

Network card settings

Two settings of the r8169 driver are worth changing anyway, they lower the CPU cost of the upload and of the LAN traffic. Put them as post-up commands of the interface in /etc/network/interfaces so that they survive a reboot :

iface enp1s0 inet dhcp
    # the driver keeps scatter-gather and TCP segmentation offload off by default because of old reports of transmit timeouts, they work fine on the RTL8168H
    post-up ethtool -K enp1s0 sg on tso on gso on || true
    # one interrupt per transmitted packet by default : coalesce them (but do NOT coalesce the receive side, it makes the receive drops worse)
    post-up ethtool -C enp1s0 tx-usecs 120 tx-frames 16 || true

If dmesg ever shows NETDEV WATCHDOG for the interface, remove the first line.

Network flow

For the following examples, we will consider that the user enters http://myapp.example.com in the browser for the first time (no DNS record found in cache).

Without VPN

Here is what happen when you try to reach a service which is open to the internet, without using any VPN, from your local network holding your homelab (on the left), or from any other location (on the right) :

From local network From outside local network

From local network	From outside local network
flowchart TB style HOSTING_PROVIDER fill: #bbfd795e style DDNS_PROVIDER fill: #e2b6ff6e style INTERNET_SERVICE_PROVIDER fill: #45d8ff57 style SINGLE_BOARD_COMPUTER fill: #febfb45e style CONTAINER_ENGINE fill: #6701022d style TRAEFIK_CONTAINER fill: #66010136 style PIHOLE_CONTAINER fill: #66010136 style UNBOUND_CONTAINER fill: #66010136 style MYAPP_CONTAINER fill: #66010136 style TRAEFIK_ROUTER fill: #defc1e37 DOMAIN(example.com) SUBDOMAIN_MYAPP(myapp.example.com) DDNS(myddns.ddns.net) ROUTER_PUBLIC_IP[public IP] ROUTER_PORT80{{80/tcp}} ROUTER_PORT443{{443/tcp}} ROUTER_DNS[DNS] DOCKER_PIHOLE_PORT53{{53/udp}} DOCKER_TRAEFIK_PORT443{{443/tcp}} DOCKER_TRAEFIK_PORT80{{80/tcp}} DOCKER_MYAPP_PORT{{port/tcp}} DOCKER_UNBOUND_PORT53{{53/udp}} TRAEFIK_ROUTER_MYAPP(myapp.example.com) TRAEFIK_MIDDLEWARE_REDIRECT(HTTPS redirect\non the web entrypoint) ROOT_DNS_SERVERS[Root DNS servers] subgraph HOSTING_PROVIDER[DOMAIN NAME REGISTRAR] DOMAIN SUBDOMAIN_MYAPP end subgraph DDNS_PROVIDER[DYNAMIC DNS PROVIDER] DDNS end subgraph INTERNET_SERVICE_PROVIDER[INTERNET SERVICE PROVIDER] ROUTER_PUBLIC_IP ROUTER_PORT80 ROUTER_PORT443 ROUTER_DNS end subgraph SINGLE_BOARD_COMPUTER[BANANA PI M5] subgraph CONTAINER_ENGINE[DOCKER] subgraph MYAPP_CONTAINER[MYAPP CONTAINER] DOCKER_MYAPP_PORT end subgraph UNBOUND_CONTAINER[UNBOUND CONTAINER] DOCKER_UNBOUND_PORT53 end subgraph PIHOLE_CONTAINER[PIHOLE CONTAINER] DOCKER_PIHOLE_PORT53 end subgraph TRAEFIK_CONTAINER[TRAEFIK CONTAINER] DOCKER_TRAEFIK_PORT443 DOCKER_TRAEFIK_PORT80 TRAEFIK_MIDDLEWARE_REDIRECT subgraph TRAEFIK_ROUTER[TRAEFIK HTTP ROUTER] TRAEFIK_ROUTER_MYAPP end end end end CLIENT((client)) --->\|" http‎://myapp.example.com "\| BROWSER BROWSER((browser)) -->\|HTTP\| ROUTER_PUBLIC_IP DOMAIN <-->\|subdomain\| SUBDOMAIN_MYAPP SUBDOMAIN_MYAPP <-->\|CNAME\| DDNS DDNS <-->\|DynDNS\| ROUTER_PUBLIC_IP ROUTER_PUBLIC_IP --> ROUTER_PORT80 ROUTER_PUBLIC_IP --> ROUTER_PORT443 ROUTER_PORT443 -->\|port forward\| DOCKER_TRAEFIK_PORT443 ROUTER_PORT80 -->\|port forward\| DOCKER_TRAEFIK_PORT80 DOCKER_TRAEFIK_PORT443 --> TRAEFIK_ROUTER DOCKER_TRAEFIK_PORT80 --> TRAEFIK_MIDDLEWARE_REDIRECT TRAEFIK_MIDDLEWARE_REDIRECT -.->\|301 to https\| DOCKER_TRAEFIK_PORT443 TRAEFIK_ROUTER_MYAPP --> DOCKER_MYAPP_PORT BROWSER((browser)) <--> LOCAL_DNS_RESOLVER[/local resolver\] LOCAL_DNS_RESOLVER <--->\|router local IP address\| ROUTER_DNS ROUTER_DNS <-->\|Banana Pi M5 static IP\| DOCKER_PIHOLE_PORT53 DOCKER_PIHOLE_PORT53 <-->\|DNS\| DOCKER_UNBOUND_PORT53 UNBOUND_CONTAINER <-----> ROOT_DNS_SERVERS linkStyle 0 stroke-width: 4px, stroke: red linkStyle 1 stroke-width: 4px, stroke: red linkStyle 2 stroke-width: 4px, stroke: #d4a017, stroke-dasharray: 5 linkStyle 3 stroke-width: 4px, stroke: #d4a017, stroke-dasharray: 5 linkStyle 4 stroke-width: 4px, stroke: #d4a017, stroke-dasharray: 5 linkStyle 5 stroke-width: 4px, stroke: red linkStyle 8 stroke-width: 4px, stroke: red linkStyle 9 stroke-width: 4px, stroke: red linkStyle 10 stroke-width: 4px, stroke: red linkStyle 11 stroke-width: 4px, stroke: red linkStyle 12 stroke-width: 4px, stroke: red linkStyle 13 stroke-width: 4px, stroke: #d4a017, stroke-dasharray: 5 linkStyle 14 stroke-width: 4px, stroke: #d4a017, stroke-dasharray: 5 linkStyle 15 stroke-width: 4px, stroke: #d4a017, stroke-dasharray: 5 linkStyle 16 stroke-width: 4px, stroke: #d4a017, stroke-dasharray: 5 linkStyle 17 stroke-width: 4px, stroke: #d4a017, stroke-dasharray: 5	flowchart TB style HOSTING_PROVIDER fill: #bbfd795e style DDNS_PROVIDER fill: #e2b6ff6e style INTERNET_SERVICE_PROVIDER fill: #45d8ff57 style INTERNET_SERVICE_PROVIDER2 fill: #45d8ff57 style SERVER_DEVICE fill: #febfb45e style CONTAINER_ENGINE fill: #6701022d style TRAEFIK_CONTAINER fill: #66010136 style MYAPP_CONTAINER fill: #66010136 style TRAEFIK_ROUTER fill: #defc1e37 style DNS_RESOLVER fill: #fd95b07a DOMAIN(example.com) SUBDOMAIN_MYAPP(myapp.example.com) DDNS(myddns.ddns.net) ROUTER_PUBLIC_IP[public IP] ROUTER_PORT80{{80/tcp}} ROUTER_PORT443{{443/tcp}} ROUTER2_DNS[DNS] DOCKER_TRAEFIK_PORT443{{443/tcp}} DOCKER_TRAEFIK_PORT80{{80/tcp}} DOCKER_MYAPP_PORT{{port/tcp}} TRAEFIK_ROUTER_MYAPP(myapp.example.com) TRAEFIK_MIDDLEWARE_REDIRECT(HTTPS redirect\non the web entrypoint) ROOT_DNS_SERVERS[Root DNS servers] CLOUDFLARE(Cloudflare, etc.) subgraph HOSTING_PROVIDER[DOMAIN NAME REGISTRAR] DOMAIN SUBDOMAIN_MYAPP end subgraph DDNS_PROVIDER[DYNAMIC DNS PROVIDER] DDNS end subgraph INTERNET_SERVICE_PROVIDER[ISP ROUTER] ROUTER_PUBLIC_IP ROUTER_PORT80 ROUTER_PORT443 end subgraph INTERNET_SERVICE_PROVIDER2[CLIENT ISP ROUTER] ROUTER2_DNS end subgraph DNS_RESOLVER[DNS RESOLVER] CLOUDFLARE end subgraph SERVER_DEVICE[MINI PC] subgraph CONTAINER_ENGINE[DOCKER] subgraph MYAPP_CONTAINER[MYAPP CONTAINER] DOCKER_MYAPP_PORT end subgraph TRAEFIK_CONTAINER[TRAEFIK CONTAINER] DOCKER_TRAEFIK_PORT443 DOCKER_TRAEFIK_PORT80 TRAEFIK_MIDDLEWARE_REDIRECT subgraph TRAEFIK_ROUTER[TRAEFIK HTTP ROUTER] TRAEFIK_ROUTER_MYAPP end end end end CLIENT((client)) ---->\|" http://myapp.example.com "\| BROWSER BROWSER((browser)) ---> ROUTER_PUBLIC_IP DOMAIN <-->\|subdomain\| SUBDOMAIN_MYAPP SUBDOMAIN_MYAPP <-->\|CNAME\| DDNS DDNS <--->\|DynDNS\| ROUTER_PUBLIC_IP ROUTER_PUBLIC_IP --> ROUTER_PORT80 ROUTER_PUBLIC_IP --> ROUTER_PORT443 ROUTER_PORT443 -->\|port forward\| DOCKER_TRAEFIK_PORT443 ROUTER_PORT80 -->\|port forward\| DOCKER_TRAEFIK_PORT80 DOCKER_TRAEFIK_PORT443 --> TRAEFIK_ROUTER DOCKER_TRAEFIK_PORT80 --> TRAEFIK_MIDDLEWARE_REDIRECT TRAEFIK_MIDDLEWARE_REDIRECT -.->\|301 to https\| DOCKER_TRAEFIK_PORT443 TRAEFIK_ROUTER_MYAPP --> DOCKER_MYAPP_PORT BROWSER((browser)) <--> LOCAL_DNS_RESOLVER[/local resolver\] LOCAL_DNS_RESOLVER <--->\|router local IP address\| ROUTER2_DNS ROUTER2_DNS <--> CLOUDFLARE CLOUDFLARE <---> ROOT_DNS_SERVERS linkStyle 0 stroke-width: 4px, stroke: red linkStyle 1 stroke-width: 4px, stroke: red linkStyle 2 stroke-width: 4px, stroke: #d4a017, stroke-dasharray: 5 linkStyle 3 stroke-width: 4px, stroke: #d4a017, stroke-dasharray: 5 linkStyle 4 stroke-width: 4px, stroke: #d4a017, stroke-dasharray: 5 linkStyle 5 stroke-width: 4px, stroke: red linkStyle 8 stroke-width: 4px, stroke: red linkStyle 9 stroke-width: 4px, stroke: red linkStyle 10 stroke-width: 4px, stroke: red linkStyle 11 stroke-width: 4px, stroke: red linkStyle 12 stroke-width: 4px, stroke: red linkStyle 13 stroke-width: 4px, stroke: #d4a017, stroke-dasharray: 5 linkStyle 14 stroke-width: 4px, stroke: #d4a017, stroke-dasharray: 5 linkStyle 15 stroke-width: 4px, stroke: #d4a017, stroke-dasharray: 5 linkStyle 16 stroke-width: 4px, stroke: #d4a017, stroke-dasharray: 5

flowchart TB
    style HOSTING_PROVIDER fill: #bbfd795e
    style DDNS_PROVIDER fill: #e2b6ff6e
    style INTERNET_SERVICE_PROVIDER fill: #45d8ff57
    style SINGLE_BOARD_COMPUTER fill: #febfb45e
    style CONTAINER_ENGINE fill: #6701022d
    style TRAEFIK_CONTAINER fill: #66010136
    style PIHOLE_CONTAINER fill: #66010136
    style UNBOUND_CONTAINER fill: #66010136
    style MYAPP_CONTAINER fill: #66010136
    style TRAEFIK_ROUTER fill: #defc1e37
    DOMAIN(example.com)
    SUBDOMAIN_MYAPP(myapp.example.com)
    DDNS(myddns.ddns.net)
    ROUTER_PUBLIC_IP[public IP]
    ROUTER_PORT80{{80/tcp}}
    ROUTER_PORT443{{443/tcp}}
    ROUTER_DNS[DNS]
    DOCKER_PIHOLE_PORT53{{53/udp}}
    DOCKER_TRAEFIK_PORT443{{443/tcp}}
    DOCKER_TRAEFIK_PORT80{{80/tcp}}
    DOCKER_MYAPP_PORT{{port/tcp}}
    DOCKER_UNBOUND_PORT53{{53/udp}}
    TRAEFIK_ROUTER_MYAPP(myapp.example.com)
    TRAEFIK_MIDDLEWARE_REDIRECT(HTTPS redirect\non the web entrypoint)
    ROOT_DNS_SERVERS[Root DNS servers]

    subgraph HOSTING_PROVIDER[DOMAIN NAME REGISTRAR]
        DOMAIN
        SUBDOMAIN_MYAPP
    end

    subgraph DDNS_PROVIDER[DYNAMIC DNS PROVIDER]
        DDNS
    end

    subgraph INTERNET_SERVICE_PROVIDER[INTERNET SERVICE PROVIDER]
        ROUTER_PUBLIC_IP
        ROUTER_PORT80
        ROUTER_PORT443
        ROUTER_DNS
    end

    subgraph SINGLE_BOARD_COMPUTER[BANANA PI M5]
        subgraph CONTAINER_ENGINE[DOCKER]
            subgraph MYAPP_CONTAINER[MYAPP CONTAINER]
                DOCKER_MYAPP_PORT
            end

            subgraph UNBOUND_CONTAINER[UNBOUND CONTAINER]
                DOCKER_UNBOUND_PORT53
            end

            subgraph PIHOLE_CONTAINER[PIHOLE CONTAINER]
                DOCKER_PIHOLE_PORT53
            end

            subgraph TRAEFIK_CONTAINER[TRAEFIK CONTAINER]
                DOCKER_TRAEFIK_PORT443
                DOCKER_TRAEFIK_PORT80
                TRAEFIK_MIDDLEWARE_REDIRECT

                subgraph TRAEFIK_ROUTER[TRAEFIK HTTP ROUTER]
                    TRAEFIK_ROUTER_MYAPP
                end
            end

        end

    end

    CLIENT((client)) --->|" http‎://myapp.example.com "| BROWSER
    BROWSER((browser)) -->|HTTP| ROUTER_PUBLIC_IP
    DOMAIN <-->|subdomain| SUBDOMAIN_MYAPP
    SUBDOMAIN_MYAPP <-->|CNAME| DDNS
    DDNS <-->|DynDNS| ROUTER_PUBLIC_IP
    ROUTER_PUBLIC_IP --> ROUTER_PORT80
    ROUTER_PUBLIC_IP --> ROUTER_PORT443
    ROUTER_PORT443 -->|port forward| DOCKER_TRAEFIK_PORT443
    ROUTER_PORT80 -->|port forward| DOCKER_TRAEFIK_PORT80
    DOCKER_TRAEFIK_PORT443 --> TRAEFIK_ROUTER
    DOCKER_TRAEFIK_PORT80 --> TRAEFIK_MIDDLEWARE_REDIRECT
    TRAEFIK_MIDDLEWARE_REDIRECT -.->|301 to https| DOCKER_TRAEFIK_PORT443
    TRAEFIK_ROUTER_MYAPP --> DOCKER_MYAPP_PORT
    BROWSER((browser)) <--> LOCAL_DNS_RESOLVER[/local resolver\]
    LOCAL_DNS_RESOLVER <--->|router local IP address| ROUTER_DNS
    ROUTER_DNS <-->|Banana Pi M5 static IP| DOCKER_PIHOLE_PORT53
    DOCKER_PIHOLE_PORT53 <-->|DNS| DOCKER_UNBOUND_PORT53
    UNBOUND_CONTAINER <-----> ROOT_DNS_SERVERS
    linkStyle 0 stroke-width: 4px, stroke: red
    linkStyle 1 stroke-width: 4px, stroke: red
    linkStyle 2 stroke-width: 4px, stroke: #d4a017, stroke-dasharray: 5
    linkStyle 3 stroke-width: 4px, stroke: #d4a017, stroke-dasharray: 5
    linkStyle 4 stroke-width: 4px, stroke: #d4a017, stroke-dasharray: 5
    linkStyle 5 stroke-width: 4px, stroke: red
    linkStyle 8 stroke-width: 4px, stroke: red
    linkStyle 9 stroke-width: 4px, stroke: red
    linkStyle 10 stroke-width: 4px, stroke: red
    linkStyle 11 stroke-width: 4px, stroke: red
    linkStyle 12 stroke-width: 4px, stroke: red
    linkStyle 13 stroke-width: 4px, stroke: #d4a017, stroke-dasharray: 5
    linkStyle 14 stroke-width: 4px, stroke: #d4a017, stroke-dasharray: 5
    linkStyle 15 stroke-width: 4px, stroke: #d4a017, stroke-dasharray: 5
    linkStyle 16 stroke-width: 4px, stroke: #d4a017, stroke-dasharray: 5
    linkStyle 17 stroke-width: 4px, stroke: #d4a017, stroke-dasharray: 5

flowchart TB
    style HOSTING_PROVIDER fill: #bbfd795e
    style DDNS_PROVIDER fill: #e2b6ff6e
    style INTERNET_SERVICE_PROVIDER fill: #45d8ff57
    style INTERNET_SERVICE_PROVIDER2 fill: #45d8ff57
    style SERVER_DEVICE fill: #febfb45e
    style CONTAINER_ENGINE fill: #6701022d
    style TRAEFIK_CONTAINER fill: #66010136
    style MYAPP_CONTAINER fill: #66010136
    style TRAEFIK_ROUTER fill: #defc1e37
    style DNS_RESOLVER fill: #fd95b07a
    DOMAIN(example.com)
    SUBDOMAIN_MYAPP(myapp.example.com)
    DDNS(myddns.ddns.net)
    ROUTER_PUBLIC_IP[public IP]
    ROUTER_PORT80{{80/tcp}}
    ROUTER_PORT443{{443/tcp}}
    ROUTER2_DNS[DNS]
    DOCKER_TRAEFIK_PORT443{{443/tcp}}
    DOCKER_TRAEFIK_PORT80{{80/tcp}}
    DOCKER_MYAPP_PORT{{port/tcp}}
    TRAEFIK_ROUTER_MYAPP(myapp.example.com)
    TRAEFIK_MIDDLEWARE_REDIRECT(HTTPS redirect\non the web entrypoint)
    ROOT_DNS_SERVERS[Root DNS servers]
    CLOUDFLARE(Cloudflare, etc.)

    subgraph HOSTING_PROVIDER[DOMAIN NAME REGISTRAR]
        DOMAIN
        SUBDOMAIN_MYAPP
    end

    subgraph DDNS_PROVIDER[DYNAMIC DNS PROVIDER]
        DDNS
    end

    subgraph INTERNET_SERVICE_PROVIDER[ISP ROUTER]
        ROUTER_PUBLIC_IP
        ROUTER_PORT80
        ROUTER_PORT443
    end

    subgraph INTERNET_SERVICE_PROVIDER2[CLIENT ISP ROUTER]
        ROUTER2_DNS
    end

    subgraph DNS_RESOLVER[DNS RESOLVER]
        CLOUDFLARE
    end

    subgraph SERVER_DEVICE[MINI PC]
        subgraph CONTAINER_ENGINE[DOCKER]
            subgraph MYAPP_CONTAINER[MYAPP CONTAINER]
                DOCKER_MYAPP_PORT
            end

            subgraph TRAEFIK_CONTAINER[TRAEFIK CONTAINER]
                DOCKER_TRAEFIK_PORT443
                DOCKER_TRAEFIK_PORT80
                TRAEFIK_MIDDLEWARE_REDIRECT

                subgraph TRAEFIK_ROUTER[TRAEFIK HTTP ROUTER]
                    TRAEFIK_ROUTER_MYAPP
                end
            end

        end

    end

    CLIENT((client)) ---->|" http://myapp.example.com "| BROWSER
    BROWSER((browser)) ---> ROUTER_PUBLIC_IP
    DOMAIN <-->|subdomain| SUBDOMAIN_MYAPP
    SUBDOMAIN_MYAPP <-->|CNAME| DDNS
    DDNS <--->|DynDNS| ROUTER_PUBLIC_IP
    ROUTER_PUBLIC_IP --> ROUTER_PORT80
    ROUTER_PUBLIC_IP --> ROUTER_PORT443
    ROUTER_PORT443 -->|port forward| DOCKER_TRAEFIK_PORT443
    ROUTER_PORT80 -->|port forward| DOCKER_TRAEFIK_PORT80
    DOCKER_TRAEFIK_PORT443 --> TRAEFIK_ROUTER
    DOCKER_TRAEFIK_PORT80 --> TRAEFIK_MIDDLEWARE_REDIRECT
    TRAEFIK_MIDDLEWARE_REDIRECT -.->|301 to https| DOCKER_TRAEFIK_PORT443
    TRAEFIK_ROUTER_MYAPP --> DOCKER_MYAPP_PORT
    BROWSER((browser)) <--> LOCAL_DNS_RESOLVER[/local resolver\]
    LOCAL_DNS_RESOLVER <--->|router local IP address| ROUTER2_DNS
    ROUTER2_DNS <--> CLOUDFLARE
    CLOUDFLARE <---> ROOT_DNS_SERVERS
    linkStyle 0 stroke-width: 4px, stroke: red
    linkStyle 1 stroke-width: 4px, stroke: red
    linkStyle 2 stroke-width: 4px, stroke: #d4a017, stroke-dasharray: 5
    linkStyle 3 stroke-width: 4px, stroke: #d4a017, stroke-dasharray: 5
    linkStyle 4 stroke-width: 4px, stroke: #d4a017, stroke-dasharray: 5
    linkStyle 5 stroke-width: 4px, stroke: red
    linkStyle 8 stroke-width: 4px, stroke: red
    linkStyle 9 stroke-width: 4px, stroke: red
    linkStyle 10 stroke-width: 4px, stroke: red
    linkStyle 11 stroke-width: 4px, stroke: red
    linkStyle 12 stroke-width: 4px, stroke: red
    linkStyle 13 stroke-width: 4px, stroke: #d4a017, stroke-dasharray: 5
    linkStyle 14 stroke-width: 4px, stroke: #d4a017, stroke-dasharray: 5
    linkStyle 15 stroke-width: 4px, stroke: #d4a017, stroke-dasharray: 5
    linkStyle 16 stroke-width: 4px, stroke: #d4a017, stroke-dasharray: 5

From the local network (left), Pi-Hole answers with its local DNS record (yellow dotted line), so the browser gets the mini PC's internal IP and reaches the reverse proxy directly on the LAN, without going through the public IP (no port forwarding, no NAT loopback). From any other location (right), the name is resolved publicly through the client's DNS resolver and the request reaches the mini PC on port 80 (HTTP) after being port forwarded by the ISP router. In both cases the web entrypoint of the reverse proxy redirects the request to port 443 (HTTPS), where the router of the application finally routes it to the target application (red line).

With VPN

If you try to reach the service through the WireGuard VPN, the flow will look like the following :

flowchart TB
    style HOSTING_PROVIDER fill: #bbfd795e
    style DDNS_PROVIDER fill: #e2b6ff6e
    style INTERNET_SERVICE_PROVIDER fill: #45d8ff57
    style SERVER_DEVICE fill: #febfb45e
    style CONTAINER_ENGINE fill: #6701022d
    style TRAEFIK_CONTAINER fill: #66010136
    style PIHOLE_CONTAINER fill: #66010136
    style UNBOUND_CONTAINER fill: #66010136
    style WIREGUARD_HOST fill: #66010159
    style MYAPP_CONTAINER fill: #66010136
    style PIHOLE_DNS_RECORDS fill: #defc1e37
    style TRAEFIK_ROUTER fill: #defc1e37
    style TRAEFIK_MIDDLEWARE fill: #defc1e37
    style VPN_CLIENT fill: #18fdb144
    DOMAIN(example.com)
    SUBDOMAIN_MYAPP(myapp.example.com)
    SUBDOMAIN_WIREGUARD(wireguard.example.com)
    DDNS(myddns.ddns.net)
    ROUTER_PUBLIC_IP[public IP]
    ROUTER_PORT51820{{51820/udp}}
    WIREGUARD_PORT{{51820/udp}}
    ROUTER_DNS[DNS 1]
    DOCKER_PIHOLE_PORT53{{53/udp}}
    DOCKER_TRAEFIK_PORT443{{443/tcp}}
    DOCKER_TRAEFIK_PORT80{{80/tcp}}
    DOCKER_MYAPP_PORT{{port/tcp}}
    DOCKER_UNBOUND_PORT53{{53/udp}}
    TRAEFIK_ROUTER_MYAPP(myapp.example.com)
    TRAEFIK_MIDDLEWARE_REDIRECT(HTTPS redirect\non the web entrypoint)
    TRAEFIK_MIDDLEWARE_WHITELIST(IP whitelist)
    ROOT_DNS_SERVERS[Root DNS servers]
    PIHOLE_DNS_MYAPP(myapp.example.com)

    subgraph VPN_CLIENT[VPN CLIENT]
        WIREGUARD_CLIENT_ENDPOINT[Endpoint]
        WIREGUARD_CLIENT_DNS[DNS]
    end

    subgraph HOSTING_PROVIDER[DOMAIN NAME REGISTRAR]
        DOMAIN
        SUBDOMAIN_MYAPP
        SUBDOMAIN_WIREGUARD
    end

    subgraph DDNS_PROVIDER[DYNAMIC DNS PROVIDER]
        DDNS
    end

    subgraph INTERNET_SERVICE_PROVIDER[INTERNET SERVICE PROVIDER]
        ROUTER_PUBLIC_IP
        ROUTER_PORT51820
        ROUTER_DNS
    end

    subgraph SERVER_DEVICE[MINI PC]
        subgraph CONTAINER_ENGINE[DOCKER]
            subgraph MYAPP_CONTAINER[MYAPP CONTAINER]
                DOCKER_MYAPP_PORT
            end

            subgraph UNBOUND_CONTAINER[UNBOUND CONTAINER]
                DOCKER_UNBOUND_PORT53
            end

            subgraph PIHOLE_CONTAINER[PIHOLE CONTAINER]
                DOCKER_PIHOLE_PORT53
                subgraph PIHOLE_DNS_RECORDS[LOCAL DNS RECORDS]
                    PIHOLE_DNS_MYAPP
                end
            end

            subgraph TRAEFIK_CONTAINER[TRAEFIK CONTAINER]
                DOCKER_TRAEFIK_PORT443
                DOCKER_TRAEFIK_PORT80
                TRAEFIK_MIDDLEWARE_REDIRECT

                subgraph TRAEFIK_ROUTER[TRAEFIK HTTP ROUTER]
                    TRAEFIK_ROUTER_MYAPP
                end

                subgraph TRAEFIK_MIDDLEWARE[TRAEFIK MIDDLEWARE]
                    TRAEFIK_MIDDLEWARE_WHITELIST
                end
            end

        end

        subgraph WIREGUARD_HOST[WIREGUARD - on the host]
            WIREGUARD_PORT
        end

    end

    CLIENT((client)) --> VPN_CLIENT
    WIREGUARD_CLIENT_ENDPOINT --> SUBDOMAIN_WIREGUARD
    WIREGUARD_CLIENT_DNS -->|Server tunnel address| DOCKER_PIHOLE_PORT53
    VPN_CLIENT -->|" http://myapp.example.com "| BROWSER
    BROWSER((browser)) --> ROUTER_PUBLIC_IP
    DOMAIN -->|subdomain| SUBDOMAIN_MYAPP
    DOMAIN -->|subdomain| SUBDOMAIN_WIREGUARD
    SUBDOMAIN_MYAPP -->|CNAME| DDNS
    SUBDOMAIN_WIREGUARD -->|CNAME| DDNS
    DDNS -->|DynDNS| ROUTER_PUBLIC_IP
    ROUTER_PUBLIC_IP --> ROUTER_PORT51820
    ROUTER_PORT51820 ----->|port forward| WIREGUARD_PORT
    PIHOLE_DNS_MYAPP -->|mini PC internal IP| DOCKER_TRAEFIK_PORT80
    DOCKER_TRAEFIK_PORT443 --> TRAEFIK_ROUTER
    DOCKER_TRAEFIK_PORT80 --> TRAEFIK_MIDDLEWARE_REDIRECT
    TRAEFIK_MIDDLEWARE_REDIRECT -.->|301 to https| DOCKER_TRAEFIK_PORT443
    TRAEFIK_ROUTER_MYAPP --> TRAEFIK_MIDDLEWARE_WHITELIST
    TRAEFIK_MIDDLEWARE_WHITELIST --> DOCKER_MYAPP_PORT
    ROUTER_DNS <---->|mini PC static IP| DOCKER_PIHOLE_PORT53
    DOCKER_PIHOLE_PORT53 <-->|DNS| DOCKER_UNBOUND_PORT53
    UNBOUND_CONTAINER <------> ROOT_DNS_SERVERS
    linkStyle 0 stroke-width: 4px, stroke: red
    linkStyle 1 stroke-width: 4px, stroke: #d4a017, stroke-dasharray: 5
    linkStyle 2 stroke-width: 4px, stroke: #d4a017, stroke-dasharray: 5
    linkStyle 3 stroke-width: 4px, stroke: red
    linkStyle 4 stroke-width: 4px, stroke: red
    linkStyle 5 stroke-width: 4px, stroke: #d4a017, stroke-dasharray: 5
    linkStyle 6 stroke-width: 4px, stroke: #d4a017, stroke-dasharray: 5
    linkStyle 7 stroke-width: 4px, stroke: #d4a017, stroke-dasharray: 5
    linkStyle 8 stroke-width: 4px, stroke: #d4a017, stroke-dasharray: 5
    linkStyle 9 stroke-width: 4px, stroke: #d4a017, stroke-dasharray: 5
    linkStyle 10 stroke-width: 4px, stroke: red
    linkStyle 11 stroke-width: 4px, stroke: red
    linkStyle 12 stroke-width: 4px, stroke: red
    linkStyle 13 stroke-width: 4px, stroke: red
    linkStyle 14 stroke-width: 4px, stroke: red
    linkStyle 15 stroke-width: 4px, stroke: red
    linkStyle 16 stroke-width: 4px, stroke: red
    linkStyle 17 stroke-width: 4px, stroke: red
    linkStyle 19 stroke-width: 4px, stroke: #d4a017, stroke-dasharray: 5
    linkStyle 20 stroke-width: 4px, stroke: #d4a017, stroke-dasharray: 5

Here, first the client needs to connect to the VPN server through his preferred VPN client. The request for VPN connection reaches the mini PC on UDP port 51820 (VPN port) after being routed by the CNAME record to our dynamic DNS and then port forwarded by our router.

The DNS resolving always go through Pi-Hole and Unbound (yellow dotted line), to resolve the VPN server and the application.

The request for the application is handled by Pi-Hole DNS local record which route it to the mini PC IP address, to be handled by the reverse proxy, and is then redirected to port 443 (HTTPS) by its web entrypoint, where the router of the application finally routes it to the target application (red line).

If in any way the request arrives to Traefik with an unauthorized IP address, it will be rejected thanks to the IP whitelist middleware.

Install services

PocketID

We will use PocketID to add a single sign-on in front of the services that don't have a proper authentication of their own (Pi-Hole, the Traefik dashboard), and as identity provider for the services that support OpenID Connect natively (Arcane, Grafana, ...).

PocketID is a small self-hosted OpenID Connect (OIDC) provider with a twist : users don't have passwords, they authenticate with passkeys only (a hardware key, or the passkey manager of the phone, the browser or a password manager). Nothing to remember, nothing to phish, and one login for every service.

There are two ways to plug a service on it :

services that speak OIDC natively (Arcane, Grafana, ...) get their own OIDC client in PocketID and show a "login with PocketID" button
services that don't (Pi-Hole, the Traefik dashboard) are put behind the traefik-oidc-auth➚ Traefik plugin : a middleware that redirects the browser to PocketID, checks the token it comes back with and keeps a session cookie, so that the service behind never sees an unauthenticated request

Here is an overview of the network flow when a service is protected by the middleware :

flowchart LR
    style INCOMING_REQUEST fill: #45d8ff57
    style TRAEFIK_CONTAINER fill: #6601013c
    style APP_CONTAINER fill: #6601013c
    style POCKETID_CONTAINER fill: #6601013c
    style TRAEFIK_ROUTER fill: #defc1e37
    style TRAEFIK_MIDDLEWARE fill: #defc1e37
    style SERVER_DEVICE fill: #fec9bf5e
    style CONTAINER_ENGINE fill: #67010030
    DOCKER_TRAEFIK_PORT443{{443/tcp}}
    DOCKER_TRAEFIK_PORT80{{80/tcp}}
    DOCKER_APP_PORT{{80/tcp}}
    DOCKER_POCKETID_PORT{{1411/tcp}}
    TRAEFIK_ROUTER_APP(pihole)
    TRAEFIK_ROUTER_POCKETID(pocketid)
    TRAEFIK_MIDDLEWARE_REDIRECT(HTTPS redirect\non the web entrypoint)
    TRAEFIK_MIDDLEWARE_IP_WHITELIST(IP whitelist)
    TRAEFIK_MIDDLEWARE_OIDC(OIDC auth\npihole-auth)
    INCOMING_REQUEST((INCOMING\nREQUEST))

    subgraph SERVER_DEVICE[MINI PC]
        subgraph CONTAINER_ENGINE[DOCKER]
            subgraph TRAEFIK_CONTAINER[TRAEFIK CONTAINER]
                TRAEFIK_MIDDLEWARE_REDIRECT
                DOCKER_TRAEFIK_PORT443
                DOCKER_TRAEFIK_PORT80

                subgraph TRAEFIK_MIDDLEWARE[TRAEFIK MIDDLEWARES]
                    TRAEFIK_MIDDLEWARE_IP_WHITELIST
                    TRAEFIK_MIDDLEWARE_OIDC
                end

                subgraph TRAEFIK_ROUTER[TRAEFIK HTTP ROUTERS]
                    TRAEFIK_ROUTER_APP
                    TRAEFIK_ROUTER_POCKETID
                end
            end

            subgraph APP_CONTAINER[PI-HOLE CONTAINER]
                DOCKER_APP_PORT
            end

            subgraph POCKETID_CONTAINER[POCKETID CONTAINER]
                DOCKER_POCKETID_PORT
            end
        end
    end

    INCOMING_REQUEST --> DOCKER_TRAEFIK_PORT443
    INCOMING_REQUEST --> DOCKER_TRAEFIK_PORT80
    DOCKER_TRAEFIK_PORT443 --> TRAEFIK_ROUTER_APP
    DOCKER_TRAEFIK_PORT80 --> TRAEFIK_MIDDLEWARE_REDIRECT
    TRAEFIK_MIDDLEWARE_REDIRECT -.->|301 to https, before any router| INCOMING_REQUEST
    TRAEFIK_ROUTER_APP --> TRAEFIK_MIDDLEWARE_IP_WHITELIST
    TRAEFIK_MIDDLEWARE_IP_WHITELIST --> TRAEFIK_MIDDLEWARE_OIDC
    TRAEFIK_MIDDLEWARE_OIDC -->|authenticated| DOCKER_APP_PORT
    TRAEFIK_MIDDLEWARE_OIDC -.->|not authenticated : browser redirected to the login page| TRAEFIK_ROUTER_POCKETID
    TRAEFIK_MIDDLEWARE_OIDC -.->|token validation through the Docker network| DOCKER_POCKETID_PORT
    TRAEFIK_ROUTER_POCKETID --> DOCKER_POCKETID_PORT

Setting up

Create the folders and the encryption key (PocketID encrypts its secrets at rest with it : keep that file with your backups, without it the database is unusable). The container runs as user 1000:1001 (see the .env file), so give it the ownership of the data folder and of the key :

sudo mkdir -p /opt/apps/pocketid/data
openssl rand -base64 32 | sudo tee /opt/apps/pocketid/encryption_key > /dev/null
sudo chown -R 1000:1001 /opt/apps/pocketid/data /opt/apps/pocketid/encryption_key
sudo chmod 600 /opt/apps/pocketid/encryption_key

Then :

copy the .env and docker-compose.yml files from this project's pocketid directory into the /opt/apps/pocketid directory, and adapt the .env file to your domain
copy the pocketid.yml file from this project's traefik/dynamic directory into the /opt/apps/traefik/dynamic directory
declare the traefik-oidc-auth plugin in the traefik.yml static configuration (see below) and restart Traefik, plugins are downloaded when it starts

Run the Compose file (see Run), then open https://pocketid.example.com : on first start the setup page (/setup) creates the administrator account and registers its first passkey.

Now create one OIDC client per service to protect (OIDC Clients -> Add) :

for a service put behind the Traefik middleware, the callback URL is the service URL followed by /oidc/callback (the default CallbackUri of the plugin), for example https://pihole.example.com/oidc/callback, and PKCE enabled. Copy the generated client ID and secret into the ClientId / ClientSecret fields of the corresponding middleware in pocketid.yml, and give the middleware a random 32 characters Secret (openssl rand -base64 48 | tr -dc 'A-Za-z0-9' | head -c 32; echo) : this one is not a PocketID secret, it is the key the plugin uses to encrypt its own session cookie. The plugin expects exactly 32 characters, and each middleware must have its own. Traefik picks up the change without restart
for a service with native OIDC support, use the callback URL it documents, and disable PKCE on the PocketID side if the application does not send a code_challenge (a confidential client is protected by its secret anyway). Some applications ask for each endpoint separately instead of an issuer URL : the authorization URL is always the public one (https://pocketid.example.com/authorize, the browser follows it), while the token and user info URLs are called by the container itself. They can be the internal ones (http://pocketid:1411/api/oidc/token and http://pocketid:1411/api/oidc/userinfo), or the public ones thanks to the alias and the pocketid-whitelist middleware described below (this is what Grafana does)
most OIDC libraries, however, verify that the issuer announced by the provider matches the URL they queried (Homebox and its go-oidc for instance), so they cannot use the internal URL at all : querying http://pocketid:1411 returns https://pocketid.example.com as issuer and they refuse. Those applications must use the public issuer URL, which means their container has to reach it. Two small additions make that work, and they serve every future application :
- Traefik carries a network alias with the provider's public name on the private network (see Service definition), so that the containers resolve it to Traefik itself, without any hard coded IP address and without depending on Pi-Hole for the container DNS
- the PocketID router uses the pocketid-whitelist middleware instead of vpn-whitelist : same ranges plus the private Docker network, so that a container is allowed to fetch the discovery document and to exchange the token. The public Docker network is deliberately left out, an application exposed to the internet must not reach the provider this way
```
flowchart LR
    APP[application container] -->|1 . resolves<br>pocketid.example.com| DNS[[Docker DNS : alias on Traefik]]
    APP -->|2 . HTTPS,<br>source 172.21.x.x| TRAEFIK[Traefik]
    TRAEFIK -->|3 . pocketid-whitelist<br>accepts the private network| POCKETID[PocketID]
```

Finally, to protect a service with the middleware, add it to the middlewares list of its router, after the IP whitelist, as done for Pi-Hole :

      middlewares:
        - vpn-whitelist@file
        - pihole-auth@file

[!NOTE] The pocketid router is itself behind the vpn-whitelist middleware because every service I protect with it is only reachable from the local network or the VPN. If one day a public service is put behind the middleware, the login page must be reachable from the internet too : remove the whitelist from the pocketid router only, the login page is designed to be public (passkeys cannot be brute-forced or phished). PocketID itself stays on the private network (see Network segmentation).

Details

Service definition

:page_facing_up: docker-compose.yml :

services:

  pocketid:
    image: ghcr.io/pocket-id/pocket-id:v2
    container_name: pocketid
    restart: unless-stopped
    env_file: .env
    volumes:
      - ./data:/app/data
      - /opt/apps/pocketid/encryption_key:/opt/pocket-id/encryption_key:ro
    networks:
      - pocketid-net
      - traefik-private-net

networks:

  pocketid-net:
    name: pocketid-net

  traefik-private-net:
    name: traefik-private-net
    external: true

:page_facing_up: pocketid.yml :

http:
  services:
    pocketid:
      loadBalancer:
        servers:
          - url: http://pocketid:1411

  routers:
    pocketid:
      rule: 'Host(`pocketid.example.com`)'
      entryPoints:
        - websecure
      tls:
        certResolver: default
      service: pocketid
      # Wider whitelist than the other private services : the containers must reach the provider too (see below)
      middlewares:
        - pocketid-whitelist@file

  middlewares:
    # Same as vpn-whitelist, plus the private Docker network : an application authenticating natively against
    # PocketID fetches the discovery document and exchanges the token itself, from its container.
    # The public Docker network is NOT included : an application exposed to the internet must never reach the provider this way.
    pocketid-whitelist:
      ipAllowList:
        sourceRange:
          - "192.168.0.0/24" # your LAN
          - "10.0.0.0/24" # Wireguard subnet
          - "172.21.0.0/16" # traefik-private-net, check it with : docker network inspect traefik-private-net
    traefik-auth:
      plugin:
        traefik-oidc-auth:
          Secret: "<secret>"
          Provider:
            Url: "http://pocketid:1411/"
            ClientId: "<oidc_client_id>"
            ClientSecret: "<oidc_client_secret>"
            UsePkce: true
          Scopes: [ "openid", "profile", "email" ]
    pihole-auth:
      plugin:
        traefik-oidc-auth:
          Secret: "<secret>"
          Provider:
            Url: "http://pocketid:1411/"
            ClientId: "<oidc_client_id>"
            ClientSecret: "<oidc_client_secret>"
            UsePkce: true
          Scopes: [ "openid", "profile", "email" ]
    goatcounter-auth:
      plugin:
        traefik-oidc-auth:
          Secret: "<secret>"
          Provider:
            Url: "http://pocketid:1411/"
            ClientId: "<oidc_client_id>"
            ClientSecret: "<oidc_client_secret>"
            UsePkce: true
          Scopes: [ "openid", "profile", "email" ]
    prometheus-auth:
      plugin:
        traefik-oidc-auth:
          Secret: "<secret>"
          Provider:
            Url: "http://pocketid:1411/"
            ClientId: "<oidc_client_id>"
            ClientSecret: "<oidc_client_secret>"
            UsePkce: true
          Scopes: [ "openid", "profile", "email" ]
    backrest-auth:
      plugin:
        traefik-oidc-auth:
          Secret: "<secret>"
          Provider:
            Url: "http://pocketid:1411/"
            ClientId: "<oidc_client_id>"
            ClientSecret: "<oidc_client_secret>"
            UsePkce: true
          Scopes: [ "openid", "profile", "email" ]
    phpmyadmin-auth:
      plugin:
        traefik-oidc-auth:
          Secret: "<secret>"
          Provider:
            Url: "http://pocketid:1411/"
            ClientId: "<oidc_client_id>"
            ClientSecret: "<oidc_client_secret>"
            UsePkce: true
          Scopes: [ "openid", "profile", "email" ]

:page_facing_up: traefik.yml (plugin declaration, in the static configuration) :

experimental:
  plugins:
    traefik-oidc-auth:
      moduleName: "github.com/sevensolutions/traefik-oidc-auth"
      version: "v0.18.0"

Things to notice :

PocketID's data (SQLite database, uploaded logos) lives in the data folder, and the encryption key is mounted read-only from the host
the settings come from the .env file (see Environment variables)
it runs in its own network (pocketid-net) but must also share the same network as Traefik (traefik-private-net), both to be reachable by the reverse proxy and so that the plugin can talk to it directly by container name
the Traefik dynamic config file :
- creates a service which will point to our container application running on port 1411
- creates an HTTP router that will match pocketid.example.com URL on our websecure entrypoint to point to our service
- assigns the pocketid-whitelist middleware rather than the vpn-whitelist used by every other private service. It allows the same ranges (local network and VPN) plus the private Docker network, because an application that authenticates natively does not only send your browser to the provider : it also fetches the discovery document and exchanges the authorization code from its own container, and would therefore reach Traefik with a Docker source address that vpn-whitelist rejects. The public Docker network is deliberately left out, so that an application exposed to the internet can never reach the identity provider this way
- adds a TLS configuration that will use our default certificates resolver, so it can generate Let's encrypt certificates
- defines one middleware per protected service, named after it (pihole-auth, phpmyadmin-auth, ...). Each one needs its own OIDC client in PocketID, since the callback URL carries the host of the service it protects, and its own 32 characters Secret, which encrypts its session cookie : two middlewares sharing a secret would share their sessions. Add one block per service you put behind the single sign-on
- all of them reach PocketID through the internal URL http://pocketid:1411/ : being a Traefik plugin, the token exchange is done from the Traefik container and stays inside the Docker network, instead of looping through the reverse proxy. This is also why the plugin accepts plain HTTP here, where an application doing its own OIDC may not
the plugin itself is declared once in the static configuration, Traefik downloads it from its plugin catalog at start

Environment variables

:page_facing_up: .env :

APP_URL=https://pocketid.example.com
ENCRYPTION_KEY_FILE=/opt/pocket-id/encryption_key
# These variables are optional but recommended to review:
TRUST_PROXY=true
MAXMIND_LICENSE_KEY=
PUID=1000
PGID=1000

APP_URL is the public URL, it is also the OIDC issuer written in every token, so it must match the router's host exactly
there is no INTERNAL_APP_URL here on purpose : it makes the discovery document advertise the token and userinfo endpoints as http://pocketid:1411/..., for every client and whatever URL the document was fetched from. Clients whose library refuses plain HTTP (CrowdSec Web UI) then break on the token exchange. With the network alias and pocketid-whitelist, the containers reach the public HTTPS endpoints directly, so it is no longer needed
ENCRYPTION_KEY_FILE points to the key mounted read-only in the container
TRUST_PROXY makes PocketID take the client IP addresses from the headers set by Traefik (audit log, rate limiting), which is required behind a reverse proxy
MAXMIND_LICENSE_KEY is optional, with a free MaxMind license key the audit log shows where the logins come from
PUID / PGID are the user and group the application runs as, hence the ownership of the data folder and of the key

Run

Finally, simply run the Compose file :

sudo docker-compose -f /opt/apps/pocketid/docker-compose.yml up -d

You should end-up with a running pocketid container.

It should also have generated the needed Let's Encrypt certificates in the acme.json file in the Traefik folder.

The application is available at https://pocketid.example.com, where the first visit creates the administrator account and its passkey (see Setting up).

CrowdSec

We will use CrowdSec to detect and block the attackers knocking on the reverse proxy : scanners looking for /.env or /wp-login.php, brute force attempts, known exploits, bad bots.

CrowdSec is an open source, collaborative intrusion prevention system : a security engine reads logs, matches them against scenarios from a community hub and takes decisions (ban an IP address for a few hours), and a bouncer enforces them where the traffic enters. In return for the signals it shares, the engine also receives the community blocklist : IP addresses currently attacking other CrowdSec users are blocked before they even try anything here.

In our setup the only door open to the internet is Traefik, so everything happens there :

the security engine runs in a container on the private Traefik network and reads the Traefik access log through a shared folder (no Docker socket involved)
the bouncer is a Traefik plugin, declared as a middleware on the websecure entrypoint : every HTTPS request is checked against the current decisions before reaching any router, private services included (harmless : the local network and the VPN peers are trusted and never blocked)

Here is an overview of the network flow (when trying to reach Lychee app) :

flowchart LR
    style INCOMING_REQUEST fill: #45d8ff57
    style TRAEFIK_CONTAINER fill: #6601013c
    style APP_CONTAINER fill: #6601013c
    style CROWDSEC_CONTAINER fill: #6601013c
    style TRAEFIK_ROUTER fill: #defc1e37
    style TRAEFIK_MIDDLEWARE fill: #defc1e37
    style SERVER_DEVICE fill: #fec9bf5e
    style CONTAINER_ENGINE fill: #67010030
    style HUB fill: #bbfd795e
    DOCKER_TRAEFIK_PORT443{{443/tcp}}
    DOCKER_TRAEFIK_PORT80{{80/tcp}}
    DOCKER_APP_PORT{{80/tcp}}
    DOCKER_CROWDSEC_PORT{{8080/tcp\nlocal API}}
    TRAEFIK_ROUTER_APP(lychee)
    TRAEFIK_MIDDLEWARE_REDIRECT(HTTPS redirect on<br>the web entrypoint)
    TRAEFIK_MIDDLEWARE_CROWDSEC(CrowdSec bouncer on<br>the websecure entrypoint)
    INCOMING_REQUEST((INCOMING\nREQUEST))
    HUB((CrowdSec hub\nand community))

    subgraph SERVER_DEVICE[MINI PC]
        subgraph CONTAINER_ENGINE[DOCKER]
            subgraph TRAEFIK_CONTAINER[TRAEFIK CONTAINER]
                DOCKER_TRAEFIK_PORT443
                DOCKER_TRAEFIK_PORT80
                
                subgraph TRAEFIK_MIDDLEWARE[TRAEFIK MIDDLEWARES]
                    TRAEFIK_MIDDLEWARE_REDIRECT
                    TRAEFIK_MIDDLEWARE_CROWDSEC
                end

                subgraph TRAEFIK_ROUTER[TRAEFIK HTTP ROUTER]
                    TRAEFIK_ROUTER_APP
                end
            end

            subgraph APP_CONTAINER[APP CONTAINER]
                DOCKER_APP_PORT
            end

            subgraph CROWDSEC_CONTAINER[CROWDSEC CONTAINER]
                DOCKER_CROWDSEC_PORT
            end
        end

        ACCESS_LOG
    end

    INCOMING_REQUEST --> DOCKER_TRAEFIK_PORT443
    INCOMING_REQUEST --> DOCKER_TRAEFIK_PORT80
    DOCKER_TRAEFIK_PORT443 --> TRAEFIK_MIDDLEWARE_CROWDSEC
    DOCKER_TRAEFIK_PORT80 --> TRAEFIK_MIDDLEWARE_REDIRECT
    TRAEFIK_MIDDLEWARE_REDIRECT -.->|301 to https, before any router| INCOMING_REQUEST
    TRAEFIK_MIDDLEWARE_CROWDSEC -->|IP not banned| TRAEFIK_ROUTER_APP
    TRAEFIK_MIDDLEWARE_CROWDSEC -.->|IP banned : 403| INCOMING_REQUEST
    ACCESS_LOG[(access.log)]
    TRAEFIK_ROUTER_APP --> DOCKER_APP_PORT
    TRAEFIK_CONTAINER --->|writes| ACCESS_LOG
    CROWDSEC_CONTAINER --->|reads| ACCESS_LOG
    TRAEFIK_MIDDLEWARE_CROWDSEC <-.->|pulls the decisions<br>every minute| DOCKER_CROWDSEC_PORT
    CROWDSEC_CONTAINER <-->|scenarios, signals, community blocklist| HUB

Setting up

Create the folders, and a random key that will be shared between the security engine and the bouncer :

sudo mkdir -p /opt/apps/crowdsec /opt/apps/traefik/logs
openssl rand -base64 48

Then :

copy the .env, docker-compose.yml and acquis.yml files from this project's crowdsec directory into the /opt/apps/crowdsec directory, and put the key in BOUNCER_KEY_traefik of the .env file
put the same key in CROWDSEC_BOUNCER_KEY of Traefik's .env file, and copy the crowdsec.yml file from this project's traefik/dynamic directory into the /opt/apps/traefik/dynamic directory
update Traefik : the access log now goes to a file with the User-Agent header kept, the plugin is declared and the crowdsec@file middleware is set on the websecure entrypoint in traefik.yml (see Static configuration file), and the logs folder is bound in Traefik's docker-compose.yml (see Service definition)
copy the logrotate file from this project's traefik directory to /etc/logrotate.d/traefik : the access log is rotated daily and kept 7 days, Traefik reopens it on the USR1 signal

Details

Service definition

:page_facing_up: crowdsec/docker-compose.yml :

services:

  crowdsec:
    image: crowdsecurity/crowdsec:latest
    container_name: crowdsec
    restart: unless-stopped
    # Holds BOUNCER_KEY_traefik : registers the Traefik bouncer with this key at start (same value in traefik/.env)
    env_file: .env
    environment:
      TZ: "Europe/Zurich"
      # Hub items installed at start : Traefik log parser + HTTP scenarios, known CVE exploits, private IP ranges whitelist
      COLLECTIONS: "crowdsecurity/traefik crowdsecurity/http-cve"
      PARSERS: "crowdsecurity/whitelists"
    volumes:
      - ./acquis.yml:/etc/crowdsec/acquis.yaml:ro   # acquis.yaml is the path expected by CrowdSec's config.yaml
      - ./config:/etc/crowdsec
      - ./data:/var/lib/crowdsec/data
      - /opt/apps/traefik/logs:/var/log/traefik:ro
    networks:
      - traefik-private-net

networks:

  traefik-private-net:
    name: traefik-private-net
    external: true

:page_facing_up: crowdsec/.env :

# Key shared with the Traefik bouncer (same value as CROWDSEC_BOUNCER_KEY in traefik/.env), generate it with : openssl rand -base64 48
BOUNCER_KEY_traefik=<bouncer_key>

:page_facing_up: crowdsec/acquis.yml :

# Log sources read by the CrowdSec agent : the Traefik access log (bind mount shared with the Traefik container).
# A glob pattern, so that the file is picked up when it appears (Traefik may start after CrowdSec) or is recreated by logrotate.
filenames:
  - /var/log/traefik/*.log
labels:
  type: traefik

:page_facing_up: traefik/dynamic/crowdsec.yml :

http:
  middlewares:
    crowdsec:
      plugin:
        crowdsec-bouncer-traefik-plugin:
          enabled: true
          logLevel: INFO
          # stream mode : the plugin pulls the decisions from the CrowdSec local API every updateIntervalSeconds
          # and answers from its cache, nothing is called on the request path
          crowdsecMode: stream
          updateIntervalSeconds: 60
          crowdsecLapiScheme: http
          crowdsecLapiHost: crowdsec:8080
          # Dynamic files are Go templates : the key is read from the CROWDSEC_BOUNCER_KEY variable of the Traefik container (traefik/.env),
          # same value as BOUNCER_KEY_traefik in crowdsec/.env
          crowdsecLapiKey: '{{ env "CROWDSEC_BOUNCER_KEY" }}'
          # never block the local network and the VPN peers, whatever the decisions say
          clientTrustedIPs:
            - 192.168.0.0/24
            - 10.0.0.0/24

:page_facing_up: traefik/logrotate (to copy to /etc/logrotate.d/traefik) :

# Rotation of the Traefik access log (copy this file to /etc/logrotate.d/traefik on the host).
# Traefik reopens its log files when it receives the USR1 signal, no restart needed.
/opt/apps/traefik/logs/access.log {
    daily
    rotate 7
    compress
    delaycompress
    missingok
    notifempty
    create 0644 root root
    postrotate
        /usr/bin/docker kill --signal=USR1 traefik >/dev/null 2>&1 || true
    endscript
}

Things to notice :

the security engine only joins traefik-private-net : the bouncer reaches its local API at crowdsec:8080 by name, nothing is published on the host
COLLECTIONS and PARSERS are installed from the hub at the first start : crowdsecurity/traefik (the access log parser and the base HTTP scenarios), crowdsecurity/http-cve (known exploits) and crowdsecurity/whitelists (private IP ranges are never banned, so a misbehaving device at home cannot lock you out)
BOUNCER_KEY_traefik (from the .env file) registers the traefik bouncer with the given key at start, no manual cscli bouncers add needed, the middleware reads the same key from Traefik's own .env file through a template, so that the key never appears in a configuration file
the volumes hold the acquisition file (which log to read, and which parser applies to it â€” mounted as /etc/crowdsec/acquis.yaml, the path CrowdSec expects), the configuration (hub items, local API and community API credentials, all created automatically) and the data (SQLite database of alerts and decisions, downloaded blocklists), the Traefik logs folder is mounted read-only
the middleware runs in stream mode : it pulls the decisions from the local API every updateIntervalSeconds and answers from its cache, nothing is called on the request path. If the local API becomes unreachable, the plugin keeps serving with the decisions it already has and logs errors
clientTrustedIPs makes the bouncer skip the local network and the VPN peers entirely, in addition to the CrowdSec side whitelist
as the middleware sits on the entrypoint, it runs before the routers and their own middlewares (IP whitelist, authentication) for every request on 443, present and future services alike

[!NOTE] Your own public IP is not a private range. If some of your traffic reached Traefik through the NAT loopback of the router (a name resolving to the public IP, see IP whitelisting), a noisy test could ban you from your own services : cscli decisions delete --ip <your_public_ip> lifts it. With local DNS records for the private and the public services (see Pi-hole), the devices at home never take that path.

The engine shares the alerts it raises (attacking IP address and scenario) with CrowdSec's central API, that is what feeds the community blocklist everybody benefits from. If you don't want that, remove the api.server.online_client section from config.yaml.

Run

Start the security engine first, so that the bouncer finds its local API, then recreate Traefik (the static configuration changed, and the plugin is downloaded at that moment) :

sudo docker-compose -f /opt/apps/crowdsec/docker-compose.yml up -d
sudo docker-compose -f /opt/apps/traefik/docker-compose.yml up -d --force-recreate

You should end-up with a running crowdsec container. Check that everything talks to everything :

sudo docker exec crowdsec cscli bouncers list          # the "traefik" bouncer, with a recent "last pull"
sudo docker exec crowdsec cscli collections list       # crowdsecurity/traefik and http-cve installed
sudo docker exec crowdsec cscli metrics                # "Acquisition Metrics" : lines read and parsed from access.log (browse a site first)
sudo docker logs traefik 2>&1 | grep -i crowdsec       # plugin loaded, no error

To test the bouncer independently of the scenarios, ban an outside address (your phone on 4G for example) for a few minutes and try to reach a public service from it :

sudo docker exec crowdsec cscli decisions add --ip <phone_public_ip> --duration 5m --reason "bouncer test"
sudo docker exec crowdsec cscli decisions list
sudo docker exec crowdsec cscli decisions delete --ip <phone_public_ip>

The phone must get a 403 from Traefik while the decision is active. To test the scenarios, from the same phone request a dozen pages a scanner would try (/.env, /wp-login.php, /phpmyadmin/, /.git/config, ...) on a public service : after a few of them cscli alerts list shows a http-probing or http-sensitive-files alert and the phone is banned for four hours (the default duration), lift it with cscli decisions delete.

CrowdSec Web UI

CrowdSec is driven from the command line with cscli, which is fine for a check now and then but tedious to browse. CrowdSec Web UI is a small third-party dashboard that reads the same local API and shows the alerts, the active decisions, the bouncers and the metrics, with the country and the AS of every attacker, filters, and the ability to ban or unban an address in two clicks.

It is a plain HTTP application, it holds no Docker socket and no privilege : it only needs a machine account on the CrowdSec local API, so it sits on the private network like the other administration tools. It authenticates its users against PocketID with its own OIDC support.

Here is an overview of the network flow :

flowchart LR
    style INCOMING_REQUEST fill: #45d8ff57
    style TRAEFIK_CONTAINER fill: #6601013c
    style APP_CONTAINER fill: #6601013c
    style CROWDSEC_CONTAINER fill: #6601013c
    style POCKETID_CONTAINER fill: #6601013c
    style TRAEFIK_ROUTER fill: #defc1e37
    style TRAEFIK_MIDDLEWARE fill: #defc1e37
    style SERVER_DEVICE fill: #fec9bf5e
    style CONTAINER_ENGINE fill: #67010030
    DOCKER_TRAEFIK_PORT443{{443/tcp}}
    DOCKER_TRAEFIK_PORT80{{80/tcp}}
    DOCKER_APP_PORT{{3000/tcp}}
    DOCKER_CROWDSEC_PORT{{8080/tcp\nlocal API}}
    DOCKER_POCKETID_PORT{{1411/tcp}}
    TRAEFIK_ROUTER_APP(crowdsec-web-ui)
    TRAEFIK_MIDDLEWARE_REDIRECT(HTTPS redirect on<br>the web entrypoint)
    TRAEFIK_MIDDLEWARE_IP_WHITELIST(IP whitelist)
    INCOMING_REQUEST((INCOMING\nREQUEST))

    subgraph SERVER_DEVICE[MINI PC]
        subgraph CONTAINER_ENGINE[DOCKER]
            subgraph TRAEFIK_CONTAINER[TRAEFIK CONTAINER]
                DOCKER_TRAEFIK_PORT443
                DOCKER_TRAEFIK_PORT80
                
                subgraph TRAEFIK_ROUTER[TRAEFIK HTTP ROUTER]
                    TRAEFIK_ROUTER_APP
                end

                subgraph TRAEFIK_MIDDLEWARE[TRAEFIK MIDDLEWARES]
                    TRAEFIK_MIDDLEWARE_REDIRECT
                    TRAEFIK_MIDDLEWARE_IP_WHITELIST
                end
            end

            subgraph APP_CONTAINER[CROWDSEC WEB UI CONTAINER]
                DOCKER_APP_PORT
            end

            subgraph CROWDSEC_CONTAINER[CROWDSEC CONTAINER]
                DOCKER_CROWDSEC_PORT
            end

            subgraph POCKETID_CONTAINER[POCKETID CONTAINER]
                DOCKER_POCKETID_PORT
            end
        end
    end

    INCOMING_REQUEST --> DOCKER_TRAEFIK_PORT443
    INCOMING_REQUEST --> DOCKER_TRAEFIK_PORT80
    DOCKER_TRAEFIK_PORT80 ---> TRAEFIK_MIDDLEWARE_REDIRECT
    TRAEFIK_MIDDLEWARE_REDIRECT -.->|301 to https| INCOMING_REQUEST
    DOCKER_TRAEFIK_PORT443 --> TRAEFIK_ROUTER_APP
    TRAEFIK_ROUTER_APP --> TRAEFIK_MIDDLEWARE_IP_WHITELIST
    TRAEFIK_MIDDLEWARE_IP_WHITELIST --> DOCKER_APP_PORT
    DOCKER_APP_PORT --->|machine account :<br>alerts, decisions, metrics| DOCKER_CROWDSEC_PORT
    DOCKER_APP_PORT -..->|OIDC SSO,<br>through the Traefik alias| DOCKER_POCKETID_PORT

Setting up

Create the folders, then register the machine account the UI will use to read the local API :

sudo mkdir -p /opt/apps/crowdsec-web-ui/data
PW=$(openssl rand -base64 32); echo "machine password : $PW"
sudo docker exec crowdsec cscli machines add crowdsec-web-ui --password "$PW" -f /dev/null

Then :

copy the .env and docker-compose.yml files from this project's crowdsec-web-ui directory into the /opt/apps/crowdsec-web-ui directory, and put the generated password in CONFIG_INSTANCE_LAPI_AUTH_PASSWORD
copy the crowdsec-web-ui.yml file from this project's traefik/dynamic directory into the /opt/apps/traefik/dynamic directory
create an OIDC client in PocketID with the callback URL of the application : https://crowdsec.example.com/api/auth/oidc/callback, and PKCE disabled, as the application does not send a code_challenge (it is a confidential client, the client secret is what protects the code exchange). Then put its client ID in CONFIG_AUTH_OIDC_CLIENT_ID and its secret in CONFIG_AUTH_OIDC_CLIENT_SECRET. No middleware on the router : the application talks to PocketID itself
add a local DNS record crowdsec.example.com pointing to the mini PC (see Pi-hole), the service is not published on the internet

[!TIP] If PocketID answers access_denied, "you are not allowed to access this service" right after the login, the problem is not in the middleware : the OIDC client restricts access to some user groups and your account is not in them. Remove the restriction on the client, or add your group. The error comes from PocketID (look at the domain in the address bar), the application is not even reached.

[!NOTE] This service uses the native OIDC support of the application rather than the Traefik plugin used by Pi-Hole, so that the UI knows who is connected and can apply its admin / read-only roles. Its OIDC library only accepts HTTPS issuers (only requests to HTTPS are allowed), so the internal http://pocketid:1411 URL cannot be used : it goes through the public issuer, reachable from the container thanks to the Traefik network alias and the pocketid-whitelist middleware described in PocketID.

CONFIG_AUTH_ENABLED stays on auto : the built-in account (password, TOTP, passkeys) created on the first visit remains available and is your way back in if the OIDC login ever breaks.

Roles are decided by group mapping, and CONFIG_AUTH_OIDC_UNMATCHED_ROLE defaults to deny : without any group configured, a user who authenticates perfectly is still rejected with OIDC user is not authorized, and nothing is written in the logs since it is a decision, not an error. Either declare the groups as above, or set CONFIG_AUTH_OIDC_UNMATCHED_ROLE to admin and let PocketID alone decide who may use the client.

Details

Service definition

:page_facing_up: docker-compose.yml :

services:

  crowdsec-web-ui:
    image: ghcr.io/theduffman85/crowdsec-web-ui:latest
    container_name: crowdsec-web-ui
    restart: unless-stopped
    # Holds the password of the CrowdSec machine account (see .env)
    env_file: .env
    environment:
      TZ: "Europe/Zurich"
      # Built-in authentication stays enabled : the local account (password, TOTP, passkeys) is the fallback
      # if the OIDC login ever fails, and it is what gives the UI a real identity and admin / read-only roles
      CONFIG_AUTH_ENABLED: "auto"
      # Single sign-on against PocketID, handled by the application itself (no middleware on the router).
      # The issuer is the PUBLIC URL, no trailing slash : the container reaches it through the Traefik network
      # alias and the pocketid-whitelist middleware, see the PocketID section
      CONFIG_AUTH_OIDC_ISSUER_URL: https://pocketid.example.com
      CONFIG_AUTH_OIDC_CLIENT_ID: <oidc_client_id>
      # CONFIG_AUTH_OIDC_CLIENT_SECRET comes from the .env file
      # Role given to a user matching no group. It defaults to "deny", which rejects every OIDC user with
      # "OIDC user is not authorized" as long as no group is mapped. With a single administrator, "admin" is
      # enough : PocketID already decides who may use the client, through the allowed groups of the client itself.
      # For real admin / read-only roles, set it back to "deny" and map the groups :
      #   CONFIG_AUTH_OIDC_SCOPE: "openid profile email groups"
      #   CONFIG_AUTH_OIDC_GROUPS_CLAIM: groups
      #   CONFIG_AUTH_OIDC_ADMIN_GROUPS_0: <admin_group>
      #   CONFIG_AUTH_OIDC_READ_ONLY_GROUPS_0: <read_only_group>
      CONFIG_AUTH_OIDC_UNMATCHED_ROLE: admin
      # CrowdSec local API, reached by container name on the private Traefik network
      CONFIG_INSTANCE_LAPI_URL: http://crowdsec:8080
      CONFIG_INSTANCE_LAPI_AUTH_TYPE: password
      CONFIG_INSTANCE_LAPI_AUTH_USERNAME: crowdsec-web-ui
      # CONFIG_INSTANCE_LAPI_AUTH_PASSWORD comes from the .env file
    volumes:
      # SQLite database of the UI (its own users, notification rules, GeoNames data)
      - ./data:/app/data
    networks:
      - traefik-private-net

networks:

  traefik-private-net:
    name: traefik-private-net
    external: true

:page_facing_up: .env :

# Password of the CrowdSec machine account the UI uses to read the local API.
# Generate it with `openssl rand -base64 32`, then register the machine in the CrowdSec container :
#   sudo docker exec crowdsec cscli machines add crowdsec-web-ui --password '<password>' -f /dev/null
CONFIG_INSTANCE_LAPI_AUTH_PASSWORD=<lapi_machine_password>

# Secret of the PocketID OIDC client used for the single sign-on
CONFIG_AUTH_OIDC_CLIENT_SECRET=<oidc_client_secret>

:page_facing_up: crowdsec-web-ui.yml :

http:
  services:
    crowdsec-web-ui:
      loadBalancer:
        servers:
          - url: http://crowdsec-web-ui:3000

  routers:
    crowdsec-web-ui:
      rule: 'Host(`crowdsec.example.com`)'
      entryPoints:
        - websecure
      tls:
        certResolver: default
      service: crowdsec-web-ui
      # Only the IP whitelist : the application handles the PocketID single sign-on itself (native OIDC),
      # so no authentication middleware here, otherwise you would log in twice
      middlewares:
        - vpn-whitelist@file

Things to notice :

it only joins traefik-private-net : it reaches the CrowdSec local API at crowdsec:8080 by container name, and nothing is published on the host
CONFIG_INSTANCE_LAPI_AUTH_* are the credentials of the machine account registered with cscli machines add. A machine account is required : a bouncer API key like the one used by the Traefik plugin can only read the decisions, not the alerts
the data volume holds the UI's own SQLite database (its accounts, its notification rules, the GeoNames data used to locate the attackers), not CrowdSec data
the router only carries the IP whitelist : the single sign-on is done by the application itself, adding an authentication middleware would mean logging in twice
deleting alerts from the UI additionally requires its source IP to be trusted by CrowdSec, see the note below

[!WARNING] To allow alert deletion, the UI's IP must be listed in api.server.trusted_ips of CrowdSec's config.yaml (in /opt/apps/crowdsec/config/), then restart the container. Use the private network range only, never the 172.16.0.0/12 the project suggests : that range also covers traefik-public-net, so the applications exposed to the internet would be trusted too, which is exactly what Network segmentation avoids.
api:
  server:
    trusted_ips:
      - 127.0.0.1
      - ::1
      - 172.21.0.0/16 # traefik-private-net, check it with : docker network inspect traefik-private-net
Everything else (reading the alerts, adding or lifting a ban) works without it.

Run

Simply run the Compose file :

sudo docker-compose -f /opt/apps/crowdsec-web-ui/docker-compose.yml up -d

You should end-up with a running crowdsec-web-ui container, and Traefik picks up the dynamic configuration file without restarting.

The application is available at https://crowdsec.example.com. On the first visit it asks you to create the local administrator account, then the PocketID button appears on the login page.

[!NOTE] This is a third-party project, unrelated to the CrowdSec company, and it only publishes a latest tag : keep an eye on it when you pull the images.

Arcane

We will use Arcane to easily manage our Docker containers.

Arcane is an open source web interface to start, stop, restart, update and inspect the containers, read their logs, open a shell in them, and manage the images, volumes, networks and Compose projects, with image update checks and vulnerability scanning.

It needs the Docker socket, which means full control over the Docker daemon, i.e. root on the host : it sits on the private network only, reachable from the local network and the VPN, see Network segmentation.

Here is an overview of the network flow :

flowchart LR
    style INCOMING_REQUEST fill: #45d8ff57
    style TRAEFIK_CONTAINER fill: #6601013c
    style APP_CONTAINER fill: #6601013c
    style POCKETID_CONTAINER fill: #6601013c
    style TRAEFIK_ROUTER fill: #defc1e37
    style TRAEFIK_MIDDLEWARE fill: #defc1e37
    style SERVER_DEVICE fill: #fec9bf5e
    style CONTAINER_ENGINE fill: #67010030
    DOCKER_TRAEFIK_PORT443{{443/tcp}}
    DOCKER_TRAEFIK_PORT80{{80/tcp}}
    DOCKER_APP_PORT{{3552/tcp}}
    DOCKER_POCKETID_PORT{{1411/tcp}}
    DOCKER_SOCKET[(Docker socket)]
    TRAEFIK_ROUTER_APP(arcane)
    TRAEFIK_MIDDLEWARE_REDIRECT(HTTPS redirect on<br>the web entrypoint)
    TRAEFIK_MIDDLEWARE_IP_WHITELIST(IP whitelist)
    INCOMING_REQUEST((INCOMING\nREQUEST))

    subgraph SERVER_DEVICE[MINI PC]
        DOCKER_SOCKET

        subgraph CONTAINER_ENGINE[DOCKER]
            subgraph TRAEFIK_CONTAINER[TRAEFIK CONTAINER]
                DOCKER_TRAEFIK_PORT443
                DOCKER_TRAEFIK_PORT80
                
                subgraph TRAEFIK_ROUTER[TRAEFIK HTTP ROUTER]
                    TRAEFIK_ROUTER_APP
                end

                subgraph TRAEFIK_MIDDLEWARE[TRAEFIK MIDDLEWARES]
                    TRAEFIK_MIDDLEWARE_REDIRECT
                    TRAEFIK_MIDDLEWARE_IP_WHITELIST
                end
            end

            subgraph APP_CONTAINER[ARCANE CONTAINER]
                DOCKER_APP_PORT
            end

            subgraph POCKETID_CONTAINER[POCKETID CONTAINER]
                DOCKER_POCKETID_PORT
            end
        end
    end

    INCOMING_REQUEST --> DOCKER_TRAEFIK_PORT443
    INCOMING_REQUEST --> DOCKER_TRAEFIK_PORT80
    DOCKER_TRAEFIK_PORT80 ---> TRAEFIK_MIDDLEWARE_REDIRECT
    TRAEFIK_MIDDLEWARE_REDIRECT -.->|301 to https| INCOMING_REQUEST
    DOCKER_TRAEFIK_PORT443 --> TRAEFIK_ROUTER_APP
    TRAEFIK_ROUTER_APP --> TRAEFIK_MIDDLEWARE_IP_WHITELIST
    TRAEFIK_MIDDLEWARE_IP_WHITELIST --> DOCKER_APP_PORT
    DOCKER_APP_PORT -.->|OIDC SSO,<br>through the Traefik alias| DOCKER_POCKETID_PORT
    DOCKER_APP_PORT -->|containers, images,<br>volumes, . . .| DOCKER_SOCKET

Setting up

Create a folder to hold the configuration :

sudo mkdir /opt/apps/arcane

Then :

copy the .env and docker-compose.yml files from this project's arcane directory into the /opt/apps/arcane directory, and generate the encryption key in the .env file (openssl rand -hex 32). Keep it with your backups : it encrypts the secrets Arcane stores (registry credentials, ...)
copy the arcane.yml file from this project's traefik/dynamic directory into the /opt/apps/traefik/dynamic directory
create an OIDC client in PocketID with the callback URL of the application : https://arcane.example.com/auth/oidc/callback, and PKCE disabled (it is a confidential client, the client secret protects the code exchange). Restrict it to your administrators group (Allowed user groups), then put its client ID and secret in OIDC_CLIENT_ID and OIDC_CLIENT_SECRET of the .env file. No middleware on the router : the application talks to PocketID itself
add a local DNS record arcane.example.com pointing to the mini PC (see Pi-hole), the service is not published on the internet

Then start the service (see below) and finish the configuration in this order, the local admin account is needed until the OIDC login works :

for the first start, set OIDC_AUTO_REDIRECT_TO_PROVIDER to "false" in the docker-compose.yml file (the file of this project holds the final value, "true"), otherwise the login page redirects to PocketID before any role is mapped
log in with the default account, arcane / arcane-admin, and change the password as requested
in Settings -> Authentication, map the PocketID group super_admins to the Admin role, Global scope
log out, log in through PocketID, and check that you are an admin
disable the local login in Settings -> Authentication, set OIDC_AUTO_REDIRECT_TO_PROVIDER back to "true" and recreate the container : the login page then redirects straight to PocketID

[!IMPORTANT] Configure the role mapping before relying on the OIDC login : Arcane creates the OIDC users automatically on their first login, but without a matching mapping they get no role, and therefore no permission at all. The groups are read again at every login, PocketID is the source of truth.

The mapping can also be declared in the docker-compose.yml file with OIDC_ROLE_MAPPINGS, a JSON array such as [{"claimValue":"super_admins","roleId":"<admin_role_id>"}], but it references the role by its ID (see Settings -> Roles), not by its name.

[!NOTE] The default account is arcane / arcane-admin, not admin / admin as some pages of the documentation say. It is only created when the database holds no user : if the login fails on a fresh install, an earlier attempt already initialized the arcane-data volume. As long as nothing is configured, delete it and start again : sudo docker-compose -f /opt/apps/arcane/docker-compose.yml down -v.

Details

Service definition

:page_facing_up: docker-compose.yml :

services:

  arcane:
    image: ghcr.io/getarcaneapp/manager:latest
    container_name: arcane
    restart: unless-stopped
    # Encryption key and OIDC client credentials (see .env)
    env_file: .env
    environment:
      TZ: "Europe/Zurich"
      APP_URL: https://arcane.example.com
      # X-Forwarded-* headers are only trusted from the private Traefik network. Never the 172.16.0.0/12 suggested by
      # the documentation : that range also covers traefik-public-net. Check it with : docker network inspect traefik-private-net
      TRUSTED_PROXIES: 172.21.0.0/16
      ANALYTICS_DISABLED: "true"

      # Native OIDC authentication against PocketID (callback URL : https://arcane.example.com/auth/oidc/callback)
      # The issuer is the PUBLIC URL, no trailing slash : the container resolves it to Traefik thanks to the alias on
      # traefik-private-net, and Traefik lets it through with the pocketid-whitelist middleware (see PocketID)
      OIDC_ENABLED: "true"
      OIDC_ISSUER_URL: https://pocketid.example.com
      OIDC_SCOPES: openid email profile groups
      OIDC_GROUPS_CLAIM: groups
      OIDC_PROVIDER_NAME: PocketID
      # Set it to "false" for the first start, until the role mapping is configured and the OIDC login validated
      # (the local admin is needed for that), then back to "true" once the local login is disabled in Settings -> Authentication
      OIDC_AUTO_REDIRECT_TO_PROVIDER: "true"
      # Roles can also be mapped declaratively (role referenced by its ID, see Settings -> Roles) :
      # OIDC_ROLE_MAPPINGS: '[{"claimValue":"super_admins","roleId":"<admin_role_id>"}]'
    volumes:
      # Full access to the Docker daemon, i.e. root on the host : private network only
      - /var/run/docker.sock:/var/run/docker.sock
      # SQLite database, projects, settings, session signing key
      - arcane-data:/app/data
    # Host cgroup namespace, so that Arcane reliably detects its own container
    cgroup: host
    healthcheck:
      test: [ "CMD", "./arcane", "health", "--timeout", "2s" ]
      interval: 30s
      timeout: 3s
      retries: 5
      start_period: 15s
    networks:
      - traefik-private-net

volumes:

  arcane-data:
    name: arcane-data

networks:

  traefik-private-net:
    name: traefik-private-net
    external: true

Environment variables

:page_facing_up: .env :

# Key encrypting the secrets stored by Arcane (registry credentials, ...), 32 bytes : openssl rand -hex 32
# Keep it with your backups, without it the encrypted data is lost
ENCRYPTION_KEY=<encryption_key>

# PocketID OIDC client (callback URL : https://arcane.example.com/auth/oidc/callback)
OIDC_CLIENT_ID=<oidc_client_id>
OIDC_CLIENT_SECRET=<oidc_client_secret>

Traefik routing

:page_facing_up: arcane.yml :

http:
  services:
    arcane:
      loadBalancer:
        servers:
          - url: http://arcane:3552

  routers:
    arcane:
      rule: 'Host(`arcane.example.com`)'
      entryPoints:
        - websecure
      tls:
        certResolver: default
      service: arcane
      # Only the IP whitelist : the application handles the PocketID single sign-on itself (native OIDC),
      # so no authentication middleware here, otherwise you would log in twice
      middlewares:
        - vpn-whitelist@file

Things to notice :

it only joins traefik-private-net, nothing is published on the host
the data (SQLite database, settings, the key Arcane generates to sign its sessions) lives in the arcane-data Docker volume, see Volumes to back it up
TRUSTED_PROXIES is restricted to the private Traefik network, so that Arcane sees the real client IP in the X-Forwarded-For header set by Traefik. The documentation suggests 172.16.0.0/12, which also covers traefik-public-net
the stacks started with docker-compose from /opt/apps show up in Arcane as they are, through the labels Compose puts on the containers : nothing to import
the router only carries the IP whitelist : the single sign-on is done by the application itself, adding an authentication middleware would mean logging in twice
the Traefik documentation of Arcane also describes a gRPC router and a readtimeout=0s on the entrypoint : they are only needed for the edge agents, used to manage remote Docker hosts. The WebSockets (logs, container shell) go through the regular router without any special configuration
the arcane health command of the health check calls /api/health, which Gatus uses too

Run

Simply run the Compose file :

sudo docker-compose -f /opt/apps/arcane/docker-compose.yml up -d

You should end-up with a running arcane container, and Traefik picks up the dynamic configuration file without restarting.

The application is available at https://arcane.example.com, finish the configuration in the order described in Setting up above.

PhpMyAdmin

As our services will use some MySQL/MariaDB databases, we will use PhpMyAdmin to easily manage our databases.

PhpMyAdmin is a free software tool intended to handle the administration of MySQL over the Web, it supports a wide range of operations on MySQL and MariaDB (managing databases, tables, columns, relations, indexes, users, permissions, etc.).

Here is an overview of the network flow :

flowchart LR
    style INCOMING_REQUEST fill: #45d8ff57
    style TRAEFIK_CONTAINER fill: #6601013c
    style APP_CONTAINER fill: #6601013c
    style POCKETID_CONTAINER fill: #6601013c
    style TRAEFIK_ROUTER fill: #defc1e37
    style TRAEFIK_MIDDLEWARE fill: #defc1e37
    style SERVER_DEVICE fill: #fec9bf5e
    style CONTAINER_ENGINE fill: #67010030
    DOCKER_TRAEFIK_PORT443{{443/tcp}}
    DOCKER_TRAEFIK_PORT80{{80/tcp}}
    DOCKER_APP_PORT{{80/tcp}}
    DOCKER_POCKETID_PORT{{1411/tcp}}
    TRAEFIK_ROUTER_APP(phpmyadmin)
    TRAEFIK_MIDDLEWARE_REDIRECT(HTTPS redirect on<br>the web entrypoint)
    TRAEFIK_MIDDLEWARE_IP_WHITELIST(IP whitelist)
    INCOMING_REQUEST((INCOMING\nREQUEST))

    subgraph SERVER_DEVICE[MINI PC]
        subgraph CONTAINER_ENGINE[DOCKER]
            subgraph APP_CONTAINER[PHPMYADMIN CONTAINER]
                DOCKER_APP_PORT
            end

            subgraph TRAEFIK_CONTAINER[TRAEFIK CONTAINER]
                DOCKER_TRAEFIK_PORT443
                DOCKER_TRAEFIK_PORT80
                
                subgraph TRAEFIK_ROUTER[TRAEFIK HTTP ROUTER]
                    TRAEFIK_ROUTER_APP
                end

                subgraph TRAEFIK_MIDDLEWARE[TRAEFIK MIDDLEWARES]
                    TRAEFIK_MIDDLEWARE_REDIRECT
                    TRAEFIK_MIDDLEWARE_IP_WHITELIST
                end
            end

            subgraph POCKETID_CONTAINER[POCKETID CONTAINER]
                DOCKER_POCKETID_PORT
            end
        end
    end

    INCOMING_REQUEST --> DOCKER_TRAEFIK_PORT443
    INCOMING_REQUEST --> DOCKER_TRAEFIK_PORT80
    DOCKER_TRAEFIK_PORT80 ---> TRAEFIK_MIDDLEWARE_REDIRECT
    TRAEFIK_MIDDLEWARE_REDIRECT -.->|301 to https| INCOMING_REQUEST
    DOCKER_TRAEFIK_PORT443 --> TRAEFIK_ROUTER_APP
    TRAEFIK_MIDDLEWARE_IP_WHITELIST --> DOCKER_APP_PORT
    TRAEFIK_ROUTER_APP --> TRAEFIK_MIDDLEWARE_IP_WHITELIST
    DOCKER_APP_PORT -..->|OIDC| DOCKER_POCKETID_PORT

Setting up

Create a folder to hold the configuration :

sudo mkdir /opt/apps/phpmyadmin

Then :

copy the docker-compose.yml file from this project's phpmyadmin directory into the /opt/apps/phpmyadmin directory
copy the phpmyadmin.yml file from this project's traefik/dynamic directory into the /opt/apps/traefik/dynamic directory
create an OIDC client in PocketID with the callback URL https://phpmyadmin.example.com/oidc/callback (the plugin's default CallbackUri) and PKCE enabled, then fill the phpmyadmin-auth middleware in pocketid.yml with its client ID, its secret and its own 32 characters Secret
add a local DNS record phpmyadmin.example.com pointing to the mini PC (see Pi-hole), the service is not published on the internet

Details

Service definition

:page_facing_up: docker-compose.yml :

services:

  phpmyadmin:
    image: phpmyadmin:latest
    container_name: phpmyadmin
    environment:
      - PMA_ARBITRARY=1
    restart: unless-stopped
    volumes:
      - ./darkwolf/:/var/www/html/themes/darkwolf/
    networks:
      - phpmyadmin-net
      - traefik-private-net

networks:

  phpmyadmin-net:
    name: phpmyadmin-net

  traefik-private-net:
    name: traefik-private-net
    external: true

:page_facing_up: phpmyadmin.yml :

http:
  services:
    phpmyadmin:
      loadBalancer:
        servers:
          - url: http://phpmyadmin:80

  routers:
    phpmyadmin:
      rule: 'Host(`phpmyadmin.example.com`)'
      entryPoints:
        - websecure
      tls:
        certResolver: default
      service: phpmyadmin
      # PhpMyAdmin keeps its own authentication (the MySQL credentials), so this is a second, independent gate :
      # worth the two logins for a tool that gives direct access to the databases
      middlewares:
        - vpn-whitelist@file
        - phpmyadmin-auth@file

Things to notice :

We mount a theme directory to use a custom theme (dark theme named darkwolf), so just copy the theme data from official repository https://www.phpmyadmin.net/themes/
It uses Traefik dynamic config file to :
- create a service which will point to our container application running on port 80
- create an HTTP router that will match phpmyadmin.example.com URL on our websecure entrypoint to point to our service
- assign the vpn-whitelist middleware so that the traffic will be restricted to allowed IPs only (application reachable only from local network or through VPN)
- add a TLS configuration that will use our default certificates resolver, so it can generate Let's encrypt certificates
It runs in its own network (phpmyadmin-net) but must also share the same network as Traefik (traefik-private-net) so it can be auto discovered
The phpmyadmin network will have to be added to any MySQL/MariaDB database container that we want to make reachable from PhpMyAdmin
We set the environment variable PMA_ARBITRARY to 1 to tell PhpMyAdmin to allow connection to any arbitrary database server (we will be able to specify the server on login screen)

Run

Finally, simply run the Compose file :

sudo docker-compose -f /opt/apps/phpmyadmin/docker-compose.yml up -d

You should end-up with a running phpmyadmin container.

It should also have generated the needed Let's Encrypt certificates in the acme.json file in the Traefik folder.

The application is available at https://phpmyadmin.example.com.

[!IMPORTANT] You will have to use the database service name as host to connect to a database

Homer

Homer is a simple application that allows to generate a static homepage from a simple yaml configuration file.

We will use it as a dashboard to list our services.

flowchart LR
    style INCOMING_REQUEST fill: #45d8ff57
    style TRAEFIK_CONTAINER fill: #6601013c
    style APP_CONTAINER fill: #6601013c
    style TRAEFIK_ROUTER fill: #defc1e37
    style TRAEFIK_MIDDLEWARE fill: #defc1e37
    style SERVER_DEVICE fill: #fec9bf5e
    style CONTAINER_ENGINE fill: #67010030
    DOCKER_TRAEFIK_PORT443{{443/tcp}}
    DOCKER_TRAEFIK_PORT80{{80/tcp}}
    DOCKER_APP_PORT{{8080/tcp}}
    TRAEFIK_ROUTER_APP(dashboard)
    TRAEFIK_MIDDLEWARE_REDIRECT(HTTPS redirect on<br>the web entrypoint)
    TRAEFIK_MIDDLEWARE_IP_WHITELIST(IP whitelist)
    INCOMING_REQUEST((INCOMING\nREQUEST))

    subgraph SERVER_DEVICE[MINI_PC]
        subgraph CONTAINER_ENGINE[DOCKER]
            subgraph APP_CONTAINER[HOMER CONTAINER]
                DOCKER_APP_PORT
            end

            subgraph TRAEFIK_CONTAINER[TRAEFIK CONTAINER]
                DOCKER_TRAEFIK_PORT443
                DOCKER_TRAEFIK_PORT80
                
                subgraph TRAEFIK_ROUTER[TRAEFIK HTTP ROUTER]
                    TRAEFIK_ROUTER_APP
                end

                subgraph TRAEFIK_MIDDLEWARE[TRAEFIK MIDDLEWARES]
                    TRAEFIK_MIDDLEWARE_REDIRECT
                    TRAEFIK_MIDDLEWARE_IP_WHITELIST
                end
            end
        end
    end

    INCOMING_REQUEST --> DOCKER_TRAEFIK_PORT443
    INCOMING_REQUEST --> DOCKER_TRAEFIK_PORT80
    DOCKER_TRAEFIK_PORT80 ---> TRAEFIK_MIDDLEWARE_REDIRECT
    TRAEFIK_MIDDLEWARE_REDIRECT -.->|301 to https| INCOMING_REQUEST
    DOCKER_TRAEFIK_PORT443 --> TRAEFIK_ROUTER_APP
    TRAEFIK_MIDDLEWARE_IP_WHITELIST --> DOCKER_APP_PORT
    TRAEFIK_ROUTER_APP --> TRAEFIK_MIDDLEWARE_IP_WHITELIST

Setting up

First, create a folder to hold the configuration :

sudo mkdir /opt/apps/homer

Also create an assets directory to hold the application assets and the main configuration file, it will be mounted in the container.

mkdir /opt/apps/homer/assets

By default, on first run, it installs in this directory some example configuration files and assets (favicons, ...), we have disabled this by setting the environment variable INIT_ASSETS to 0 (default 1).

Note that this assets directory must have the same gid / uid that the container user have (default 1000:1000), so make sure to execute :

chown -R 1000:1000 /opt/apps/homer/assets/

Then copy :

the docker-compose.yml file from this project's homer directory into the /opt/apps/homer directory
the config.yml file from this project's homer directory into the /opt/apps/homer/assets directory
the logos used by the tiles (the logo-* files of this project's images directory) into the /opt/apps/homer/assets/logos directory
the homer.yml file from this project's traefik/dynamic directory into the /opt/apps/traefik/dynamic directory

I have decided to not protect it behind PocketID, as it is only reachable from the local network or through VPN, and it is a read-only dashboard, links to the services themselves are protected by PocketID.

Details

Service definition

:page_facing_up: docker-compose.yml :

services:

  homer:
    image: b4bz/homer:latest
    container_name: homer
    volumes:
      - ./assets/:/www/assets
    user: 1000:1000
    restart: unless-stopped
    environment:
      - INIT_ASSETS=0
      - IPV6_DISABLE=1
    networks:
      - homer-net
      - traefik-private-net

networks:

  homer-net:
    name: homer-net

  traefik-private-net:
    name: traefik-private-net
    external: true

:page_facing_up: homer.yml :

http:
  services:
    homer:
      loadBalancer:
        servers:
          - url: http://homer:8080

  routers:
    homer:
      rule: 'Host(`dashboard.example.com`)'
      entryPoints:
        - websecure
      tls:
        certResolver: default
      service: homer
      middlewares:
        - vpn-whitelist@file

Things to notice :

Homer's assets data is bound to a local directory named assets
It sets the INIT_ASSETS environment variable to 0 to avoid generating default example data
It sets the IPV6_DISABLE environment variable to 1to disable listening on IPv6 (we don't use IPv6)
It sets a user with uid and gid 1000 to run the application in the container
It uses Traefik dynamic config file to :
- create a service which will point to our container application running on port 8080
- create an HTTP router that will match dashboard.example.com URL on our websecure entrypoint to point to our service
- assign the vpn-whitelist middleware so that the traffic will be restricted to allowed IPs only (application reachable only from local network or through VPN)
- add TLS configuration that will use our default certificates resolver, so it can generate Let's encrypt certificates
It runs in its own network (homer-net) but must also share the same network as Traefik (traefik-private-net) so it can be auto discovered

Configuration file

:page_facing_up: config.yml :

---
header: false
footer: '<p>Created with <span class="has-text-danger">❤️</span> with <a href="https://bulma.io/">bulma</a>, <a href="https://vuejs.org/">vuejs</a> & <a href="https://fontawesome.com/">font awesome</a> // Fork me on <a href="https://github.com/bastienwirtz/homer"><i class="fab fa-github-alt"></i></a></p>' # set false if you want to hide it.

columns: 3

# Optional theme customization
theme: default
colors:
  light:
    highlight-primary: "#3367d6"
    highlight-secondary: "#4285f4"
    highlight-hover: "#5a95f5"
    background: "#f5f5f5"
    card-background: "#ffffff"
    text: "#363636"
    text-header: "#ffffff"
    text-title: "#303030"
    text-subtitle: "#424242"
    card-shadow: rgba(0, 0, 0, 0.1)
    link: "#3273dc"
    link-hover: "#363636"
  dark:
    highlight-primary: "#3367d6"
    highlight-secondary: "#2b2b2b"
    highlight-hover: "#131313"
    background: "#131313"
    card-background: "#2b2b2b"
    text: "#eaeaea"
    text-header: "#ffffff"
    text-title: "#fafafa"
    text-subtitle: "#f5f5f5"
    card-shadow: rgba(0, 0, 0, 0.4)
    link: "#3273dc"
    link-hover: "#ffdd57"

links:
  - name: "GitHub"
    icon: "fab fa-github"
    url: "https://github.com/Yann39"
    target: "_blank"

services:
  - name: "Admin tools"
    icon: "fas fa-shield"
    items:
      - name: "Dashdot"
        logo: "assets/logos/logo-dashdot.png"
        subtitle: "Minimal server monitoring"
        tag: "monitoring"
        url: "https://dashdot.example.com"
      - name: "Traefik"
        logo: "assets/logos/logo-traefik.svg"
        subtitle: "HTTP reverse proxy"
        tag: "network"
        url: "https://traefik.example.com"
      - name: "Arcane"
        logo: "assets/logos/logo-arcane.svg"
        subtitle: "Container management platform"
        tag: "tool"
        url: "https://arcane.example.com"
      - name: "Backrest"
        logo: "assets/logos/logo-backrest.svg"
        subtitle: "Backup solution built on top of restic"
        tag: "tool"
        url: "https://backrest.example.com"
      - name: "Pi-Hole"
        logo: "assets/logos/logo-pihole.svg"
        subtitle: "Network-wide ad blocking"
        tag: "network"
        url: "https://pihole.example.com/admin"
      - name: "GoatCounter"
        logo: "assets/logos/logo-goatcounter.svg"
        subtitle: "Privacy-friendly web analytics"
        tag: "analytics"
        url: "https://goatcounter.example.com"
      - name: "PhpMyAdmin"
        logo: "assets/logos/logo-phpmyadmin.svg"
        subtitle: "MySQL database management"
        tag: "tool"
        url: "https://phpmyadmin.example.com"
      - name: "CrowdSec Web UI"
        logo: "assets/logos/logo-crowdsec-web-ui.svg"
        subtitle: "Dashboard for CrowdSec"
        tag: "security"
        url: "https://crowdsec.example.com"
      - name: "WG Dashboard"
        logo: "assets/logos/logo-wgdashboard.png"
        subtitle: "Dashboard for WireGuard"
        tag: "tool"
        url: "https://wgdashboard.example.com"
      - name: "Prometheus"
        logo: "assets/logos/logo-prometheus.svg"
        subtitle: "Metrics and monitoring"
        tag: "Monitoring"
        url: "https://prometheus.example.com"
      - name: "Grafana"
        logo: "assets/logos/logo-grafana.svg"
        subtitle: "Analytics and visualization"
        tag: "Monitoring"
        url: "https://grafana.example.com"
      - name: "Gatus"
        logo: "assets/logos/logo-gatus.svg"
        subtitle: "Real-time uptime monitoring"
        tag: "monitoring"
        url: "https://gatus.example.com"
      - name: "Homelable"
        logo: "assets/logos/logo-homelable.svg"
        subtitle: "Homelab infrastructure map"
        tag: "network"
        url: "https://homelable.example.com"
      - name: "Speedtest Tracker"
        logo: "assets/logos/logo-speedtest-tracker.png"
        subtitle: "Internet speed tests and tracking"
        tag: "Monitoring"
        url: "https://speedtest-tracker.example.com"
  - name: "Applications"
    icon: "fas fa-globe"
    items:
      - name: "Motoclub GraphQL API"
        logo: "assets/logos/logo-ccteam.svg"
        subtitle: "GraphQL API for our motoclub mobile application"
        tag: "app"
        url: "https://ccteam.example.com/ccteam-gql/graphql"
      - name: "Defrag-life"
        logo: "https://cdn2.steamgriddb.com/file/sgdb-cdn/icon_thumb/946af3555203afdb63e571b873e419f6.png"
        subtitle: "Quake 3 arena Defrag website"
        tag: "app"
        url: "https://quake.example.com"
      - name: "Lychee"
        logo: "assets/logos/logo-lychee.png"
        subtitle: "Photo management tool"
        tag: "app"
        url: "https://lychee.example.com"
      - name: "Homebox"
        logo: "assets/logos/logo-homebox.svg"
        subtitle: "Home inventory management"
        tag: "app"
        url: "https://homebox.example.com"
      - name: "Ghostfolio"
        logo: "assets/logos/logo-ghostfolio.svg"
        subtitle: "Wealth management and portfolio tracking"
        tag: "app"
        url: "https://ghostfolio.example.com"
      - name: "Medusa"
        logo: "assets/logos/logo-medusa.svg"
        subtitle: "E-commerce platform, admin dashboard"
        tag: "app"
        url: "https://medusa.example.com/app"
      - name: "Medusa storefront"
        logo: "assets/logos/logo-medusa.svg"
        subtitle: "Online shop built on Medusa"
        tag: "app"
        url: "https://shop.example.com"
  - name: "Internal"
    icon: "fas fa-microchip"
    items:
      - name: "WireGuard"
        logo: "assets/logos/logo-wireguard.svg"
        subtitle: "Simple yet fast and modern VPN"
        tag: "network"
      - name: "Sablier"
        logo: "assets/logos/logo-sablier.svg"
        subtitle: "Workload scaling on demand"
        tag: "tool"
      - name: "Unbound"
        logo: "assets/logos/logo-unbound.svg"
        subtitle: "Validating, recursive, and caching DNS resolver"
        tag: "network"
      - name: "Pocket ID"
        logo: "assets/logos/logo-pocketid.svg"
        subtitle: "Simple OIDC provider"
        tag: "authentication"
        url: "https://pocketid.example.com"
      - name: "CrowdSec"
        logo: "assets/logos/logo-crowdsec.svg"
        subtitle: "Security automation engine"
        tag: "security"

This is simply the configuration file that is used by the application to display the dashboard page.

Run

Finally, simply run the Compose file :

sudo docker-compose -f /opt/apps/homer/docker-compose.yml up -d

You should end-up with a running homer container.

It should also have generated the needed Let's Encrypt certificates in the acme.json file in the Traefik folder.

The application will be available at https://dashboard.example.com.

Dashdot

Dashdot is a modern application to monitor server resources through a basic UI.

flowchart LR
    style INCOMING_REQUEST fill: #45d8ff57
    style TRAEFIK_CONTAINER fill: #6601013c
    style APP_CONTAINER fill: #6601013c
    style TRAEFIK_ROUTER fill: #defc1e37
    style TRAEFIK_MIDDLEWARE fill: #defc1e37
    style SERVER_DEVICE fill: #fec9bf5e
    style CONTAINER_ENGINE fill: #67010030
    DOCKER_TRAEFIK_PORT443{{443/tcp}}
    DOCKER_TRAEFIK_PORT80{{80/tcp}}
    DOCKER_APP_PORT{{3001/tcp}}
    TRAEFIK_ROUTER_APP(dashdot)
    TRAEFIK_MIDDLEWARE_REDIRECT(HTTPS redirect on<br>the web entrypoint)
    TRAEFIK_MIDDLEWARE_IP_WHITELIST(IP whitelist)
    INCOMING_REQUEST((INCOMING\nREQUEST))

    subgraph SERVER_DEVICE[MINI PC]
        subgraph CONTAINER_ENGINE[DOCKER]
            subgraph APP_CONTAINER[DASHDOT CONTAINER]
                DOCKER_APP_PORT
            end

            subgraph TRAEFIK_CONTAINER[TRAEFIK CONTAINER]
                DOCKER_TRAEFIK_PORT443
                DOCKER_TRAEFIK_PORT80
                
                subgraph TRAEFIK_ROUTER[TRAEFIK HTTP ROUTER]
                    TRAEFIK_ROUTER_APP
                end

                subgraph TRAEFIK_MIDDLEWARE[TRAEFIK MIDDLEWARES]
                    TRAEFIK_MIDDLEWARE_REDIRECT
                    TRAEFIK_MIDDLEWARE_IP_WHITELIST
                end
            end
        end
    end

    INCOMING_REQUEST --> DOCKER_TRAEFIK_PORT443
    INCOMING_REQUEST --> DOCKER_TRAEFIK_PORT80
    DOCKER_TRAEFIK_PORT80 ---> TRAEFIK_MIDDLEWARE_REDIRECT
    TRAEFIK_MIDDLEWARE_REDIRECT -.->|301 to https| INCOMING_REQUEST
    DOCKER_TRAEFIK_PORT443 --> TRAEFIK_ROUTER_APP
    TRAEFIK_MIDDLEWARE_IP_WHITELIST --> DOCKER_APP_PORT
    TRAEFIK_ROUTER_APP --> TRAEFIK_MIDDLEWARE_IP_WHITELIST

Setting up

Create a folder to hold the configuration :

sudo mkdir /opt/apps/dashdot

Then :

copy the docker-compose.yml file from this project's dashdot directory into the /opt/apps/dashdot directory
copy the dashdot.yml file from this project's traefik/dynamic directory into the /opt/apps/traefik/dynamic directory

I decided to not protect Dashdot behind PocketID, this is a read-only dashboard, that displays no sensitive information, and it is only reachable from the local network or through VPN.

Details

Service definition

:page_facing_up: docker-compose.yml :

services:

  dashdot:
    image: mauricenino/dashdot:latest
    container_name: dashdot
    restart: unless-stopped
    volumes:
      - /:/mnt/host:ro
    networks:
      - dashdot-net
      - traefik-private-net

networks:

  dashdot-net:
    name: dashdot-net

  traefik-private-net:
    name: traefik-private-net
    external: true

:page_facing_up: dashdot.yml :

http:
  services:
    dashdot:
      loadBalancer:
        servers:
          - url: http://dashdot:3001

  routers:
    dashdot:
      rule: 'Host(`dashdot.example.com`)'
      entryPoints:
        - websecure
      tls:
        certResolver: default
      service: dashdot
      middlewares:
        - vpn-whitelist@file
        - sablier-dashdot@file

Things to notice :

Dashdot's data is bound to the current directory (read-only)
It uses Traefik dynamic config file to :
- create a service which will point to our container application running on port 3001
- create an HTTP router that will match dashdot.example.com URL on our websecure entrypoint to point to our service
- add a TLS configuration that will use our default certificates resolver, so it can generate Let's encrypt certificates
- assign the vpn-whitelist middleware so that the traffic will be restricted to allowed IPs only (application reachable only from local network or through VPN)
- assign the sablier-dashdot middleware so that on-demand stop/start of the container can be done through Sablier
It runs in its own network (dashdot-net) but must also share the same network as Traefik (traefik-private-net) so it can be auto discovered

Run

Finally, simply run the Compose file :

sudo docker-compose -f /opt/apps/dashdot/docker-compose.yml up -d

You should end-up with a running dashdot container.

It should also have generated the needed Let's Encrypt certificates in the acme.json file in the Traefik folder.

The application is available at https://dashdot.example.com.

Lychee

Lychee is a robust, locally hosted web-based photo management tool. It enables you to carry out various operations on photos, including uploading, organizing, sharing, and more.

flowchart LR
    style INCOMING_REQUEST fill: #45d8ff57
    style TRAEFIK_CONTAINER fill: #6601013c
    style APP_CONTAINER fill: #6601013c
    style TRAEFIK_ROUTER fill: #defc1e37
    style TRAEFIK_MIDDLEWARE fill: #defc1e37
    style SERVER_DEVICE fill: #fec9bf5e
    style CONTAINER_ENGINE fill: #67010030
    DOCKER_TRAEFIK_PORT443{{443/tcp}}
    DOCKER_TRAEFIK_PORT80{{80/tcp}}
    DOCKER_APP_PORT{{80/tcp}}
    TRAEFIK_ROUTER_APP(lychee)
    TRAEFIK_MIDDLEWARE_REDIRECT(HTTPS redirect on<br>the web entrypoint)
    TRAEFIK_MIDDLEWARE_IP_WHITELIST(IP whitelist)
    INCOMING_REQUEST((INCOMING\nREQUEST))

    subgraph SERVER_DEVICE[MINI_PC]
        subgraph CONTAINER_ENGINE[DOCKER]
            subgraph APP_CONTAINER[LYCHEE CONTAINER]
                DOCKER_APP_PORT
            end

            subgraph TRAEFIK_CONTAINER[TRAEFIK CONTAINER]
                DOCKER_TRAEFIK_PORT443
                DOCKER_TRAEFIK_PORT80
                
                subgraph TRAEFIK_ROUTER[TRAEFIK HTTP ROUTER]
                    TRAEFIK_ROUTER_APP
                end

                subgraph TRAEFIK_MIDDLEWARE[TRAEFIK MIDDLEWARES]
                    TRAEFIK_MIDDLEWARE_REDIRECT
                    TRAEFIK_MIDDLEWARE_IP_WHITELIST
                end
            end
        end
    end

    INCOMING_REQUEST --> DOCKER_TRAEFIK_PORT443
    INCOMING_REQUEST --> DOCKER_TRAEFIK_PORT80
    DOCKER_TRAEFIK_PORT80 ---> TRAEFIK_MIDDLEWARE_REDIRECT
    TRAEFIK_MIDDLEWARE_REDIRECT -.->|301 to https| INCOMING_REQUEST
    DOCKER_TRAEFIK_PORT443 --> TRAEFIK_ROUTER_APP
    TRAEFIK_MIDDLEWARE_IP_WHITELIST --> DOCKER_APP_PORT
    TRAEFIK_ROUTER_APP --> TRAEFIK_MIDDLEWARE_IP_WHITELIST

Setting up

First, create a folder to hold the configuration :

sudo mkdir /opt/apps/lychee

Then copy :

the docker-compose.yml file from this project's lychee directory into the /opt/apps/lychee directory.
the lychee.yml file from this project's traefik/dynamic directory into the /opt/apps/traefik/dynamic directory.

Details

Service definition

:page_facing_up: docker-compose.yml :

services:

  lychee:
    image: lycheeorg/lychee:latest
    container_name: lychee
    volumes:
      - ./lychee/conf:/conf
      - ./lychee/uploads:/uploads
      - ./lychee/sym:/sym
      - ./lychee/logs:/logs
    environment:
      - PHP_TZ=UTC
      - TIMEZONE=UTC
      - DB_CONNECTION=mysql
      - DB_HOST=lychee-db
      - DB_PORT=3306
      - DB_DATABASE=lychee
      - DB_USERNAME=$MYSQL_USERNAME
      - DB_PASSWORD=$MYSQL_PASSWORD
      - STARTUP_DELAY=30
      - ADMIN_USER=$ADMIN_USER
      - ADMIN_PASSWORD=$ADMIN_PASSWORD
      - APP_URL=https://lychee.example.com
      - TRUSTED_PROXIES=*
    depends_on:
      - lychee-db
    restart: unless-stopped
    networks:
      - lychee-net
      - traefik-public-net

  lychee-db:
    container_name: lychee-db
    image: mariadb:latest
    restart: unless-stopped
    environment:
      - MARIADB_AUTO_UPGRADE=1
      - MYSQL_ROOT_PASSWORD=$MYSQL_ROOT_PASSWORD
      - MYSQL_DATABASE=lychee
      - MYSQL_USER=$MYSQL_USERNAME
      - MYSQL_PASSWORD=$MYSQL_PASSWORD
    volumes:
      - lychee-db-vol:/var/lib/mysql
    networks:
      - lychee-net

volumes:

  lychee-db-vol:
    name: lychee-db-vol

networks:

  lychee-net:
    name: lychee-net

  traefik-public-net:
    name: traefik-public-net
    external: true

:page_facing_up: lychee.yml :

http:
  services:
    lychee:
      loadBalancer:
        servers:
          - url: http://lychee:80

  routers:
    lychee:
      rule: 'Host(`lychee.example.com`)'
      entryPoints:
        - websecure
      tls:
        certResolver: default
      service: lychee

Things to notice :

It binds some volumes for configuration, uploads, symbolic links and logs
It sets some environment variables for timezone, database connection, admin user and password, application URL and trusted proxies
It uses Traefik dynamic config file to :
- create a service which will point to our container application running on port 80
- create an HTTP router that will match lychee.example.com URL on our websecure entrypoint to point to our service
- add TLS configuration that will use our default certificates resolver, so it can generate Let's encrypt certificates
It runs in its own network (lychee-net) but must also join the public network of Traefik (traefik-public-net) to be reachable by the reverse proxy, as it is exposed to the internet (see Network segmentation)

Run

Finally, simply run the Compose file :

sudo docker-compose -f /opt/apps/lychee/docker-compose.yml up -d

You should end-up with a running lychee container.

It should also have generated the needed Let's Encrypt certificates in the acme.json file in the Traefik folder.

The application will be available at https://lychee.example.com.

Homebox

Homebox is a simple inventory for the house : what you own, where it is stored, when it was bought, the warranty, the receipts and the manuals attached to it, with labels, a QR code per item and a full text search. Useful when the insurance asks for a list, or just to remember in which box something ended up.

It is a small Go application with an embedded database, it needs nothing else. It is reachable from the local network and the VPN only, and it is our example of an application doing OIDC natively against PocketID, with its public issuer URL.

Here is an overview of the network flow :

flowchart LR
    style INCOMING_REQUEST fill: #45d8ff57
    style TRAEFIK_CONTAINER fill: #6601013c
    style APP_CONTAINER fill: #6601013c
    style POCKETID_CONTAINER fill: #6601013c
    style TRAEFIK_ROUTER fill: #defc1e37
    style TRAEFIK_MIDDLEWARE fill: #defc1e37
    style SERVER_DEVICE fill: #fec9bf5e
    style CONTAINER_ENGINE fill: #67010030
    DOCKER_TRAEFIK_PORT443{{443/tcp}}
    DOCKER_TRAEFIK_PORT80{{80/tcp}}
    DOCKER_APP_PORT{{7745/tcp}}
    DOCKER_POCKETID_PORT{{1411/tcp}}
    TRAEFIK_ROUTER_APP(homebox)
    TRAEFIK_MIDDLEWARE_REDIRECT(HTTPS redirect on<br>the web entrypoint)
    TRAEFIK_MIDDLEWARE_IP_WHITELIST(IP whitelist)
    INCOMING_REQUEST((INCOMING\nREQUEST))

    subgraph SERVER_DEVICE[MINI PC]
        subgraph CONTAINER_ENGINE[DOCKER]
            subgraph TRAEFIK_CONTAINER[TRAEFIK CONTAINER]
                DOCKER_TRAEFIK_PORT443
                DOCKER_TRAEFIK_PORT80
                
                subgraph TRAEFIK_ROUTER[TRAEFIK HTTP ROUTER]
                    TRAEFIK_ROUTER_APP
                end

                subgraph TRAEFIK_MIDDLEWARE[TRAEFIK MIDDLEWARES]
                    TRAEFIK_MIDDLEWARE_REDIRECT
                    TRAEFIK_MIDDLEWARE_IP_WHITELIST
                end

            end

            subgraph APP_CONTAINER[HOMEBOX CONTAINER]
                DOCKER_APP_PORT
            end

            subgraph POCKETID_CONTAINER[POCKETID CONTAINER]
                DOCKER_POCKETID_PORT
            end
        end
    end

    INCOMING_REQUEST --> DOCKER_TRAEFIK_PORT443
    INCOMING_REQUEST --> DOCKER_TRAEFIK_PORT80
    DOCKER_TRAEFIK_PORT80 ---> TRAEFIK_MIDDLEWARE_REDIRECT
    TRAEFIK_MIDDLEWARE_REDIRECT -.->|301 to https| INCOMING_REQUEST
    DOCKER_TRAEFIK_PORT443 --> TRAEFIK_ROUTER_APP
    TRAEFIK_ROUTER_APP --> TRAEFIK_MIDDLEWARE_IP_WHITELIST
    TRAEFIK_MIDDLEWARE_IP_WHITELIST --> DOCKER_APP_PORT
    DOCKER_APP_PORT -.->|OIDC SSO,<br>through the Traefik alias| DOCKER_POCKETID_PORT

Setting up

Create a folder to hold the configuration :

sudo mkdir /opt/apps/homebox

Then :

copy the .env and docker-compose.yml files from this project's homebox directory into the /opt/apps/homebox directory
copy the homebox.yml file from this project's traefik/dynamic directory into the /opt/apps/traefik/dynamic directory
create an OIDC client in PocketID with the callback URL Homebox documents, then fill HBOX_OIDC_CLIENT_ID and HBOX_OIDC_CLIENT_SECRET
generate the pepper used to hash the API keys (openssl rand -base64 32) and put it in HBOX_AUTH_API_KEY_PEPPER
add a local DNS record homebox.example.com pointing to the mini PC (see Pi-hole), the service is not published on the internet

[!IMPORTANT] HBOX_OIDC_ISSUER_URL must be the public URL, without a trailing slash (Homebox is sensitive to it➚), and it must match character for character the issuer returned by the provider : its OIDC library refuses any difference. The internal URL http://pocketid:1411 therefore cannot be used, it answers with the public issuer and Homebox rejects it with issuer URL provided to client ... did not match. That the container can nonetheless reach the public URL is exactly what the Traefik network alias and the pocketid-whitelist middleware are for, see PocketID. Without them the container does not even resolve the name, since the private services have no public DNS record.

Details

Service definition

:page_facing_up: docker-compose.yml :

services:

  homebox:
    image: ghcr.io/sysadminsmedia/homebox:latest
    container_name: homebox
    restart: always
    env_file: .env
    environment:
      - HBOX_LOG_LEVEL=debug
      - HBOX_LOG_FORMAT=text
      - HBOX_WEB_MAX_UPLOAD_SIZE=10
      - HBOX_OIDC_ENABLED=true
      - HBOX_OIDC_ISSUER_URL=https://pocketid.example.com
      - HBOX_OIDC_CLIENT_ID=f1644c44-4f50-458f-9043-2bad9224e09c
      #- HBOX_OIDC_AUTO_REDIRECT=true
      #- HBOX_OPTIONS_ALLOW_LOCAL_LOGIN=false
      - HBOX_OPTIONS_TRUST_PROXY=true
      # Please consider allowing analytics to help us improve Homebox (basic computer information, no personal data)
      - HBOX_OPTIONS_ALLOW_ANALYTICS=true
    volumes:
      - homebox-data:/data/
    networks:
      - homebox-net
      - traefik-private-net

volumes:

  homebox-data:
    name: homebox-data-vol

networks:

  homebox-net:
    name: homebox-net

  traefik-private-net:
    name: traefik-private-net
    external: true

:page_facing_up: homebox.yml :

http:
  services:
    homebox:
      loadBalancer:
        servers:
          - url: http://homebox:7745

  routers:
    homebox:
      rule: 'Host(`homebox.example.com`)'
      entryPoints:
        - websecure
      tls:
        certResolver: default
      service: homebox
      # Only the IP whitelist : Homebox handles the PocketID single sign-on itself (native OIDC)
      middlewares:
        - vpn-whitelist@file

Things to notice :

the data (SQLite database, uploaded receipts and pictures) lives in a Docker volume named homebox-data-vol
it runs in its own network (homebox-net) but must also join the private network of Traefik (traefik-private-net) to be reachable by the reverse proxy, which is also what lets it reach PocketID, see Network segmentation
the Traefik dynamic config file creates the service pointing to the container on port 7745, the HTTP router matching homebox.example.com on the websecure entrypoint with a Let's Encrypt certificate, and applies the IP whitelist. No authentication middleware : Homebox does the single sign-on itself
HBOX_OPTIONS_TRUST_PROXY makes Homebox read the client address and the protocol from the headers set by Traefik, which is required behind a reverse proxy
HBOX_OPTIONS_ALLOW_LOCAL_LOGIN and HBOX_OIDC_AUTO_REDIRECT are commented out : the first one disables the local accounts once the single sign-on works, the second one sends the user straight to PocketID without showing the login page. Enable them only when you are sure the OIDC login works, otherwise you lock yourself out

Environment variables

:page_facing_up: .env :

HBOX_OIDC_CLIENT_SECRET=<client_secret>
HBOX_AUTH_API_KEY_PEPPER=<pepper_auth_api_key>

HBOX_OIDC_CLIENT_SECRET is the secret of the PocketID client
HBOX_AUTH_API_KEY_PEPPER is the value Homebox mixes into the hash of the API keys it issues. Set it once and keep it : changing it invalidates every existing key

Run

Simply run the Compose file :

sudo docker-compose -f /opt/apps/homebox/docker-compose.yml up -d

You should end-up with a running homebox container, and Traefik picks up the dynamic configuration file without restarting.

The application is available at https://homebox.example.com, with a button to log in through PocketID.

GoatCounter

GoatCounter counts the visits on the public websites. It is deliberately minimal : no cookies, no tracking across sites, no personal data stored (the visitor IP is only used to derive the country and to compute a daily hash, it is never kept), which also means no consent banner to display. A single Go binary with an embedded SQLite database, a few megabytes of memory.

Unlike every other tool of this guide, it is exposed to the internet : the tracking script and the endpoint that collects the hits must be reachable by the visitors of the public websites. It therefore sits on the public network and relies on CrowdSec like the other public applications. Only the collecting endpoints are open, the dashboard is put behind PocketID with a second router, see below.

Here is an overview of the network flow :

flowchart LR
    style INCOMING_REQUEST fill: #45d8ff57
    style TRAEFIK_CONTAINER fill: #6601013c
    style APP_CONTAINER fill: #6601013c
    style WEBSITE_CONTAINER fill: #6601013c
    style TRAEFIK_ROUTER fill: #defc1e37
    style TRAEFIK_MIDDLEWARE fill: #defc1e37
    style SERVER_DEVICE fill: #fec9bf5e
    style CONTAINER_ENGINE fill: #67010030
    DOCKER_TRAEFIK_PORT443{{443/tcp}}
    DOCKER_TRAEFIK_PORT80{{80/tcp}}
    DOCKER_APP_PORT{{8080/tcp}}
    DOCKER_WEBSITE_PORT{{80/tcp}}
    TRAEFIK_ROUTER_APP(goatcounter\n/count, /loader, ...)
    TRAEFIK_ROUTER_DASH(goatcounter\ndashboard)
    TRAEFIK_ROUTER_SITE(quake)
    TRAEFIK_MIDDLEWARE_REDIRECT(HTTPS redirect on<br>the web entrypoint)
    TRAEFIK_MIDDLEWARE_CROWDSEC(CrowdSec bouncer)
    INCOMING_REQUEST((Incoming<br>request))

    subgraph SERVER_DEVICE[MINI PC]
        subgraph CONTAINER_ENGINE[DOCKER]
            subgraph TRAEFIK_CONTAINER[TRAEFIK CONTAINER]
                DOCKER_TRAEFIK_PORT443
                DOCKER_TRAEFIK_PORT80

                subgraph TRAEFIK_MIDDLEWARE[TRAEFIK MIDDLEWARES]
                    TRAEFIK_MIDDLEWARE_REDIRECT
                    TRAEFIK_MIDDLEWARE_CROWDSEC
                end

                subgraph TRAEFIK_ROUTER[TRAEFIK HTTP ROUTERS]
                    TRAEFIK_ROUTER_SITE
                    TRAEFIK_ROUTER_APP
                    TRAEFIK_ROUTER_DASH
                end

                TRAEFIK_MIDDLEWARE_OIDC(PocketID auth)
            end

            subgraph WEBSITE_CONTAINER[WEBSITE CONTAINER]
                DOCKER_WEBSITE_PORT
            end

            subgraph APP_CONTAINER[GOATCOUNTER CONTAINER]
                DOCKER_APP_PORT
            end
        end
    end

    INCOMING_REQUEST -->|1 . loads the page| DOCKER_TRAEFIK_PORT443
    INCOMING_REQUEST -.->|2 . the script reports the visit| DOCKER_TRAEFIK_PORT443
    INCOMING_REQUEST --> DOCKER_TRAEFIK_PORT80
    DOCKER_TRAEFIK_PORT80 --> TRAEFIK_MIDDLEWARE_REDIRECT
    TRAEFIK_MIDDLEWARE_REDIRECT -.->|301 to https| INCOMING_REQUEST
    DOCKER_TRAEFIK_PORT443 --> TRAEFIK_MIDDLEWARE_CROWDSEC
    TRAEFIK_MIDDLEWARE_CROWDSEC --> TRAEFIK_ROUTER_DASH
    TRAEFIK_ROUTER_SITE --> DOCKER_WEBSITE_PORT
    TRAEFIK_ROUTER_APP -->|X - Forwarded - For : the visitor IP| DOCKER_APP_PORT
    TRAEFIK_ROUTER_DASH --> TRAEFIK_MIDDLEWARE_OIDC
    TRAEFIK_MIDDLEWARE_OIDC --> DOCKER_APP_PORT

Setting up

Create a folder to hold the configuration :

sudo mkdir /opt/apps/goatcounter

Then :

copy the docker-compose.yml file from this project's goatcounter directory into the /opt/apps/goatcounter directory
copy the goatcounter.yml file from this project's traefik/dynamic directory into the /opt/apps/traefik/dynamic directory
add a public DNS record for goatcounter.example.com (a CNAME to your dynamic DNS, like the other public services, see Domain and subdomains), and a local DNS record pointing to the mini PC (see Pi-hole) so that your own devices do not go through the NAT loopback of the router
create an OIDC client and its goatcounter-auth middleware as described in PocketID, with the callback URL https://goatcounter.example.com/oidc/callback
start the service (see Run), then create the site and its administrator account :
```
sudo docker exec -it goatcounter goatcounter db create site -vhost=goatcounter.example.com [email protected]
```
It asks for a password. The -vhost must match the Host() of the router : GoatCounter is multi-site and dispatches on the Host header.

Finally, add the tracking script to the websites you want to count, just before </body> :


<script data-goatcounter="https://goatcounter.example.com/count"
        async src="https://goatcounter.example.com/count.js"></script>

[!TIP] The script must run on every page, but there is no need to edit them one by one : put it once in the shared header or footer that all the pages already include (include 'header.php' and friends). If the site has no such common template, PHP-FPM can append a file to every script without touching a single page, with php_value[auto_append_file] = /var/www/html/goatcounter.php in its pool configuration, beware that it appends to every PHP response, which would corrupt the ones that are not HTML (JSON, generated images, downloads).

Serving the script from your own domain rather than from a third party CDN also makes it far less likely to be stopped by ad blockers.

Details

Service definition

:page_facing_up: docker-compose.yml :

services:

  goatcounter:
    image: arp242/goatcounter:latest
    container_name: goatcounter
    restart: unless-stopped
    environment:
      TZ: "Europe/Zurich"
    # The image entrypoint is the goatcounter binary, its default command being "serve -automigrate".
    # We keep -automigrate (pending migrations are applied at start, useful when pulling a new image) and add :
    #   -tls=http  serve plain HTTP, TLS is terminated by Traefik. This one is NOT optional : without it
    #              goatcounter defaults to "acme" in production and tries to get its own certificates
    #   -listen    the address Traefik forwards to
    command: [ "serve", "-automigrate", "-tls=http", "-listen=:8080" ]
    volumes:
      # SQLite database and uploaded data, in a named volume : the container runs as a non-root user,
      # a bind mount would need the right ownership on the host
      - goatcounter-data:/home/goatcounter/goatcounter-data
    networks:
      - goatcounter-net
      - traefik-public-net

volumes:

  goatcounter-data:
    name: goatcounter-data-vol

networks:

  goatcounter-net:
    name: goatcounter-net

  traefik-public-net:
    name: traefik-public-net
    external: true

:page_facing_up: goatcounter.yml :

http:
  services:
    goatcounter:
      loadBalancer:
        servers:
          - url: http://goatcounter:8080

  routers:
    # Public part : everything a visitor's browser needs to report a hit. No whitelist, no authentication,
    # otherwise the collection silently stops. PathPrefix(`/count`) covers /count, /count.js and /counter/...
    # The higher priority makes this router win over the one below, which matches the whole host.
    goatcounter-public:
      rule: 'Host(`goatcounter.example.com`) && (PathPrefix(`/count`) || PathPrefix(`/loader`) || PathPrefix(`/load-widget`) || Path(`/jserr`) || Path(`/csp`) || Path(`/robots.txt`) || Path(`/security.txt`))'
      priority: 100
      entryPoints:
        - websecure
      tls:
        certResolver: default
      service: goatcounter

    # Everything else : the dashboard, the settings, the login page, behind PocketID authentication
    goatcounter:
      rule: 'Host(`goatcounter.example.com`)'
      priority: 1
      entryPoints:
        - websecure
      tls:
        certResolver: default
      service: goatcounter
      # IP whitelist first : the dashboard is only meant to be used from the local network or through the VPN,
      # so it does not depend on the authentication middleware alone. This matters because the host is public
      # and GoatCounter's own login is disabled (its site is set to public) : without this, a mistake in the
      # public router's path rule above would expose the statistics to the internet.
      middlewares:
        - vpn-whitelist@file
        - goatcounter-auth@file

Things to notice :

-tls=http is not optional : GoatCounter defaults to acme in production and would try to obtain its own Let's Encrypt certificates, in competition with Traefik. Here TLS is terminated by the reverse proxy and GoatCounter only serves plain HTTP on its port
-automigrate comes from the image's default command and is kept : pending schema migrations are applied at start, which matters when pulling a new image
the data (SQLite database) lives in a named volume rather than a bind mount : the container runs as a non-root user, a bind mount would need the matching ownership on the host
there are two routers on the same host, split by path and separated by an explicit priority. The public one carries no middleware : an IP whitelist would block the visitors and an authentication middleware would block the collection. The other one, matching everything else, carries the IP whitelist and the PocketID middleware : the dashboard is only used from the local network or the VPN, and since the host is public it must not depend on the authentication middleware alone. The CrowdSec bouncer applies to both, since it sits on the websecure entrypoint
it joins traefik-public-net, so it cannot reach the private services, see Network segmentation
GoatCounter reads the visitor address from the X-Forwarded-For header set by Traefik, there is nothing to configure for that

[!WARNING] Splitting a host between a public router and an authenticated one is effective but unforgiving : if a collecting path ends up on the wrong side, the tracking request is answered with a redirection to PocketID, the browser reports nothing and you silently lose visits. Check the paths your visitors really request before and after the change, the Traefik access log gives them :
grep '"RequestHost":"goatcounter.example.com"' /opt/apps/traefik/logs/access.log | grep -oE '"RequestPath":"[^"]+"' | sort | uniq -c | sort -rn
Note also that GoatCounter has its own login : behind the middleware you would authenticate twice. To keep a single login, mark the site as public in its settings so that the statistics no longer require a GoatCounter account, and let PocketID be the only gate, at the cost of world readable statistics should a path rule ever leak.

[!NOTE] About the locations shown in the dashboard :

a Countries database is built into GoatCounter, countries work out of the box. Regions need the Cities version of the MaxMind database, given with the -geodb flag (-geodb maxmind:<account_id>:<license_key> downloads and refreshes it automatically, or drop any .mmdb file in the data volume and it is picked up)

the location is resolved when the visit is recorded, and stored. It is never recomputed : the visits collected before you enable or fix anything stay Unknown forever

your own visits are always Unknown, since they come from the local network or from the VPN and a private address has no location. Testing from a phone only proves something if the VPN is turned off on it, otherwise the visit arrives from the tunnel with a 10.0.0.x address

Run

Simply run the Compose file :

sudo docker-compose -f /opt/apps/goatcounter/docker-compose.yml up -d

You should end-up with a running goatcounter container, and Traefik picks up the dynamic configuration file without restarting.

The dashboard is available at https://goatcounter.example.com, with the account created above.

Prometheus

Prometheus collects metrics : at a regular interval it calls (scrapes) an HTTP endpoint exposed by each monitored application, and stores the values in its own time series database, queried with the PromQL language. It has no real dashboard, that is the job of Grafana, which reads its data.

Here it monitors the CCTeam GraphQL API (see CCTeam) : request rate, errors and response times, per operation and per GraphQL field, plus the JVM, the database connection pool, ... that Spring Boot exposes out of the box through Actuator and Micrometer.

It is an administration tool, so it sits on the private network and is only reachable from the local network and the VPN. Prometheus has no native OIDC support (only basic authentication and TLS, through a web configuration file), so its web interface is put behind PocketID with the Traefik plugin, like Pi-Hole.

Here is an overview of the network flow :

flowchart LR
    style INCOMING_REQUEST fill: #45d8ff57
    style TRAEFIK_CONTAINER fill: #6601013c
    style APP_CONTAINER fill: #6601013c
    style CCTEAM_CONTAINER fill: #6601013c
    style GRAFANA_CONTAINER fill: #6601013c
    style TRAEFIK_ROUTER fill: #defc1e37
    style TRAEFIK_MIDDLEWARE fill: #defc1e37
    style SERVER_DEVICE fill: #fec9bf5e
    style CONTAINER_ENGINE fill: #67010030
    DOCKER_TRAEFIK_PORT443{{443/tcp}}
    DOCKER_TRAEFIK_PORT80{{80/tcp}}
    DOCKER_APP_PORT{{9090/tcp}}
    DOCKER_CCTEAM_PORT{{5001/tcp\nGraphQL API}}
    DOCKER_CCTEAM_MANAGEMENT_PORT{{8081/tcp\nactuator}}
    DOCKER_GRAFANA_PORT{{3000/tcp}}
    TRAEFIK_ROUTER_APP(prometheus)
    TRAEFIK_ROUTER_CCTEAM(ccteam)
    TRAEFIK_MIDDLEWARE_REDIRECT(HTTPS redirect on<br>the web entrypoint)
    TRAEFIK_MIDDLEWARE_IP_WHITELIST(IP whitelist)
    TRAEFIK_MIDDLEWARE_OIDC(PocketID auth)
    INCOMING_REQUEST((INCOMING\nREQUEST))

    subgraph SERVER_DEVICE[MINI PC]
        subgraph CONTAINER_ENGINE[DOCKER]
            subgraph TRAEFIK_CONTAINER[TRAEFIK CONTAINER]
                DOCKER_TRAEFIK_PORT443
                DOCKER_TRAEFIK_PORT80
                
                subgraph TRAEFIK_ROUTER[TRAEFIK HTTP ROUTERS]
                    TRAEFIK_ROUTER_APP
                    TRAEFIK_ROUTER_CCTEAM
                end

                subgraph TRAEFIK_MIDDLEWARE[TRAEFIK MIDDLEWARES]
                    TRAEFIK_MIDDLEWARE_REDIRECT
                    TRAEFIK_MIDDLEWARE_IP_WHITELIST
                    TRAEFIK_MIDDLEWARE_OIDC
                end
            end

            subgraph APP_CONTAINER[PROMETHEUS CONTAINER]
                DOCKER_APP_PORT
            end

            subgraph CCTEAM_CONTAINER[CCTEAM CONTAINER]
                DOCKER_CCTEAM_PORT
                DOCKER_CCTEAM_MANAGEMENT_PORT
            end

            subgraph GRAFANA_CONTAINER[GRAFANA CONTAINER]
                DOCKER_GRAFANA_PORT
            end
        end
    end

    INCOMING_REQUEST --> DOCKER_TRAEFIK_PORT443
    INCOMING_REQUEST --> DOCKER_TRAEFIK_PORT80
    DOCKER_TRAEFIK_PORT80 ---> TRAEFIK_MIDDLEWARE_REDIRECT
    TRAEFIK_MIDDLEWARE_REDIRECT -.->|301 to https| INCOMING_REQUEST
    DOCKER_TRAEFIK_PORT443 --> TRAEFIK_ROUTER_APP
    TRAEFIK_ROUTER_APP --> TRAEFIK_MIDDLEWARE_IP_WHITELIST
    TRAEFIK_MIDDLEWARE_IP_WHITELIST --> TRAEFIK_MIDDLEWARE_OIDC
    TRAEFIK_MIDDLEWARE_OIDC --> DOCKER_APP_PORT
    TRAEFIK_ROUTER_CCTEAM --> DOCKER_CCTEAM_PORT
    DOCKER_APP_PORT -->|scrape every 15 s,<br>prometheus - ccteam - net| DOCKER_CCTEAM_MANAGEMENT_PORT
    DOCKER_GRAFANA_PORT -->|PromQL queries,<br>prometheus - net| DOCKER_APP_PORT

The monitoring stack uses its own networks, on top of the Traefik ones :

Network	Who	Why
`traefik-private-net`	Traefik, Prometheus, Grafana	web interfaces behind Traefik, Grafana reaches PocketID
`prometheus-net`	Prometheus, Grafana	Grafana queries Prometheus directly
`prometheus-ccteam-net`	Prometheus, the CCTeam application container only	Prometheus scrapes the API

The scraping network is dedicated on purpose. CCTeam is exposed to the internet, it is the container most likely to be compromised one day :

if CCTeam joined prometheus-net, it could reach Grafana, and the Prometheus API which has no authentication
if Prometheus joined ccteam-net, it could reach the database of CCTeam

With prometheus-ccteam-net, the API and Prometheus only see each other. The database stays on ccteam-net, and Grafana is not reachable from the application. This is a deliberate, limited exception to the rules of Network segmentation : a compromised CCTeam container could query Prometheus directly, but only Prometheus, and only to read metrics (the admin and lifecycle APIs are disabled, see below). A monitored application never joins prometheus-net, each one gets its own prometheus-<app>-net.

Setting up

Create a folder to hold the configuration :

sudo mkdir /opt/apps/prometheus

Then :

copy the docker-compose.yml and prometheus.yml files from this project's prometheus directory into the /opt/apps/prometheus directory
copy the prometheus.yml file from this project's traefik/dynamic directory into the /opt/apps/traefik/dynamic directory
create an OIDC client and its prometheus-auth middleware as described in PocketID, with the callback URL https://prometheus.example.com/oidc/callback and PKCE enabled. Restrict the client to your administrators group (Allowed user groups), nobody else has anything to do there
add a local DNS record prometheus.example.com pointing to the mini PC (see Pi-hole), the service is not published on the internet

On the application side, the Spring Boot API needs the spring-boot-starter-actuator and micrometer-registry-prometheus dependencies, and the following properties :

# Actuator on a dedicated port, not routed by Traefik : the metrics are never exposed to the internet
management.server.port=8081
management.endpoints.web.exposure.include=health,prometheus
# Tag added to every metric, used by the Grafana dashboard to select the application
management.metrics.tags.application=ccteam-graphql
# Histogram buckets, needed to compute percentiles (p95, p99) in Prometheus
management.metrics.distribution.percentiles-histogram.graphql.request=true
management.metrics.distribution.percentiles-histogram.graphql.datafetcher=true

And its container joins the scraping network, see the docker-compose.yml file of the ccteam directory :

    networks:
      - ccteam-net
      - traefik-public-net
      # Scraped by Prometheus on the management port (8081), which is not routed by Traefik
      - prometheus-ccteam-net

[!IMPORTANT] The management port is what keeps the metrics private. Without it, the actuator is served on the application port, the one Traefik routes to the internet, and https://ccteam.example.com/ccteam-gql/actuator/prometheus would be readable by anyone : GraphQL operation names, response times, JVM version, ...

With a dedicated port, the metrics endpoint is only reachable from the Docker network, so it does not need any authentication : the application does not need a Spring Security filter chain (basic authentication for instance) for it anymore. If your application has a catch-all filter chain (anyRequest().authenticated()), permit the actuator endpoints explicitly (EndpointRequest.toAnyEndpoint()), otherwise Prometheus gets a 401.

Note that the servlet context-path does not apply to the management port : the endpoint is /actuator/prometheus, not /ccteam-gql/actuator/prometheus.

[!NOTE] The histograms are what allow computing percentiles, but each one produces a few dozen series (one per bucket), per operation, per GraphQL field and per outcome. That is fine for an API of this size, but keep an eye on the number of series from time to time (scrape_samples_scraped{job="ccteam-graphql"}) : if it keeps growing, a label has too many distinct values.

Details

Service definition

:page_facing_up: docker-compose.yml :

services:

  prometheus:
    image: prom/prometheus:latest
    container_name: prometheus
    restart: unless-stopped
    command:
      - --config.file=/etc/prometheus/prometheus.yml
      - --storage.tsdb.path=/prometheus
      - --storage.tsdb.retention.time=30d
    volumes:
      - ./prometheus.yml:/etc/prometheus/prometheus.yml:ro
      - prometheus-data:/prometheus
    networks:
      # Queried by Grafana
      - prometheus-net
      # To be reachable by Traefik
      - traefik-private-net
      # To scrape the CCTeam API, shared with ccteam-app only (not with its database)
      - prometheus-ccteam-net

networks:

  prometheus-net:
    name: prometheus-net

  traefik-private-net:
    name: traefik-private-net
    external: true

  # Created by this stack, the ccteam stack joins it : start Prometheus first
  prometheus-ccteam-net:
    name: prometheus-ccteam-net

volumes:

  prometheus-data:

Configuration file

:page_facing_up: prometheus.yml :

global:
  scrape_interval: 15s
  evaluation_interval: 15s

scrape_configs:

  - job_name: prometheus
    static_configs:
      - targets: [ localhost:9090 ]

  # CCTeam GraphQL (Spring Boot)
  # The actuator listens on a dedicated management port (management.server.port=8081), not routed by Traefik,
  # so the metrics are never exposed to the internet. The servlet context-path does not apply on this port.
  - job_name: ccteam-graphql
    metrics_path: /actuator/prometheus
    static_configs:
      - targets: [ ccteam-app:8081 ]

Traefik routing

:page_facing_up: prometheus.yml (Traefik dynamic configuration) :

http:
  services:
    prometheus:
      loadBalancer:
        servers:
          - url: http://prometheus:9090

  routers:
    prometheus:
      rule: 'Host(`prometheus.example.com`)'
      entryPoints:
        - websecure
      tls:
        certResolver: default
      service: prometheus
      # Prometheus has no native OIDC support, the authentication is done by Traefik (see PocketID)
      middlewares:
        - vpn-whitelist@file
        - prometheus-auth@file

And the middleware, in pocketid.yml :

    prometheus-auth:
      plugin:
        traefik-oidc-auth:
          Secret: "<secret>"
          Provider:
            Url: "http://pocketid:1411/"
            ClientId: "<oidc_client_id>"
            ClientSecret: "<oidc_client_secret>"
            UsePkce: true
          Scopes: [ "openid", "profile", "email" ]

Things to notice :

the targets are reached by container name (ccteam-app:8081) on the shared network, nothing is published on the host
the data is kept 30 days (--storage.tsdb.retention.time), in the prometheus-data volume
--web.enable-lifecycle is deliberately not set : it would let anyone who reaches port 9090 stop Prometheus (POST /-/quit) without any authentication. To reload the configuration after a change, send it a signal instead : sudo docker kill -s HUP prometheus
the admin API (deleting series, snapshots) is not enabled either, the API only allows reading
Grafana does not go through Traefik nor through the PocketID middleware : it queries http://prometheus:9090 directly on prometheus-net
adding an application to monitor means : a scrape job in prometheus.yml, and a new prometheus-<app>-net network shared between Prometheus and that application only

Run

Prometheus creates the prometheus-net and prometheus-ccteam-net networks, so start it before CCTeam and Grafana, which join them as external networks :

sudo docker-compose -f /opt/apps/prometheus/docker-compose.yml up -d

You should end-up with a running prometheus container, and Traefik picks up the dynamic configuration file without restarting.

The web interface is available at https://prometheus.example.com, after the PocketID login. Check the Status -> Targets page : the ccteam-graphql job must be UP.

Then check that the metrics are not reachable from the internet : from a phone on mobile data (VPN turned off), https://ccteam.example.com/ccteam-gql/actuator/prometheus must answer 404.

Grafana

Grafana is the dashboard tool on top of Prometheus : it runs the PromQL queries and displays the results as graphs, tables, gauges, ... with alerting capabilities if needed.

Nothing is configured by hand : the Prometheus data source and the dashboards are provisioned from files at startup, so the whole configuration lives in this repository and a fresh container comes up ready to use.

Like the other administration tools it sits on the private network. Unlike Prometheus, it supports OIDC natively (generic OAuth), so it authenticates its users against PocketID itself, with roles mapped from the PocketID groups. The local login form is disabled : PocketID is the only way in.

Here is an overview of the network flow :

flowchart LR
    style INCOMING_REQUEST fill: #45d8ff57
    style TRAEFIK_CONTAINER fill: #6601013c
    style APP_CONTAINER fill: #6601013c
    style PROMETHEUS_CONTAINER fill: #6601013c
    style POCKETID_CONTAINER fill: #6601013c
    style TRAEFIK_ROUTER fill: #defc1e37
    style TRAEFIK_MIDDLEWARE fill: #defc1e37
    style SERVER_DEVICE fill: #fec9bf5e
    style CONTAINER_ENGINE fill: #67010030
    DOCKER_TRAEFIK_PORT443{{443/tcp}}
    DOCKER_TRAEFIK_PORT80{{80/tcp}}
    DOCKER_APP_PORT{{3000/tcp}}
    DOCKER_PROMETHEUS_PORT{{9090/tcp}}
    DOCKER_POCKETID_PORT{{1411/tcp}}
    TRAEFIK_ROUTER_APP(grafana)
    TRAEFIK_MIDDLEWARE_REDIRECT(HTTPS redirect on<br>the web entrypoint)
    TRAEFIK_MIDDLEWARE_IP_WHITELIST(IP whitelist)
    INCOMING_REQUEST((INCOMING\nREQUEST))

    subgraph SERVER_DEVICE[MINI PC]
        subgraph CONTAINER_ENGINE[DOCKER]
            subgraph TRAEFIK_CONTAINER[TRAEFIK CONTAINER]
                DOCKER_TRAEFIK_PORT443
                DOCKER_TRAEFIK_PORT80
                
                subgraph TRAEFIK_ROUTER[TRAEFIK HTTP ROUTER]
                    TRAEFIK_ROUTER_APP
                end

                subgraph TRAEFIK_MIDDLEWARE[TRAEFIK MIDDLEWARES]
                    TRAEFIK_MIDDLEWARE_REDIRECT
                    TRAEFIK_MIDDLEWARE_IP_WHITELIST
                end
            end

            subgraph APP_CONTAINER[GRAFANA CONTAINER]
                DOCKER_APP_PORT
            end

            subgraph PROMETHEUS_CONTAINER[PROMETHEUS CONTAINER]
                DOCKER_PROMETHEUS_PORT
            end

            subgraph POCKETID_CONTAINER[POCKETID CONTAINER]
                DOCKER_POCKETID_PORT
            end
        end
    end

    INCOMING_REQUEST --> DOCKER_TRAEFIK_PORT443
    INCOMING_REQUEST --> DOCKER_TRAEFIK_PORT80
    DOCKER_TRAEFIK_PORT80 ---> TRAEFIK_MIDDLEWARE_REDIRECT
    TRAEFIK_MIDDLEWARE_REDIRECT -.->|301 to https| INCOMING_REQUEST
    DOCKER_TRAEFIK_PORT443 --> TRAEFIK_ROUTER
    TRAEFIK_ROUTER_APP --> TRAEFIK_MIDDLEWARE_IP_WHITELIST
    TRAEFIK_MIDDLEWARE_IP_WHITELIST --> DOCKER_APP_PORT
    DOCKER_APP_PORT -->|PromQL queries,<br>prometheus - net| DOCKER_PROMETHEUS_PORT
    DOCKER_APP_PORT -.->|OIDC SSO, through<br>the Traefik alias| DOCKER_POCKETID_PORT

Setting up

Create a folder to hold the configuration :

sudo mkdir /opt/apps/grafana

Then :

copy the .env and docker-compose.yml files, and the provisioning and dashboards folders, from this project's grafana directory into the /opt/apps/grafana directory
copy the grafana.yml file from this project's traefik/dynamic directory into the /opt/apps/traefik/dynamic directory
create an OIDC client in PocketID with the callback URL of the application : https://grafana.example.com/login/generic_oauth, and PKCE enabled. Restrict it to your administrators group (Allowed user groups), then put its client ID and secret in OIDC_CLIENT_ID and OIDC_CLIENT_SECRET of the .env file. No middleware on the router : the application talks to PocketID itself
give the local admin account a strong random password in the .env file (openssl rand -base64 32), you will never have to type it
add a local DNS record grafana.example.com pointing to the mini PC (see Pi-hole), the service is not published on the internet

[!NOTE] The roles come from the PocketID groups, through GF_AUTH_GENERIC_OAUTH_ROLE_ATTRIBUTE_PATH : the members of super_admins are Grafana server admins, everybody else is a viewer.

Grafana has two levels of permissions : the organization roles (Viewer, Editor, Admin), which manage the dashboards, the data sources and the members of an organization, and the server admin, which manages the whole instance (organizations, all the users, server settings). The GrafanaAdmin value gives both, and requires GF_AUTH_GENERIC_OAUTH_ALLOW_ASSIGN_GRAFANA_ADMIN. Admin alone would not be enough here, since the local admin account is not usable anymore.

A few things to know about it :

the groups claim contains the name of the PocketID group, not its friendly name

the role is computed again at every login : a role changed from the Grafana interface is overwritten at the next login, PocketID is the source of truth

if the group does not match (wrong name, groups scope not allowed on the client), the login still works, but as a viewer, with no admin left to fix it : check the group before the first login

[!NOTE] Grafana always creates a local admin account on its first start, it cannot be removed. It is made unusable instead : no login form (GF_AUTH_DISABLE_LOGIN_FORM) and no basic authentication on the API (GF_AUTH_BASIC_ENABLED), otherwise admin:<password> would still open /api/.... GF_AUTH_OAUTH_AUTO_LOGIN redirects straight to PocketID, without an intermediate login page.

If the OIDC login ever breaks, comment these three lines, recreate the container, and log in with the local admin. Also note that GF_SECURITY_ADMIN_PASSWORD is only applied on the first start, it is then stored in the Grafana database.

[!NOTE] The token and user info endpoints are the public URLs of PocketID : Grafana calls them from its container, through the Traefik network alias and the pocketid-whitelist middleware described in PocketID.

Details

Service definition

:page_facing_up: docker-compose.yml :

services:

  grafana:
    image: grafana/grafana:latest
    container_name: grafana
    restart: unless-stopped
    environment:
      # Grafana always creates a local admin account on the first start, it cannot be removed.
      # It is unusable since the login form and basic authentication are disabled (see below), but it must
      # still have a strong password : it is only applied on the first start (then stored in the Grafana database)
      GF_SECURITY_ADMIN_USER: ${GRAFANA_ADMIN_USER:-admin}
      GF_SECURITY_ADMIN_PASSWORD: ${GRAFANA_ADMIN_PASSWORD:?set GRAFANA_ADMIN_PASSWORD in .env}
      GF_USERS_ALLOW_SIGN_UP: "false"
      GF_ANALYTICS_REPORTING_ENABLED: "false"
      GF_SERVER_ROOT_URL: https://grafana.example.com

      # Native OIDC authentication against PocketID (callback URL : https://grafana.example.com/login/generic_oauth)
      # The container resolves pocketid.example.com to Traefik thanks to the alias on traefik-private-net (see PocketID)
      GF_AUTH_GENERIC_OAUTH_ENABLED: "true"
      GF_AUTH_GENERIC_OAUTH_NAME: PocketID
      GF_AUTH_GENERIC_OAUTH_CLIENT_ID: ${OIDC_CLIENT_ID:?set OIDC_CLIENT_ID in .env}
      GF_AUTH_GENERIC_OAUTH_CLIENT_SECRET: ${OIDC_CLIENT_SECRET:?set OIDC_CLIENT_SECRET in .env}
      GF_AUTH_GENERIC_OAUTH_SCOPES: openid email profile groups
      GF_AUTH_GENERIC_OAUTH_AUTH_URL: https://pocketid.example.com/authorize
      GF_AUTH_GENERIC_OAUTH_TOKEN_URL: https://pocketid.example.com/api/oidc/token
      GF_AUTH_GENERIC_OAUTH_API_URL: https://pocketid.example.com/api/oidc/userinfo
      GF_AUTH_GENERIC_OAUTH_USE_PKCE: "true"
      GF_AUTH_GENERIC_OAUTH_ALLOW_SIGN_UP: "true"
      # Members of the PocketID group "super_admins" are Grafana server admins (the local admin is not usable),
      # everybody else is a viewer
      GF_AUTH_GENERIC_OAUTH_ROLE_ATTRIBUTE_PATH: "contains(groups[*], 'super_admins') && 'GrafanaAdmin' || 'Viewer'"
      GF_AUTH_GENERIC_OAUTH_ALLOW_ASSIGN_GRAFANA_ADMIN: "true"

      # PocketID only : no local login form, no basic authentication on the API, direct redirection to PocketID.
      # If the OIDC login is broken, comment these three lines temporarily to log in with the local admin
      GF_AUTH_DISABLE_LOGIN_FORM: "true"
      GF_AUTH_BASIC_ENABLED: "false"
      GF_AUTH_OAUTH_AUTO_LOGIN: "true"
    volumes:
      - ./provisioning:/etc/grafana/provisioning:ro
      - ./dashboards:/var/lib/grafana/dashboards:ro
      - grafana-data:/var/lib/grafana
    networks:
      # To query Prometheus directly, without going through Traefik
      - prometheus-net
      # To be reachable by Traefik, and to reach PocketID
      - traefik-private-net

networks:

  prometheus-net:
    name: prometheus-net
    external: true

  traefik-private-net:
    name: traefik-private-net
    external: true

volumes:

  grafana-data:

Environment variables

:page_facing_up: .env :

# Local admin account, unusable (no login form) but it must have a strong password : openssl rand -base64 32
GRAFANA_ADMIN_USER=<username>
GRAFANA_ADMIN_PASSWORD=<password>

# PocketID OIDC client
OIDC_CLIENT_ID=<oidc_client_id>
OIDC_CLIENT_SECRET=<oidc_client_secret>

Provisioning

:page_facing_up: provisioning/datasources/prometheus.yml :

apiVersion: 1

datasources:
  - name: Prometheus
    uid: prometheus
    type: prometheus
    access: proxy
    url: http://prometheus:9090
    isDefault: true
    editable: false

:page_facing_up: provisioning/dashboards/dashboards.yml :

apiVersion: 1

providers:
  - name: homelab
    folder: Homelab
    type: file
    allowUiUpdates: true
    options:
      path: /var/lib/grafana/dashboards

The dashboards folder holds the dashboards as JSON files, loaded into the Homelab folder of Grafana. The spring-graphql.json dashboard shows, for the selected application :

the GraphQL requests : requests per second, error rate, average and p95 response time, requests by outcome and operation type, p50 / p95 / p99 percentiles over time
the GraphQL fields (data fetchers) : the most called fields, the slowest ones (average and p95), calls, response time and errors per field

Traefik routing

:page_facing_up: grafana.yml :

http:
  services:
    grafana:
      loadBalancer:
        servers:
          - url: http://grafana:3000

  routers:
    grafana:
      rule: 'Host(`grafana.example.com`)'
      entryPoints:
        - websecure
      tls:
        certResolver: default
      service: grafana
      middlewares:
        - vpn-whitelist@file

Things to notice :

the router only carries the IP whitelist : the single sign-on is done by the application itself, adding an authentication middleware would mean logging in twice
the data source reaches Prometheus by container name on prometheus-net, with access: proxy : the queries are made by the Grafana server, the browser never talks to Prometheus
the data source has a fixed uid (prometheus), referenced by the dashboards : keep it if you import other dashboards, or adapt their JSON
editable: false locks the data source in the interface, the file is the only place to change it
with allowUiUpdates: true a provisioned dashboard can be modified and saved from the interface, but the change only lives in the Grafana database : to keep it, export the JSON (Share -> Export) and replace the file in the dashboards folder
the provisioning and dashboards folders are mounted read-only, the grafana-data volume holds the database (users, preferences, sessions)

Run

Prometheus must be started first, it creates the prometheus-net network (see Prometheus) :

sudo docker-compose -f /opt/apps/grafana/docker-compose.yml up -d

You should end-up with a running grafana container, and Traefik picks up the dynamic configuration file without restarting.

Open https://grafana.example.com : you are redirected straight to PocketID, then back to Grafana, where the Spring GraphQL dashboard is waiting in the Homelab folder. Check your role in your profile : it must be Grafana Admin.

Gatus

Gatus checks at a regular interval that the services answer, and sends an alert when one of them stops answering, and again when it comes back. Each check is an HTTP request, a DNS query or a TCP connection, validated by conditions (status code, body content, response time, days left on the TLS certificate, ...), and the history is displayed on a small status page. Where Prometheus and Grafana tell how well an application behaves, Gatus only answers is it up ?, and tells you as soon as it is not.

A single Go binary, configured with one YAML file, with its history in an SQLite database. It sits on the private network and authenticates its users against PocketID with its own OIDC support.

The checks do not all take the same path, to respect the Network segmentation :

the private services are checked directly, by container name on traefik-private-net, on their health endpoint when they have one
the public services are checked through Traefik, with their real public name : Gatus never joins traefik-public-net. Traefik carries these names as network aliases on the private network (like pocketid.example.com), so Gatus reaches it directly, without depending on Pi-hole nor on the NAT loopback of the router. It also covers the routing, the TLS certificate and CrowdSec
the DNS servers are queried on pihole-net, with their fixed IP addresses

Here is an overview of the network flow :

flowchart LR
    style INCOMING_REQUEST fill: #45d8ff57
    style TRAEFIK_CONTAINER fill: #6601013c
    style APP_CONTAINER fill: #6601013c
    style POCKETID_CONTAINER fill: #6601013c
    style PRIVATE_CONTAINER fill: #6601013c
    style PUBLIC_CONTAINER fill: #6601013c
    style DNS_CONTAINERS fill: #6601013c
    style TRAEFIK_ROUTER fill: #defc1e37
    style TRAEFIK_MIDDLEWARE fill: #defc1e37
    style SERVER_DEVICE fill: #fec9bf5e
    style CONTAINER_ENGINE fill: #67010030
    DOCKER_APP_PORT{{8080/tcp}}
    DOCKER_TRAEFIK_PORT443{{443/tcp}}
    DOCKER_TRAEFIK_PORT80{{80/tcp}}
    DOCKER_TRAEFIK_PORT8080{{8080/tcp\nping}}
    DOCKER_POCKETID_PORT{{1411/tcp}}
    DOCKER_PRIVATE_PORT{{health endpoint}}
    DOCKER_PUBLIC_PORT{{app port}}
    DOCKER_PIHOLE_PORT{{10.2.0.100:53}}
    DOCKER_UNBOUND_PORT{{10.2.0.200:53}}
    TRAEFIK_ROUTER_APP(gatus.example.com)
    TRAEFIK_MIDDLEWARE_REDIRECT(HTTPS redirect on<br>the web entrypoint)
    TRAEFIK_MIDDLEWARE_IP_WHITELIST(IP whitelist)
    INCOMING_REQUEST((INCOMING\nREQUEST))
    SMTP[SMTP server]
    ADMIN((ADMIN))

    subgraph SERVER_DEVICE[MINI PC]
        subgraph CONTAINER_ENGINE[DOCKER]
            subgraph APP_CONTAINER[GATUS CONTAINER]
                DOCKER_APP_PORT
            end

            subgraph TRAEFIK_CONTAINER[TRAEFIK CONTAINER]
                DOCKER_TRAEFIK_PORT443
                DOCKER_TRAEFIK_PORT80
                DOCKER_TRAEFIK_PORT8080

                subgraph TRAEFIK_ROUTER[TRAEFIK HTTP ROUTER]
                    TRAEFIK_ROUTER_APP
                end

                subgraph TRAEFIK_MIDDLEWARE[TRAEFIK MIDDLEWARES]
                    TRAEFIK_MIDDLEWARE_REDIRECT
                    TRAEFIK_MIDDLEWARE_IP_WHITELIST
                end
            end

            subgraph POCKETID_CONTAINER[POCKETID CONTAINER]
                DOCKER_POCKETID_PORT
            end

            subgraph PRIVATE_CONTAINER[PRIVATE SERVICES]
                DOCKER_PRIVATE_PORT
            end

            subgraph PUBLIC_CONTAINER[PUBLIC SERVICES]
                DOCKER_PUBLIC_PORT
            end

            subgraph DNS_CONTAINERS[PI-HOLE / UNBOUND]
                DOCKER_PIHOLE_PORT
                DOCKER_UNBOUND_PORT
            end

        end
    end

    INCOMING_REQUEST --> DOCKER_TRAEFIK_PORT443
    INCOMING_REQUEST --> DOCKER_TRAEFIK_PORT80
    DOCKER_TRAEFIK_PORT80 --> TRAEFIK_MIDDLEWARE_REDIRECT
    TRAEFIK_MIDDLEWARE_REDIRECT -.->|301 to https| INCOMING_REQUEST
    DOCKER_TRAEFIK_PORT443 --> TRAEFIK_ROUTER_APP
    TRAEFIK_ROUTER_APP --> TRAEFIK_MIDDLEWARE_IP_WHITELIST
    TRAEFIK_MIDDLEWARE_IP_WHITELIST -->|status page| DOCKER_APP_PORT
    DOCKER_APP_PORT -.->|OIDC SSO,<br>through the Traefik alias| DOCKER_POCKETID_PORT
    DOCKER_APP_PORT -->|traefik - private - net| DOCKER_TRAEFIK_PORT8080
    DOCKER_APP_PORT -->|by container name, traefik - private - net| DOCKER_PRIVATE_PORT
    DOCKER_APP_PORT -->|public name = Traefik alias| DOCKER_TRAEFIK_PORT443
    DOCKER_TRAEFIK_PORT443 -->|traefik - public - net| DOCKER_PUBLIC_PORT
    DOCKER_APP_PORT -->|DNS queries, pihole - net| DOCKER_PIHOLE_PORT
    DOCKER_APP_PORT -->|DNS queries, pihole - net| DOCKER_UNBOUND_PORT
    DOCKER_APP_PORT -.->|alert e - mail| SMTP
    SMTP -.-> ADMIN

Setting up

Create the folders :

sudo mkdir -p /opt/apps/gatus/config /opt/apps/gatus/data

Then :

copy the .env and docker-compose.yml files from this project's gatus directory into the /opt/apps/gatus directory, and the config.yaml file into /opt/apps/gatus/config. Adapt the domain names and the IP address of the mini PC in config.yaml, and fill the SMTP account used to send the alerts in the .env file
copy the gatus.yml file from this project's traefik/dynamic directory into the /opt/apps/traefik/dynamic directory
enable the ping endpoint in the Traefik static configuration (ping: {}), and add the public names to the network aliases of Traefik (see Configuration files details), then recreate the Traefik container
create an OIDC client in PocketID with the callback URL of the application : https://gatus.example.com/authorization-code/callback, and PKCE disabled, as the application does not send a code_challenge. Restrict it to your administrators group (Allowed user groups) : Gatus has no roles, every authenticated user sees everything. Then put its client ID and secret in OIDC_CLIENT_ID and OIDC_CLIENT_SECRET of the .env file. No middleware on the router : the application talks to PocketID itself
add a local DNS record gatus.example.com pointing to the mini PC (see Pi-hole), the service is not published on the internet

[!NOTE] Dashdot is not checked : it is started on demand by Sablier, a request every minute would wake it up and keep it running forever. The same goes for any service put behind Sablier.

[!NOTE] The DNS checks query Pi-hole and Unbound on pihole-net, not the port 53 published on the host : Gatus and Pi-hole share traefik-private-net, and a container reaching a port published by a container of the same Docker network (hairpin) is not reliable, the query times out or the answer comes from an unexpected address. Your devices on the local network are not affected, they come through the real network interface.

Checking Unbound directly, and not only through Pi-hole, tells which one of the two is down.

[!TIP] An application without a dedicated health endpoint can still be checked on any page, but avoid a page that logs every call : a check every minute is 1440 lines a day. This is why the CCTeam API is checked on its readiness probe rather than on its GraphQL endpoint, where every request without a token wrote four lines of authentication logs. Spring Boot serves the liveness and readiness probes on the main port (/livez, /readyz) with these properties, while the rest of the actuator stays on the management port (see Prometheus) :
management.endpoint.health.probes.enabled=              true
management.endpoint.health.probes.add-additional-paths= true
management.endpoint.health.group.readiness.include=     readinessState,db
/readyz answers 503 if the application or its database is not ready, and only exposes {"status":"UP"}, without any detail. No Spring Security filter chain matches this path, so it is reachable without a token.

Details

Service definition

:page_facing_up: docker-compose.yml :

services:

  gatus:
    image: twinproduction/gatus:latest
    container_name: gatus
    restart: unless-stopped
    # OIDC client and SMTP credentials, substituted into config/config.yaml (see .env)
    env_file: .env
    environment:
      TZ: "Europe/Zurich"
    volumes:
      # Endpoints, alerting and OIDC configuration
      - ./config:/config:ro
      # SQLite database : history of the checks, so that the uptime survives a restart
      - ./data:/data
    networks:
      # Private network : the private services are checked by container name, the public ones through Traefik
      # (their public names resolve to Traefik thanks to its network aliases), and PocketID is reached the same way
      - traefik-private-net
      # To query Pi-hole and Unbound directly on their fixed IP addresses (DNS checks)
      - pihole-net

networks:

  traefik-private-net:
    name: traefik-private-net
    external: true

  pihole-net:
    name: pihole-net
    external: true

Environment variables

:page_facing_up: .env :

# PocketID OIDC client (callback URL : https://gatus.example.com/authorization-code/callback)
OIDC_CLIENT_ID=<oidc_client_id>
OIDC_CLIENT_SECRET=<oidc_client_secret>

# SMTP account used to send the alerts
SMTP_HOST=<smtp_host>
SMTP_PORT=465
SMTP_USERNAME=<smtp_username>
SMTP_PASSWORD=<smtp_password>
ALERT_FROM=<sender_address>
ALERT_TO=<recipient_address>

Configuration file

:page_facing_up: config/config.yaml (shortened, one endpoint of each kind, see the full file in this project's gatus directory) :

# Gatus configuration
# The ${...} variables come from the .env file : Gatus substitutes the environment variables in the whole file
# before parsing it, so never write a literal "$" here.

storage:
  type: sqlite
  path: /data/data.db

ui:
  title: Status | example.com
  header: Status

# Native OIDC authentication against PocketID, protects the dashboard and the API.
# The issuer is the PUBLIC URL : the container resolves it to Traefik thanks to the alias on traefik-private-net,
# and Traefik lets it through with the pocketid-whitelist middleware (see PocketID)
security:
  oidc:
    issuer-url: https://pocketid.example.com
    redirect-url: https://gatus.example.com/authorization-code/callback
    client-id: ${OIDC_CLIENT_ID}
    client-secret: ${OIDC_CLIENT_SECRET}
    scopes: [ "openid" ]

alerting:
  email:
    from: ${ALERT_FROM}
    username: ${SMTP_USERNAME}
    password: ${SMTP_PASSWORD}
    host: ${SMTP_HOST}
    port: ${SMTP_PORT}
    to: ${ALERT_TO}
    # Applied to every endpoint declaring an "email" alert
    default-alert:
      enabled: true
      # 3 failed checks in a row (3 minutes) before alerting, to ignore a restart or a short hiccup
      failure-threshold: 3
      success-threshold: 2
      send-on-resolved: true

endpoints:

  # Needs "ping: {}" in the Traefik static configuration, served on the internal "traefik" entrypoint (8080)
  - name: Traefik
    group: Infrastructure
    url: http://traefik:8080/ping
    interval: 1m
    conditions:
      - "[STATUS] == 200"
    alerts:
      - type: email

  # Local API of the security engine : a TCP check is enough, the bouncer in Traefik fails open if it is down
  - name: CrowdSec
    group: Infrastructure
    url: tcp://crowdsec:8080
    interval: 1m
    conditions:
      - "[CONNECTED] == true"
    alerts:
      - type: email

  # DNS servers queried on pihole-net with their fixed IP addresses. Not through the port 53 published on the host :
  # a container reaching a port published by a container of the same Docker network (hairpin) is not reliable.
  # A local record checks Pi-hole itself...
  - name: Pi-hole (DNS)
    group: Infrastructure
    url: 10.2.0.100
    interval: 1m
    dns:
      query-name: pihole.example.com
      query-type: A
    conditions:
      - "[DNS_RCODE] == NOERROR"
      - "[BODY] == 192.168.0.16"
    alerts:
      - type: email

  # ... and Unbound is asked directly for an external name, which checks the recursive resolution
  - name: Unbound
    group: Infrastructure
    url: 10.2.0.200
    interval: 1m
    dns:
      query-name: debian.org
      query-type: A
    conditions:
      - "[DNS_RCODE] == NOERROR"
    alerts:
      - type: email

  # Private services : checked by container name on traefik-private-net
  # Same endpoint as the "arcane health" command of its Docker health check
  - name: Arcane
    group: Private
    url: http://arcane:3552/api/health
    method: HEAD
    interval: 1m
    conditions:
      - "[STATUS] == 200"
    alerts:
      - type: email

  # Dashdot is NOT checked : it is started on demand by Sablier, a check every minute would keep it running forever

  # Public services : checked through Traefik with their public name (network alias of Traefik), which also covers
  # the routing, the TLS certificate and CrowdSec. Gatus is never on traefik-public-net.
  - name: Lychee
    group: Public
    url: https://lychee.example.com
    interval: 1m
    conditions:
      - "[STATUS] == 200"
      # Let's Encrypt renews 30 days before expiry : less than 10 days left means the renewal is broken
      - "[CERTIFICATE_EXPIRATION] > 240h"
    alerts:
      - type: email

  # Readiness probe served on the main port (the rest of the actuator is on the management port), it includes the
  # database : 503 if the application or its database is not ready. Only exposes {"status":"UP"}, no details
  - name: CCTeam API
    group: Public
    url: https://ccteam.example.com/ccteam-gql/readyz
    interval: 1m
    conditions:
      - "[STATUS] == 200"
      - "[BODY].status == UP"
      - "[CERTIFICATE_EXPIRATION] > 240h"
    alerts:
      - type: email

The full file checks :

Group	Services	How
Infrastructure	Traefik, PocketID, CrowdSec, Pi-hole (DNS), Unbound, WGDashboard	ping, discovery document, TCP, DNS queries, host IP
Private	Pi-hole, Arcane, Backrest, CrowdSec Web UI, PhpMyAdmin, Homer, Homebox, Prometheus, Grafana, Ghostfolio, Homelable, Speedtest Tracker	container name, health endpoint when there is one
Public	Lychee, Defrag-life, GoatCounter, CCTeam API, Medusa API, Medusa storefront	public name through Traefik, TLS certificate expiry

Traefik routing

:page_facing_up: gatus.yml :

http:
  services:
    gatus:
      loadBalancer:
        servers:
          - url: http://gatus:8080

  routers:
    gatus:
      rule: 'Host(`gatus.example.com`)'
      entryPoints:
        - websecure
      tls:
        certResolver: default
      service: gatus
      # Only the IP whitelist : the application handles the PocketID single sign-on itself (native OIDC),
      # so no authentication middleware here, otherwise you would log in twice
      middlewares:
        - vpn-whitelist@file

Things to notice :

Gatus joins traefik-private-net and pihole-net only, never traefik-public-net : the public services are reached through Traefik, like any visitor would
a dedicated network shared with Pi-hole only would bring nothing : pihole-net only holds Pi-hole and Unbound, the two containers Gatus has to query, and Gatus already shares traefik-private-net with Pi-hole
WGDashboard runs on the host network (network_mode: host), it is reached through the IP address of the mini PC, without any Docker port mapping, so the hairpin problem of the DNS checks does not apply
the CrowdSec local API is checked with a simple TCP connection : if it is down, the bouncer in Traefik lets the requests through, the protection silently disappears, hence the check
the GoatCounter check fetches /count.js, which does not record any visit. Gatus does not run the JavaScript of the pages it checks, so it does not pollute the statistics of the websites either
the Medusa storefront has no health endpoint, its check loads the home page. It costs little, the data of the page comes from the cache of Next.js, but for the same reason it does not see an outage of the backend : the Medusa API check does
the public checks also verify the TLS certificate : less than 10 days left means that the Let's Encrypt renewal, which normally happens 30 days before expiry, is broken
Gatus substitutes the environment variables in the whole configuration file (${...}), a literal $ would be interpreted as a variable. The .env values themselves are not affected
Gatus watches its configuration file and reloads it by itself when it changes, no restart needed

[!WARNING] Gatus checks from inside the network : it does not see an outage of the internet connection, of the router, of the dynamic DNS or of the port forwarding, and in that case the alert e-mail could not leave anyway. Catching these needs an external probe, from another network (a second Gatus instance hosted elsewhere, or a free external uptime service) checking a public URL such as https://quake.example.com.

Run

Traefik must have been recreated with the ping endpoint and the aliases first (see above), then run the Compose file :

sudo docker-compose -f /opt/apps/gatus/docker-compose.yml up -d

You should end-up with a running gatus container, and Traefik picks up the dynamic configuration file without restarting.

The status page is available at https://gatus.example.com, after the PocketID login. Every check should turn green within a minute. Hover a red bar to see the failed condition, for instance [BODY] () == 192.168.0.16 : an empty answer, the query name does not match any local DNS record.

Ghostfolio

Ghostfolio is a wealth management application : it tracks stocks, ETFs, cryptocurrencies, savings accounts, ... across several accounts and brokers, with the performance over time, the allocation by asset class, region or currency, the dividends, and the fees.

It runs as three containers : the application, a PostgreSQL database, and Redis (cache and job queue, nothing to persist). It sits on the private network and authenticates its users against PocketID with its own OIDC support (still flagged experimental by Ghostfolio at the time of writing).

Here is an overview of the network flow :

flowchart LR
    style INCOMING_REQUEST fill: #45d8ff57
    style TRAEFIK_CONTAINER fill: #6601013c
    style APP_CONTAINER fill: #6601013c
    style DB_CONTAINER fill: #6601013c
    style REDIS_CONTAINER fill: #6601013c
    style POCKETID_CONTAINER fill: #6601013c
    style TRAEFIK_ROUTER fill: #defc1e37
    style TRAEFIK_MIDDLEWARE fill: #defc1e37
    style SERVER_DEVICE fill: #fec9bf5e
    style CONTAINER_ENGINE fill: #67010030
    DOCKER_TRAEFIK_PORT443{{443/tcp}}
    DOCKER_TRAEFIK_PORT80{{80/tcp}}
    DOCKER_APP_PORT{{3333/tcp}}
    DOCKER_DB_PORT{{5432/tcp}}
    DOCKER_REDIS_PORT{{6379/tcp}}
    DOCKER_POCKETID_PORT{{1411/tcp}}
    TRAEFIK_ROUTER_APP(ghostfolio)
    TRAEFIK_MIDDLEWARE_REDIRECT(HTTPS redirect on<br>the web entrypoint)
    TRAEFIK_MIDDLEWARE_IP_WHITELIST(IP whitelist)
    MARKET_DATA[Market data providers]
    INCOMING_REQUEST((INCOMING\nREQUEST))

    subgraph SERVER_DEVICE[MINI PC]
        subgraph CONTAINER_ENGINE[DOCKER]
            subgraph TRAEFIK_CONTAINER[TRAEFIK CONTAINER]
                DOCKER_TRAEFIK_PORT443
                DOCKER_TRAEFIK_PORT80
                
                subgraph TRAEFIK_ROUTER[TRAEFIK HTTP ROUTER]
                    TRAEFIK_ROUTER_APP
                end

                subgraph TRAEFIK_MIDDLEWARE[TRAEFIK MIDDLEWARES]
                    TRAEFIK_MIDDLEWARE_REDIRECT
                    TRAEFIK_MIDDLEWARE_IP_WHITELIST
                end

            end

            subgraph APP_CONTAINER[GHOSTFOLIO CONTAINER]
                DOCKER_APP_PORT
            end

            subgraph DB_CONTAINER[POSTGRESQL CONTAINER]
                DOCKER_DB_PORT
            end

            subgraph REDIS_CONTAINER[REDIS CONTAINER]
                DOCKER_REDIS_PORT
            end

            subgraph POCKETID_CONTAINER[POCKETID CONTAINER]
                DOCKER_POCKETID_PORT
            end
        end
    end

    INCOMING_REQUEST --> DOCKER_TRAEFIK_PORT443
    INCOMING_REQUEST --> DOCKER_TRAEFIK_PORT80
    DOCKER_TRAEFIK_PORT80 ---> TRAEFIK_MIDDLEWARE_REDIRECT
    TRAEFIK_MIDDLEWARE_REDIRECT -.->|301 to https| INCOMING_REQUEST
    DOCKER_TRAEFIK_PORT443 --> TRAEFIK_ROUTER_APP
    TRAEFIK_ROUTER_APP --> TRAEFIK_MIDDLEWARE_IP_WHITELIST
    TRAEFIK_MIDDLEWARE_IP_WHITELIST --> DOCKER_APP_PORT
    DOCKER_APP_PORT -->|ghostfolio - net| DOCKER_DB_PORT
    DOCKER_APP_PORT -->|ghostfolio - net| DOCKER_REDIS_PORT
    DOCKER_APP_PORT -.->|OIDC SSO, through<br>the Traefik alias| DOCKER_POCKETID_PORT
    DOCKER_APP_PORT -.->|quotes, exchange rates| MARKET_DATA

Setting up

Create a folder to hold the configuration :

sudo mkdir /opt/apps/ghostfolio

Then :

copy the .env and docker-compose.yml files from this project's ghostfolio directory into the /opt/apps/ghostfolio directory, and generate every secret of the .env file with openssl rand -hex 32 (hexadecimal on purpose : the database password goes into the DATABASE_URL, special characters would have to be URL-encoded)
copy the ghostfolio.yml file from this project's traefik/dynamic directory into the /opt/apps/traefik/dynamic directory
create an OIDC client in PocketID with the callback URL of the application : https://ghostfolio.example.com/api/auth/oidc/callback, and PKCE disabled, as the application does not send a code_challenge. Restrict it to your administrators group (Allowed user groups), then put its client ID and secret in OIDC_CLIENT_ID and OIDC_CLIENT_SECRET of the .env file
add a local DNS record ghostfolio.example.com pointing to the mini PC (see Pi-hole), the service is not published on the internet
add the /sources/volumes/ghostfolio-db-vol exclude to the backup plan : the PostgreSQL database is dumped every evening, see Database dumps

[!IMPORTANT] The first user created gets the ADMIN role : sign in through PocketID right after the first start, before anybody else. Then disable the sign-up in Admin Control -> Settings (User Signup).

Details

Service definition

:page_facing_up: docker-compose.yml :

services:

  ghostfolio:
    image: docker.io/ghostfolio/ghostfolio:latest
    container_name: ghostfolio
    restart: unless-stopped
    init: true
    # Hardening of the official Compose file : no Linux capability, no privilege escalation
    cap_drop:
      - ALL
    security_opt:
      - no-new-privileges:true
    # Database and Redis credentials, secrets, OIDC client (see .env)
    env_file: .env
    environment:
      TZ: "Europe/Zurich"
      # Public URL, used to build the OIDC callback URL (${ROOT_URL}/api/auth/oidc/callback)
      ROOT_URL: https://ghostfolio.example.com

      # Native OIDC authentication against PocketID (experimental in Ghostfolio).
      # The issuer is the PUBLIC URL : the container resolves it to Traefik thanks to the alias on traefik-private-net,
      # and Traefik lets it through with the pocketid-whitelist middleware (see PocketID)
      ENABLE_FEATURE_AUTH_OIDC: "true"
      OIDC_ISSUER: https://pocketid.example.com
      # OIDC_CLIENT_ID / OIDC_CLIENT_SECRET come from the .env file
    depends_on:
      ghostfolio-db:
        condition: service_healthy
      ghostfolio-redis:
        condition: service_healthy
    healthcheck:
      test: [ "CMD-SHELL", "curl -f http://localhost:3333/api/v1/health" ]
      interval: 30s
      timeout: 5s
      retries: 5
    networks:
      - ghostfolio-net
      - traefik-private-net

  ghostfolio-db:
    # Major version pinned : a new major version of PostgreSQL cannot start on the data of the previous one,
    # the upgrade (dump and restore) is done on purpose, never by a pull
    image: docker.io/library/postgres:15-alpine
    container_name: ghostfolio-db
    restart: unless-stopped
    cap_drop:
      - ALL
    cap_add:
      - CHOWN
      - DAC_READ_SEARCH
      - FOWNER
      - SETGID
      - SETUID
    security_opt:
      - no-new-privileges:true
    env_file: .env
    healthcheck:
      test: [ "CMD-SHELL", "pg_isready -d \"$${POSTGRES_DB}\" -U $${POSTGRES_USER}" ]
      interval: 10s
      timeout: 5s
      retries: 5
    volumes:
      - ghostfolio-db-vol:/var/lib/postgresql/data
    networks:
      - ghostfolio-net

  # Cache and job queue only, nothing to persist
  ghostfolio-redis:
    image: docker.io/library/redis:alpine
    container_name: ghostfolio-redis
    restart: unless-stopped
    user: "999:1000"
    cap_drop:
      - ALL
    security_opt:
      - no-new-privileges:true
    env_file: .env
    command:
      - /bin/sh
      - -c
      - redis-server --requirepass "$${REDIS_PASSWORD:?REDIS_PASSWORD variable is not set}"
    healthcheck:
      test: [ "CMD-SHELL", "redis-cli --pass \"$${REDIS_PASSWORD}\" ping | grep PONG" ]
      interval: 10s
      timeout: 5s
      retries: 5
    networks:
      - ghostfolio-net

volumes:

  ghostfolio-db-vol:
    name: ghostfolio-db-vol

networks:

  # The database and Redis stay on the private network of the stack, never on a Traefik network
  ghostfolio-net:
    name: ghostfolio-net

  traefik-private-net:
    name: traefik-private-net
    external: true

Environment variables

:page_facing_up: .env :

# Generate every secret with : openssl rand -hex 32
# (hexadecimal on purpose : the database password goes into a URL, special characters would have to be URL-encoded)

# Redis (cache)
REDIS_HOST=ghostfolio-redis
REDIS_PORT=6379
REDIS_PASSWORD=<redis_password>

# PostgreSQL
POSTGRES_DB=ghostfolio-db
POSTGRES_USER=ghostfolio
POSTGRES_PASSWORD=<postgres_password>
DATABASE_URL=postgresql://${POSTGRES_USER}:${POSTGRES_PASSWORD}@ghostfolio-db:5432/${POSTGRES_DB}?connect_timeout=300

# Secrets of the application
ACCESS_TOKEN_SALT=<random_string>
JWT_SECRET_KEY=<random_string>

# PocketID OIDC client (callback URL : https://ghostfolio.example.com/api/auth/oidc/callback)
OIDC_CLIENT_ID=<oidc_client_id>
OIDC_CLIENT_SECRET=<oidc_client_secret>

Traefik routing

:page_facing_up: ghostfolio.yml :

http:
  services:
    ghostfolio:
      loadBalancer:
        servers:
          - url: http://ghostfolio:3333

  routers:
    ghostfolio:
      rule: 'Host(`ghostfolio.example.com`)'
      entryPoints:
        - websecure
      tls:
        certResolver: default
      service: ghostfolio
      # Only the IP whitelist : the application handles the PocketID single sign-on itself (native OIDC),
      # so no authentication middleware here, otherwise you would log in twice
      middlewares:
        - vpn-whitelist@file

Things to notice :

the database and Redis only join ghostfolio-net, the private network of the stack : only the application reaches them, and the application alone joins traefik-private-net
the hardening of the official Compose file is kept : no Linux capability (only the few PostgreSQL needs to manage its files), no privilege escalation, Redis running as a non-root user
the major version of PostgreSQL is pinned (15-alpine) : unlike MariaDB, which upgrades its system tables at startup (MARIADB_AUTO_UPGRADE), PostgreSQL refuses to start on the data of a previous major version. The upgrade is a dump and a restore, done on purpose
Redis only holds a cache and the queue of the data gathering jobs : no volume, it starts empty after a restart
the containers wait for the database and Redis to be healthy (depends_on with condition: service_healthy) before starting the application, which would otherwise fail its first connection
the application needs an outgoing internet access to fetch the quotes and exchange rates
Gatus checks the /api/v1/health endpoint, which answers {"status":"OK"} without authentication

[!NOTE] The OIDC support is still experimental in Ghostfolio. The native method remains available : Get Started creates an account identified by a security token, which is then its only credential (keep it in your password manager). If the OIDC login ever breaks, putting the Traefik OIDC plugin in front of the router (like for Prometheus) keeps the application behind PocketID.

Run

Simply run the Compose file :

sudo docker-compose -f /opt/apps/ghostfolio/docker-compose.yml up -d

You should end-up with three running containers, ghostfolio, ghostfolio-db and ghostfolio-redis, and Traefik picks up the dynamic configuration file without restarting.

The application is available at https://ghostfolio.example.com : Sign in -> Sign in with OpenID Connect, then disable the sign-up as described above.

Homelable

Homelable draws the map of the homelab : an interactive diagram of the machines, devices and services with their live status (ping, HTTP, TCP, ...), a rack view for the physical layout, and a Markdown space to document the whole thing. A network scanner (nmap) speeds up the inventory : it finds the hosts and their open ports, and they only have to be placed on the canvas.

It runs as two containers : a backend (API, scanner, status checks, SQLite database) and a frontend (the interface, whose nginx proxies /api to the backend). It sits on the private network and authenticates its users against PocketID with its own OIDC support, in exclusive mode : there is no local password at all.

Here is an overview of the network flow :

flowchart LR
    style INCOMING_REQUEST fill: #45d8ff57
    style TRAEFIK_CONTAINER fill: #6601013c
    style FRONTEND_CONTAINER fill: #6601013c
    style BACKEND_CONTAINER fill: #6601013c
    style POCKETID_CONTAINER fill: #6601013c
    style PRIVATE_CONTAINER fill: #6601013c
    style TRAEFIK_ROUTER fill: #defc1e37
    style TRAEFIK_MIDDLEWARE fill: #defc1e37
    style SERVER_DEVICE fill: #fec9bf5e
    style CONTAINER_ENGINE fill: #67010030
    DOCKER_TRAEFIK_PORT443{{443/tcp}}
    DOCKER_TRAEFIK_PORT80{{80/tcp}}
    DOCKER_FRONTEND_PORT{{80/tcp}}
    DOCKER_BACKEND_PORT{{8000/tcp}}
    DOCKER_POCKETID_PORT{{1411/tcp}}
    DOCKER_PRIVATE_PORT{{service port}}
    TRAEFIK_ROUTER_APP(homelable.example.com)
    TRAEFIK_MIDDLEWARE_REDIRECT(HTTPS redirect on<br>the web entrypoint)
    TRAEFIK_MIDDLEWARE_IP_WHITELIST(IP whitelist)
    LAN[Local network\n192.168.0.0/24]
    INCOMING_REQUEST((INCOMING\nREQUEST))

    subgraph SERVER_DEVICE[MINI PC]
        subgraph CONTAINER_ENGINE[DOCKER]
            subgraph TRAEFIK_CONTAINER[TRAEFIK CONTAINER]

                subgraph TRAEFIK_ROUTER[TRAEFIK HTTP ROUTER]
                    TRAEFIK_ROUTER_APP
                end

                subgraph TRAEFIK_MIDDLEWARE[TRAEFIK MIDDLEWARES]
                    TRAEFIK_MIDDLEWARE_REDIRECT
                    TRAEFIK_MIDDLEWARE_IP_WHITELIST
                end
            end

            subgraph FRONTEND_CONTAINER[FRONTEND CONTAINER]
                DOCKER_FRONTEND_PORT
            end

            subgraph BACKEND_CONTAINER[BACKEND CONTAINER]
                DOCKER_BACKEND_PORT
            end

            subgraph POCKETID_CONTAINER[POCKETID CONTAINER]
                DOCKER_POCKETID_PORT
            end

            subgraph PRIVATE_CONTAINER[PRIVATE SERVICES]
                DOCKER_PRIVATE_PORT
            end
        end
    end

   INCOMING_REQUEST --> DOCKER_TRAEFIK_PORT443
   INCOMING_REQUEST --> DOCKER_TRAEFIK_PORT80
   DOCKER_TRAEFIK_PORT80 --> TRAEFIK_MIDDLEWARE_REDIRECT
   TRAEFIK_MIDDLEWARE_REDIRECT -.->|301 to https| INCOMING_REQUEST
   DOCKER_TRAEFIK_PORT443 --> TRAEFIK_ROUTER_APP
   TRAEFIK_ROUTER_APP --> TRAEFIK_MIDDLEWARE_IP_WHITELIST
   TRAEFIK_MIDDLEWARE_IP_WHITELIST --> DOCKER_FRONTEND_PORT
   DOCKER_FRONTEND_PORT -->|/api, homelable - net| DOCKER_BACKEND_PORT
   DOCKER_BACKEND_PORT -.->|OIDC SSO,<br>through the Traefik alias| DOCKER_POCKETID_PORT
   DOCKER_BACKEND_PORT -->|status checks, scan<br>traefik - private - net| DOCKER_PRIVATE_PORT
   DOCKER_BACKEND_PORT -->|nmap scan| LAN

Setting up

Create a folder to hold the configuration :

sudo mkdir /opt/apps/homelable

Then :

copy the .env and docker-compose.yml files from this project's homelable directory into the /opt/apps/homelable directory, generate the SECRET_KEY of the .env file (openssl rand -hex 32, at least 32 bytes in OIDC mode), and adapt the scanned ranges (see below)
copy the homelable.yml file from this project's traefik/dynamic directory into the /opt/apps/traefik/dynamic directory
create an OIDC client in PocketID with the callback URL of the application : https://homelable.example.com/api/v1/auth/oidc/callback, and PKCE disabled. Restrict it to your administrators group (Allowed user groups) : this is mandatory here, Homelable has no roles, anybody PocketID lets in has a full access. Then put its client ID and secret in OIDC_CLIENT_ID and OIDC_CLIENT_SECRET of the .env file. No middleware on the router : the application talks to PocketID itself
add a local DNS record homelable.example.com pointing to the mini PC (see Pi-hole), the service is not published on the internet

[!NOTE] Scanning the containers : the containers are not on the local network but on the internal networks of Docker (172.x.x.x), behind the mini PC. A scan of 192.168.0.0/24 only finds the physical devices (and the mini PC with its published ports), never the containers. Homelable has no Docker integration (no import through the Docker socket), the scanner can only reach the networks the backend is attached to. This is why SCANNER_RANGES also holds the subnet of traefik-private-net (docker network inspect traefik-private-net | grep Subnet), as a /24 : Docker hands out the addresses from the start of the range, and scanning the whole /16 (65 000 addresses) would take ages.

The public services (on traefik-public-net) and the databases (on the private network of their stack) stay out of reach, on purpose : the backend joining them would break the Network segmentation. Add them to the map by hand, under the mini PC.

[!TIP] For the status checks of the containers, use their container name (http://grafana:3000, tcp://crowdsec:8080, ...) rather than their IP address, which changes when the container is recreated. A check on the public URL of a private service (https://grafana.example.com) answers 403 : the backend reaches Traefik with a Docker address, refused by vpn-whitelist. The public services can be checked on their public URL, their routers have no whitelist.

Details

Service definition

:page_facing_up: docker-compose.yml :

services:

  backend:
    image: ghcr.io/pouzor/homelable-backend:latest
    container_name: homelable-backend
    restart: unless-stopped
    # Secrets, OIDC client and scanner settings (see .env)
    env_file: .env
    environment:
      TZ: "Europe/Zurich"
      SQLITE_PATH: /app/data/homelab.db
      # The browser origin only : wildcard CORS is rejected in OIDC mode
      CORS_ORIGINS: '["https://homelable.example.com"]'

      # Native OIDC authentication against PocketID, exclusive : no local password in this mode.
      # The discovery URL is the PUBLIC one : the container resolves it to Traefik thanks to the alias on
      # traefik-private-net, and Traefik lets it through with the pocketid-whitelist middleware (see PocketID)
      AUTH_MODE: oidc
      OIDC_DISCOVERY_URL: https://pocketid.example.com/.well-known/openid-configuration
      OIDC_REDIRECT_URI: https://homelable.example.com/api/v1/auth/oidc/callback
      OIDC_SCOPES: "openid profile email"
      OIDC_COOKIE_SECURE: "true"
      # OIDC_CLIENT_ID / OIDC_CLIENT_SECRET come from the .env file
    volumes:
      # SQLite database and uploaded media (floor plans)
      - homelable-data-vol:/app/data
    # Raw sockets for nmap (host discovery, port scan)
    cap_add:
      - NET_RAW
    networks:
      # Reached by the frontend only
      - homelable-net
      # To reach PocketID through the Traefik alias, and the private services by container name for the status checks
      - traefik-private-net

  frontend:
    image: ghcr.io/pouzor/homelable-frontend:latest
    container_name: homelable-frontend
    restart: unless-stopped
    environment:
      # nginx of the frontend proxies /api to the backend. Container name rather than the default "backend" : a service
      # name is also resolvable on traefik-private-net, where another stack could use the same one
      BACKEND_UPSTREAM: "homelable-backend:8000"
    depends_on:
      - backend
    networks:
      - homelable-net
      # To be reachable by Traefik
      - traefik-private-net

  # The MCP server (AI assistants integration) is not deployed. To add it, see the official docker-compose.prebuilt.yml

volumes:

  homelable-data-vol:
    name: homelable-data-vol

networks:

  homelable-net:
    name: homelable-net

  traefik-private-net:
    name: traefik-private-net
    external: true

Environment variables

:page_facing_up: .env :

# Signs the sessions, at least 32 bytes in OIDC mode : openssl rand -hex 32
SECRET_KEY=<secret_key>

# PocketID OIDC client (callback URL : https://homelable.example.com/api/v1/auth/oidc/callback)
OIDC_CLIENT_ID=<oidc_client_id>
OIDC_CLIENT_SECRET=<oidc_client_secret>

# Scanner : JSON array of the CIDR ranges to scan. The local network, and traefik-private-net to find the private
# containers (check its subnet with : docker network inspect traefik-private-net). A /24 only : Docker hands out the
# addresses from the start of the range, scanning the whole /16 (65 000 addresses) would take ages
SCANNER_RANGES=["192.168.0.0/24","172.21.0.0/24"]
SCANNER_HTTP_RANGES=[]
SCANNER_HTTP_PROBE_ENABLED=false
SCANNER_HTTP_VERIFY_TLS=false

# Status checker interval, in seconds
STATUS_CHECKER_INTERVAL=60

Traefik routing

:page_facing_up: homelable.yml :

http:
  services:
    homelable:
      loadBalancer:
        servers:
          # The frontend serves the interface and proxies /api to the backend
          - url: http://homelable-frontend:80

  routers:
    homelable:
      rule: 'Host(`homelable.example.com`)'
      entryPoints:
        - websecure
      tls:
        certResolver: default
      service: homelable
      # Only the IP whitelist : the application handles the PocketID single sign-on itself (native OIDC),
      # so no authentication middleware here, otherwise you would log in twice
      middlewares:
        - vpn-whitelist@file

Things to notice :

Traefik only reaches the frontend, the backend is never routed : the nginx of the frontend proxies /api to it
the backend also joins traefik-private-net : it needs to reach PocketID through the Traefik alias (the OIDC discovery URL is the public one), and it lets the status checks reach the private services by container name
BACKEND_UPSTREAM targets the container name rather than the default service name backend : a service name is resolvable on every network the container joins, another stack using the same name on traefik-private-net would get the requests
the backend gets the NET_RAW capability only, enough for the nmap host discovery and port scans. It stays on a bridge network : the scan cannot see the MAC addresses (layer 2), so a device that changes its IP address through DHCP is not recognized. The host network would fix it, at the cost of exposing the port 8000 of the backend on the local network
CORS_ORIGINS is restricted to the public URL, a wildcard is rejected in OIDC mode, and the session cookie is Secure
the MCP server of the official Compose file (AI assistants integration) is not deployed
the SQLite database lives in the homelable-data-vol volume, part of the Backrest backup like the other volumes

Run

Simply run the Compose file :

sudo docker-compose -f /opt/apps/homelable/docker-compose.yml up -d

You should end-up with two running containers, homelable-backend and homelable-frontend, and Traefik picks up the dynamic configuration file without restarting.

The application is available at https://homelable.example.com, after the PocketID login.

Medusa

Medusa is an open source e-commerce platform (Node.js / TypeScript) : products and variants, stock, prices per region and currency, carts, orders, customers, promotions, ..., managed in an admin dashboard and exposed through a store API to a separate storefront. I am evaluating it to build an online shop for a friend.

Unlike the other applications of this guide, Medusa does not come as a ready-made image : a shop is a project, code that you customize, then build. The starting point is the official DTC starter (https://github.com/medusajs/dtc-starter), a monorepo holding the backend (apps/backend) and a Next.js storefront (apps/storefront), from which two images are built on the mini PC.

It runs as four containers :

the backend : the store API, the admin dashboard (served under /app) and its API, and the background jobs (events, scheduled jobs, workflows), in a single process
the storefront : the shop itself, a Next.js server
a PostgreSQL database
Redis : event bus, workflow engine, locks and sessions of the dashboard

The shop is exposed to the internet, on the public network, with two hosts : shop.example.com for the storefront and medusa.example.com for the backend. The store API has to be public, the browsers of the visitors call it directly (search, product images), but the administration part of the same host (/app, /admin, /auth/user) is restricted to the local network and the VPN by a second router.

Here is an overview of the network flow :

flowchart LR
    style VISITOR fill: #45d8ff57
    style ADMINISTRATOR fill: #45d8ff57
    style TRAEFIK_CONTAINER fill: #6601013c
    style STOREFRONT_CONTAINER fill: #6601013c
    style APP_CONTAINER fill: #6601013c
    style DB_CONTAINER fill: #6601013c
    style REDIS_CONTAINER fill: #6601013c
    style TRAEFIK_ROUTER fill: #defc1e37
    style TRAEFIK_MIDDLEWARE fill: #defc1e37
    style SERVER_DEVICE fill: #fec9bf5e
    style CONTAINER_ENGINE fill: #67010030
    DOCKER_TRAEFIK_PORT443{{443/tcp}}
    DOCKER_TRAEFIK_PORT80{{80/tcp}}
    DOCKER_STOREFRONT_PORT{{8000/tcp}}
    DOCKER_APP_PORT{{9000/tcp}}
    DOCKER_DB_PORT{{5432/tcp}}
    DOCKER_REDIS_PORT{{6379/tcp}}
    TRAEFIK_ROUTER_STOREFRONT(shop)
    TRAEFIK_ROUTER_API(medusa\n/store, /static, /hooks, ...)
    TRAEFIK_ROUTER_ADMIN(medusa\n/app, /admin, /auth/user)
    TRAEFIK_MIDDLEWARE_REDIRECT(HTTPS redirect on<br>the web entrypoint)
    TRAEFIK_MIDDLEWARE_CROWDSEC(CrowdSec bouncer on<br>the websecure entrypoint)
    TRAEFIK_MIDDLEWARE_IP_WHITELIST(IP whitelist)
    VISITOR((VISITOR))
    ADMINISTRATOR((ADMINISTRATOR\nLAN or VPN))

    subgraph SERVER_DEVICE[MINI PC]
        subgraph CONTAINER_ENGINE[DOCKER]
            subgraph TRAEFIK_CONTAINER[TRAEFIK CONTAINER]
                DOCKER_TRAEFIK_PORT443
                DOCKER_TRAEFIK_PORT80

                subgraph TRAEFIK_MIDDLEWARE[TRAEFIK MIDDLEWARES]
                    TRAEFIK_MIDDLEWARE_REDIRECT
                    TRAEFIK_MIDDLEWARE_CROWDSEC
                    TRAEFIK_MIDDLEWARE_IP_WHITELIST
                end

                subgraph TRAEFIK_ROUTER[TRAEFIK HTTP ROUTERS]
                    TRAEFIK_ROUTER_STOREFRONT
                    TRAEFIK_ROUTER_API
                    TRAEFIK_ROUTER_ADMIN
                end
            end

            subgraph STOREFRONT_CONTAINER[STOREFRONT CONTAINER]
                DOCKER_STOREFRONT_PORT
            end

            subgraph APP_CONTAINER[MEDUSA CONTAINER]
                DOCKER_APP_PORT
            end

            subgraph DB_CONTAINER[POSTGRESQL CONTAINER]
                DOCKER_DB_PORT
            end

            subgraph REDIS_CONTAINER[REDIS CONTAINER]
                DOCKER_REDIS_PORT
            end
        end
    end

    VISITOR -->|pages, search,<br>product images| DOCKER_TRAEFIK_PORT443
    VISITOR --> DOCKER_TRAEFIK_PORT80
    ADMINISTRATOR -->|dashboard| DOCKER_TRAEFIK_PORT443
    DOCKER_TRAEFIK_PORT80 --> TRAEFIK_MIDDLEWARE_REDIRECT
    TRAEFIK_MIDDLEWARE_REDIRECT -.->|301 to https| VISITOR
    DOCKER_TRAEFIK_PORT443 --> TRAEFIK_MIDDLEWARE_CROWDSEC
    TRAEFIK_MIDDLEWARE_CROWDSEC --> TRAEFIK_ROUTER_STOREFRONT
    TRAEFIK_MIDDLEWARE_CROWDSEC --> TRAEFIK_ROUTER_API
    TRAEFIK_MIDDLEWARE_CROWDSEC --> TRAEFIK_ROUTER_ADMIN
    TRAEFIK_ROUTER_STOREFRONT -->|traefik - public - net| DOCKER_STOREFRONT_PORT
    TRAEFIK_ROUTER_API -->|traefik - public - net| DOCKER_APP_PORT
    TRAEFIK_ROUTER_ADMIN --> TRAEFIK_MIDDLEWARE_IP_WHITELIST
    TRAEFIK_MIDDLEWARE_IP_WHITELIST --> DOCKER_APP_PORT
    DOCKER_STOREFRONT_PORT -.->|store API calls of the server<br>side, through the Traefik alias| DOCKER_TRAEFIK_PORT443
    DOCKER_APP_PORT -->|medusa - net| DOCKER_DB_PORT
    DOCKER_APP_PORT -->|medusa - net| DOCKER_REDIS_PORT

Setting up

Create a folder to hold the configuration, and clone the project into its src subfolder :

sudo mkdir /opt/apps/medusa
sudo git clone --depth 1 https://github.com/medusajs/dtc-starter.git /opt/apps/medusa/src

Then :

copy the .env, .dockerignore, docker-compose.yml, backend.Dockerfile, storefront.Dockerfile and medusa-config.ts files from this project's medusa directory into the /opt/apps/medusa directory, and generate every secret of the .env file with openssl rand -hex 32 (hexadecimal on purpose : the passwords go into the DATABASE_URL and the REDIS_URL, special characters would have to be URL-encoded). Leave MEDUSA_PUBLISHABLE_KEY aside for now, it is created by the first start of the backend (see Run)
replace the configuration of the backend in the project with the production one (see Backend configuration) :
```
sudo cp /opt/apps/medusa/medusa-config.ts /opt/apps/medusa/src/apps/backend/medusa-config.ts
```
copy the medusa.yml file from this project's traefik/dynamic directory into the /opt/apps/traefik/dynamic directory
add medusa.example.com to the network aliases of Traefik on traefik-public-net, for the storefront, and shop.example.com and medusa.example.com to its aliases on traefik-private-net, for Gatus (see Configuration files details), then recreate the Traefik container
add public DNS records for shop.example.com and medusa.example.com (a CNAME to your dynamic DNS, like the other public services, see Domain and subdomains), and local DNS records pointing to the mini PC (see Pi-hole) : without them, the dashboard is reached through the NAT loopback of the router and rejected by the IP whitelist
add the /sources/volumes/medusa-db-vol and /sources/volumes/medusa-redis-vol excludes to the backup plan (see Backup plan) : the database is dumped every evening, the medusa-db container is part of the list of the dump script (see Database dumps)

[!NOTE] The src folder holds the project, it is not part of this repository (medusa/.gitignore). For a real shop it becomes a fork of the starter, with its own customizations (theme, pages, modules, and the medusa-config.ts above), cloned there instead of the starter.

Details

Service definition

:page_facing_up: docker-compose.yml :

services:

  # Backend : store API, admin dashboard (/app) and background jobs, built from the project cloned in ./src
  medusa:
    build:
      context: .
      dockerfile: backend.Dockerfile
    container_name: medusa
    restart: unless-stopped
    init: true
    cap_drop:
      - ALL
    security_opt:
      - no-new-privileges:true
    # Database and Redis credentials, secrets (see .env)
    env_file: .env
    environment:
      TZ: "Europe/Zurich"
      NODE_ENV: production
      MEDUSA_DISABLE_TELEMETRY: "true"
      # Public URL of the backend, used to build the URL of the uploaded files (see medusa-config.ts)
      MEDUSA_BACKEND_URL: https://medusa.example.com
      # A single process serves the API and runs the background jobs, enough for a small shop
      MEDUSA_WORKER_MODE: shared
      # Origins allowed to call the API from a browser : the storefront for the store API, the dashboard for the admin API
      STORE_CORS: https://shop.example.com
      ADMIN_CORS: https://medusa.example.com
      AUTH_CORS: https://shop.example.com,https://medusa.example.com
    depends_on:
      medusa-db:
        condition: service_healthy
      medusa-redis:
        condition: service_healthy
    healthcheck:
      test: [ "CMD-SHELL", "wget -qO /dev/null http://127.0.0.1:9000/health || exit 1" ]
      interval: 30s
      timeout: 5s
      retries: 5
      # The migrations run before the server starts, the first ones also seed the store (regions, publishable key, ...)
      start_period: 180s
    volumes:
      # Uploaded files (product images, ...), served by the backend under /static
      - medusa-static-vol:/server/static
    networks:
      - medusa-net
      - traefik-public-net

  # Storefront : the Next.js shop of the project. It only talks to the backend through its public URL
  # (from the visitors' browsers and from its own server-side code), so it does not join the private network of the stack
  medusa-storefront:
    build:
      context: .
      dockerfile: storefront.Dockerfile
      # The NEXT_PUBLIC_ variables are written into the code at build time : changing one means rebuilding the image
      args:
        NEXT_PUBLIC_MEDUSA_BACKEND_URL: https://medusa.example.com
        NEXT_PUBLIC_BASE_URL: https://shop.example.com
        # Country used when the visitor's one is not part of any region, must belong to a region of the store
        NEXT_PUBLIC_DEFAULT_REGION: fr
        NEXT_PUBLIC_MEDUSA_PUBLISHABLE_KEY: ${MEDUSA_PUBLISHABLE_KEY}
    container_name: medusa-storefront
    restart: unless-stopped
    init: true
    cap_drop:
      - ALL
    security_opt:
      - no-new-privileges:true
    environment:
      TZ: "Europe/Zurich"
      NEXT_TELEMETRY_DISABLED: "1"
      # Also checked when the server starts (next.config.js), not only at build time
      NEXT_PUBLIC_MEDUSA_PUBLISHABLE_KEY: ${MEDUSA_PUBLISHABLE_KEY}
    depends_on:
      medusa:
        condition: service_healthy
    healthcheck:
      # A static file : the pages themselves depend on the backend
      test: [ "CMD-SHELL", "wget -qO /dev/null http://127.0.0.1:8000/favicon.ico || exit 1" ]
      interval: 30s
      timeout: 5s
      retries: 5
    networks:
      - traefik-public-net

  medusa-db:
    # Major version pinned : a new major version of PostgreSQL cannot start on the data of the previous one,
    # the upgrade (dump and restore) is done on purpose, never by a pull
    image: docker.io/library/postgres:17-alpine
    container_name: medusa-db
    restart: unless-stopped
    cap_drop:
      - ALL
    cap_add:
      - CHOWN
      - DAC_READ_SEARCH
      - FOWNER
      - SETGID
      - SETUID
    security_opt:
      - no-new-privileges:true
    env_file: .env
    healthcheck:
      test: [ "CMD-SHELL", "pg_isready -d \"$${POSTGRES_DB}\" -U $${POSTGRES_USER}" ]
      interval: 10s
      timeout: 5s
      retries: 5
    volumes:
      - medusa-db-vol:/var/lib/postgresql/data
    networks:
      - medusa-net

  # Event bus, workflow engine, locks and admin sessions : jobs in progress are kept on disk so that they survive a restart
  medusa-redis:
    image: docker.io/library/redis:alpine
    container_name: medusa-redis
    restart: unless-stopped
    user: "999:1000"
    cap_drop:
      - ALL
    security_opt:
      - no-new-privileges:true
    env_file: .env
    command:
      - /bin/sh
      - -c
      - redis-server --appendonly yes --requirepass "$${REDIS_PASSWORD:?REDIS_PASSWORD variable is not set}"
    healthcheck:
      test: [ "CMD-SHELL", "redis-cli --pass \"$${REDIS_PASSWORD}\" --no-auth-warning ping | grep PONG" ]
      interval: 10s
      timeout: 5s
      retries: 5
    volumes:
      - medusa-redis-vol:/data
    networks:
      - medusa-net

volumes:

  medusa-db-vol:
    name: medusa-db-vol

  medusa-redis-vol:
    name: medusa-redis-vol

  medusa-static-vol:
    name: medusa-static-vol

networks:

  # The database and Redis stay on the private network of the stack, never on a Traefik network
  medusa-net:
    name: medusa-net

  traefik-public-net:
    name: traefik-public-net
    external: true

Images

:page_facing_up: backend.Dockerfile :

# Medusa backend (API + admin dashboard), built from the project cloned in ./src (pnpm monorepo of the DTC starter)

FROM node:22-alpine AS build

# pnpm, at the version pinned by the "packageManager" field of the project
RUN corepack enable

WORKDIR /app
COPY src/ ./

# Dependencies of the backend only, at the versions of the lockfile
RUN pnpm install --frozen-lockfile --filter ./apps/backend

# Compiles the backend and the admin dashboard into apps/backend/.medusa/server
WORKDIR /app/apps/backend
RUN pnpm build


FROM node:22-alpine

# The application files belong to the runtime user : the migrations write into node_modules
# (they create an empty "migrations" folder in every module that has none)
WORKDIR /server
RUN chown node:node /server
USER node

COPY --from=build --chown=node:node /app/apps/backend/.medusa/server ./

# Production dependencies of the compiled application
RUN npm install --omit=dev --no-audit --no-fund && npm cache clean --force

# Directory of the uploaded files (volume)
RUN mkdir static

EXPOSE 9000

# The pending migrations are applied at every start (the first ones also seed the store), then the server starts
CMD [ "sh", "-c", "npx medusa db:migrate && exec npx medusa start" ]

:page_facing_up: storefront.Dockerfile :

# Next.js storefront, built from the project cloned in ./src (pnpm monorepo of the DTC starter)

FROM node:22-alpine AS build

# pnpm, at the version pinned by the "packageManager" field of the project
RUN corepack enable

WORKDIR /app
COPY src/ ./

# Dependencies of the storefront only, at the versions of the lockfile
RUN pnpm install --frozen-lockfile --filter ./apps/storefront

# The NEXT_PUBLIC_ variables are written into the code by the build (see docker-compose.yml)
ARG NEXT_PUBLIC_MEDUSA_BACKEND_URL
ARG NEXT_PUBLIC_MEDUSA_PUBLISHABLE_KEY
ARG NEXT_PUBLIC_BASE_URL
ARG NEXT_PUBLIC_DEFAULT_REGION
ENV NEXT_PUBLIC_MEDUSA_BACKEND_URL=$NEXT_PUBLIC_MEDUSA_BACKEND_URL \
    NEXT_PUBLIC_MEDUSA_PUBLISHABLE_KEY=$NEXT_PUBLIC_MEDUSA_PUBLISHABLE_KEY \
    NEXT_PUBLIC_BASE_URL=$NEXT_PUBLIC_BASE_URL \
    NEXT_PUBLIC_DEFAULT_REGION=$NEXT_PUBLIC_DEFAULT_REGION \
    NEXT_TELEMETRY_DISABLED=1

WORKDIR /app/apps/storefront
RUN pnpm build


FROM node:22-alpine

ENV NODE_ENV=production

# The whole workspace : "next start" reads next.config.js, which needs some development dependencies of the starter.
# Owned by the runtime user, Next.js writes its cache into .next
COPY --from=build --chown=node:node /app /app

WORKDIR /app/apps/storefront
USER node
EXPOSE 8000

CMD [ "node_modules/.bin/next", "start", "-p", "8000" ]

:page_facing_up: .dockerignore :

# Only the sources of the project go into the images, never what was installed or built outside of Docker, nor the secrets
.env
**/.env
**/.env.*
**/node_modules
**/.next
**/.medusa
**/.turbo
**/.git

Backend configuration

:page_facing_up: medusa-config.ts :

import { loadEnv, defineConfig } from '@medusajs/framework/utils'

loadEnv(process.env.NODE_ENV || 'development', process.cwd())

// Production configuration of the backend : copy this file over apps/backend/medusa-config.ts in the project.
// Everything comes from the environment variables (see docker-compose.yml and .env)
module.exports = defineConfig({
  projectConfig: {
    databaseUrl: process.env.DATABASE_URL,
    // Sessions of the admin dashboard, kept in Redis instead of memory
    redisUrl: process.env.REDIS_URL,
    // shared : one process for the API and the background jobs ; server / worker to run them in two containers
    workerMode: process.env.MEDUSA_WORKER_MODE as 'shared' | 'worker' | 'server',
    http: {
      storeCors: process.env.STORE_CORS!,
      adminCors: process.env.ADMIN_CORS!,
      authCors: process.env.AUTH_CORS!,
      jwtSecret: process.env.JWT_SECRET,
      cookieSecret: process.env.COOKIE_SECRET,
    },
  },
  modules: [
    // Redis versions of the infrastructure modules : the default ones keep the events, the workflows in progress
    // and the locks in memory, lost at every restart (development only)
    {
      resolve: '@medusajs/medusa/event-bus-redis',
      options: {
        redisUrl: process.env.REDIS_URL,
      },
    },
    {
      resolve: '@medusajs/medusa/workflow-engine-redis',
      options: {
        redis: {
          redisUrl: process.env.REDIS_URL,
        },
      },
    },
    {
      resolve: '@medusajs/medusa/locking',
      options: {
        providers: [
          {
            resolve: '@medusajs/medusa/locking-redis',
            id: 'locking-redis',
            is_default: true,
            options: {
              redisUrl: process.env.REDIS_URL,
            },
          },
        ],
      },
    },
    // Uploaded files stored on disk (a volume) and served by the backend under /static.
    // backend_url must be the PUBLIC address : it is saved as is in the database and loaded by the visitors' browsers
    // (the default one, http://localhost:9000/static, only works on a development machine)
    {
      resolve: '@medusajs/medusa/file',
      options: {
        providers: [
          {
            resolve: '@medusajs/medusa/file-local',
            id: 'local',
            options: {
              upload_dir: 'static',
              backend_url: `${process.env.MEDUSA_BACKEND_URL}/static`,
            },
          },
        ],
      },
    },
  ],
})

Environment variables

:page_facing_up: .env :

# Generate every secret with : openssl rand -hex 32
# (hexadecimal on purpose : the passwords go into URLs, special characters would have to be URL-encoded)

# PostgreSQL
POSTGRES_DB=medusa
POSTGRES_USER=medusa
POSTGRES_PASSWORD=<postgres_password>
# sslmode=disable is required : Medusa turns SSL on by itself for any database that is not on localhost,
# and the database of the stack has none (private network). Without it the migrations hang, then time out
DATABASE_URL=postgres://${POSTGRES_USER}:${POSTGRES_PASSWORD}@medusa-db:5432/${POSTGRES_DB}?sslmode=disable

# Redis
REDIS_PASSWORD=<redis_password>
REDIS_URL=redis://:${REDIS_PASSWORD}@medusa-redis:6379

# Secrets of the backend (signature of the tokens and of the session cookies)
JWT_SECRET=<random_string>
COOKIE_SECRET=<random_string>

# Publishable API key of the storefront (pk_...), created by the first start of the backend :
# copy it from the admin dashboard (Settings > Publishable API Keys), then build the storefront.
# Only read by docker-compose.yml (build argument of the storefront)
MEDUSA_PUBLISHABLE_KEY=<publishable_key>

Traefik routing

:page_facing_up: medusa.yml :

http:
  services:
    medusa:
      loadBalancer:
        servers:
          - url: http://medusa:9000

    medusa-storefront:
      loadBalancer:
        servers:
          - url: http://medusa-storefront:8000

  routers:
    # The shop itself
    medusa-storefront:
      rule: 'Host(`shop.example.com`)'
      entryPoints:
        - websecure
      tls:
        certResolver: default
      service: medusa-storefront

    # Administration part of the backend : the dashboard (/app), its API (/admin) and the login of the administrators (/auth/user),
    # from the local network or through the VPN only. The dashboard keeps its own login on top of it.
    # The higher priority makes this router win over the one below, which matches the whole host.
    medusa-admin:
      rule: 'Host(`medusa.example.com`) && (PathPrefix(`/app`) || PathPrefix(`/admin`) || PathPrefix(`/auth/user`))'
      priority: 100
      entryPoints:
        - websecure
      tls:
        certResolver: default
      service: medusa
      middlewares:
        - vpn-whitelist@file

    # Everything else is public : the store API (/store), the customer login (/auth/customer, /auth/session, /auth/token),
    # the uploaded files (/static) and the webhooks of the payment providers (/hooks). It is called by the visitors' browsers
    # (search, product images) and by the storefront's server-side code.
    # Excluding the administration paths rather than listing the public ones : Medusa keeps all of them under the three prefixes above,
    # whereas a forgotten public path would silently break the shop (customer login, password reset, payment).
    medusa-api:
      rule: 'Host(`medusa.example.com`)'
      priority: 1
      entryPoints:
        - websecure
      tls:
        certResolver: default
      service: medusa

Things to notice :

the images are built on the mini PC from the project in src (build instead of image), in two stages : the first one installs the dependencies at the versions of the lockfile and compiles, the second one only keeps what runs. The backend is compiled with its admin dashboard into .medusa/server, where its production dependencies are then installed. The .dockerignore keeps out of the images what may have been installed or built outside of Docker in src, and every .env file
the backend applies the pending migrations at every start, before the server starts. The first ones also run the seed script of the starter (src/migration-scripts) : a region Europe in EUR (no Switzerland, no CHF), a few products, a stock location, shipping options, and the publishable API key the storefront needs
the NEXT_PUBLIC_ variables of the storefront are written into its code at build time (build arguments) : the storefront must be rebuilt when one of them changes. The publishable key is also checked when the server starts, hence the same variable in environment
the storefront calls the store API by its public URL, from the browsers of the visitors and from its own server side rendering. Inside its container, medusa.example.com resolves to Traefik thanks to its network alias on traefik-public-net : the requests stay on the mini PC instead of looping through the router, and the storefront does not need to join medusa-net
two routers share medusa.example.com : medusa-admin (priority 100, IP whitelist) takes the administration paths, medusa-api (priority 1) everything else. Here the administration paths are listed rather than the public ones (unlike GoatCounter) : Medusa keeps all of them under three prefixes, while the public side is wider (customer login and password reset, files, webhooks of the payment providers, ...) and a forgotten path would silently break the shop
the dashboard keeps its own login behind the IP whitelist, no PocketID : the administrators are Medusa users, with their own roles
the database and Redis only join medusa-net, only the backend reaches them
Redis writes its data to disk (--appendonly yes), the events not handled yet and the workflows in progress survive a restart. Its volume is excluded from the backup though : restored next to an older dump of the database, it would replay jobs that no longer match the data
the uploaded files (product images, ...) are stored in the medusa-static-vol volume and served by the backend under /static, with their public URL saved in the database (backend_url in medusa-config.ts, the default one points to localhost). The volume is part of the backup like the other volumes
the shared worker mode runs the API and the background jobs in the same process, enough for a small shop. A busier one splits them into two containers of the same image (MEDUSA_WORKER_MODE set to server and worker), see the deployment documentation of Medusa
the major version of PostgreSQL is pinned (17-alpine), as for Ghostfolio
no Linux capability and no privilege escalation, the backend and the storefront run as the non-root node user. The files of the backend belong to that user : the migrations write into node_modules (an empty migrations folder in every module that has none) and fail with EACCES otherwise
the telemetry of Medusa and Next.js is disabled (MEDUSA_DISABLE_TELEMETRY, NEXT_TELEMETRY_DISABLED). Medusa still prints its banner about it at startup, but sends nothing
Gatus checks https://medusa.example.com/health, which answers OK, and the home page of the shop

[!IMPORTANT] ?sslmode=disable is required at the end of DATABASE_URL : Medusa turns SSL on by itself for any database that is not on localhost, and the PostgreSQL of the stack has no TLS (it never leaves medusa-net). Without it the migrations hang, then fail after 10 seconds with Could not connect to the database while running migrations.

[!WARNING] All the server side requests of the storefront reach Traefik from one address, the one of its container, and go through CrowdSec like any visitor. A scenario triggered by that traffic (a crawler browsing the whole shop, for instance) would ban the storefront itself, and the shop goes down for everybody. Before opening a real shop, make the server side of the storefront call the backend directly, on medusa-net (http://medusa:9000), the browsers keeping the public URL : a small change in the project, where the backend URL is read (apps/storefront/src/lib/config.ts and apps/storefront/src/middleware.ts).

[!NOTE] Also missing before a real shop :

payments : the storefront handles Stripe (NEXT_PUBLIC_STRIPE_KEY), the Stripe provider has to be added to the payment module of the backend

e-mails : Medusa sends none by default (order confirmation, password reset, ...), a notification provider has to be added

Switzerland : create a region in CHF with the ch country, set NEXT_PUBLIC_DEFAULT_REGION to ch and rebuild the storefront. The search index of the starter only holds prices in EUR and USD : add chf to PRICE_CURRENCIES (apps/backend/src/search/helpers/pricing.ts) and to SEARCH_PRICE_CURRENCIES (apps/storefront/src/lib/search-client.ts)

administration : the dashboard is only reachable from the local network and the VPN, the shop owner needs a VPN access, or the administration routes have to be opened to the internet

Run

The storefront needs the publishable key created by the first start of the backend : start the backend alone first, its database and Redis come with it :

sudo docker-compose -f /opt/apps/medusa/docker-compose.yml up -d --build medusa

The first build takes a few minutes, then the backend applies the migrations and seeds the store : wait for Server is ready on port: 9000 in sudo docker logs -f medusa.

Create the administrator account :

sudo docker exec medusa npx medusa user -e [email protected] -p <password>

The dashboard is available at https://medusa.example.com/app.

Copy the publishable API key (pk_...) from Settings -> Publishable API Keys, or straight from the database :

sudo docker exec medusa-db psql -U medusa -d medusa -tAc "select token from api_key where type = 'publishable'"

Put it in MEDUSA_PUBLISHABLE_KEY in the .env file, then build and start the storefront :

sudo docker-compose -f /opt/apps/medusa/docker-compose.yml up -d --build medusa-storefront

You should end-up with four running containers, medusa, medusa-storefront, medusa-db and medusa-redis, and Traefik picks up the dynamic configuration file without restarting.

The shop is available at https://shop.example.com.

After a change in the project (or to update Medusa : git pull in src), rebuild the images :

sudo docker-compose -f /opt/apps/medusa/docker-compose.yml up -d --build

Defrag-life

Defrag-life is a PHP / MySQL website I made in the early 2000's, about the DeFRaG mod of the Quake 3 arena game. The project can be found here : https://github.com/Yann39/defrag-life I simply keep hosting it as a souvenir, but it is a static snapshot, it is not intended to be used anymore.

We will use Nginx as HTTP server along with the PHP-FPM module (FastCGI Process Manager) for processing PHP files. At the time of writing this is the preferred method of processing PHP pages with Nginx and is faster than traditional CGI-based methods.

The website also requires a MySQL or MariaDB database, we will use MariaDB.

flowchart LR
    style INCOMING_REQUEST fill: #45d8ff57
    style TRAEFIK_CONTAINER fill: #6601013c
    style NGINX_CONTAINER fill: #6601013c
    style PHP_CONTAINER fill: #6601013c
    style MARIADB_CONTAINER fill: #6601013c
    style TRAEFIK_ROUTER fill: #defc1e37
    style TRAEFIK_MIDDLEWARE fill: #defc1e37
    style SERVER_DEVICE fill: #fec9bf5e
    style CONTAINER_ENGINE fill: #67010030
    DOCKER_TRAEFIK_PORT443{{443/tcp}}
    DOCKER_TRAEFIK_PORT80{{80/tcp}}
    DOCKER_NGINX_PORT{{80/tcp}}
    DOCKER_PHP_PORT{{9000/tcp}}
    DOCKER_MARIADB_PORT{{3306/tcp}}
    TRAEFIK_ROUTER_APP(quake.example.com)
    TRAEFIK_MIDDLEWARE_REDIRECT(HTTPS redirect\non the web entrypoint)
    INCOMING_REQUEST((INCOMING\nREQUEST))

    subgraph SERVER_DEVICE[MINI_PC]
        subgraph CONTAINER_ENGINE[DOCKER]
            subgraph NGINX_CONTAINER[NGINX CONTAINER]
                DOCKER_NGINX_PORT
            end

            subgraph PHP_CONTAINER[PHP-FPM CONTAINER]
                DOCKER_PHP_PORT
            end

            subgraph MARIADB_CONTAINER[MARIADB CONTAINER]
                DOCKER_MARIADB_PORT
            end

            subgraph TRAEFIK_CONTAINER[TRAEFIK CONTAINER]
                DOCKER_TRAEFIK_PORT443
                DOCKER_TRAEFIK_PORT80

                subgraph TRAEFIK_ROUTER[TRAEFIK HTTP ROUTER]
                    TRAEFIK_ROUTER_APP
                end

                subgraph TRAEFIK_MIDDLEWARE[TRAEFIK MIDDLEWARES]
                    TRAEFIK_MIDDLEWARE_REDIRECT
                end
            end
        end
    end

    INCOMING_REQUEST --> DOCKER_TRAEFIK_PORT443
    INCOMING_REQUEST --> DOCKER_TRAEFIK_PORT80
    DOCKER_TRAEFIK_PORT443 --> TRAEFIK_ROUTER_APP
    DOCKER_TRAEFIK_PORT80 ---> TRAEFIK_MIDDLEWARE_REDIRECT
    TRAEFIK_MIDDLEWARE_REDIRECT -.->|301 to https| INCOMING_REQUEST
    TRAEFIK_ROUTER_APP --> DOCKER_NGINX_PORT
    DOCKER_NGINX_PORT --> DOCKER_PHP_PORT
    DOCKER_PHP_PORT --> DOCKER_MARIADB_PORT

Setting up

First, create a folder to hold the configuration :

sudo mkdir /opt/apps/defrag-life

Also create a data directory to hold the application files (PHP, HTML, CSS, JavaScript files) :

mkdir /opt/apps/defrag-life/data

and copy inside that folder the content from https://github.com/Yann39/defrag-life.

Then copy the following files from this project's defrag-life directory into the /opt/apps/defrag-life directory :

Dockerfile : The file responsible for building image of PHP-FPM
docker-compose.yml : The definition of the services
.env : The environment variables (for database connection)
default.conf : The Nginx configuration
www.conf : The PHP-FPM pool configuration

And copy the defrag-life.yml file from this project's traefik/dynamic directory into the /opt/apps/traefik/dynamic directory.

Details

Dockerfile

:page_facing_up: Dockerfile :

FROM php:8.2-fpm-alpine

LABEL maintainer="Yann39"

# Install mysqli extension
RUN docker-php-ext-install mysqli

RUN addgroup --gid 1000 --system phpuser && \
    adduser --uid 1000 --system phpuser --ingroup phpuser && \
    chown -R phpuser:phpuser /var/www/html && \
    chmod -R 644 /var/www/html

USER 1000

# Start PHP-FPM
CMD ["php-fpm"]

This Dockerfile allows us to create the Docker image for the PHP-FPM service, the image is based on the popular Alpine Linux project, which is much smaller than most distribution base images.

In this Dockerfile we also install the mysqli extension which will be required to connect to the MariaDB database from PHP.

Finally, we create a phpuser user that will run the process (for security purposes we always ensure that our images run as non-root).

In order to use this image, an HTTP server or a reverse proxy (such as Nginx, Apache, or other tool which speaks the FastCGI protocol) is required, that's why we use Nginx.

Service definition

:page_facing_up: docker-compose.yml :

services:

  nginx:
    image: nginx:latest
    container_name: defrag-life-nginx
    volumes:
      - ./data/:/var/www/html/
      - ./default.conf:/etc/nginx/conf.d/default.conf
    restart: unless-stopped
    networks:
      - defrag-life-net
      - traefik-public-net

  php-fpm:
    build:
      context: .
      dockerfile: ./Dockerfile
    container_name: defrag-life-php
    restart: unless-stopped
    networks:
      - defrag-life-net
    volumes:
      - ./data/:/var/www/html/
      - ./www.conf:/usr/local/etc/php-fpm.d/www.conf

  mariadb:
    image: mariadb:latest
    container_name: defrag-life-db
    restart: unless-stopped
    env_file: .env
    environment:
      - MARIADB_AUTO_UPGRADE="1"
      - MARIADB_ROOT_PASSWORD=$MARIADB_ROOT_PASSWORD
      - MARIADB_DATABASE=$MARIADB_DATABASE
      - MARIADB_USER=$MARIADB_USER
      - MARIADB_PASSWORD=$MARIADB_PASSWORD
    volumes:
      - defrag-life-db-vol:/var/lib/mysql
    networks:
      - defrag-life-net
      - phpmyadmin-net

volumes:

  defrag-life-db-vol:
    name: defrag-life-db-vol

networks:

  defrag-life-net:
    name: defrag-life-net

  traefik-public-net:
    name: traefik-public-net
    external: true

  phpmyadmin-net:
    name: phpmyadmin-net
    external: true

:page_facing_up: defrag-life.yml :

http:
  services:
    defrag-life:
      loadBalancer:
        servers:
          - url: http://defrag-life:80

  routers:
    defrag-life:
      rule: 'Host(`quake.example.com`)'
      entryPoints:
        - websecure
      tls:
        certResolver: default
      service: defrag-life

Here we define 3 services :

nginx : the HTTP server which will speak with the PHP-FPM service to interpret PHP files (whenever the server gets a PHP script request, it utilizes a proxy, FastCGI connection to pass that request on to the PHP-FPM service)
- It defines 2 volumes to bind the website files and the Nginx configuration file (see Nginx configuration file)
php-fpm : the PHP-FPM service responsible for processing PHP scripts
- It uses our own Dockerfile, see Dockerfile
- It defines 2 volumes to bind the website files and the PHP-FPM pool configuration file (see PHP-FPM configuration file)
- It will run by default on port 9000
mariadb : The MariaDB database that will hold the application data
- It uses our .env file to retrieve environment variables values for database connection
- It defines a named volume defrag-life-db-vol that will hold the database data
- It will run by default on port 3306

Then we use Traefik dynamic config file to :

create a service which will point to our container application running on port 80
create an HTTP router that will match quake.example.com URL on our websecure entrypoint to point to our service
add TLS configuration that will use our default certificates resolver, so it can generate Let's encrypt certificates

All services run in a defrag-life-net network, the nginx front must also join the public network of Traefik (traefik-public-net) to be reachable by the reverse proxy, as the website is exposed to the internet (see Network segmentation), and phpmyadmin-net so that the database is reachable from PhpMyAdmin, see PhpMyAdmin.

Environment variables

:page_facing_up: .env :

MARIADB_ROOT_PASSWORD=<root_password>
MARIADB_DATABASE=<db_name>
MARIADB_USER=<username>
MARIADB_PASSWORD=<password>

It simply set environment variable values (used in the mariadb service in the Compose file).

Nginx configuration file

:page_facing_up: www.conf :

; Start a new pool named 'www'.
[www]
; Unix user/group of the child processes.
user = www-data
group = www-data
; The address on which to accept FastCGI requests.
listen = 127.0.0.1:9000
; Choose how the process manager will control the number of child processes.
pm = dynamic
pm.max_children = 5
pm.start_servers = 2
pm.min_spare_servers = 1
pm.max_spare_servers = 3
; The ping URI to call the monitoring page of FPM.
ping.path = /ping
; This directive may be used to customize the response of a ping request.
ping.response = pong

This is a quite basic configuration file for Nginx, we just enabled ping, so we can ping the service from any monitoring tool.

PHP-FPM configuration file

:page_facing_up: default.conf :

server {
    listen 80 default_server;
    listen [::]:80 default_server;
    server_name _;

    root /var/www/html;
    index index.php index.html;

    error_log  /var/log/nginx/error.log;
    access_log /var/log/nginx/access.log;

    location ~ \.php$ {
        include fastcgi_params;
        try_files $uri =404;
        fastcgi_index index.php;
        fastcgi_param SCRIPT_FILENAME $document_root$fastcgi_script_name;
        fastcgi_param PATH_INFO $fastcgi_path_info;
        fastcgi_pass php-fpm:9000;
        fastcgi_split_path_info ^(.+\.php)(/.+)$;
    }

    location ~ ^/ping$ {
        access_log off;
        include fastcgi_params;
        fastcgi_param SCRIPT_FILENAME $document_root$fastcgi_script_name;
        fastcgi_pass php-fpm:9000;
    }
}

This is also a quite basic configuration file for PHP-FPM :

it listens for incoming requests on port 80 (and set it as default server, although it is the only one)
it sets the server name to an invalid server names which never intersect with any real name (as they are only one server blocks for port 80, Nginx will not even bother comparing server_name with a request's Host header, all the requests would be directed there anyway)
it sets the root directory to /var/www/html for static content in the file system
it defines files that will be used as an index (index.php and index.html)
it defines the location of error and access log files
it sets a location to match PHP files (.php$ will match files ending with .php, .php3, etc.) to be processed by our PHP-FPM service
it sets a location to match /ping endpoint to ping our PHP-FPM service

Run

Finally, simply run the Compose file :

sudo docker-compose -f /opt/apps/defrag-life/docker-compose.yml up -d

You should end-up with 3 running containers :

defrag-life-nginx : The HTTP server
defrag-life-php : The PHP-FPM service
defrag-life-mariadb : The MariaDB database

It should also have generated the needed Let's Encrypt certificates in the acme.json file in the Traefik folder.

The application will be available at https://quake.example.com.

CCTeam

CCTeam is the GraphQL API (Java / Spring Boot) of a Flutter mobile application, with its MariaDB database. It is exposed to the internet, the mobile application calls it from anywhere : it sits on the public network, behind the CrowdSec bouncer, and the requests are authenticated by the application itself (JWT tokens).

Here is an overview of the network flow :

flowchart LR
    style INCOMING_REQUEST fill: #45d8ff57
    style TRAEFIK_CONTAINER fill: #6601013c
    style APP_CONTAINER fill: #6601013c
    style DB_CONTAINER fill: #6601013c
    style PROMETHEUS_CONTAINER fill: #6601013c
    style PHPMYADMIN_CONTAINER fill: #6601013c
    style TRAEFIK_ROUTER fill: #defc1e37
    style TRAEFIK_MIDDLEWARE fill: #defc1e37
    style SERVER_DEVICE fill: #fec9bf5e
    style CONTAINER_ENGINE fill: #67010030
    DOCKER_TRAEFIK_PORT443{{443/tcp}}
    DOCKER_TRAEFIK_PORT80{{80/tcp}}
    DOCKER_APP_PORT{{5001/tcp\nGraphQL API}}
    DOCKER_APP_MANAGEMENT_PORT{{8081/tcp\nactuator}}
    DOCKER_DB_PORT{{3306/tcp}}
    DOCKER_PROMETHEUS_PORT{{9090/tcp}}
    DOCKER_PHPMYADMIN_PORT{{80/tcp}}
    TRAEFIK_ROUTER_APP(ccteam.example.com)
    TRAEFIK_MIDDLEWARE_REDIRECT(HTTPS redirect on<br>the web entrypoint)
    TRAEFIK_MIDDLEWARE_CROWDSEC(CrowdSec bouncer on<br>the websecure entrypoint)
    FIREBASE[Firebase\npush notifications]
    INCOMING_REQUEST((MOBILE\nAPPLICATION))

    subgraph SERVER_DEVICE[MINI PC]
        subgraph CONTAINER_ENGINE[DOCKER]
            subgraph TRAEFIK_CONTAINER[TRAEFIK CONTAINER]
                DOCKER_TRAEFIK_PORT443
                DOCKER_TRAEFIK_PORT80

                subgraph TRAEFIK_MIDDLEWARE[TRAEFIK MIDDLEWARES]
                    TRAEFIK_MIDDLEWARE_REDIRECT
                    TRAEFIK_MIDDLEWARE_CROWDSEC
                end

                subgraph TRAEFIK_ROUTER[TRAEFIK HTTP ROUTER]
                    TRAEFIK_ROUTER_APP
                end
            end

            subgraph APP_CONTAINER[CCTEAM APP CONTAINER]
                DOCKER_APP_PORT
                DOCKER_APP_MANAGEMENT_PORT
            end

            subgraph DB_CONTAINER[MARIADB CONTAINER]
                DOCKER_DB_PORT
            end

            subgraph PROMETHEUS_CONTAINER[PROMETHEUS CONTAINER]
                DOCKER_PROMETHEUS_PORT
            end

            subgraph PHPMYADMIN_CONTAINER[PHPMYADMIN CONTAINER]
                DOCKER_PHPMYADMIN_PORT
            end
        end
    end

   INCOMING_REQUEST --> DOCKER_TRAEFIK_PORT443
   INCOMING_REQUEST --> DOCKER_TRAEFIK_PORT80
   DOCKER_TRAEFIK_PORT80 --> TRAEFIK_MIDDLEWARE_REDIRECT
   TRAEFIK_MIDDLEWARE_REDIRECT -.->|301 to https| INCOMING_REQUEST
   DOCKER_TRAEFIK_PORT443 --> TRAEFIK_MIDDLEWARE_CROWDSEC
   TRAEFIK_MIDDLEWARE_CROWDSEC -->|IP not banned| TRAEFIK_ROUTER_APP
   TRAEFIK_ROUTER_APP -->|traefik - public - net,<br>JWT checked by the API| DOCKER_APP_PORT
   DOCKER_APP_PORT -->|ccteam - net| DOCKER_DB_PORT
   DOCKER_PROMETHEUS_PORT -->|metrics, prometheus - ccteam - net| DOCKER_APP_MANAGEMENT_PORT
   DOCKER_PHPMYADMIN_PORT -->|phpmyadmin - net| DOCKER_DB_PORT
   DOCKER_APP_PORT -.->|push notifications| FIREBASE

Things to notice :

only the port 5001 is routed by Traefik : the actuator (metrics, health) listens on the management port 8081, only reachable by Prometheus through prometheus-ccteam-net (see Prometheus), except the readiness probe /readyz served on the main port for Gatus
the database is never on a Traefik network : the application reaches it on ccteam-net, and PhpMyAdmin on phpmyadmin-net
no IP whitelist on the router, the mobile application is used from anywhere : CrowdSec filters the known attackers, and the API rejects any request without a valid JWT token

Create a directory to hold the app :

mkdir /opt/apps/ccteam
cd /opt/apps/ccteam

Create the Dockerfile and docker-compose.yml files based on the files in the ccteam folder in this project.

[!NOTE] The application container joins the prometheus-ccteam-net network, created by the Prometheus stack : start Prometheus first, or remove that network from the docker-compose.yml file if you do not monitor the API.

In the same directory, create a .env file to hold the environment variables :

MARIADB_ROOT_PASSWORD=<root_password>
MARIADB_DATABASE=<db_name>
MARIADB_USER=<username>
MARIADB_PASSWORD=<password>
MAIL_SERVER_HOST=<mail_server_host>
MAIL_SERVER_PORT=<mail_server_port>
MAIL_SERVER_USERNAME=<mail_server_username>
MAIL_SERVER_PASSWORD=<mail_server_password>
JWT_SECRET=<jwt_secret>
JWT_EXPIRATION_TIME=<jwt_expiration_time>

Move the application JAR file (ccteam-graphql.jar) into the current directory.

Start :

sudo docker-compose up -d

This will create 2 containers :

A container holding the MariaDB database
A container holding the Java application (based on the provided Dockerfile), exposed on port 5001

Then the API is available at : https://ccteam.example.com/ccteam-gql/graphql

You will get access denied as you need a valid JWT token, but it confirms that the service is running correctly :

{
  "errors": [
    {
      "cause": null,
      "stackTrace": null,
      "extensions": {
        "errorCode": "no_token"
      },
      "errorType": "DataFetchingException",
      "locations": null,
      "message": "Full authentication is required to access this resource",
      "path": null,
      "suppressed": [],
      "localizedMessage": "Full authentication is required to access this resource"
    }
  ],
  "data": null
}

Scale to zero with Sablier

Some of our services will be accessed quite rarely (i.e. UIs of monitoring tools, websites open only to family through VPN, etc.), it would be a shame to leave them running for days and waste resources while there are no requests, wouldn't it ?

That's why we're going to use Sablier, a little tool that lets you start / stop containers on demand (also known as "scale-to-zero"). Basically it allows to start a container when a request arrives, and stop it after a period of inactivity.

Sablier provides 2 strategies, a dynamic strategy which provides a waiting page while the container is not ready, and a blocking strategy which hangs the request until the container is ready.

We will use the dynamic strategy, well suited for a user that would access a frontend directly and expects to see a loading page. The blocking strategy is better suited for API communication.

Basically here is how it works when using the dynamic strategy with Traefik :

flowchart LR
    style INCOMING_REQUEST fill: #45d8ff57
    style TRAEFIK_CONTAINER fill: #fe73685e
    style SABLIER_CONTAINER fill: #fe73685e
    style APP_CONTAINER fill: #fe73685e
    style TRAEFIK_MIDDLEWARE fill: #dffc1e37
    DOCKER_SABLIER_PORT{{10000/tcp}}
    DOCKER_APP_PORT{{myapp port}}
    WAITING_PAGE(waiting page)
    TRAEFIK_MIDDLEWARE_APP(sablier-myapp)
    INCOMING_REQUEST((INCOMING<br/>REQUEST))
    INCOMING_REQUEST --> TRAEFIK_MIDDLEWARE_APP

    subgraph SABLIER_CONTAINER[SABLIER CONTAINER]
        DOCKER_SABLIER_PORT
        WAITING_PAGE
    end

    subgraph APP_CONTAINER[APP CONTAINER]
        DOCKER_APP_PORT
    end

    subgraph TRAEFIK_CONTAINER[TRAEFIK CONTAINER]
        subgraph TRAEFIK_MIDDLEWARE[TRAEFIK MIDDLEWARES]
            TRAEFIK_MIDDLEWARE_APP
        end

        TRAEFIK_MIDDLEWARE_APP -.->|request session status| DOCKER_SABLIER_PORT
        DOCKER_SABLIER_PORT -.->|" return status header "| TRAEFIK_MIDDLEWARE_APP
        TRAEFIK_MIDDLEWARE_APP -->|" ready "| DOCKER_APP_PORT
        TRAEFIK_MIDDLEWARE_APP -->|" not ready "| WAITING_PAGE
        DOCKER_APP_PORT -.->|return instance status| DOCKER_SABLIER_PORT
        DOCKER_SABLIER_PORT -.->|" check instance status "| DOCKER_APP_PORT
    end

When a request arrives, a Traefik middleware is responsible to contact Sablier to know if the target container is ready or not. Sablier asks for the container status to the Docker provider, then return the result to the proxy, to either serve the waiting page or redirect to the application. It is done through a X-Sablier-Status request header value :

sequenceDiagram
    User ->> Proxy: Website Request
    Proxy ->> Sablier: Reverse Proxy Plugin Request Session Status
    Sablier ->> Provider: Request Instance Status
    Provider -->> Sablier: Response Instance Status
    Sablier -->> Proxy: Returns the X-Sablier-Status Header
    alt X-Sablier-Status` value is `not-ready`
        Proxy -->> User: Serve the waiting page
        loop until `X-Sablier-Status` value is `ready`
            User ->> Proxy: Self-Reload Waiting Page
            Proxy ->> Sablier: Reverse Proxy Plugin Request Session Status
            Sablier ->> Provider: Request Instance Status
            Provider -->> Sablier: Response Instance Status
            Sablier -->> Proxy: Returns the waiting page
            Proxy -->> User: Serve the waiting page
        end
    end
    Proxy -->> User: Content

As you see it continuously checks for instance status until it is ready, and will intend to start the underlying container if not started, or shut it down if it has reached the configured period of inactivity.

[!NOTE] Note that you need one plugin configuration (one middleware) per application set if you want to start/stop them independently or if you want to have different theme, display name, loading strategy or session duration. In the flow chart above, sablier-app is a dedicated middleware for "myapp" application, but you could have several of them.

Install Sablier

Sablier can be installed using the binary distribution, or through Docker. We will use the Docker image.

Setting up

Create a folder to hold the configuration :

sudo mkdir /opt/apps/sablier

Then copy the docker-compose.yml file from this project's sablier directory into the /opt/apps/sablier directory.

Details

Service definition

:page_facing_up: docker-compose.yml :

version: "3.7"

services:
  sablier:
    image: sablierapp/sablier:latest
    container_name: sablier
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock
    restart: unless-stopped
    command:
      - start
      - --provider.name=docker
    networks:
      - sablier-net
      - traefik-net
    labels:
      - "traefik.enable=true"
      # here we will add middleware configuration, see later in this guide

networks:

  sablier-net:
    name: sablier-net

  traefik-net:
    name: traefik-net
    external: true

Essentially :

We bind the Docker socket to the container because the Docker provider communicates with the docker.sock socket to start and stop containers on demand
We specify the command to start the server with the parameter to set the provider name (docker)
It runs in its own network (sablier-net) but must also share the same network as Traefik (traefik-net) so it can be discovered

Install Traefik plugin

There are plugins available for easier integration with major reverse proxies, Traefik in particular. Sablier is designed as an API that can be used on its own, reverse proxy integrations acts as a client of that API.

Thus, simply add the following into the Traefik static configuration file (traefik.yml) to load the plugin :

experimental:
  plugins:
    sablier:
      moduleName: "github.com/sablierapp/sablier"
      version: "v1.7.0"

You can take a look at the apps/traefik/traefik.yml file from this repository.

Configure target applications

In order for Sablier to be able to contact the containers to start and stop them, we need to change the configuration of the target service to use a dynamic configuration file instead of Docker labels. Indeed, Traefik no longer has access to container labels when a container is not running.

We will configure Sablier for the Dashdot application as an example, but it can be applied to any container :

flowchart LR
    style INCOMING_REQUEST fill: #45d8ff57
    style TRAEFIK_CONTAINER fill: #6601013c
    style SABLIER_CONTAINER fill: #6601013c
    style DASHDOT_CONTAINER fill: #6601013c
    style TRAEFIK_ROUTER fill: #defc1e37
    style TRAEFIK_MIDDLEWARE fill: #defc1e37
    style SERVER_DEVICE fill: #fec9bf5e
    style CONTAINER_ENGINE fill: #67010030
    DOCKER_TRAEFIK_PORT443{{443/tcp}}
    DOCKER_TRAEFIK_PORT80{{80/tcp}}
    DOCKER_SABLIER_PORT{{10000/tcp}}
    DOCKER_DASHDOT_PORT{{3001/tcp}}
    WAITING_PAGE(Waiting page)
    TRAEFIK_ROUTER_APP(dashdot.example.com)
    TRAEFIK_MIDDLEWARE_REDIRECT(HTTPS redirect<br/>on the web entrypoint)
    TRAEFIK_MIDDLEWARE_DASHDOT(sablier-dashdot)
    INCOMING_REQUEST((INCOMING<br/>REQUEST))
    INCOMING_REQUEST --> DOCKER_TRAEFIK_PORT443
    INCOMING_REQUEST --> DOCKER_TRAEFIK_PORT80

    subgraph SERVER_DEVICE[MINI_PC]
        subgraph CONTAINER_ENGINE[DOCKER]
            subgraph SABLIER_CONTAINER[SABLIER CONTAINER]
                DOCKER_SABLIER_PORT
                WAITING_PAGE
            end

            subgraph DASHDOT_CONTAINER[DASHDOT CONTAINER]
                DOCKER_DASHDOT_PORT
            end

            subgraph TRAEFIK_CONTAINER[TRAEFIK CONTAINER]
                DOCKER_TRAEFIK_PORT443 --> TRAEFIK_ROUTER
                DOCKER_TRAEFIK_PORT80 ---> TRAEFIK_MIDDLEWARE_REDIRECT

                subgraph TRAEFIK_ROUTER[TRAEFIK HTTP ROUTER]
                    TRAEFIK_ROUTER_APP
                end

                subgraph TRAEFIK_MIDDLEWARE[TRAEFIK MIDDLEWARES]
                    TRAEFIK_MIDDLEWARE_REDIRECT
                    TRAEFIK_MIDDLEWARE_DASHDOT
                end

                TRAEFIK_MIDDLEWARE_REDIRECT -.->|301 to https| DOCKER_TRAEFIK_PORT443
                TRAEFIK_ROUTER_APP --> TRAEFIK_MIDDLEWARE_DASHDOT
                TRAEFIK_MIDDLEWARE_DASHDOT -->|check status| DOCKER_SABLIER_PORT
                DOCKER_SABLIER_PORT -->|return status| TRAEFIK_MIDDLEWARE_DASHDOT
                TRAEFIK_MIDDLEWARE_DASHDOT -->|ready| DOCKER_DASHDOT_PORT
                TRAEFIK_MIDDLEWARE_DASHDOT -->|not ready| WAITING_PAGE
            end

        end
    end

In the above flow chart we have named the Traefik middleware sablier-dashdot because it is specific to the Dashdot application, but you can absolutely create one to manage several services, or one for each service.

So let's transfer the labels from the service configuration file to a file in our Traefik dynamic configuration. I personally use a file per service, for example for Dashdot, the configuration will be held in a file dashdot.yml in the dynamic folder :

Note that you need to define a volume to bind Traefik dynamic configuration to the container, in addition to the static configuration (simply create a dynamic folder in /opt/apps/traefik) :

sudo mkdir /opt/apps/traefik/dynamic
sudo vi /opt/apps/traefik/dynamic/dashdot.yml

Then add the volume in the Traefik service configuration (docker-compose.yml) :

volumes:
  - ./dynamic:/etc/traefik/dynamic:ro

So Traefik will handle every YAML file placed in the dynamic directory.

That's it, the following labels :

    labels:
      - "traefik.enable=true"
      - "traefik.http.routers.dashdot.rule=Host(`dashdot.example.com`)"
      - "traefik.http.routers.dashdot.entrypoints=websecure"
      - "traefik.http.routers.dashdot.tls.certresolver=default"
      - "traefik.http.routers.dashdot.middlewares=vpn-whitelist"
      - "traefik.http.services.dashdot.loadbalancer.server.port=3001"
      - "traefik.docker.network=traefik-net"

becomes the following inside dashdot.yml :

http:
  services:
    dashdot:
      loadBalancer:
        servers:
          - url: http://dashdot:3001

  routers:
    dashdot:
      rule: 'Host(`dashdot.example.com`)'
      entryPoints:
        - websecure
      tls:
        certResolver: default
      service: dashdot
      middlewares:
        - vpn-whitelist@docker
        - sablier-dashdot@docker

As you see, the only thing we added is the sablier-dashdot middleware reference, which defines the strategy to use to respond to any incoming HTTP request when the corresponding container is not running :

Then add the labels for the sablier-dashdot middleware configuration into the sablier service configuration (docker_compose.yml file) :

    labels:
      - "traefik.enable=true"
      # Dashdot
      - "traefik.http.middlewares.sablier-dashdot.plugin.sablier.names=dashdot"
      - "traefik.http.middlewares.sablier-dashdot.plugin.sablier.sablierUrl=http://sablier:10000"
      - "traefik.http.middlewares.sablier-dashdot.plugin.sablier.sessionDuration=5m"
      - "traefik.http.middlewares.sablier-dashdot.plugin.sablier.dynamic.theme=hacker-terminal"
      - "traefik.http.middlewares.sablier-dashdot.plugin.sablier.dynamic.displayName=Dashdot"
      - "traefik.http.middlewares.sablier-dashdot.plugin.sablier.dynamic.refreshFrequency=1s"
      - "traefik.http.middlewares.sablier-dashdot.plugin.sablier.dynamic.showDetails=true"

Add here any other middleware that would need a different configuration for other services.

Basically, it defines a Traefik middleware to configure the Sablier plugin to :

Provide the name of the service (s) to be checked
The URL to Sablier
The session duration (will stop the container after that period)
The theme for the waiting page
The display name of the target service (s), to be displayed on the waiting page
The refresh frequency of the waiting page
Show the loading instances details

Here is how the "hacker-terminal" waiting page looks like while starting the Dashdot container :

Backup

We have so far set up a structure with a folder per stack/container (in /opt/apps). That way each stack definition (Docker Compose file) and bind mount data is fully contained in that single folder.

The only exception is named volumes, which store data in the /var/lib/docker/volumes directory. This includes the databases of some applications.

This is the only data that really concerns us, thanks to Docker, the system has hardly been modified at all, so there's no need to back it up completely (like doing entire system image backup).

So there are three things we have to worry about in terms of backup :

the content of the /opt/apps directory, holding services configuration and containers bound data (including the secrets : .env files, acme.json, the PocketID encryption key, ...)
the content of the /var/lib/docker/volumes, holding the Docker container named volumes data
the databases (MariaDB for CCTeam, Lychee and Defrag-life, PostgreSQL for Ghostfolio and Medusa) : copying the files of a running database gives an inconsistent copy, they must be dumped first

The backups are done by Backrest every evening, to a Windows PC on the local network, which then synchronizes them to a cloud. That gives the classic 3-2-1 rule : the data on the mini PC, a copy on the Windows PC, and an off-site copy in the cloud.

flowchart LR
    style N100 fill: #fec9bf5e
    style PC fill: #45d8ff57
    style CLOUD fill: #bbfd795e
    TIMER[systemd timer\n19:45] -->|mariadb - dump, pg_dump| DUMPS[(database dumps)]

    subgraph N100[MINI PC]
        APPS[(/opt/apps)]
        VOLUMES[(Docker volumes)]
        DUMPS
        BACKREST[Backrest\nrestic]
        TIMER
    end

    subgraph PC[WINDOWS PC]
        REPO[(restic repository\nD:\Backups\N100)]
        CLOUD_CLIENT[Cloud client]
    end

    subgraph CLOUD[CLOUD]
        CLOUD_COPY[(off-site copy)]
    end

    APPS -->|read - only| BACKREST
    VOLUMES -->|read - only| BACKREST
    DUMPS -->|read - only| BACKREST
    BACKREST -->|20:00, SFTP, encrypted snapshots| REPO
    REPO --> CLOUD_CLIENT
    CLOUD_CLIENT -->|sync| CLOUD_COPY

The Manual backups section below keeps the one-off commands and tools (rsync, FreeFileSync, volume archives, database dumps), useful before a migration or to copy a single volume.

Backrest

Backrest is a web interface on top of restic, a proven backup tool. Unlike a synchronization (rsync, a mirror), restic keeps snapshots : every backup is a complete, dated point in time, and restoring the state of last Tuesday is always possible, even if a file got corrupted or deleted since (a mirror would have propagated the damage).

deduplicated : only the blocks that changed since the last snapshot are sent and stored, a daily backup of the whole server takes a few seconds and a few megabytes
encrypted on the mini PC, before leaving it : the repository is unreadable without its password, which is also what makes it safe to push it to a cloud storage
retention policy : keep the last 7 daily, 4 weekly and 6 monthly snapshots, for instance, the older ones are removed automatically
restore from the interface : browse any snapshot like a file explorer, download a file or a folder, or restore it to a folder
scheduling, hooks before and after each operation, notifications on error

It is an administration tool : it sits on the private network, behind the IP whitelist. Backrest has no native OIDC support, it is put behind PocketID with the Traefik plugin, and its own login is kept on top : it reads everything (secrets included) and holds the repository password, and without its own login any container of traefik-private-net could use it directly, without going through Traefik. Its session lasts long, the double login is rare.

Here is an overview of the network flow :

flowchart LR
    style INCOMING_REQUEST fill: #45d8ff57
    style TRAEFIK_CONTAINER fill: #6601013c
    style APP_CONTAINER fill: #6601013c
    style TRAEFIK_ROUTER fill: #defc1e37
    style TRAEFIK_MIDDLEWARE fill: #defc1e37
    style SERVER_DEVICE fill: #fec9bf5e
    style CONTAINER_ENGINE fill: #67010030
    style WINDOWS_PC fill: #45d8ff57
    DOCKER_TRAEFIK_PORT443{{443/tcp}}
    DOCKER_TRAEFIK_PORT80{{80/tcp}}
    DOCKER_APP_PORT{{9898/tcp}}
    TRAEFIK_ROUTER_APP(backrest)
    TRAEFIK_MIDDLEWARE_REDIRECT(HTTPS redirect on<br>the web entrypoint)
    TRAEFIK_MIDDLEWARE_IP_WHITELIST(IP whitelist)
    TRAEFIK_MIDDLEWARE_OIDC(PocketID auth)
    SOURCES[(/opt/apps\nDocker volumes)]
    INCOMING_REQUEST((INCOMING\nREQUEST))

    subgraph SERVER_DEVICE[MINI PC]
        SOURCES

        subgraph CONTAINER_ENGINE[DOCKER]
            subgraph TRAEFIK_CONTAINER[TRAEFIK CONTAINER]
                DOCKER_TRAEFIK_PORT80
                DOCKER_TRAEFIK_PORT443

                subgraph TRAEFIK_ROUTER[TRAEFIK HTTP ROUTER]
                    TRAEFIK_ROUTER_APP
                end

                subgraph TRAEFIK_MIDDLEWARE[TRAEFIK MIDDLEWARES]
                    TRAEFIK_MIDDLEWARE_REDIRECT
                    TRAEFIK_MIDDLEWARE_IP_WHITELIST
                    TRAEFIK_MIDDLEWARE_OIDC
                end

            end

            subgraph APP_CONTAINER[BACKREST CONTAINER]
                DOCKER_APP_PORT
            end

        end

    end

    subgraph WINDOWS_PC[WINDOWS PC 192.168.0.12]
        SSHD{{OpenSSH 22/tcp\nSFTP only}}
        REPO[(D:\Backups\N100)]
    end

    INCOMING_REQUEST --> DOCKER_TRAEFIK_PORT443
    INCOMING_REQUEST --> DOCKER_TRAEFIK_PORT80
    DOCKER_TRAEFIK_PORT443 --> TRAEFIK_ROUTER
    DOCKER_TRAEFIK_PORT80 --> TRAEFIK_MIDDLEWARE_REDIRECT
    TRAEFIK_MIDDLEWARE_REDIRECT -.->|301 to https| INCOMING_REQUEST
    TRAEFIK_ROUTER_APP --> TRAEFIK_MIDDLEWARE_IP_WHITELIST
    TRAEFIK_MIDDLEWARE_IP_WHITELIST --> TRAEFIK_MIDDLEWARE_OIDC
    TRAEFIK_MIDDLEWARE_OIDC -->|then Backrest login| DOCKER_APP_PORT
    SOURCES -->|read - only mounts| DOCKER_APP_PORT
    SSHD --> REPO
    DOCKER_APP_PORT -->|SFTP, key authentication| SSHD

Windows target

The repository lives on a Windows PC of the local network, reached through SFTP with the OpenSSH server built into Windows. SFTP is preferred over a Windows share (SMB) mounted on the mini PC :

	SFTP (OpenSSH)	Windows share (SMB)
On the mini PC	nothing to mount, restic opens the connection for the backup and closes it	a permanent CIFS mount (`fstab`, credentials file) passed to the container
PC switched off	the backup fails cleanly with an explicit error, and is retried at the next run	the mount goes stale and blocks. Worse, if the mount is missing, restic writes the repository into the empty mount point, on the mini PC disk, silently
Authentication	dedicated SSH key, no password stored	Windows password in a file on the mini PC
Other quirks	few	file locks, NTFS permissions, SMB versions, antivirus

A switched off PC will happen regularly : with SFTP it is only a missed backup, never a hung or misplaced one.

The PowerShell script backrest/windows/setup-sftp-target.ps1 of this project prepares the PC in one go. Run it once in an elevated PowerShell (Run as administrator), with the public key generated on the mini PC (see Setting up below) :

.\setup-sftp-target.ps1 -PublicKey "ssh-ed25519 AAAA... backrest@n100" -N100Ip 192.168.0.16

It :

installs and starts the OpenSSH server
creates a local backup user, not administrator, hidden from the logon screen, with a random password nobody needs (key authentication only)
creates the repository folder D:\Backups\N100, writable by backup only, readable by your own account (so that the cloud client can read it)
restricts backup to SFTP (ForceCommand internal-sftp, no shell, no forwarding), with its authorized key in C: \ProgramData\ssh\backup_authorized_keys
restricts the firewall rule of OpenSSH to the IP address of the mini PC

[!NOTE] A few Windows specific traps the script avoids :

the key of an administrator account is not read from ~\.ssh\authorized_keys but from administrators_authorized_keys, with very strict permissions : a dedicated non administrator user avoids it, and limits the damage if the key ever leaks

the profile folder of a user (C:\Users\backup) only exists after its first logon, and a folder created by hand beforehand is not used by Windows : the key is declared in ProgramData with a Match User block instead

the firewall rule created by the OpenSSH installation accepts everybody

Give the PC a fixed IP address (DHCP reservation on the router), and avoid spaces in the repository path : restic accepts them, but they complicate the quoting in the SFTP path and on the command line.

[!TIP] The repository is on a mechanical drive : it may need a few seconds to spin up, the first write simply waits. The SSH configuration keeps the connection alive meanwhile (ServerAliveInterval).

Database dumps

A running database (MariaDB, PostgreSQL) copied file by file gives an inconsistent copy, which may not even start once restored. The databases are therefore dumped by the host just before the backup, and the raw database volumes are excluded from the backup plan.

The backrest/scripts/dump-databases.sh script runs on the host (it needs docker exec, which Backrest must not have : the Docker socket would give it root on the host), started by a systemd timer at 19:45, 15 minutes before the backup. For each database container, it runs mariadb-dump or pg_dump inside the container, with the credentials of its own environment, and writes a compressed dump in /opt/apps/backrest/dumps, which is part of the backup. A new database is one line in the MARIADB_CONTAINERS or POSTGRES_CONTAINERS list of the script, plus its volume in the excludes of the backup plan.

[!WARNING] The database images use the mariadb:latest tag : a pull may start a new major version on existing data. The system tables must then be upgraded, which MARIADB_AUTO_UPGRADE=1 does automatically at startup. Without it, the application keeps working, but the dump fails, as it did for Lychee :
mariadb-dump: Couldn't execute 'SHOW FUNCTION STATUS WHERE Db = 'lychee'': Column count of mysql.proc is wrong.
Expected 22, found 21. Created with MariaDB 110402, now running 120303. Please use mariadb-upgrade to fix this error
Fix it by upgrading the system tables once (after a dump without --routines, just in case), then add MARIADB_AUTO_UPGRADE=1 to the Compose file :
sudo docker exec lychee-db sh -c 'MYSQL_PWD="$MYSQL_ROOT_PASSWORD" mariadb-upgrade -uroot'
Pinning the major version (mariadb:12) is another option : the upgrade then happens when you decide it.

The SQLite databases (PocketID, Gatus, Grafana, Arcane, GoatCounter, Homebox, CrowdSec, ...) are copied as they are : they write very little at that time of the day and the risk is low, but not zero. A sqlite3 .backup can be added to the script for the most critical ones.

Setting up Backrest

Create the folders and the SSH key Backrest will use to reach the PC :

sudo mkdir -p /opt/apps/backrest/ssh
sudo ssh-keygen -t ed25519 -N "" -C backrest@n100 -f /opt/apps/backrest/ssh/id_ed25519
sudo cat /opt/apps/backrest/ssh/id_ed25519.pub

Prepare the Windows PC with the displayed public key (see Windows target), then record its host key, so that restic can verify it is talking to the right machine :

sudo sh -c 'ssh-keyscan -H 192.168.0.12 > /opt/apps/backrest/ssh/known_hosts'

Then :

copy the docker-compose.yml file and the ssh/config, scripts and systemd folders from this project's backrest directory into the /opt/apps/backrest directory, and protect the SSH folder :
```
sudo chmod 700 /opt/apps/backrest/ssh && sudo chmod 600 /opt/apps/backrest/ssh/*
sudo chmod +x /opt/apps/backrest/scripts/dump-databases.sh
```

install the systemd units, enable the timer, and run a first dump to check it :

sudo cp /opt/apps/backrest/systemd/backrest-db-dump.* /etc/systemd/system/
sudo systemctl daemon-reload
sudo systemctl enable --now backrest-db-dump.timer
sudo /opt/apps/backrest/scripts/dump-databases.sh

copy the backrest.yml file from this project's traefik/dynamic directory into the /opt/apps/traefik/dynamic directory
create an OIDC client and its backrest-auth middleware as described in PocketID, with the callback URL https://backrest.example.com/oidc/callback and PKCE enabled, restricted to your administrators group
add a local DNS record backrest.example.com pointing to the mini PC (see Pi-hole), the service is not published on the internet

[!TIP] If the dump fails, run the script directly rather than through systemd, to see its output : a && after a failed systemctl start never runs the journalctl that would show it. A script edited on Windows may also come with CRLF line endings, which bash does not understand ($'\r': command not found). The .gitattributes file of this project forces LF on the .sh, .service and .timer files, and sed -i 's/\r$//' <file> fixes a file already copied.

Backup plan

Start the service (see Run), open https://backrest.example.com, and create the Backrest account on the first visit. Then check that the container reaches the PC :

sudo docker exec -it backrest sftp windows-pc

ls /D:/Backups/N100 must work (the drive letter is written with a leading / in SFTP paths).

Add the repository :

URI : sftp:windows-pc:/D:/Backups/N100 (windows-pc is the host alias of the SSH configuration)
password : generate it and store it in your password manager. Without it the backups are unreadable, by anybody, forever
prune every week, check every month

Then the plan :

paths : /sources/apps and /sources/volumes
schedule : 0 20 * * * (every day at 20:00, when the PC is usually on)
retention : 7 daily, 4 weekly, 6 monthly

excludes :

Exclude	Why
`/sources/apps/backrest/cache`, `/sources/apps/backrest/tmp`, `/sources/apps/backrest/restore`	cache and temporary files of Backrest itself, restored files
`/sources/volumes/ccteam-db-vol`, `/sources/volumes/lychee-db-vol`, `/sources/volumes/defrag-life-db-vol`, `/sources/volumes/ghostfolio-db-vol`, `/sources/volumes/medusa-db-vol`	raw database files, inconsistent when copied hot : the dumps are what gets restored
`/sources/volumes/medusa-redis-vol`	events and jobs of Medusa : restored next to an older dump, they would be replayed on data that no longer match
`/sources/volumes/prometheus-data`	Prometheus time series, inconsistent when copied hot, and only metrics
`/sources/volumes/metadata.db`, `/sources/volumes/backingFsBlockDev`	internal files of Docker, not volumes

a hook on CONDITION_ANY_ERROR, of type Shoutrrr, to be notified of a failed backup (the PC was switched off, for instance), for instance by e-mail with the same SMTP account as Gatus : smtp://<user>:<password>@<host>:465/?from=<sender>&to=<recipient>

Run the plan once manually (Backup now), and test a restore right away (see Restore) : a backup that was never restored is not a backup.

Details

Service definition

:page_facing_up: docker-compose.yml :

services:

  backrest:
    image: ghcr.io/garethgeorge/backrest:latest
    container_name: backrest
    # restic records the host name in every snapshot : keep it stable, or the retention policy sees a new host
    hostname: backrest
    restart: unless-stopped
    environment:
      TZ: "Europe/Zurich"
      BACKREST_DATA: /data
      BACKREST_CONFIG: /config/config.json
      XDG_CACHE_HOME: /cache
      TMPDIR: /tmp
    volumes:
      # Backrest configuration (repositories, plans, users) and operation history
      - ./config:/config
      - ./data:/data
      # restic cache and temporary files, excluded from the backup
      - ./cache:/cache
      - ./tmp:/tmp
      # SSH key, config and known_hosts to reach the SFTP repository on the Windows PC
      - ./ssh:/root/.ssh:ro
      # What to back up, READ-ONLY : the configuration of every service (including the database dumps written in
      # /opt/apps/backrest/dumps by the host timer) and the Docker volumes
      - /opt/apps:/sources/apps:ro
      - /var/lib/docker/volumes:/sources/volumes:ro
      # Restores land here, never over the sources (which are read-only anyway)
      - ./restore:/restore
    networks:
      - traefik-private-net

networks:

  traefik-private-net:
    name: traefik-private-net
    external: true

SSH configuration

:page_facing_up: ssh/config :

# SSH configuration used by restic (SFTP repository), mounted as /root/.ssh in the Backrest container.
# Repository URI in Backrest : sftp:windows-pc:/D:/Backups/N100
Host windows-pc
    HostName 192.168.0.12
    User backup
    Port 22
    IdentityFile /root/.ssh/id_ed25519
    IdentitiesOnly yes
    # Generated with : ssh-keyscan -H 192.168.0.12 > known_hosts
    StrictHostKeyChecking yes
    UserKnownHostsFile /root/.ssh/known_hosts
    # The repository is on a mechanical drive that may need a few seconds to spin up : keep the connection alive
    # instead of dropping it
    ServerAliveInterval 30
    ServerAliveCountMax 6

Database dumps

:page_facing_up: scripts/dump-databases.sh :

#!/bin/bash
#
# Dumps the databases into /opt/apps/backrest/dumps, before the Backrest backup.
# Copying the files of a running database gives an inconsistent copy : the dumps are what gets restored, the raw
# database volumes are excluded from the backup plan.
#
# Run on the HOST (it needs docker exec), by the backrest-db-dump systemd timer.
# The credentials are read from the environment of each database container, nothing is stored here.

set -uo pipefail

DUMP_DIR=/opt/apps/backrest/dumps
MARIADB_CONTAINERS=(ccteam-db lychee-db defrag-life-db)
POSTGRES_CONTAINERS=(ghostfolio-db medusa-db)

# The password goes through MYSQL_PWD rather than the command line, the MYSQL_* names are the legacy aliases used by
# some of the stacks
MARIADB_DUMP='MYSQL_PWD="${MARIADB_ROOT_PASSWORD:-$MYSQL_ROOT_PASSWORD}" exec mariadb-dump -uroot \
    --single-transaction --routines --events \
    --databases "${MARIADB_DATABASE:-$MYSQL_DATABASE}"'

# Local socket inside the container, trusted by the official image : no password needed.
# --clean --if-exists : the dump drops the existing objects before recreating them, it can be restored over a database
POSTGRES_DUMP='exec pg_dump -U "$POSTGRES_USER" --clean --if-exists "$POSTGRES_DB"'

mkdir -p "$DUMP_DIR"
chmod 700 "$DUMP_DIR"

status=0

# dump <container> <command run inside the container, writing the dump on its standard output>
dump() {
    local container=$1 command=$2

    if ! docker ps --format '{{.Names}}' | grep -qx "$container"; then
        echo "$container is not running, skipped" >&2
        status=1
        return
    fi

    # Written to a temporary file first : a failed dump never replaces the previous good one
    if docker exec "$container" sh -c "$command" | gzip > "$DUMP_DIR/$container.sql.gz.tmp"; then
        mv "$DUMP_DIR/$container.sql.gz.tmp" "$DUMP_DIR/$container.sql.gz"
        echo "$container dumped ($(du -h "$DUMP_DIR/$container.sql.gz" | cut -f1))"
    else
        rm -f "$DUMP_DIR/$container.sql.gz.tmp"
        echo "$container dump FAILED" >&2
        status=1
    fi
}

for container in "${MARIADB_CONTAINERS[@]}"; do
    dump "$container" "$MARIADB_DUMP"
done

for container in "${POSTGRES_CONTAINERS[@]}"; do
    dump "$container" "$POSTGRES_DUMP"
done

exit $status

:page_facing_up: systemd/backrest-db-dump.service :

[Unit]
Description=Dump the databases before the Backrest backup
After=docker.service
Requires=docker.service

[Service]
Type=oneshot
ExecStart=/opt/apps/backrest/scripts/dump-databases.sh

:page_facing_up: systemd/backrest-db-dump.timer :

# Runs 15 minutes before the backup plan of Backrest (20:00), keep both schedules in sync
[Unit]
Description=Dump the databases before the Backrest backup

[Timer]
OnCalendar=*-*-* 19:45:00

[Install]
WantedBy=timers.target

Traefik routing

:page_facing_up: backrest.yml :

http:
  services:
    backrest:
      loadBalancer:
        servers:
          - url: http://backrest:9898

  routers:
    backrest:
      rule: 'Host(`backrest.example.com`)'
      entryPoints:
        - websecure
      tls:
        certResolver: default
      service: backrest
      # Backrest has no native OIDC support, the authentication is done by Traefik (see PocketID).
      # Its own login is kept on top : it reads everything and holds the repository password
      middlewares:
        - vpn-whitelist@file
        - backrest-auth@file

And the middleware, in pocketid.yml :

    backrest-auth:
      plugin:
        traefik-oidc-auth:
          Secret: "<secret>"
          Provider:
            Url: "http://pocketid:1411/"
            ClientId: "<oidc_client_id>"
            ClientSecret: "<oidc_client_secret>"
            UsePkce: true
          Scopes: [ "openid", "profile", "email" ]

Things to notice :

the sources are mounted read-only : Backrest can read everything but modify nothing, a restore goes to /restore (/opt/apps/backrest/restore on the host), from where you copy the files back yourself
Backrest does not hold the Docker socket, this is why the database dumps are done by the host
hostname is fixed : restic records it in every snapshot, and the retention policy applies per host. The default host name of a container is its random ID, which would change at every recreation
the private key, the generated files and the dumps are ignored by Git (backrest/.gitignore), never commit them
the Backrest configuration (config/config.json) holds the repository password : it is part of the backup, inside the encrypted repository, but keep the password in your password manager as well, it is the only way to read the backups if the mini PC is lost
Gatus checks that Backrest answers, and the error hook reports a failed backup

Run Backrest

Prepare the Windows PC first, then run the Compose file :

sudo docker-compose -f /opt/apps/backrest/docker-compose.yml up -d

You should end-up with a running backrest container, and Traefik picks up the dynamic configuration file without restarting. Configure the repository and the plan as described in Backup plan.

Restore from Backrest

From the interface : open the plan or the repository, pick a snapshot and browse it. For a file or a folder you can :

download it straight from the browser (a folder comes as an archive)
restore it to /restore (/opt/apps/backrest/restore on the host), then compare it or copy it back

From the Windows PC, without the mini PC (the day it is dead is precisely when you need it) : the repository is a plain folder on the PC, the single restic.exe binary is enough, with the repository password :

restic -r D:\Backups\N100 snapshots
restic -r D:\Backups\N100 dump latest /sources/apps/traefik/traefik.yml > traefik.yml
restic -r D:\Backups\N100 restore latest --target D:\restore --include /sources/apps/pocketid

Replace latest with the ID of an older snapshot, from the first command. To compare two versions :

restic -r D:\Backups\N100 diff <snapshot_1> <snapshot_2> lists the files added, removed or modified between two snapshots
the content of a file : dump the old version and compare it with the current one in any diff tool (WinMerge, code --diff old.yml current.yml, ...)
on Linux, restic mount exposes every snapshot as a regular folder, where a simple diff works (not available on Windows)

To restore a database, load its dump into the (running) database container. For MariaDB :

gunzip -c ccteam-db.sql.gz | sudo docker exec -i ccteam-db sh -c 'MYSQL_PWD="$MARIADB_ROOT_PASSWORD" mariadb -uroot'

The dump contains the CREATE DATABASE / USE statements (--databases), the target database is recreated as it was.

For PostgreSQL :

gunzip -c ghostfolio-db.sql.gz | sudo docker exec -i ghostfolio-db sh -c 'psql -U "$POSTGRES_USER" -d "$POSTGRES_DB"'

The dump drops the existing objects before recreating them (--clean --if-exists), it can be loaded over the current database.

[!NOTE] The cloud copy is a mirror of the repository on the PC : if the repository gets corrupted or deleted there, the cloud copy follows at the next synchronization, only the file history of the cloud (Rewind) keeps an older version. Avoid running the synchronization while a backup or a prune writes to the repository, shifting the schedules is enough.

Manual backups

The following commands and tools are not scheduled, they are useful for a one-off copy : before a migration, to move a volume to another machine, or to export a database by hand.

Files

Rsync

The simplest way to copy the content of our N100 server is by using rsync.

rsync (remote sync) is a utility for transferring and synchronizing files between a computer and a storage drive and across networked computers by comparing the modification times and sizes of files.

We can use it to copy the file system (actually only required files) to another machine (such as a Windows computer on the local network, or any external drive connected to it) through a mount point.

First, make sure to have a folder on the machine that will hold the backup (Windows in my case) that is shared and have enough storage for the server backup :
- Create a folder to hold the backup data (i.e. E:\data\N100 backup),
- Right-click on the folder
- Select Properties > Sharing tab
- Click Share... and choose the user with whom you want to share the folder (you can either use your default Windows user or create a specific user for that)
- Assign the appropriate permissions (at least Read access).
- Click Share, then Done.
- Take note of the network path of the share (i.e. \DESKTOP-ABCDEF\N100 backup).
Secondly, mount the shared Windows folder on the server :

To mount a Windows share, you need to install the cifs-utils package :
```
sudo apt install cifs-utils
```
Then create a directory where you will mount the shared folder. For example:
```
sudo mkdir /mnt/windows
```
And mount the shared folder from the Windows machine to the server :
```
sudo mount -t cifs -o username=my_windows_user "//DESKTOP-ABCDEF/N100 backup" /mnt/windows
```
Explanation:
- -t cifs: Specifies that you’re using the CIFS protocol
- //DESKTOP-ABCDEF/N100 backup: The network path to the Windows share
- /mnt/windows: The mount point on the server
[!NOTE] You can create a file to hold the credentials for authentication, instead of specifying it in the command line (so you can protect the credentials file by setting the appropriate permissions), this can be done by using the -o credentials option of the mount command
Finally, use rsync to synchronize the files to the mount point :

Install rsync:
```
sudo apt install rsync
```
Then either sync all the file system or only some folders :
```
# all file system with some exceptions
sudo rsync -aAXv --exclude={"/dev/*","/proc/*","/sys/*","/tmp/*","/run/*","/mnt/*","/media/*","/lost+found"} / /mnt/windows
# only specified paths
sudo rsync -aAXv /home /opt/apps /var/lib/docker/volumes /var/log /mnt/windows
```
Explanation:
- -aAXv: Preserve permissions, ownership, timestamps, and device files, with verbose output
- --exclude: Exclude certain directories
- /: The root of the server, to be backed up (without excluded directories)
- /home /opt/apps /var/lib/docker/volumes /var/log: The 4 directories to be backed up
- /mnt/windows: The mount point on the server

Once the backup is complete, you can verify that the backup files are on the destination machine and that they contain all your server data.

[!NOTE] If you want to make a bit-for-bit clone of your entire disk, you can use the dd command. However, it requires more storage and time, for the time being I prefer rsync for flexibility, file-based backups, and faster cloning of only necessary files

FreeFileSync

Another solution than rsync is to simply use a tool from the Windows machine, to copy the /opt/apps folder regularly through SFTP.

One awesome tool which I've been using for years for synchronizing my disks, is named FreeFileSync.

FreeFileSync is a folder comparison and synchronization software that creates and manages backup copies of target files. Instead of copying every file every time, FreeFileSync determines the differences between a source and a target folder and transfers only the minimum amount of data needed.

Source and target folders can be remote folders (support for Google Drive and FTP/SFTP).

FreeFileSync is Open Source software, available for Windows, macOS, and Linux.

I will install the Windows version on my home Windows machine, which will be used as client to connect to the mini PC through SFTP (SSH File Transfer Protocol, allows secure file transfer trough SSH encrypted connections).

To do a mirror synchronization :

Download the software for your operating system at https://freefilesync.org/
Install and start it
Choose left and right folders :

Click the cloud icon to connect to the mini PC via SFTP and select the /opt/apps folder
Compare them :
Adapt synchronization settings if needed :
Start synchronization :

Refer to the documentation and tutorials on the software's website for more information.

Volumes

Backup

We can back up Docker volumes using docker run and tar command. This method involves creating a temporary container that mounts the named volume we want to back up, then using tar to produce an archive of the volume content.

For example to back up the Arcane volume arcane-data to the current directory :

sudo docker run --rm --mount source=arcane-data,target=/mybackup -v $(pwd):/backup busybox tar cvf /backup/arcane-data-backup.tar /mybackup

--rm will remove the container when it exits
--mount source=arcane-data,target=/mybackup will mount the arcane-data volume to the container mount point /mybackup
-v $(pwd):/backup bind mount the current directory into the container's backup directory to write the tar file to
busybox is an image of a lightweight Linux distribution with basic Unix utilities, good for that kind of quick maintenance
tar cvf /backup/arcane-data-backup.tar /mybackup will create an uncompressed tar file of all the files in the /mybackup directory

This will create a arcane-data-backup.tar archive in the current directory. The tar will contain a mybackup directory containing all volume data.

Then feel free to move it to the /opt/apps/arcane directory if you want to back it up along with that directory when using FreeFileSync (see Files), or simply move the backup file to an external server.

[!IMPORTANT] Some services may need to be stopped during backup or restore to ensure data consistency

Restore

To restore the volume :

Create a new container (this represents the container in which you wish to restore the backup) :
```
sudo docker create -v /data --name newcontainer busybox /bin/bash
```

Untar the backup files into the new container volume :

sudo docker run --rm --volumes-from newcontainer -v $(pwd):/backup busybox tar -xvf /backup/arcane-data-backup.tar --strip 1 -C /data

--rm will remove the container when it exits
--volumes-from newcontainer mounts all the volumes from the newcontainer container into the new container being started
-v $(pwd):/backup bind mount the current directory into the container's /backup directory to write the tar file to
busybox is an image of a lightweight Linux distribution with basic Unix utilities, good for that kind of quick maintenance
tar xvf /backup/arcane-data-backup.tar --strip 1 -C /data will extract the files from the tar archive in the /data directory of the container's filesystem (without the parent directory thanks to --strip 1)

Finally, you can compare the 2 volumes content to check that everything has been copied correctly :

sudo diff -qr /var/lib/docker/volumes/arcane-data /var/lib/docker/volumes/0862be139e8b9e8137c02005739071d2338fd04f6090b8a89d6b5012fc5fb33a

Databases

When applicable, we can also back up the database directly.

MySQL

For MySQL, we can use mysqldump, a command-line utility that is used to generate or restore logical backups of MySQL databases.

To export data :

mysqldump --complete-insert --skip-comments --skip-tz-utc --skip-opt --hex-blob --no-set-names --set-charset --column-statistics=0 --set-gtid-purged=OFF -P 3306 -h localhost -u <user> -p <dbname> > db_backup.sql

If you don't have the mysqldump utility installed on your environment, you can use the one embedded in the MySQL container :

docker exec <container_id> /usr/bin/mysqldump --complete-insert --skip-comments --skip-tz-utc --skip-opt --hex-blob --no-set-names --set-charset --column-statistics=0 --set-gtid-purged=OFF -P 3306 -h localhost -u <user> --password=<password_here> <dbname> > db_backup.sql

[!IMPORTANT] Again there is a slight chance that a database gets inconsistent when backing up hot files, so prefer to stop services before proceeding, but in a home lab with minimal load this is usually not an issue

To import data :

mysql -P 3306 -h localhost -u <user> -p <dbname> < db_backup.sql

Or again if you don't have the mysqldump utility installed on your environment, you can use the one from the MySQL container :

docker exec -i <container_id> /usr/bin/mysql -P 3306 -h localhost -u <user> --password=<password_here> <dbname> < db_backup.sql

Contributing

You are invited to contribute fixes or updates. I'm also open to any criticism or suggestion for improvement.

Please familiarize yourself with the README file before attempting a pull request. Also make sure to not include any sensitive information.

You can also simply fork the project and continue on your own.

Acknowledgments

Mainly :

Some cool GitHub projects related to self-hosting :
Various threads on Reddit, but especially in :
- r/selfhosted➚
- r/homelab➚
- r/pihole➚
- r/raspberry_pi➚
StackExchange network (particularly Stack Overflow, Superuser, and Server Fault):
- Q&A communities➚
Blog post about WireGuard performance tuning :
- https://www.procustodibus.com/blog/2022/12/wireguard-performance-tuning/
Lots of Google searches
Recently some AI for WireGuard and CrowdSec tweaks, and for backup setup, mainly Claude (Opus/Fable)

Of course every upstream project (especially the ones with good documentation :grin:) also deserve credit :beer:

License

Please refer to the license of each product mentioned in this guide.

Otherwise, the Apache 2.0 license applies.

Apache License, Version 2.0

Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at https://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.

The logos and names of the products belong to their respective owners, the license does not grant any right on them.

If this guide helped you, a mention of it (a link to this repository) in your own project, article or setup is always appreciated. This is a simple request, not a condition of the license.