Work

Free time · in progress

My homelab and its CI

Two small servers at home and one in the cloud, joined by my own private network. They run my git server and CI, which test and ship my websites and side projects, this one included.

  • Self-hosting
  • Networking
  • CI/CD security
  • Operations
  • Monitoring
  • Proxmox VE
  • OpenWrt
  • WireGuard
  • Headscale
  • Forgejo Actions
  • Caddy
  • Grafana
  • Docker
  • Podman

Situation

My homelab is a handful of computers I run like a company’s servers, just for me. It’s where I try the things I don’t get to do at work. I’m its only admin, so when something breaks, the alert lands on my phone.

Here’s what’s in it:

  • Two small servers at home, a Dell desktop (Intel i5-8500T, 16 GB of RAM) and a mini PC (Celeron N5105). Both run Proxmox, which splits one machine into several virtual ones. They stay two separate hosts, not a cluster: below three nodes a cluster has no real quorum, and running each one on its own is enough for me right now.
  • On the desktop: my git server (Forgejo) and the CI runners that test and ship my code, a DNS filter that blocks ads and trackers for every device in the house, Grafana dashboards and alerts, and a few web apps for the home.
  • On the mini PC: the NAS, the shared storage the other machines keep their files and backups on.
  • One server in the cloud (a VPS), the lab’s public side. It serves my websites, this one included, and coordinates the private network.
  • The router runs OpenWrt, open-source firmware that lets me run the home network’s addresses and names myself.
  • My own private network joins all of it, plus phones, laptops and a travel router, wherever they are. It’s WireGuard, coordinated by Headscale, which I host myself.

The lab also builds and ships my side projects. My public repositories live on Codeberg, the private ones on the lab’s Forgejo, and these days most of them are private.

Approach

One private network. WireGuard links each device straight to the others, encrypted. My devices run the Tailscale app, but they answer to Headscale, the open-source coordinator, which I host with my own relay for devices that can’t connect directly. The travel router has a physical switch: one side sends all its traffic home, the other only the lab’s.

Decision

Host the coordinator and the relay myself, so the part that decides who joins the network is mine.

Turned down: Tailscale’s hosted coordinator and its public relays

One certificate for internal names. Every publicly trusted HTTPS certificate goes into public logs that anyone can search (Certificate Transparency). A certificate per service would publish the name of everything I run. So all the internal services share one wildcard certificate, obtained through a DNS challenge.

Decision

One wildcard certificate for all internal names.

Turned down: one certificate per service, which publishes every name

Two CI runners. The check runner builds and tests every push and pull request, and holds no keys. The deploy runner works only on the main branch and ships the exact build that passed. Its one key can only copy files into one folder on the web server. Code from a pull request, or from a package it pulls in, never runs where a key lives.

The homelab CI: a check runner without keys, then a deploy runner with one narrow keyA push reaches the git forge. The check runner, which holds no keys, runs lint, type checks, unit tests, the build and browser tests. Only on the main branch, the deploy runner takes the exact build that was tested, copies it with rsync into one folder on the static host using its single restricted key, then runs a smoke test.Git forgeCheck runnerDeploy runnerStatic hostevery push andpull requestno keys at alllint, types, tests,build, four browsersmain only; one key,rsync into one folder,smoke test afterserves the exact buildthat was testedtested build(artifact)
How the lab’s CI tests and ships a site, and where the one key lives.

Decision

Two runners as two users: one runs untrusted code without keys, the other deploys without running package code.

Turned down: one runner whose keys every job could read, my first setup

Backups. The apps’ configuration goes to the NAS every night, and the last seven copies are kept. The private network’s coordinator gets a database copy before each upgrade, and a machine gets a snapshot before a big change, like the runner split.

Day to day.

  • I keep a dated copy of every file before I edit it, and the web server’s configuration (Caddy) is validated before every reload.
  • Versions are pinned, and each upgrade has a written rollback.
  • Health checks run every 15 minutes and report to my phone. Grafana alerts on disks and services, and a watchdog restarts the network client when it gets stuck.
  • Every deploy ends with a smoke test.
  • A read-only drift check compares the machines and containers with the state I last accepted.

The exact configurations, with a note per container, live in a private repository.

Result

  • The web apps for the home run on the same machines, with the same backups, checks and rollbacks.
  • One DNS change fixed two bugs at once. Putting the fallback resolvers in a strict order closed a race that let some lookups skip the ad filter, and ended a reverse-lookup loop. Lookups went from 2 seconds to 1 or 2 milliseconds.
  • This site is built and deployed by the lab’s CI, so if you’re reading this, the deploy runner did its job.

Next time

I’d compartmentalize the CI from day one. I’d also invest much more in redundant hardware, a reliable NAS and 3-2-1 backups (three copies, on two kinds of storage, one of them off-site). A good NAS already felt expensive a few years ago; in 2026 it’s so much worse.