Setu

Your binaries. Your servers. One control panel.

Setu 0.2: runs anything, loses nothing

Date: 2026-10-10 · Status: approved in conversation (owner decisions recorded below) · Builds on the 0.1 design (2026-10-09-setu-design.md, revision 4). Where this document is silent, 0.1 holds.

Owner decisions (2026-10-10)

  • Roadmap order after 0.2: 0.3 hosted control plane, 0.4 catalogue, 0.5 providers and billing.
  • Container runtime: rootless Podman through systemd Quadlet, one container per site running as the site's own user. Not Docker, not nspawn.
  • Off-box backups: one S3-compatible target per panel, archives encrypted with age under a key derived from the panel secret.
  • Cloudflare tunnel ingress is in 0.2.

Intent

A site is just as safe on a box that may die as on one that will not: its backups leave the server, it can be moved to another server, and a server can be drained for maintenance. Apps that ship only as images run under the same unit, limits and helper model as binaries. Servers without a public IP join through a Cloudflare tunnel. Operators get the guard rails a real deployment needs: enforced disk quotas, a deletion grace period, two-factor sign-in and alerts.

Success: on two Ubuntu 24.04 servers (one behind a tunnel, no public IP) the e2e provisions an image-based app (Miniflux's image) and a binary app, backs both up to a MinIO target with encryption, restores one from the bucket on the other server (a move), drains the first server, proves the deleted-site grace and undo, enforces a 100 MB disk quota, and signs an operator in with TOTP, all from the panel, in CI.

1. Container runtime (QuadletRuntime)

  • Manifest: run.kind = "image", run.image = "ghcr.io/org/app@sha256:…" (digest pinning required; a tag is a validation error), optional run.command/run.args, run.workdir ignored; needs.dirs are bind-mounted at /data/<dir>; the app reads its port from bind_addr/port as before and is published to 127.0.0.1:<port>.
  • Unit: the helper writes /etc/containers/systemd/users/<uid>/setu-site- <short>.container (Quadlet, rootless, User=/Group= inherited from the site user's systemd user instance) or, to keep one unit name and one helper verb set, a system-level setu-site@<short>.service whose ExecStart is podman run --rm --userns=keep-id … --user <uid>:<gid> run by the site user through User=: the design picks the system unit with podman run as the site user (ruling: one unit namespace, the same unit verb, cgroup limits from the same drop-in, logs in the same journal; Quadlet's generator is used only to validate the container file we render, not as the unit source).
  • Images are pulled by the agent as the site user (podman pull with the site's XDG_RUNTIME_DIR), stored under /srv/setu/sites/<short>/containers/ (site-owned), never shared between sites in 0.2; install for images = pull by digest and record; switch_version = rewrite the unit to the new digest.
  • Health, env, backups, update and rollback are unchanged: env goes in through --env-file, dirs are the same data/ tree, restore swaps the same directories.
  • Install prerequisites on agent install: podman, uidmap, slirp4netns/pasta, fuse-overlayfs; /etc/subuid and /etc/subgid ranges allocated per site user by the helper's site-create (65536 ids each, from a per-server pool starting at 100000), and lingering enabled for the site user so rootless Podman has a user runtime dir without a login session.
  • Security: no daemon, no socket; the container runs with the site user's uid, --cap-drop=all, --security-opt no-new-privileges, read-only root filesystem with /data and /tmp writable, memory and CPU from the unit's cgroup, the same nftables and Caddy rules. SECURITY.md gains a section; the e2e asserts the container cannot reach the Caddy socket or another site's data.

2. Off-box backups

  • Settings → Backups: one S3 target (endpoint, region, bucket, path style, key id, secret sealed) with Test; the control plane never sees archive bytes, it hands the agent a short-lived presigned PUT/GET URL per object (15 minutes), so servers need no S3 credentials.
  • Layout: s3://<bucket>/<prefix>/<server_short>/<site_short>/<backup_id>/ {db.dump.age,dirs.tar.zst.age,manifest.json}.
  • Encryption: age with a recipient derived from the panel secret (X25519 key from HKDF(secret, "setu-backup-v1")); the agent encrypts on the way out and decrypts on restore with a per-restore identity the control plane sends inside the restore command (never stored on the server). Losing the panel secret loses the backups: setu backup of the control plane includes the key material, and OPERATIONS.md says so in bold.
  • Nightly backup_site now ends with upload; local copies keep the 0.1 retention (7); remote retention: 30 daily, 12 weekly, configurable; delete_backup deletes both. Weekly restore check: the scheduler restores the newest backup of one site per server into a scratch database setu_check_<short>, runs the app's health against it when the manifest declares health_db_check = true, otherwise only pg_restore --list, and records pass/fail on the backup row.
  • Setu's own database: nightly to the same target under <prefix>/panel/.

3. Site move and server drain

  • POST /sites/{id}/move { server_id } → job move_site: pre_move backup (uploaded), provision on the target with the same env, port allocated there, restore from the bucket, health on the target, set_route on the target, remove_route on the source, stop and site_remove on the source after 24 h (grace), DNS: when both servers are behind the same tunnel nothing changes; otherwise the UI shows the new address and keeps the old server serving until the operator confirms DNS moved (confirm_dns action).
  • POST /servers/{id}/drain moves every site off (one at a time, most memory first) to the online server with the most free memory, then cordons; uncordon re-enables placement. Agent self-update: the control plane sends UpdateAgent { version, url, sha256 }; the agent downloads, verifies, replaces /opt/setu/setu through a new helper verb self-update (root-owned swap with rename), and restarts setu-agent.service; the control plane refuses to talk to agents more than one minor version behind.

4. Cloudflare tunnel ingress

  • A server may be ingress = "tunnel": setu agent install --tunnel- token <t> installs cloudflared (official package, verified) as cloudflared.service with a per-server tunnel token (sealed on the control plane, written 0600 on the server); the agent manages the tunnel's ingress rules through cloudflared's local config file (rewritten atomically, cloudflared reloads on SIGHUP) with one rule per site hostname → https://127.0.0.1:443 with originServerName and noTLSVerify (Caddy keeps terminating TLS with its internal CA on tunnel servers), plus the panel rule on a control-plane box. DNS records are created through the Cloudflare API with an API token the operator enters once in Settings (zone-scoped), or printed as the CNAME to add by hand when no token is given.
  • Reconcile covers tunnel rules like Caddy routes; doctor checks the tunnel is connected.

5. Quotas, grace, 2FA, alerts, metrics

  • Disk quotas enforced where the sites filesystem is XFS with prjquota (installer offers to create /srv/setu as an XFS image file mounted with project quotas when the root filesystem is ext4; otherwise quotas stay report-only and the UI says so); over quota → the site's data/ becomes read-only to the app (quota hard limit), the site shows a warning, an alert fires.
  • Deletion grace: delete_site stops the site and keeps everything 24 h (status = deleting, delete_at); POST /sites/{id}/undo-delete restores it; the scheduler finalises after the grace.
  • Two-factor: TOTP (RFC 6238) per operator, enforced panel-wide by a setting; recovery codes (10, hashed); login limiter counts TOTP failures.
  • Alerts: rules site failed, server offline > 5 min, backup failed, restore check failed, disk > 90 %, quota exceeded; channels email (SMTP in Settings) and webhook (JSON POST with HMAC signature); de-duplicated per rule and target with a 1 h cool-down.
  • Metrics: 90 days at 1-minute, rolled up to hourly after 7 days; the site page shows 24 h sparklines.

6. API, UI, agent protocol additions

New routes: settings/backups, settings/alerts, settings/tunnel, sites/{id}/move, sites/{id}/undo-delete, sites/{id}/confirm-dns, servers/{id}/drain|uncordon|update-agent, operators/{id}/totp, backups/{id}/verify. New commands: Pull, UploadBackup, DownloadBackup, UpdateAgent, SetTunnelRule, RemoveTunnelRule, SetQuota; new helper verbs: self-update, quota-set, subid-alloc (the only additions to the verb table; each follows the 0.1 rules). UI: Settings gains Backups, Alerts, Tunnel and Security (2FA); Servers gains Drain and Update agent; Sites gains Move and Undo delete; a Containers badge on image-based sites.

7. Compatibility and migration

Schema: migrations 0002_… onward are forward-only from 0.1.1. A 0.1.x agent keeps working against a 0.2 control plane for everything that existed in 0.1 (commands it does not know are rejected as unsupported and the job fails with a clear message to update the agent). setu agent install re-run upgrades an agent in place.

8. Testing

Unit and integration as in 0.1 for every new repo, machine and verb. The e2e grows to two agent containers (one --network none except the tunnel, using a cloudflared stub that records ingress rules, since real Cloudflare needs an account) and a MinIO container; assertions per the success paragraph. The installer test adds the XFS quota image path.

9. Open questions settled as rulings (changeable by the owner)

  • A move does not keep the port; the target allocates its own.
  • Quotas are hard limits on data/, not on the database.
  • 2FA is TOTP only (no WebAuthn in 0.2).
  • Alerts have no paging integration; webhook covers it.
  • Image pulls are per site, not shared; dedup is phase 4's concern.
  • Real Cloudflare is not exercised in CI; the stub records what the agent would configure, and the owner runs the live walkthrough before tagging, as with 0.1.