# Setu 0.2: runs anything, loses nothing Published 2026-10-21 · Canonical: https://setucp.com/docs/specs/10-10-setu-0-2-design --- Date: 2026-10-10 · Status: approved in conversation (owner decisions recorded below) · Builds on the 0.1 design (`2026-10-09-setu-design.md`, revision 4). Where this document is silent, 0.1 holds. ## Owner decisions (2026-10-10) - Roadmap order after 0.2: 0.3 hosted control plane, 0.4 catalogue, 0.5 providers and billing. - Container runtime: **rootless Podman through systemd Quadlet**, one container per site running as the site's own user. Not Docker, not nspawn. - Off-box backups: **one S3-compatible target per panel**, archives **encrypted with age** under a key derived from the panel secret. - **Cloudflare tunnel ingress is in 0.2.** ## Intent A site is just as safe on a box that may die as on one that will not: its backups leave the server, it can be moved to another server, and a server can be drained for maintenance. Apps that ship only as images run under the same unit, limits and helper model as binaries. Servers without a public IP join through a Cloudflare tunnel. Operators get the guard rails a real deployment needs: enforced disk quotas, a deletion grace period, two-factor sign-in and alerts. Success: on two Ubuntu 24.04 servers (one behind a tunnel, no public IP) the e2e provisions an image-based app (Miniflux's image) and a binary app, backs both up to a MinIO target with encryption, restores one from the bucket on the **other** server (a move), drains the first server, proves the deleted-site grace and undo, enforces a 100 MB disk quota, and signs an operator in with TOTP, all from the panel, in CI. ## 1. Container runtime (`QuadletRuntime`) - Manifest: `run.kind = "image"`, `run.image = "ghcr.io/org/app@sha256:…"` (digest pinning required; a tag is a validation error), optional `run.command`/`run.args`, `run.workdir` ignored; `needs.dirs` are bind-mounted at `/data/`; the app reads its port from `bind_addr`/`port` as before and is published to `127.0.0.1:`. - Unit: the helper writes `/etc/containers/systemd/users//setu-site- .container` (Quadlet, rootless, `User=`/`Group=` inherited from the site user's systemd user instance) **or**, to keep one unit name and one helper verb set, a system-level `setu-site@.service` whose `ExecStart` is `podman run --rm --userns=keep-id … --user :` run by the site user through `User=`: the design picks the **system unit with `podman` run as the site user** (ruling: one unit namespace, the same `unit` verb, cgroup limits from the same drop-in, logs in the same journal; Quadlet's generator is used only to validate the container file we render, not as the unit source). - Images are pulled by the agent as the site user (`podman pull` with the site's `XDG_RUNTIME_DIR`), stored under `/srv/setu/sites//containers/` (site-owned), never shared between sites in 0.2; `install` for images = pull by digest and record; `switch_version` = rewrite the unit to the new digest. - Health, env, backups, update and rollback are unchanged: env goes in through `--env-file`, dirs are the same `data/` tree, restore swaps the same directories. - Install prerequisites on `agent install`: `podman`, `uidmap`, `slirp4netns`/`pasta`, `fuse-overlayfs`; `/etc/subuid` and `/etc/subgid` ranges allocated per site user by the helper's `site-create` (65536 ids each, from a per-server pool starting at 100000), and `lingering` enabled for the site user so rootless Podman has a user runtime dir without a login session. - Security: no daemon, no socket; the container runs with the site user's uid, `--cap-drop=all`, `--security-opt no-new-privileges`, read-only root filesystem with `/data` and `/tmp` writable, memory and CPU from the unit's cgroup, the same nftables and Caddy rules. SECURITY.md gains a section; the e2e asserts the container cannot reach the Caddy socket or another site's data. ## 2. Off-box backups - Settings → Backups: one S3 target (endpoint, region, bucket, path style, key id, secret sealed) with Test; the control plane never sees archive bytes, it hands the agent a short-lived presigned PUT/GET URL per object (15 minutes), so servers need no S3 credentials. - Layout: `s3://///// {db.dump.age,dirs.tar.zst.age,manifest.json}`. - Encryption: **age** with a recipient derived from the panel secret (X25519 key from `HKDF(secret, "setu-backup-v1")`); the agent encrypts on the way out and decrypts on restore with a per-restore identity the control plane sends inside the restore command (never stored on the server). Losing the panel secret loses the backups: `setu backup` of the control plane includes the key material, and OPERATIONS.md says so in bold. - Nightly `backup_site` now ends with `upload`; local copies keep the 0.1 retention (7); remote retention: 30 daily, 12 weekly, configurable; `delete_backup` deletes both. Weekly **restore check**: the scheduler restores the newest backup of one site per server into a scratch database `setu_check_`, runs the app's health against it when the manifest declares `health_db_check = true`, otherwise only `pg_restore --list`, and records pass/fail on the backup row. - Setu's own database: nightly to the same target under `/panel/`. ## 3. Site move and server drain - `POST /sites/{id}/move { server_id }` → job `move_site`: `pre_move` backup (uploaded), provision on the target with the same env, port allocated there, `restore` from the bucket, health on the target, `set_route` on the target, `remove_route` on the source, stop and `site_remove` on the source after 24 h (grace), DNS: when both servers are behind the same tunnel nothing changes; otherwise the UI shows the new address and keeps the old server serving until the operator confirms DNS moved (`confirm_dns` action). - `POST /servers/{id}/drain` moves every site off (one at a time, most memory first) to the online server with the most free memory, then cordons; `uncordon` re-enables placement. Agent self-update: the control plane sends `UpdateAgent { version, url, sha256 }`; the agent downloads, verifies, replaces `/opt/setu/setu` through a new helper verb `self-update` (root-owned swap with rename), and restarts `setu-agent.service`; the control plane refuses to talk to agents more than one minor version behind. ## 4. Cloudflare tunnel ingress - A server may be `ingress = "tunnel"`: `setu agent install --tunnel- token ` installs `cloudflared` (official package, verified) as `cloudflared.service` with a per-server tunnel token (sealed on the control plane, written 0600 on the server); the agent manages the tunnel's ingress rules through cloudflared's local config file (rewritten atomically, `cloudflared` reloads on SIGHUP) with one rule per site `hostname → https://127.0.0.1:443` with `originServerName` and `noTLSVerify` (Caddy keeps terminating TLS with its internal CA on tunnel servers), plus the panel rule on a control-plane box. DNS records are created through the Cloudflare API with an API token the operator enters once in Settings (zone-scoped), or printed as the CNAME to add by hand when no token is given. - Reconcile covers tunnel rules like Caddy routes; `doctor` checks the tunnel is connected. ## 5. Quotas, grace, 2FA, alerts, metrics - Disk quotas enforced where the sites filesystem is XFS with `prjquota` (installer offers to create `/srv/setu` as an XFS image file mounted with project quotas when the root filesystem is ext4; otherwise quotas stay report-only and the UI says so); over quota → the site's `data/` becomes read-only to the app (quota hard limit), the site shows a warning, an alert fires. - Deletion grace: `delete_site` stops the site and keeps everything 24 h (`status = deleting`, `delete_at`); `POST /sites/{id}/undo-delete` restores it; the scheduler finalises after the grace. - Two-factor: TOTP (RFC 6238) per operator, enforced panel-wide by a setting; recovery codes (10, hashed); login limiter counts TOTP failures. - Alerts: rules `site failed`, `server offline > 5 min`, `backup failed`, `restore check failed`, `disk > 90 %`, `quota exceeded`; channels email (SMTP in Settings) and webhook (JSON POST with HMAC signature); de-duplicated per rule and target with a 1 h cool-down. - Metrics: 90 days at 1-minute, rolled up to hourly after 7 days; the site page shows 24 h sparklines. ## 6. API, UI, agent protocol additions New routes: `settings/backups`, `settings/alerts`, `settings/tunnel`, `sites/{id}/move`, `sites/{id}/undo-delete`, `sites/{id}/confirm-dns`, `servers/{id}/drain|uncordon|update-agent`, `operators/{id}/totp`, `backups/{id}/verify`. New commands: `Pull`, `UploadBackup`, `DownloadBackup`, `UpdateAgent`, `SetTunnelRule`, `RemoveTunnelRule`, `SetQuota`; new helper verbs: `self-update`, `quota-set`, `subid-alloc` (the only additions to the verb table; each follows the 0.1 rules). UI: Settings gains Backups, Alerts, Tunnel and Security (2FA); Servers gains Drain and Update agent; Sites gains Move and Undo delete; a Containers badge on image-based sites. ## 7. Compatibility and migration Schema: migrations `0002_…` onward are forward-only from 0.1.1. A 0.1.x agent keeps working against a 0.2 control plane for everything that existed in 0.1 (commands it does not know are rejected as `unsupported` and the job fails with a clear message to update the agent). `setu agent install` re-run upgrades an agent in place. ## 8. Testing Unit and integration as in 0.1 for every new repo, machine and verb. The e2e grows to two agent containers (one `--network none` except the tunnel, using a cloudflared stub that records ingress rules, since real Cloudflare needs an account) and a MinIO container; assertions per the success paragraph. The installer test adds the XFS quota image path. ## 9. Open questions settled as rulings (changeable by the owner) - A move does not keep the port; the target allocates its own. - Quotas are hard limits on `data/`, not on the database. - 2FA is TOTP only (no WebAuthn in 0.2). - Alerts have no paging integration; webhook covers it. - Image pulls are per site, not shared; dedup is phase 4's concern. - Real Cloudflare is not exercised in CI; the stub records what the agent would configure, and the owner runs the live walkthrough before tagging, as with 0.1.