No internet connection
  1. Home
  2. General

This deployment: reverse-proxy edge, hardening overrides & backups

By @IvanTheGeek
    2026-07-05 15:30:57.891Z

    This is a staff-only companion to the public Architecture & Internals topics. Those topics describe stock Talkyard as of v1.2026.003 and are deliberately deployment-agnostic. This page records the specifics of our install that were stripped from the public write-ups: the reverse-proxy edge, the hardening override file, and the backup schedule. If you have not read the public topics yet, start there for the container roles, the five compose networks, volumes, secrets, and the request lifecycle — everything below only covers what is different or extra here.

    Running version: v1.2026.003-f220a7d9f.

    1. The edge — Caddy in front, web with no host ports

    Stock Talkyard publishes the web container's ports 80 and 443 to the host. We do not. Instead a separate Caddy container owns the host ports and reverse-proxies to web over an external Docker network.

    The change, concretely:

    • In docker-compose.override.yml, the web service sets ports: !override [] — this removes the stock 80:80 / 443:443 publishes entirely, so web binds no host ports. It is reachable only from inside Docker.
    • A caddy container (image caddy:2, running v2.11.4, its own compose project under /srv/caddy) binds host 80/tcp and 443 tcp+udp (the udp is HTTP/3). It terminates TLS, does ACME to Let's Encrypt, and stores its certs in Caddy's own storage under /srv/caddy — not in Talkyard's web-generated volume.
    • Caddy and web share an external Docker network named ty-edge (172.18.0.0/16, not compose-managed). Caddy dials talkyard-v1-web-1:80 by DNS name; the ty-edge IPs are Docker-assigned, not pinned.
    • forum.ivanthegeek.com → proxied to web:80, stripping X-Forwarded-Host. talkyard.ivanthegeek.com → explicit 301 to the forum host. Plain :80 → Caddy's default 308 redirect to HTTPS.

    Consequence: web's own lua-resty-auto-ssl / Let's Encrypt path is dormant — traffic arrives plain on :80 from Caddy. The nginx :443 ssl listener is live but unused, and web's Redis cert-storage config (cache:6379) is still active code in init-by-lua even though nothing exercises it.

    flowchart LR
  inet(("Internet"))
  le["Let's Encrypt<br/>(ACME)"]

  subgraph caddyproj["compose project caddy · /srv/caddy"]
    caddy["caddy<br/>Caddy 2 (v2.11.4)<br/>binds host 80/tcp + 443 tcp+udp (HTTP/3)<br/>TLS termination · certs in /srv/caddy<br/>strips X-Forwarded-Host"]
  end

  edge{{"ty-edge<br/>172.18.0.0/16<br/>external net (not compose-managed)<br/>caddy + web ONLY · IPs dynamic"}}

  subgraph typroj["compose project talkyard-v1 · /srv/talkyard"]
    web["web<br/>OpenResty (nginx + Lua)<br/>ports: !override [] — NO host ports<br/>listens :80 (used) + :443 ssl (unused)<br/>auto-SSL path dormant behind Caddy"]
    app["app<br/>talkyard-server :9000"]
  end

  inet -->|"HTTPS :443 h1/h2/h3<br/>:80 → 308 HTTPS"| caddy
  caddy -.->|"ACME issue / renew"| le
  caddy -->|"HTTP → web:80"| edge
  edge --> web
  web -->|"HTTP + WebSocket → app:9000<br/>net: fe_int"| app

  classDef world fill:#2d5016,stroke:#7cb342,color:#fff
  classDef edgec fill:#1e3a5f,stroke:#4a90d9,color:#fff
  classDef net fill:#7a4a1e,stroke:#e0a060,color:#fff
  classDef svc fill:#4a2c5e,stroke:#ab7fd1,color:#fff
  class inet,le world
  class caddy edgec
  class edge net
  class web,app svc
    • inet → caddy: the only host-published ports in the whole stack live on the Caddy container. web is unreachable from the host network.
    • caddy → ty-edge → web: Caddy re-encodes nothing — it forwards plain HTTP to web:80 over the external ty-edge network, which only Caddy and web are attached to.
    • le: cert issuance/renewal is Caddy's job now; Talkyard's own ACME path never fires.

    2. The hardening override — stripping the app's baked-in debug ports

    The stock app image bakes two debug listeners into its CMD: JDWP on :9999 (Java remote debugger) and unauthenticated JMX on :3333. Both are real exposure if the app network is ever reachable. Our docker-compose.override.yml reproduces the image's launch command minus those flags, so neither port ever listens. It also patches one upstream omission on rendr.

    What the override does for app:

    • Overrides command: to re-run the same talkyard-server entrypoint the image would, but with the JDWP -agentlib:jdwp=...:9999 and the JMX -Dcom.sun.management.jmxremote...:3333 arguments removed. Verify after every upgrade: docker exec talkyard-v1-app-1 sh -c 'netstat -tlnp 2>/dev/null || ss -tlnp' should show :9000 only — no :9999, no :3333.
    • This is a genuine hardening fix, not cosmetic: the JMX listener is unauthenticated in the stock image.

    And for rendr:

    • Adds restart: always — upstream forgot it on the rendr placeholder service. (rendr is still just sleep infinity today; no traffic, but it should come back up like everything else.)

    Other override tuning that lives in the same file (for completeness, not security-critical): search gets a 512 MB heap / 1.6 G mem limit. The uniform hardening — no-new-privileges, cap_drop: ALL plus minimal per-service cap_add, local rotated log driver, egressp read_only rootfs — is stock upstream, described in the public topics.

    Re-verify the JDWP/JMX strip after every server upgrade — an image bump re-bakes the debug ports into the CMD, and the override only wins if it still matches the new launch line.

    3. Backups — timers, GPG-at-rest, and the off-box pull

    The backup service is profiles: [backup], so it never auto-starts; it is invoked one-shot by docker compose run --rm from systemd timers. See the public State & backups topic for what the stock talkyard-backup image actually archives (pg_dumpall, uploads, the config tarball, optional GPG). Here are our schedule and encryption specifics.

    • 02:10 UTC — talkyard-backup.timer → runs /ty/backup.sh: pg_dumpall of all DBs + roles, the uploads/ subdir of pub-files, the redis dump.rdb, and the whole project dir (conf, .env, compose files, and secrets/*.txt) tarred into the config archive. Because that tarball contains our secrets, GPG matters — see below.
    • 03:10 UTC — talkyard-backup-cleanup.timer → runs /ty/delete-old-backups.sh. This is a prune only, not a second backup.
    • 00:40 EDT — talkyard-backup-pull.timer (on the local machine, not the VPS) → pulls the archives off-box into backups/vps-archives/.
    • GPG AES256 at rest, but only while the backup_password docker secret is non-empty. If that secret is ever empty, backup.sh silently falls back to plaintext — so an empty passphrase means the config tarball (with secrets inside) is written unencrypted. Keep the secret populated.
    • Known gap: priv-files is mounted ro into the backup container but backup.sh never reads it — nothing from priv-files is archived. (Handoff Sec obs 7, forum p.174.)
    flowchart LR
  timer1["talkyard-backup.timer<br/>02:10 UTC daily (VPS)"]
  timer2["talkyard-backup-cleanup.timer<br/>03:10 UTC daily (VPS)"]
  backup["backup container<br/>docker compose run --rm<br/>profiles: [backup]"]
  rdb[("rdb<br/>PostgreSQL")]
  pubfiles[("pub-files<br/>uploads/ only")]
  redisdata[("redis-data<br/>dump.rdb")]
  projdir["project dir /srv/talkyard<br/>conf + .env + compose + secrets/"]
  privfiles[("priv-files<br/>mounted ro · NEVER archived")]
  bksecret["secrets/backup_password.txt<br/>GPG passphrase"]
  bkdir[("/var/opt/backups/talkyard/v1/archives/")]
  local[("local machine<br/>backups/vps-archives/")]

  timer1 ==>|"/ty/backup.sh"| backup
  rdb ==>|"pg_dumpall + gzip<br/>net: be_int"| backup
  pubfiles ==>|"ro"| backup
  redisdata ==>|"ro"| backup
  projdir ==>|"ro — tarred incl. secrets<br/>WHY GPG matters"| backup
  bksecret -->|"secret"| backup
  privfiles -.->|"never read — known gap"| backup
  backup ==>|"GPG AES256 if passphrase set<br/>else SILENT plaintext"| bkdir
  timer2 -->|"/ty/delete-old-backups.sh<br/>prune only — no backup made"| bkdir
  bkdir ==>|"talkyard-backup-pull.timer<br/>00:40 EDT daily"| local

  classDef vol fill:#4a2c5e,stroke:#ab7fd1,color:#fff
  classDef fs fill:#7a4a1e,stroke:#e0a060,color:#fff
  classDef bk fill:#2d5016,stroke:#7cb342,color:#fff
  classDef warn fill:#5e2c2c,stroke:#d17f7f,color:#fff
  class rdb,pubfiles,redisdata vol
  class privfiles warn
  class projdir,bksecret fs
  class backup,timer1,timer2,bkdir,local bk
    • timer1 → backup: the daily backup fires the one-shot container; the timer, not compose, is what ever starts it.
    • projdir → backup: the whole project dir (secrets included) goes into the config tarball — this is exactly why the GPG passphrase must stay set.
    • privfiles: mounted but never read — the dashed edge marks the known coverage gap.
    • bkdir → local: a separate timer on the local machine pulls the encrypted archives off-box nightly, so a lost VPS is recoverable.

    Everything else — container roles/versions, the five 172.26.x compose networks, named volumes, docker secrets, the render-cache chain, and the end-to-end request lifecycle — is stock and lives in the public topics. This page is only the delta.

    Reply
    • 0 replies
      Reply