Disaster Recovery

Practical recovery procedures for known failure modes. Where a complete recovery path does not exist, the gap is labeled explicitly.

PostgreSQL off-host backup and restore verified: 2026-09-08

PostgreSQL recovery levels

Staging PostgreSQL has a verified off-host backup and restore path. Two operational levels exist:

A. Routine recovery verification (restore drill)

Use the manual restore drill script to verify backup integrity without touching the live database:

sudo /usr/local/lib/bytebite/postgres-restore-drill.sh

Expected result: RESTORE DRILL VERIFIED

The drill restores the latest Restic snapshot tagged bytebite-postgres-staging into an isolated PostgreSQL 18 container (--network none, no host ports), validates core tables and staging tenant data, then cleans up.

See Backups and Validation for full details.

Scheduled automatic restore drills are not yet implemented.

B. Actual disaster recovery (database)

For a real PostgreSQL failure requiring data recovery:

1. Provision a compatible PostgreSQL 18 instance (isolated from production traffic during restore)
2. Recover the latest appropriate Restic snapshot (tag bytebite-postgres-staging) from B2
3. Validate the dump with pg_restore --list
4. Create a clean target database and run pg_restore
5. Validate schema and data (restaurants, restaurant_domains, application records)
6. Configure application DATABASE_URL separately in Dokploy — do not reuse old credentials solely because they existed in the old database
7. Run application validation suite

Prerequisites that must be available off-host:

• Restic repository encryption password (from secure credential manager)
• B2 S3-compatible credentials (from /etc/bytebite/backups/restic.env on the host, or rotated equivalents)

Important: Restoring the database alone does not restore the entire APP VM. Traefik configuration, Dokploy state, application containers, and host filesystem must be rebuilt separately. Full host disaster recovery remains incomplete.

Traefik broken after config change

Symptoms: TLS errors, routing failures, or Traefik container crash after editing host config.

Diagnosis (read-only)

docker logs dokploy-traefik --tail 100
docker inspect dokploy-traefik --format '{{json .Mounts}}' | python3 -m json.tool

Recovery (destructive — restores prior config)

1. Stop Traefik: docker stop dokploy-traefik
2. Restore files from the most recent backup:
   • Pre-wildcard: /root/bytebite-traefik-backup-20260908-172700/
   • Post-wildcard: /root/bytebite-traefik-post-wildcard-20260908-175333/
3. Copy traefik.yml, dynamic files, and ACME JSON back to /etc/dokploy/traefik/
4. Start Traefik: docker start dokploy-traefik
5. Run Validation smoke tests

Traefik recreated by Dokploy upgrade

Symptoms: Wildcard TLS fails; CF_DNS_API_TOKEN_FILE missing; letsencrypt-cloudflare resolver absent.

Diagnosis (read-only)

docker inspect dokploy-traefik --format '{{range .Config.Env}}{{println .}}{{end}}' | grep CF_DNS
docker inspect dokploy-traefik --format '{{json .Mounts}}' | python3 -m json.tool

Check for:

• CF_DNS_API_TOKEN_FILE=/run/secrets/cloudflare-dns-api-token
• Mount: /etc/dokploy/traefik/secrets → /run/secrets
• traefik.yml contains letsencrypt-cloudflare resolver

Recovery

Host files (traefik.yml, acme-cloudflare.json, token file) likely survived on disk. Re-apply container customization:

1. Verify host files exist and token file is mode 600
2. Recreate or reconfigure dokploy-traefik with secret mount and env var (via Dokploy or manual docker run matching prior inspect)
3. Use post-wildcard backup inspect output as reference
4. Run full Validation including wildcard TLS

See Operations upgrade checklist.

Wildcard TLS broken

Symptoms: Explicit hosts work; arbitrary *.staging.getbytebite.co hosts show TLS errors.

Diagnosis

TENANT_HOST="dr-test-$(date +%s).staging.getbytebite.co"
echo | openssl s_client -connect "${TENANT_HOST}:443" -servername "${TENANT_HOST}" 2>&1 | head -20
docker logs dokploy-traefik 2>&1 | grep -i acme | tail -20

Check acme-cloudflare.json exists and Cloudflare token is valid.

Recovery

1. Verify Cloudflare token file and mount (see Traefik recreate runbook above)
2. If acme-cloudflare.json is corrupt, restore from post-wildcard backup
3. Restart Traefik and wait for ACME retry or trigger re-issuance
4. Validate wildcard TLS

Cloudflare token invalid or rotated

Symptoms: ACME DNS-01 challenge failures in Traefik logs.

Recovery

1. Generate new token in Cloudflare (Zone Read + DNS Edit on getbytebite.co)
2. Write to /etc/dokploy/traefik/secrets/cloudflare-dns-api-token (mode 600)
3. Restart dokploy-traefik
4. Verify wildcard TLS

Never document or commit the token value.

APP VM lost

Status: GAP — full host recovery path is incomplete.

PostgreSQL data can be recovered from off-host Restic/B2 backups if credentials and the Restic encryption password are available. However, rebuilding the entire APP VM remains a multi-step manual process:

1. Provision new VM with 88.99.0.118 (or update DNS if IP changes)
2. Install Dokploy, recreate ByteBite staging project
3. Restore Traefik config from local backup snapshots (if accessible — snapshots are local-only and may be lost with the VM)
4. Restore PostgreSQL from off-host Restic/B2 backup (verified path — see PostgreSQL recovery levels above)
5. Redeploy application from Git
6. Reconfigure Dokploy environment variables (including new DATABASE_URL if credentials changed)
7. Run migrations if needed and full validation

Without off-host PostgreSQL backups and the Restic encryption password, database data would be unrecoverable. The Restic password must be stored in an off-host credential manager.

PostgreSQL failure

Symptoms: /health/db returns degraded; API tenant routes return 503.

Diagnosis (read-only)

# From APP VM, if Docker access is available
docker ps --filter name=postgres

Check Dokploy PostgreSQL service status in Dokploy UI. Container names are Dokploy-generated and change — do not hardcode suffixes.

Recovery

MAIL VM lost

Impact: Mail DNS points to 88.99.0.97 but mail was not fully configured. Recovery requires reprovisioning BYTEBITE-MAIL-01 and reconfiguring Stalwart (planned).

DNS mistake

Symptoms: Hostname resolves to wrong IP or does not resolve.

Diagnosis

dig +short edge.getbytebite.co
dig +short api.staging.getbytebite.co

Recovery

1. Correct record in Cloudflare dashboard (DNS-only, not proxied)
2. Wait for TTL propagation
3. Re-run DNS validation from Validation

Failed ByteBite application deployment

Symptoms: 502/503 from Traefik; containers unhealthy in Dokploy.

Diagnosis (read-only)

Check Dokploy deployment logs and container health:

docker ps --filter name=bytebite-staging
docker logs <api-container-id> --tail 50

Recovery

1. Identify failing service from logs
2. If migration needed: redeploy so the migrate one-shot runs, or node db/dist/migrate.js in the api container as fallback
3. If bad commit: redeploy previous Git commit via Dokploy
4. Validate

Related

• Backups — backup architecture and scripts
• Operations — preventive maintenance
• Known gaps — unresolved recovery paths
• Troubleshooting — symptom index