Operations

Recurring operational procedures for ByteBite infrastructure. Risky operations include pre-check, backup, action, post-validation, and rollback guidance.

Dokploy upgrades

Pre-check

• Note current Dokploy version (dokploy/dokploy:v0.30.5 as of 2026-09-08)
• Export Traefik container inspect: docker inspect dokploy-traefik > /root/traefik-pre-upgrade-inspect.json
• Confirm backup snapshots exist in /root/

Action

• Upgrade Dokploy via Dokploy's own update mechanism or host package manager (follow Dokploy documentation)

Post-validation

• Verify ByteBite staging application containers are healthy
• Critical: Run the Traefik post-upgrade checklist below
• Run Validation smoke tests

Rollback

• Restore Dokploy to previous version if application deploy breaks
• Restore Traefik from backup if ingress breaks

Traefik upgrades / recreates

Dokploy may recreate dokploy-traefik during platform upgrades. Custom DNS-01 wiring is not part of standard Dokploy and may be lost.

Traefik post-upgrade checklist

After any Dokploy upgrade or Traefik container recreation:

1. Inspect container mounts:

   docker inspect dokploy-traefik --format '{{json .Mounts}}' | python3 -m json.tool

   Verify /etc/dokploy/traefik/secrets → /run/secrets (read-only)

2. Inspect environment:

   docker inspect dokploy-traefik --format '{{range .Config.Env}}{{println .}}{{end}}' | grep CF_DNS

   Expected: CF_DNS_API_TOKEN_FILE=/run/secrets/cloudflare-dns-api-token

3. Verify static config contains letsencrypt-cloudflare resolver in /etc/dokploy/traefik/traefik.yml

4. Verify ACME files exist:

   • /etc/dokploy/traefik/dynamic/acme.json
   • /etc/dokploy/traefik/dynamic/acme-cloudflare.json

5. Test wildcard TLS:

   TENANT_HOST="ops-check-$(date +%s).staging.getbytebite.co"
   echo | openssl s_client -connect "${TENANT_HOST}:443" -servername "${TENANT_HOST}" 2>/dev/null | openssl x509 -noout -subject

6. Test wildcard API:

   curl -s -u "${STAGING_BASIC_AUTH_USER}:${STAGING_BASIC_AUTH_PASSWORD}" "https://${TENANT_HOST}/api/health"

7. Verify explicit hosts still work with BasicAuth (api, docs, dashboard, luigi, mario)

8. If any check fails, restore from /root/bytebite-traefik-post-wildcard-20260908-175333/ — see Disaster recovery

Ubuntu updates (APP VM)

Pre-check

• Confirm staging is healthy
• Schedule maintenance window if kernel update requires reboot

Action

sudo apt update && sudo apt upgrade -y

Reboot if required: sudo reboot

Post-validation

• SSH access restored
• Docker daemon running
• Dokploy UI accessible
• Full Validation suite

PostgreSQL upgrades

Managed by Dokploy. Follow Dokploy documentation for PostgreSQL version upgrades.

Pre-check

• Confirm recent off-host backup succeeded (see Backup verification)
• Note current version (postgres:18)

Post-validation

• curl -s -u "${STAGING_BASIC_AUTH_USER}:${STAGING_BASIC_AUTH_PASSWORD}" https://api.staging.getbytebite.co/health/db
• Application smoke tests

PostgreSQL backup operations

Daily backup (automated)

Runs /usr/local/lib/bytebite/postgres-backup.sh — logical dump, Restic upload, local cleanup.

Weekly maintenance (automated)

Runs /usr/local/lib/bytebite/postgres-backup-maintenance.sh — retention, prune, integrity check.

Manual restore drill

sudo /usr/local/lib/bytebite/postgres-restore-drill.sh

Expected result: RESTORE DRILL VERIFIED. Not scheduled — run manually after infrastructure changes or periodically.

Operational commands

systemctl status bytebite-postgres-backup.timer
systemctl status bytebite-postgres-backup-maintenance.timer

systemctl list-timers \
  bytebite-postgres-backup.timer \
  bytebite-postgres-backup-maintenance.timer \
  --all

journalctl -u bytebite-postgres-backup.service
journalctl -u bytebite-postgres-backup-maintenance.service

Type=oneshot services

Both backup and maintenance services are Type=oneshot. After a successful run, systemctl status shows Active: inactive (dead). This is normal — check the service result and journal logs for success or failure. Do not treat inactive (dead) as a failure.

Do not routinely run restic prune manually outside the maintenance workflow unless troubleshooting requires it.

Docker cleanup / disk monitoring

Periodic checks

docker system df
df -h /

Cleanup (destructive — removes unused images/containers)

docker system prune -f

Do not prune volumes without confirming they are unused. PostgreSQL data volumes must not be pruned.

TLS checks

Run monthly or after infrastructure changes. See Validation TLS section.

Cloudflare token rotation

1. Generate new token (same scopes: Zone Read + DNS Edit on getbytebite.co)
2. Write to /etc/dokploy/traefik/secrets/cloudflare-dns-api-token (mode 600)
3. Restart dokploy-traefik
4. Verify wildcard TLS
5. Revoke old token in Cloudflare

Secret rotation (application)

1. Update value in Dokploy environment
2. Redeploy or restart affected service
3. Verify application health

Backup verification

Traefik snapshots

Periodically confirm Traefik backup snapshots exist and are readable:

ls -la /root/bytebite-traefik-*

PostgreSQL off-host backups

Check timer and last service result:

systemctl status bytebite-postgres-backup.timer
journalctl -u bytebite-postgres-backup.service --since "24 hours ago"

For full restore validation, run the manual restore drill:

sudo /usr/local/lib/bytebite/postgres-restore-drill.sh

See Validation for the canonical backup health checks.

Dependency / runtime upgrades

Application runtime versions are pinned in Dockerfiles (Node 24, pnpm 12.3.4). Upgrade by:

1. Update Dockerfile ARG NODE_VERSION or pnpm version
2. Test locally with compose.deploy.yml
3. Deploy to staging
4. Run validation suite

Smoke tests

The canonical smoke test reference is Validation. Run after every infrastructure change.

Related

• Disaster recovery — failure runbooks
• Backups — snapshot locations
• Traefik — configuration details
• Known gaps — unresolved operational items