Backups

ByteBite has two distinct backup layers on BYTEBITE-APP-01:

1. Traefik configuration snapshots — local-only rollback points for ingress/TLS
2. PostgreSQL logical backups — automated, encrypted, off-host via Restic and Backblaze B2

Backup paths and PostgreSQL off-host restore verified: 2026-09-08

Traefik snapshots and PostgreSQL backups serve different purposes. Traefik backups do not cover database data. PostgreSQL off-host backups do not cover Traefik configuration or full VM state.

A. Traefik snapshots (local only)

Two recovery snapshots exist on the APP VM under /root/:

Pre-wildcard backup

/root/bytebite-traefik-backup-20260908-172700/

Contains:

• Traefik static configuration
• Original acme.json
• Dokploy dynamic files
• Container inspect output
• Image inspect output
• Docker network inspect output
• Checksums

Post-wildcard backup

/root/bytebite-traefik-post-wildcard-20260908-175333/

Contains current working wildcard-era configuration:

• traefik.yml
• acme.json
• acme-cloudflare.json
• Container inspect output
• Checksums

Purpose

These snapshots are configuration rollback points for Traefik/TLS recovery. They are useful when:

• Wildcard TLS breaks after a config change
• Dokploy recreates dokploy-traefik without custom wiring
• ACME state needs restoration

Security of ACME material

Backup directories contain acme.json and acme-cloudflare.json with Let's Encrypt account keys and certificates. Treat backups as sensitive:

• Restrict access to root / authorized operators
• Do not copy to unsecured locations
• Do not commit to Git

Traefik snapshot limitations

Traefik snapshots alone are not sufficient for full disaster recovery.

B. PostgreSQL logical backups (off-host)

Architecture

PostgreSQL backups are logical dumps, not raw Docker volume copies.

PostgreSQL 18 (Dokploy-managed staging container)
      |
      | pg_dump --format=custom --no-owner --no-acl
      v
temporary root-only local dump
  /var/lib/bytebite-backups/postgres/staging/bytebite.dump
      |
      | Restic (encrypted)
      v
encrypted off-host Restic repository
      |
      v
Backblaze B2 (private, S3-compatible API)

Why logical pg_dump (not volume copy)

Raw Docker volume backup is not the canonical PostgreSQL disaster-recovery method.

Dump format

pg_dump --format=custom --no-owner --no-acl

Output path (temporary, during backup run only):

/var/lib/bytebite-backups/postgres/staging/bytebite.dump

After a successful automated backup, this file is deleted. The staging backup directory normally contains no persistent dump.

Backblaze B2 and Restic

Restic snapshot tag for staging PostgreSQL backups:

bytebite-postgres-staging

Do not confuse use cases: B2 is implemented for encrypted PostgreSQL backup storage. B2 for general application media/object storage remains planned — see Storage.

Credentials and configuration

Backup credentials are stored on BYTEBITE-APP-01 under /etc/bytebite/backups/:

Both files are root-owned with restrictive permissions. Never document or reproduce their contents.

The Restic repository password must also exist in a secure off-host credential manager. Losing both the VM and that password makes encrypted backups unrecoverable from B2 alone.

Restic cache directory:

/var/cache/bytebite/restic

See Secrets for the credential inventory.

Shared backup lock

All PostgreSQL backup workflows acquire an exclusive flock on:

/run/lock/bytebite-postgres-backup.lock

Used by:

• Daily backup (postgres-backup.sh)
• Weekly maintenance (postgres-backup-maintenance.sh)
• Manual restore drill (postgres-restore-drill.sh)

If the lock cannot be acquired, scripts abort rather than overlapping operations. This is intentional safety behavior.

Backup script

/usr/local/lib/bytebite/postgres-backup.sh

Root-owned, umask 077. Behavior:

1. Acquire exclusive flock on /run/lock/bytebite-postgres-backup.lock
2. Find the currently running ByteBite staging PostgreSQL 18 container dynamically — does not rely on Dokploy-generated suffixes (e.g. bytebite-staging-postgres-vmg94p.1.z4anjftsr84p1xhkvx6hz8jne is generated and unstable)
3. Verify POSTGRES_DB=bytebite and POSTGRES_USER=bytebite without printing passwords
4. Create logical dump at /var/lib/bytebite-backups/postgres/staging/bytebite.dump
5. Validate: non-empty dump, pg_restore --list parses successfully, SHA-256 logged
6. Upload to Restic/B2 with tag bytebite-postgres-staging
7. List latest matching off-host snapshot
8. Delete local dump on exit (success or failure)

If multiple matching PostgreSQL containers are found, the backup aborts instead of guessing.

Automated backup (systemd)

Timer configuration includes Persistent=true: if the VM was down at the scheduled time, systemd can trigger a missed run after the machine becomes available again.

Both timer and service are enabled.

Type=oneshot behavior: After a successful run, systemctl status bytebite-postgres-backup.service shows Active: inactive (dead). This is normal — success is indicated by the service finishing/deactivating successfully, not by a continuously running process.

Retention and maintenance

Script: /usr/local/lib/bytebite/postgres-backup-maintenance.sh

Staging retention policy (applied to snapshots tagged bytebite-postgres-staging):

This is the staging policy. Do not assume it automatically becomes the production policy.

Maintenance performs:

1. restic forget with staging tag and retention policy
2. restic prune
3. restic check

Do not routinely run restic prune manually outside this maintenance workflow unless troubleshooting requires it.

Local directory hierarchy

After cleanup and successful automated runs:

/var/lib/bytebite-backups/
/var/lib/bytebite-backups/postgres/
/var/lib/bytebite-backups/postgres/staging/

No persistent bytebite.dump remains after a successful automated backup.

Restore verification (2026-09-08)

Off-host PostgreSQL backup and restore was verified on 2026-09-08:

1. Real ByteBite PostgreSQL logical dump created (pg_dump custom format)
2. Dump validated with pg_restore --list
3. Dump uploaded to B2 through Restic
4. Restic repository passed integrity check
5. Dump restored from B2 — byte-identical to original
6. Isolated PostgreSQL 18 container started with --network none, no host port published
7. Clean restore database created; pg_restore completed successfully
8. Core tables verified: restaurants, restaurant_domains
9. Application data present; staging tenant domains verified (luigi.staging.getbytebite.co, mario.staging.getbytebite.co)
10. Temporary restore container and storage cleaned up

The live staging PostgreSQL database was never modified during restore testing.

Restore drill script (manual)

/usr/local/lib/bytebite/postgres-restore-drill.sh

Successfully executed manually. Not scheduled by a systemd timer.

Behavior:

1. Acquire flock on /run/lock/bytebite-postgres-backup.lock
2. Load Restic configuration from /etc/bytebite/backups/
3. Restore latest snapshot tagged bytebite-postgres-staging to a temporary work directory under /var/lib/bytebite-backups/
4. Validate restored bytebite.dump archive
5. Start isolated postgres:18 container with --network none, no host ports
6. Create clean database bytebite_restore
7. Run pg_restore
8. Verify restaurants and restaurant_domains tables, non-zero data, Luigi and Mario staging domains
9. Delete temporary container and work directory

Expected terminal result: RESTORE DRILL VERIFIED

The live staging database is never targeted by the restore drill.

Current limitations

Related

• Database — PostgreSQL service details
• Disaster recovery — recovery runbooks
• Validation — backup health checks
• Operations — recurring backup tasks
• Secrets — Restic and B2 credential inventory
• Known gaps — remaining backup/DR gaps
• TLS — ACME state files