Infrastructure
Troubleshooting
Symptom-oriented infrastructure troubleshooting guide.
Symptom → likely cause → diagnosis → resolution. For recovery procedures, see Disaster recovery.
| |
|---|
| Symptom | dig returns correct IP; browser/curl shows TLS error |
| Likely cause | ACME failure, expired cert, or wrong resolver for host type |
| Diagnosis | openssl s_client -connect HOST:443 -servername HOST; check docker logs dokploy-traefik for ACME errors |
| Resolution | Explicit hosts: check HTTP-01 (acme.json). Wildcard: check DNS-01 token and acme-cloudflare.json. See TLS, Disaster recovery |
| |
|---|
| Symptom | Valid TLS on *.staging.getbytebite.co but Traefik 404 page |
| Likely cause | Container not on dokploy-network or Traefik labels missing |
| Diagnosis | docker inspect <storefront-container> --format '{{json .NetworkSettings.Networks}}' — check for dokploy-network |
| Resolution | Re-attach dokploy-network via Dokploy domain config or manual network connect. Verify labels in running container match compose.deploy.yml |
| |
|---|
| Symptom | Wildcard host serves dashboard or docs instead of storefront |
| Likely cause | Router priority conflict or missing wildcard labels |
| Diagnosis | Check Traefik router priorities; explicit Dokploy routers should have higher priority than wildcard (10/20) |
| Resolution | Verify Dokploy explicit domains still configured. Redeploy compose.deploy.yml to restore wildcard labels |
| |
|---|
| Symptom | curl https://HOST/api/health returns HTML or storefront response |
| Likely cause | Missing API wildcard router (priority 20) or StripPrefix middleware |
| Diagnosis | Check api container labels for bytebite-staging-wildcard-api router |
| Resolution | Redeploy from compose.deploy.yml. For explicit hosts, verify Dokploy /api path rule with StripPrefix |
| |
|---|
| Symptom | API logs show requests to /api/health instead of /health |
| Likely cause | StripPrefix middleware not applied |
| Diagnosis | Check Traefik middleware bytebite-staging-wildcard-strip-api on API router |
| Resolution | Verify compose.deploy.yml labels; redeploy |
| |
|---|
| Symptom | Storefront shows wrong branding for hostname |
| Likely cause | Database restaurant_domains mismatch or stale SSR cache |
| Diagnosis | curl -H "Host: EXPECTED_HOST" -u "${STAGING_BASIC_AUTH_USER}:${STAGING_BASIC_AUTH_PASSWORD}" https://api.staging.getbytebite.co/storefront/restaurant and the same Host override against /storefront/menu |
| Resolution | Check restaurant_domains table. Re-run seed-staging.js if needed. This is an application/data issue, not routing |
| |
|---|
| Symptom | Random *.staging.getbytebite.co shows "Restaurant not found" |
| Expected | Storefront: HTTP 200 with not-found page. API tenant routes: HTTP 404. /api/health: HTTP 200 |
| Note | Storefront HTTP 200 for unknown tenants is a known application gap — not an infrastructure failure |
| |
|---|
| Symptom | /health/db returns degraded |
| Likely cause | PostgreSQL container down, wrong DATABASE_URL, network issue |
| Diagnosis | Check Dokploy PostgreSQL service status; verify DATABASE_URL in api container env (name only, not value) |
| Resolution | Restart PostgreSQL via Dokploy. See Disaster recovery |
| |
|---|
| Symptom | api stays down; storefront waits on API health |
| Likely cause | migrate one-shot failed (service_completed_successfully blocks api) |
| Diagnosis | docker logs on the migrate container; confirm DATABASE_URL is set (name only). Exited (0) with ✓ migrations applied is success |
| Resolution | Fix the migration or database connectivity, then redeploy. Manual fallback: pnpm deploy:db:migrate. Do not add a Dokploy domain to migrate |
| |
|---|
| Symptom | migrate logs getaddrinfo ENOTFOUND or similar for the DATABASE_URL host |
| Likely cause | migrate is not on the same Docker network as the Dokploy PostgreSQL service |
| Diagnosis | docker inspect networks on migrate vs api; expect dokploy-network (compose external) plus bytebite and default |
| Resolution | Confirm dokploy-network exists and is declared external in compose.deploy.yml. Align migrate networks with api |
| |
|---|
| Symptom | Compose UI lists migrate as Exited (0) after a successful deploy |
| Expected | One-shot finished; restart: "no" keeps it stopped. This is success, not a failed deploy |
| Resolution | No action if logs show ✓ migrations applied. Non-zero exit means migrate failed |
| |
|---|
| Symptom | api.staging… works; random *.staging… fails |
| Likely cause | Wildcard labels missing, dokploy-network not attached, or DNS-01 cert issue |
| Diagnosis | Inspect storefront/api container labels and networks; test wildcard TLS separately |
| Resolution | Redeploy compose; verify Traefik DNS-01 wiring. See Traefik post-upgrade checklist |
| |
|---|
| Symptom | Reserved hosts serve storefront content |
| Likely cause | Wildcard priority higher than explicit Dokploy routers |
| Diagnosis | Compare router priorities in Traefik dashboard or logs |
| Resolution | Ensure Dokploy explicit Host() routers exist and have higher priority than wildcard (10/20). This was verified working post-rollout — regression indicates Dokploy config loss |
| |
|---|
| Symptom | Traefik logs: unknown cert resolver letsencrypt-cloudflare |
| Likely cause | Traefik recreated without custom traefik.yml |
| Diagnosis | Check /etc/dokploy/traefik/traefik.yml for letsencrypt-cloudflare block |
| Resolution | Restore from backup. See Disaster recovery |
| |
|---|
| Symptom | ACME DNS challenge fails; Cloudflare API errors in Traefik logs |
| Likely cause | Invalid/expired token, wrong permissions, or missing mount |
| Diagnosis | Verify token file exists at /etc/dokploy/traefik/secrets/cloudflare-dns-api-token (mode 600); check container mount |
| Resolution | Rotate token. See Operations |
| |
|---|
| Symptom | Wildcard cert not renewing or corrupt ACME state |
| Likely cause | File permissions, concurrent writes, or manual edit |
| Diagnosis | Check file exists and is valid JSON; compare with backup checksums |
| Resolution | Restore from /root/bytebite-traefik-post-wildcard-20260908-175333/ |
| |
|---|
| Symptom | All explicit TLS may work (HTTP-01) but wildcard TLS fails after Dokploy upgrade |
| Likely cause | Container env/mount customization lost on recreate |
| Diagnosis | docker inspect dokploy-traefik — check for CF_DNS_API_TOKEN_FILE and secrets mount |
| Resolution | Re-apply customization from backup inspect. See Disaster recovery |
| |
|---|
| Symptom | docker inspect fails with permission denied |
| Likely cause | User not in docker group or not root |
| Diagnosis | groups; try sudo docker inspect |
| Resolution | Use sudo or add user to docker group (requires re-login) |
| |
|---|
| Symptom | Local dev requests hang ~5s then succeed |
| Likely cause | Missing ::1 entries in /etc/hosts — macOS tries IPv6 AAAA first |
| Diagnosis | Check /etc/hosts for both 127.0.0.1 and ::1 lines per host |
| Resolution | Add IPv6 entries. See Local deployment |
| |
|---|
| Symptom | api / docs / storefront crash with Error: Cannot find module '/app/pnpm'. dashboard exits 127 with /docker-entrypoint.sh: exec: line 47: pnpm: not found |
| Likely cause | Compose started production images. Node's entrypoint turns a missing pnpm into node pnpm (cwd /app). Dashboard prod is nginx, so exec pnpm fails immediately. Common after a local docker compose -f compose.deploy.yml build without -p bytebite-staging |
| Diagnosis | docker compose -f compose.yml -f compose.dev.yml images — app images must be bytebite-<service>:dev, not :prod or untagged last-stage builds |
| Resolution | pnpm dev:down then pnpm dev (builds the :dev targets). For local staging tests always use -p bytebite-staging. See Local deployment |
| |
|---|
| Symptom | Preview Compose in Dokploy UI shows labels not in Git |
| Expected | Dokploy injects domain labels and dokploy-network — this is normal |
| Resolution | No action needed. Repository compose.deploy.yml + Dokploy injection combine at runtime. See Dokploy |
| |
|---|
| Symptom | Authenticated curl -u … https://dashboard.staging.getbytebite.co/.env returns HTTP 200 with HTML |
| Expected | Unauthenticated requests are 401. With BasicAuth, nginx SPA fallback still serves index.html for dot-paths |
| Resolution | Known gap. Fix requires nginx config change to reject dot-paths. See Security |
| |
|---|
| Symptom | bytebite-postgres-backup.timer is active; bytebite-postgres-backup.service shows inactive (dead) |
| Likely cause | Type=oneshot service finished normally — this is expected after a successful run |
| Diagnosis | systemctl status bytebite-postgres-backup.service; check Result: and journalctl -u bytebite-postgres-backup.service |
| Resolution | Failure only if result/exit code indicates failure. Successful oneshot services are always inactive (dead) |
| |
|---|
| Symptom | Backup script or journal reports no matching PostgreSQL container |
| Likely cause | Dokploy PostgreSQL service stopped; container name changed (generated suffixes are unstable) |
| Diagnosis | docker ps --filter name=postgres; check Dokploy PostgreSQL service status |
| Resolution | Restart PostgreSQL via Dokploy. Do not hardcode old container suffixes — backup script discovers dynamically |
| |
|---|
| Symptom | Backup script aborts with multiple matching containers |
| Likely cause | More than one running postgres:18 container matches ByteBite staging criteria |
| Diagnosis | docker ps --filter ancestor=postgres:18 |
| Resolution | Stop or remove unintended containers. Backup intentionally aborts instead of guessing which DB to dump |
| |
|---|
| Symptom | Backup validation fails at pg_restore --list |
| Likely cause | Corrupt or incomplete dump |
| Diagnosis | Check backup journal for dump creation errors |
| Resolution | Treat dump as invalid — do not upload or use it. Investigate PostgreSQL container health and retry backup |
| |
|---|
| Symptom | Backup or maintenance fails during Restic upload, forget, prune, or check |
| Likely cause | Network issue, B2 credentials invalid, or repository configuration problem |
| Diagnosis | Check journal for Restic error messages; verify network connectivity from APP VM |
| Resolution | Diagnose B2 credentials and repository configuration in /etc/bytebite/backups/ (do not print contents). Rotate B2 credentials if needed without changing Restic encryption password |
| |
|---|
| Symptom | Weekly maintenance reports repository check errors |
| Likely cause | Repository corruption, incomplete upload, or storage backend issue |
| Diagnosis | journalctl -u bytebite-postgres-backup-maintenance.service |
| Resolution | Treat as backup-system incident. Investigate before relying on backups for recovery |
| |
|---|
| Symptom | postgres:18 container remains after restore drill |
| Likely cause | Script cleanup/trap failure or interrupted execution |
| Diagnosis | docker ps --filter ancestor=postgres:18; check for work directories under /var/lib/bytebite-backups/ |
| Resolution | Manually remove orphaned container and temporary directories. Re-run drill to verify cleanup |
| |
|---|
| Symptom | Cannot decrypt or access Restic repository |
| Likely cause | /etc/bytebite/backups/restic-password lost and no off-host copy available |
| Resolution | Encrypted repository cannot be recovered from B2 alone. Password must be stored in off-host credential manager |
| |
|---|
| Symptom | Restic cannot authenticate to B2 |
| Likely cause | Credentials revoked, expired, or host file lost |
| Diagnosis | Check maintenance/backup journal for authentication errors |
| Resolution | Rotate or recreate B2 credentials and update /etc/bytebite/backups/restic.env. Do not change Restic encryption password unless intentionally re-initializing the repository |
| |
|---|
| Symptom | curl -sI https://HOST/ on a staging app host returns 200 instead of 401 |
| Likely cause | A lower-priority unauthenticated Dokploy UI router is winning, or compose BasicAuth labels did not load |
| Diagnosis | Compare Traefik router priorities; confirm bytebite-staging-auth exists; confirm STAGING_BASIC_AUTH_USERS is in the running container labels |
| Resolution | Redeploy after setting STAGING_BASIC_AUTH_USERS. Do not remove Dokploy domains (they attach dokploy-network). See Traefik |
| |
|---|
| Symptom | Browser/curl with the known password still gets 401 |
| Likely cause | Hash interpolation ate $ (bcrypt/apr1), wrong user string, or labels not rewritten after env change |
| Diagnosis | Inspect the basicauth.users label (hash only — do not paste into tickets). SHA {SHA} hashes must contain {SHA} and no $ |
| Resolution | Regenerate with ./infrastructure/scripts/staging-htpasswd.sh, update Dokploy, redeploy. Prefer SHA over bcrypt in Dokploy |
| |
|---|
| Symptom | Compose interpolation error: STAGING_BASIC_AUTH_USERS is required |
| Likely cause | Variable missing from the Dokploy compose environment |
| Resolution | Set the variable, then deploy. Previous containers keep running until a deploy succeeds |
| |
|---|
| Symptom | dokploy.getbytebite.co prompts with realm ByteBite Staging |
| Likely cause | Middleware attached too broadly (entrypoint or Dokploy router) |
| Resolution | Remove BasicAuth from the Dokploy router. The compose middleware must not target that host |