Backup Features and Flows

1. Documentation Evidence

2. Feature Summary

3. Actor Matrix

4. Capability Matrix

5. User-Facing Flows

5.1 Take a backup — manual ("Run now")

Summary

An operator wants a backup right now — before a risky migration, before a release, or just to confirm the feature works. They click "Run now" and see the row move from queued to running to completed on the list.

Preconditions

• backupEnabled is true in settings (defaults true).
• No other backup is currently queued or running.
• Free disk space is at least minFreeDiskMb plus 1.2× the last completed dump's size (or just minFreeDiskMb for the very first backup ever taken).
• Superadmin session with Backup_CREATE.

Main Flow

Sequence Diagram

Branches and Edge Cases

5.2 Take a backup — scheduled (nightly)

Summary

No human is involved. Every night at scheduleCron/scheduleTimezone (default 0 2 * * * Asia/Kathmandu), the system decides which grandfather-father-son (GFS) tier is due and creates that backup automatically.

Preconditions

• backupEnabled is true.
• Same disk-space and no-run-in-progress checks as the manual path — the scheduler calls the identical BackupRunCreateService.createRun.

Main Flow

Sequence Diagram

Branches and Edge Cases

5.3 Retention pruning

Summary

Old backups disappear on their own, keeping disk usage bounded, while always leaving enough history to recover from. An operator never manually deletes backups to manage disk space unless they want to remove one specific run early.

Preconditions

• At least one bucket (daily/weekly/monthly/manual/pre_restore_safety) has more completed, unpinned rows than its retain* count.

Main Flow — scheduled sweep

Main Flow — manual delete

Sequence Diagram

Branches and Edge Cases

5.4 Restore the live database

Summary

An operator needs to undo something catastrophic — replace the entire live database with a known-good backup. This is the single most irreversible operation in the product, and it is the only flow in this module gated by two independent switches: a grantable permission and a deploy-time environment variable that cannot be granted from inside the running application.

Read this before anything else in this section: requesting a restore genuinely replaces the database — preflight validates everything, maintenance engages, an automatic pre-restore safety dump is taken, linked to this specific restore, and must succeed before anything destructive runs, existing connections are drained before being terminated, a detached process runs pg_restore --clean --single-transaction against the confirmed dump, and row counts are verified against the manifest afterward. What still is not automated: verification beyond row counts (content, permissions), and a one-click "undo" that automatically resubmits the linked safety dump. See 10 below for exactly what is and is not covered.

Preconditions

• BACKUP_RESTORE_ENABLED=true at the deploy level (default false — restore is meant to be armed deliberately, after an operator has already seen a backup complete and downloaded one, never on day one).
• BACKUP_ADMIN_DATABASE_URL configured — a separate database role from the application's own DATABASE_URL, one permitted to terminate backends and drop objects. The web application's own credential must never carry that authority, so that a SQL injection anywhere in the API cannot inherit it.
• The application's own DB role (DATABASE_URL's role) must not be a Postgres superuser. This is checked at preflight, before any write: the restore fences the application out with ALTER ROLE <app> CONNECTION LIMIT 0 so pg_restore --clean is not contending for ACCESS EXCLUSIVE locks against live traffic — and PostgreSQL does not enforce rolconnlimit against a role with rolsuper. If the application ever connected as a superuser, the fence would silently do nothing.
• Superadmin session with Backup_RESTORE.
• The source backup is completed with its artifact still present locally, its recorded size and sha256 matching the file on disk, and its schemaMigrationTag equal to the running schema's tag: the live database's latest drizzle.__drizzle_migrations.hash, read at preflight with the same query a backup records its own tag with. A null or "unknown" tag on either side refuses rather than compares (no override — a mismatch means the running code and the restored schema disagree, which surfaces later as a missing column somewhere unrelated).
• No other restore is non-terminal, and no backup dump is currently queued/running.

Main Flow

Why --single-transaction is load-bearing

Probed directly, not assumed: restoring a truncated dump with --single-transaction left the target database completely untouched — the whole restore rolled back as one unit. Restoring the same truncated dump without the flag emptied the table and the original rows were gone. The automatic pre-restore safety dump is a second, independent safety net on top of this — the way back if the chosen backup itself turns out to be wrong, not just if pg_restore fails partway. This is why --single-transaction is a fixed part of pg_restore's argv in the runner script, never a configurable option.

Sequence Diagram

Branches and Edge Cases

Re-keying the restored outbox by hand

The runner gives every pending outbox row a fresh dispatch_key right after pg_restore commits, while the connection fence still holds. When that step fails (BACKUP_RESTORE_OUTBOX_REKEY_FAILED), the runner still releases the connection fence on its way out, as it does on SIGTERM. The outbox dispatcher and its scheduler do not consult the traffic lock, which gates only HTTP, so relaying resumes as soon as the fence is released. A runner killed with SIGKILL between pg_restore committing and the re-key leaves the fence set only until the supervisor marks the restore abandoned at restoring after BACKUP_RESTORE_ABANDON_SECONDS. The supervisor then releases the fence itself, and relaying resumes with the pending rows still on their restored keys.

In either case the operator runs the same statement the runner runs, against the restored database, immediately:

UPDATE public.outbox_events SET dispatch_key = gen_random_uuid() WHERE status = 'pending';

Only pending rows are re-keyed; a dispatched row keeps its key, because report maintenance looks its live job up by it.

A row that moved to dispatched between the failure and the statement may have been relayed onto a job BullMQ still retained from before the restore, and its work may not have run. Check each such row by looking up the job with id outbox-<dispatch_key> on the row's target_queue. The application sets no BullMQ key prefix, so that job's Redis hash is bull:<target_queue>:outbox-<dispatch_key>; a job whose timestamps predate the restore is the retained job, not a relay of the restored row.

5.5 Off-host replication

Summary

An operator wants a copy of every backup somewhere other than the VPS the database lives on, so a lost or compromised host does not also mean a lost backup. This is opt-in, transport-hardened, and verified by reading the bytes back rather than trusting the transfer's own exit code.

Preconditions

• BACKUP_REMOTE_ENABLED=true, BACKUP_REMOTE_TARGET (a user@host:/path), BACKUP_REMOTE_SSH_KEY_PATH and BACKUP_REMOTE_KNOWN_HOSTS_PATH all configured at deploy time.
• remoteReplicationEnabled=true in settings, and storageMode set to both or remote_only.
• At boot, BackupReplicationService refuses to start with replication enabled unless the configured rsync binary supports --secluded-args (rsync 3.2.3+) and the known-hosts file exists and is non-empty — both are asserted once, at startup, rather than discovered failing at 02:00.

Main Flow

Sequence Diagram

Branches and Edge Cases

5.6 Upload and catalogue an operator-supplied archive (SE-4)

Summary

An operator has a pg_dump archive from somewhere other than this deployment's own scheduled/manual dumps — moved from another environment, or pulled off a downed host — and wants it in the catalog so the existing restore flow can act on it. This never restores anything by itself; it only authenticates and files the archive away.

Preconditions

• Superadmin session with BackupUpload_CREATE.
• A current signing key held in the key file — see 5.8.
• The archive, its manifest.json, and its artifact.sig were all produced by a deployment holding a key this one generated — the signature is the actual authentication, not the checksum inside the manifest. An imported key does not count here: on upload the caller supplies the archive, the signature and the key, which is three operands from one actor.
• Fewer than BACKUP_UPLOAD_MAX_RETAINED completed uploaded backups already exist.

Main Flow

Sequence Diagram

Branches and Edge Cases

5.7 Release a traffic lock a failed restore left engaged

Summary

A restore engages the traffic lock when it is accepted, marked reconciliation: "pending", and lifts it itself when it completes cleanly and has reconciled the restored database. When a restore fails — the safety dump does not complete, pg_restore fails, the restored row counts do not match the manifest, the runner is abandoned, or the integration reconciliation fails — the lock stays engaged on purpose: a half-known database is not put back in front of users until an operator has looked. The operator checks the database, then releases the lock from the Backups page. No route engages the lock by hand.

A runner that has exited is not the same as a restore that has settled. The restored rows hold every integration key revoked after the dump as active, so a restore that may have replaced the database reconciles it — a new change-feed epoch, every integration key revoked, every webhook paused — before the lock can be released, whether the restore completed or failed after pg_restore committed. The lock records where that stands in reconciliation, and release refuses while it is pending or reconcile_failed.

Preconditions

• Superadmin session: Backup_READ to read the lock, Backup_RESTORE to release it.
• No backup_restore row is queued, safety_dump, restoring or verifying.
• The runnerPid recorded in restore-state.json is not a live process.
• The lock's reconciliation is reconciled or absent (null in the DTO) — not pending, not reconcile_failed.

Where reconciliation comes from

Every write compares restorePublicId first: a lock re-engaged by a later restore is never marked by an earlier one.

Main Flow

Sequence Diagram

Branches and Edge Cases

While the lock holds, the health probes, the operator login and refresh, GET /api/auth/me, GET /api/auth/permissions, every route under /api/admin/system/restores/, and every permission-gated admin route stay reachable. Every other route answers 503 SYSTEM_MAINTENANCE_ACTIVE — see the API doc's 8.15.

5.7a Getting the uploaded files back

A backup taken with "Also archive uploaded files" produces two artifacts: database.dump + manifest.json + artifact.sig in one tar, and uploads.tar.gz beside them.

Download hands out the first three only. That is deliberate — they are exactly what the upload endpoint consumes, and folding a multi-gigabyte archive into that tar would break the round trip and make every routine dump download enormous.

What was missing is that nothing said so. The operator ticked the box, the panel reported the backup complete and showed the archive's size, and the download returned a file without it.

So there are two controls, and neither is labelled just "Download":

That third condition is not padding. The retention sweep reclaims a backup to a remote copy by deleting the local files and setting local_artifact_present false — while leaving status as completed. Without checking it, the link renders on a swept run and answers 410; because the control is a plain link, the tab navigates away and shows raw JSON. A race can still reach that (the sweep can land between render and click), so the conditions make a refusal unlikely rather than impossible.

Verifying what you downloaded

The detail page shows Uploads archive sha256. Check the file you received:

sha256sum backup-<publicId>-uploads.tar.gz

The server deliberately does not hash the archive before sending it — see API 8.23. Hashing it first would read the whole file before the first header and turn every large download into a timeout, and nothing restores this archive, so there is no automated consumer to protect. The published digest is the control, and it verifies the bytes that actually arrived rather than the bytes that were on disk.

What the archive contains, and one thing to know about it

Everything under the upload root at dump time, as a gzipped tar. Two caveats:

• The artifact HMAC covers the database dump and manifest only, never uploads.tar.gz. The signature badge says "Signed (database dump)" for exactly this reason.
• The archive is taken from the local upload root regardless of the configured storage driver. On a deployment whose files live in object storage that directory may be empty or stale, and the run still records a size and a checksum. The rendered byte count next to the link is the operator's warning.

5.8 Custody of the artifact-signing key

The key that decides whether a backup archive can be verified — and therefore whether it can be restored — used to live inline in .env. It now lives in a file the application owns, with a lifecycle an operator can drive from the panel.

Rotation keeps the old key, and that is the whole design. A retired key is not a dead key: it is what still verifies every artifact signed before the rotation. Without a retired set, "rotate the signing key" would mean "make every existing backup unrestorable" — and an operator would discover that at the worst possible moment.

Reveal-once. generate and rotate return the key material in their response and nothing can retrieve it afterwards. The panel shows it in a dialog whose confirm button is disabled until the operator ticks "I have stored this key somewhere outside this server". If the value is lost, the recovery is to rotate again — not to read it back.

The migration never generates a fresh key. It copies the existing .env value byte for byte, because every artifact already on disk is signed with it. A fresh key would make them all unverifiable, and the symptom — BACKUP_UPLOAD_SIGNATURE_INVALID — reads like corruption rather than like a key change, which is the worst possible way for that to present.

Pruning a retired key, and the two refusals

Deleting a retired key is irreversible: the only way back is to import the same material again, if the operator still has it. Two independent checks refuse it, and force overrides both.

Neither count can see an archive kept off this host — it has no row here. So a 0 never means "nothing depends on this key", and the panel says so in the same sentence as the number. That is also exactly the population import exists for.

Resolving signatures

resolve-signatures walks completed runs, reads each artifact's artifact.sig, and records which held key verifies it. It is batched: call again while remaining is above zero.

It re-examines rows already resolved as signed_unknown_key, because importing a key changes that answer. signed_unknown_key is a cache, not a fact — which is why every predicate that selects work reads signature_state IS NULL OR signature_state = 'signed_unknown_key', never IS NULL alone. A predicate that checked only NULL would resolve a row once, cache "no key held", and never look again, so importing the very key that verifies it would change nothing.

5.9 Who did this, from where

The admin panel is a BFF: the browser calls the Next server, and that server calls the API. So the API's req.ip was the panel process, and every admin action in the activity log read 127.0.0.1 — literally accurate, and useless for a log whose entire purpose is telling operators apart.

Three decisions worth stating, because each has an obvious wrong answer:

• Read from the right, never split(",")[0]. Under nginx's standard $proxy_add_x_forwarded_for the leftmost entry is whatever the client sent — fully attacker-controlled. Counting in from the right, past the hops actually operated, is the only reading a caller cannot forge.
• TRUSTED_PROXY_HOPS=0 means "derive nothing", and it is the default. With one trusted hop and a single-entry inbound header — every local run, and any deployment where nothing in front rewrites x-forwarded-for — the entry counted from the right is the browser's own value. Forwarding that under a valid token would hand the caller the address the audit log records, inverting the whole exercise. A deployment must say it has an address-rewriting proxy.
• The address is not a rate-limit input. It reaches the activity log and nothing else. Feeding a client-influenced address into IpThrottlerGuard would let an unauthenticated attacker choose their own bucket on the admin credential path.

Until a deployment configures both halves, the log records the loopback address — which is merely useless, rather than wrong. That is the deliberate failure mode: an audit entry that says nothing beats one that says something false.

6. Admin Flows

6.1 Create

See 5.1 — POST admin/system/backups. Route: POST admin/system/backups. Permission: Backup_CREATE. Mutation side effects: backup_run insert, outbox enqueue. Cache invalidation: none (this module has no cached list reads). Audit: backup.create. Failure branches: 503 BACKUP_DISABLED, 409 BACKUP_ALREADY_RUNNING, 503 BACKUP_INSUFFICIENT_DISK_SPACE.

6.2 List / Read Detail

GET admin/system/backups and GET admin/system/backups/{publicId}. Permission: Backup_READ. No side effects. Failure: 404 BACKUP_NOT_FOUND on detail.

6.3 Update (pin/unpin)

PATCH admin/system/backups/{publicId} with {pinned: boolean}. Permission: Backup_UPDATE. Mutation: backup_run.pinned. Audit: backup.pin/backup.unpin with a changes[] diff. Failure: 404 BACKUP_NOT_FOUND.

6.4 Soft delete (prune)

DELETE admin/system/backups/{publicId} — see 5.3. Not a true soft delete in the usual sense: the row is never restorable back to completed, but it survives as a pruned audit record rather than being hard-deleted. Failure: 404, or 409 on any of the 3 floors.

6.5 Restore

See 5.4 in full. Route: POST admin/system/backups/{publicId}/restore. Permission: Backup_RESTORE, plus BACKUP_RESTORE_ENABLED. Takes an inline pre-restore safety dump, genuinely replaces the database via restore-runner.ts, and verifies row counts against the manifest afterward; ends at completed or failed (BACKUP_RESTORE_FAILED or, specifically for a row-count mismatch, BACKUP_RESTORE_VERIFICATION_FAILED), or at a manually force-cleared failed if the safety dump or the runner never completes.

6.6 Force-clear

POST admin/system/restores/{publicId}/force-clear — see 5.4's edge cases. Permission: Backup_RESTORE. Mutation: backup_restore.status='failed'; lifts the traffic lock unless the lock owes a post-restore reconciliation (see 5.4). Refuses while the state file names this restore's runner as a live process, and refuses a row that is already terminal — a restore that already failed is recovered by releasing the lock (6.8).

6.7 Settings (configure)

GET/PUT admin/system/backups/settings, backing a dedicated admin panel settings page — see Section 12.3 for the field-by-field editable/environment-only split. Permission: Backup_READ (get) / BackupConfigure_UPDATE (put) — a third, separate permission module from Backup_*, because authorising dump/pin/prune/restore is not the same act as authorising a change to the off-host replication destination. PUT is a full-replace write keyed by an optimistic version — a stale read is 409 BACKUP_SETTINGS_CONFLICT, not a silent overwrite.

6.8 Restore traffic lock (supporting flow, not its own feature)

GET admin/system/restores/lock (Backup_READ) and POST admin/system/restores/lock/release (Backup_RESTORE). The lock is engaged automatically when a restore is accepted and lifted automatically when it completes cleanly and has reconciled the restored database, or by force-clear when no reconciliation is owed; an operator releases it after a failed restore once no restore can be running and its reconciliation is not pending or reconcile_failed. No route engages it by hand. See 5.7 and the state diagram in Section 7.2.

6.9 Upload (catalogue)

See 5.6 in full. Route: POST admin/system/backups/upload. Permission: BackupUpload_CREATE — a fourth, separate permission module from Backup_*/BackupDownload_*/BackupConfigure_*, because introducing a database from outside is not the same act as running a dump of this one. Synchronous — 201 means the row is already catalogued, not queued.

6.10 Key custody

See 5.8 in full. Routes: GET admin/system/backup-key plus generate / import / rotate / resolve-signatures and DELETE .../retired/{fingerprint}. Permission: BackupKey_* — a fifth, separate permission module, because replacing the credential every artifact is authenticated against is not the same act as changing how often a dump runs. Audit: backup.key.generate / .import / .rotate / .prune. Every mutating route additionally re-checks the operator's password.

Failure branches: 400 BACKUP_KEY_REAUTH_REQUIRED, 401 BACKUP_KEY_REAUTH_FAILED, 409 BACKUP_KEY_NO_PASSWORD_SET, 409 BACKUP_KEY_ALREADY_EXISTS, 409 BACKUP_KEY_FINGERPRINT_UNKNOWN, 409 BACKUP_KEY_STILL_IN_USE, 422 BACKUP_KEY_PATH_NOT_CONFIGURED, 422 BACKUP_KEY_INVALID_MATERIAL, 422 BACKUP_KEY_STORE_UNREADABLE.

7. Lifecycle and State Transitions

7.1 backup_run.status

7.1a backup_run.signature_state

The only transition into signed_unknown_key is "no held key matched at that moment", and the only way out is importing a key. A resolution predicate that read signature_state IS NULL alone would resolve each row once, cache "no key held", and never revisit it — so importing the very key that verifies an archive would change nothing, and the prune refusal built on those counts would be built on a stale answer. Every predicate therefore reads signature_state IS NULL OR signature_state = 'signed_unknown_key'.

The three CHECK constraints that hold the shape (signed implies a fingerprint, unsigned implies none, any state implies status = 'completed') are written as implications with IS DISTINCT FROM, never as a closed disjunction over the enum — a CHECK passes on NULL, and a future enum value must not silently fall into the wrong branch.

7.2 Restore traffic lock state (supporting)

7.3 backup_restore.status and the live-state file's stage — two different state machines

The table column only ever holds queued or a terminal value, by design — pg_restore --clean replaces the table holding its own row mid-restore, so intermediate stages cannot live there:

The state file's stage field carries the real intermediate lifecycle (accepted → queued → draining → safety_dump → restoring → verifying → reconciling → completed/failed). safety_dump is written by BackupRestoreProcessor.takeSafetyDump; draining, restoring, verifying, completed and failed are written by restore-runner.ts as it fences, runs pg_restore, and checks row counts against the manifest. Only reconciling remains declared but never reached by any current code path. See 5.4.

9. Data and Side Effects by Flow

10. Error and Recovery Flows

10a. What the operator sees when a download is refused

Both download controls are plain <a href>. They have to be: the response is a stream, and the access token is an httpOnly cookie the browser cannot read, so the BFF route beside the page is what carries it. The consequence was that any non-2xx navigated the operator out of the dashboard and rendered the API's error envelope as raw JSON.

The BFF now answers a failed navigation with a 302 back to the page the operator started from, carrying ?downloadError=<code>. An inline notice renders the mapped message and the parameter is removed from the URL, so a refresh or a shared link cannot replay someone else's error.

Three things about this are deliberate:

• A non-navigation still gets JSON. Only a request carrying Sec-Fetch-Mode: navigate is redirected, so the route keeps an API-shaped contract for a same-origin fetch().
• A CSRF refusal is never redirected. A 302 into the dashboard would erase the signal that the guard fired, and the guard deliberately answers before the authentication check so a cross-site probe cannot tell a signed-in operator from a signed-out one.
• The message is looked up, never rendered from the URL. downloadError is attacker-supplied by construction — anyone can craft the link and send it. It only ever selects from a fixed map, and an unrecognised value gets a generic message. Rendering it would let whoever wrote the URL choose what the operator reads.

The obvious alternative — fetch the response in a click handler and show the error — was rejected: it means buffering the success case in the tab, and these artifacts are multi-gigabyte by design. That trades a bad error surface for an out-of-memory crash on the path that works.

11. Diagrams Required Per Module

• Actor capability diagram: Section 3.
• High-level module flow diagram: Section 5.1's sequence diagram, representative of the shared dump path.
• Sequence diagram for each major flow: 5.1–5.7, plus 6.4's activity diagram.
• State machine diagrams: Section 7, covering backup_run.status, the restore traffic lock, and the two-track restore state.
• Data side-effect diagram: Section 9.
• Error branch diagram: 6.4's prune flowchart, and the restore sequence diagram in 5.4.

12. Mandatory Feature and Flow Deep-Dive Pack

12.1 Feature Inventory With Minor Behaviors

12.2 Business Process Diagram Pack

See Section 5 for a sequence diagram per flow and Section 6.4 for an activity diagram. A service blueprint for the manual-backup flow:

12.3 Business Rules and Policy Traceability

12.4 Tradeoffs and Product Rationale

12.5 Flow Edge-Case Matrix

12.6 Flow-to-Data Trace

12.7 Experience Quality Checklist

• The doc explains what the actor is trying to accomplish (each flow's Summary).
• The doc explains what the backend does that the actor does not see (snapshot capture, manifest-before-row-completion, path recomputation for replication, connection fencing).
• The doc covers minor flows and branches (12.1, 12.5).
• The doc includes user (superadmin), worker/system, and the detached restore-runner actor.
• The doc explains business logic, tradeoffs, and rationale (12.3, 12.4).
• The doc maps every flow to API routes and backend side effects (Section 9, 12.6).
• The doc includes diagrams appropriate to each flow type (Section 11).
• The doc covers edge cases and failure recovery, including what the restore-safety net now genuinely does (pre-restore dump, row-count verification) and what still is not automated (linking the dump back to its restore), stated plainly rather than glossed over (Section 10, 12.5).

13. Completion Checklist

• Every feature and minor action documented is listed: backup (manual + scheduled), retention pruning, download, upload/catalogue (SE-4), restore (request through genuine execution, safety dump, and row-count verification), replication, and the restore traffic lock (read and release) as a supporting flow.
• Every actor has allowed and forbidden behavior (Section 3).
• Every major and minor flow includes steps, branches, and diagrams (Section 5, 12.1, 12.5).
• Every lifecycle has a transition table and state diagram (Section 7).
• Every flow links to the API and backend docs.
• Backend-doc dependencies are called out where they shape behavior — particularly the remaining safety-dump-linkage gap, cross-referenced to the backend doc's 16.8 Risk Register.
• The upload endpoint (admin/system/backups/upload, SE-4) is fully documented in 5.6 and 6.9.

See Also

• API doc: /docs/developer/backup/api
• Backend doc: /docs/developer/backup/backend
• TDD: not present for this module at time of writing.