Backup - API Reference

Audience: Admin panel engineers, QA, and API consumers with superadmin-level access. Scope: Admin-only. Backup catalog, backup download, upload/catalogue, restore request/live-state/force-clear, the restore traffic lock (read and release), and artifact signing-key custody — all under admin/system/*. The traffic lock itself is engaged and lifted by the restore automatically; no route turns it on. The one route that does not take an admin session is GET .../restores/{publicId}/live, which is @Public() to the session guard and authenticated by the restore's poll token instead.

1. Documentation Evidence

2. Module Summary

3. Concepts and Terminology

4. API Surface Map

Every route below is confirmed against structure.baseline.json.

Why restores/lock cannot be captured by restores/{publicId}/...

BackupRestoreAdminController declares restores/:publicId/live (GET) and restores/:publicId/force-clear (POST) on the same prefix as the two lock routes. Neither can match them: GET restores/lock has one segment after restores where live has two, and POST restores/lock/release ends in release, not force-clear. No GET restores/:publicId exists, which is the shape that would have swallowed restores/lock.

Why the key routes are backup-key, not backups/key

They were originally admin/system/backups/key*, on the same prefix as BackupAdminController's @Get(":publicId"). Express matches in registration order and module-import order decided the winner, so GET /api/admin/system/backups/key was matched by :publicId and answered 400 publicId must be a UUID — every read on the key custody page was dead.

Nothing caught it. Unit tests call the controller method directly and never build a route table; structure.baseline.json records that a route is REGISTERED, which it was; and an unauthenticated probe gets 401 from JwtAuthGuard before the parameter pipe runs, so the collision is invisible from outside a session. It was found by opening the page in a browser, and apps/api/test/structure/route-shadowing.spec.ts now fails the build on the whole class.

24 distinct route templates, matching the unique entries found in structure.baseline.json for these prefixes (the file lists each twice — once per Swagger document it appears in): 14 under admin/system/backups (including POST .../upload), 4 under admin/system/restores, and 6 under admin/system/backup-key.

5. Auth, Identity, and Permissions

No route in this module accepts a guest. Nothing in the module has a dedicated mobile surface.

6. DTO and Model Reference

6.1 CreateBackupDto (request body — POST /admin/system/backups)

An explicit false for includeUploads is honoured — only absence falls back to the configured default. Under a remote (non-local) storage driver, the effective value is forced to false regardless of what was requested; the run records includesUploads: false.

6.2 FetchBackupsDto (query — GET /admin/system/backups)

Extends the shared QueryDto (pagination, page, size, sort, order, search — search is accepted by the base class but not applied by BackupAdminService.findAll, which filters only on the three fields below).

6.3 UpdateBackupDto (request body — PATCH /admin/system/backups/{publicId})

6.4 BackupParamsDto / BackupDownloadParamsDto / RestoreParamsDto

Identical shape, three separate classes (one per controller's route params).

6.5 BackupResponseDto

Deliberately absent, by design (matching RestoreLiveStateDto's frozen contract): the integer PK, artifactDir (a filesystem path), and the database dump's own checksum — its integrity is enforced server-side before every restore rather than exposed for a caller to check.

6.6 BackupSettingsResponseDto / UpdateBackupSettingsDto

Why the environment-only fields can never be form fields: pgDumpPath/pgRestorePath/TAR_PATH (not shown in the response, but the same class) are absolute binary paths passed directly to child_process.spawn — an admin-editable path is arbitrary code execution on every scheduled backup. BACKUP_DIR (not in the response at all) is validated at boot against every statically-served root; an admin-editable value could not receive that boot-time safety check and would risk publishing every database dump over HTTP with no guard able to intervene (ServeStaticModule sits outside the guard pipeline). The remote replication destination and both replication credentials (BACKUP_REMOTE_TARGET, BACKUP_REMOTE_SSH_KEY_PATH) are a lower bar than BackupDownload_READ itself — an admin who could edit the destination could redirect every future backup to a host of their choosing, and one who could edit the key path could substitute their own key. BACKUP_RESTORE_ENABLED is a deploy-time kill switch specifically because an admin-editable kill switch is not a kill switch — its entire value is that arming the single most destructive endpoint in the product requires shell access to the box.

6.7 RestoreBackupDto (request body — POST .../restore)

6.7a RestoreAuditLogsResultDto

The outcome of the MongoDB audit_logs merge a completed restore performs — see §7.5, step 8 in the backend doc for the full merge sequence and the signature-verified anchor it runs from. It is returned as auditLogs on RestoreLiveStateDto (§6.8) once the restore has completed and the merge step has run, and is absent before that. The merge runs after every restore; its outcome is also written to the restore state file and logged.

6.8 RestoreLiveStateDto (response — POST .../restore, GET .../live)

6.9 RestoreLockDto (response — GET .../restores/lock)

Every field is always present. When the lock is not engaged, active and releasable are false and every other field except blockingRestorePublicId is null — including when the state file records an earlier release, so a consumer never reads a stale engagedAt as a live one. When maintenance.json exists but cannot be read, active is true, reason is "unreadable", the other lock fields are null, and releasable depends only on blockingRestorePublicId being null.

6.10 BackupUploadDto (multipart request body — POST .../backups/upload)

All three parts are required — there is no partial-upload path. Response is a 201 BackupResponseDto (the same shape as 6.5), with kind: "uploaded", pinned: true and status: "completed" immediately (the upload is synchronous; there is no async job).

6.11 RestoreLockReleaseDto (response — POST .../restores/lock/release)

The route takes no request body. Both DTOs live in admin/restore/dto/restore-lock.dto.ts and carry Swagger decorators only, like RestoreLiveStateDto — they are responses, so there is nothing to validate.

6.12 FetchBackupFailuresDto / BackupFailureParamsDto / BackupJobFailureDto

BackupJobFailureDto (response, list and both retry routes):

7. Enum Reference

8. Endpoint Reference

8.1 GET /api/admin/system/backups

Purpose

Lists backup runs for the admin catalog view, with optional filtering by kind, status and tier and standard offset pagination. Used to populate the backup history table and to check whether a scheduled run completed.

Source Evidence

Auth and Permissions

• Auth: JwtAuthGuard + RoleGuard
• Guard chain: JwtAuthGuard, RoleGuard, IpThrottlerGuard
• Permission: Backup_READ
• Guest support: none
• Rate limit: ADMIN_READ — 30/min, IP-keyed
• Idempotency: N/A (read)

Request

GET /api/admin/system/backups?status=completed&page=1&size=20 HTTP/1.1
Authorization: Bearer TOKEN

Response

{
  "success": true,
  "message": "Backups fetched",
  "data": [
    {
      "publicId": "018f4e2a-7b3c-7c1e-9b2a-3d4e5f6a7b8c",
      "kind": "scheduled",
      "tier": "daily",
      "status": "completed",
      "pinned": false,
      "includesUploads": false,
      "databaseDumpBytes": 23012582,
      "uploadsArchiveBytes": null,
      "totalBytes": 23012582,
      "postgresVersion": "PostgreSQL 16.4",
      "schemaMigrationTag": "a1b2c3",
      "tableCount": 42,
      "totalRows": 18342,
      "requestedByEmail": null,
      "note": null,
      "startedAt": "2026-08-30T02:00:03.000Z",
      "finishedAt": "2026-08-30T02:00:41.000Z",
      "durationMs": 38000,
      "errorCode": null,
      "errorMessage": null,
      "isRestorable": true,
      "createdAt": "2026-08-30T02:00:01.000Z"
    }
  ],
  "meta": { "count": 42, "page": 1, "size": 20 }
}

Side Effects

None — this is a read-only endpoint.

Error Cases

Edge Cases

• Empty result: data: [], meta.count: 0.
• pagination=false: meta is undefined and every matching row is returned unpaged.
• No filters: returns every run, newest first.

Example Requests

curl -X GET "$API_URL/api/admin/system/backups?status=completed" \
  -H "Authorization: Bearer TOKEN"

8.2 POST /api/admin/system/backups

Purpose

Triggers a manual backup ("Run now"). Validates that backups are enabled, no run is currently in progress, and there is enough free disk space, then inserts a queued row and returns immediately — the actual dump runs asynchronously via the outbox and QueueName.BACKUP.

Source Evidence

Auth and Permissions

• Permission: Backup_CREATE
• Rate limit: ADMIN_ASYNC_JOB_SUBMIT — 10/hour, IP-keyed
• Idempotency: none declared (@IdempotentCreate not used) — a duplicate click is refused synchronously by BACKUP_ALREADY_RUNNING as soon as the first row is queued, not only once a worker has claimed it into running.

Request

{ "includeUploads": true, "note": "before catalog migration" }

Minimal valid request: {} (both fields optional).

Response

201, same shape as 8.1's list item, status: "queued".

Side Effects

• Database write: INSERT backup_run + outbox_events (same transaction).
• Queue: backup.run reaches QueueName.BACKUP via the outbox dispatcher (~5s poll interval).
• Audit: ActivityRecordService.recordActivity (action: "backup.create"), fire-and-forget.

Error Cases

Edge Cases

• First-ever backup: the disk-space check uses a stated floor (no prior dump to multiply against), never dividing by/multiplying NULL.
• includeUploads: false explicit: honoured even though the configured default is also false — ??, not ||.
• Under a remote storage driver: includeUploads: true is silently overridden to false at dump time; the response later reports includesUploads: false.

Example Requests

curl -X POST "$API_URL/api/admin/system/backups" \
  -H "Authorization: Bearer TOKEN" -H "Content-Type: application/json" \
  -d '{"includeUploads": true, "note": "before catalog migration"}'

8.3 GET /api/admin/system/backups/settings

Purpose

Reads the full effective backup/retention configuration for the settings screen — the merge of admin-edited settings.json (or env-seeded defaults if never written) plus read-only environment fields plus computed replication health. Registered ahead of :publicId so "settings" is never parsed as a UUID.

Source Evidence

Auth and Permissions

• Permission: Backup_READ
• Rate limit: ADMIN_READ

Response

See 6.6 for every field.

{
  "version": 3,
  "backupEnabled": true,
  "scheduleCron": "0 2 * * *",
  "scheduleTimezone": "Asia/Kathmandu",
  "includeUploadsByDefault": false,
  "retainDaily": 7, "retainWeekly": 4, "retainMonthly": 6, "retainManual": 10, "retainSafety": 3,
  "minFreeDiskMb": 2048,
  "dumpTimeoutMinutes": 30, "restoreTimeoutMinutes": 60,
  "restoreDrainSeconds": 120, "restoreAbandonSeconds": 900,
  "storageMode": "local",
  "remoteReplicationEnabled": false,
  "restoreEnabled": false,
  "restoreAdminDatabaseConfigured": false,
  "pgDumpPath": "/usr/bin/pg_dump",
  "pgRestorePath": "/usr/bin/pg_restore",
  "remoteConfigured": false,
  "lastReplicationFailureAt": null,
  "replicationFailureStreak": 0
}

Side Effects

None.

Error Cases

None beyond auth/rate-limit — getEffectiveSettings never throws.

8.4 PUT /api/admin/system/backups/settings

Purpose

Updates the admin-editable backup/retention policy in one full-replace write. The admin panel reads 8.3, edits it, and sends the whole shape back with the version it read.

Source Evidence

Auth and Permissions

• Permission: BackupConfigure_UPDATE — not Backup_UPDATE, which today gates only pin/unpin on one row.
• Rate limit: ADMIN_WRITE

Request

Full valid request — every field in 6.6 except the read-only ones, plus version:

{
  "version": 3,
  "backupEnabled": true,
  "scheduleCron": "0 3 * * *",
  "scheduleTimezone": "Asia/Kathmandu",
  "includeUploadsByDefault": false,
  "retainDaily": 7, "retainWeekly": 4, "retainMonthly": 6, "retainManual": 10, "retainSafety": 3,
  "minFreeDiskMb": 2048,
  "dumpTimeoutMinutes": 30, "restoreTimeoutMinutes": 60,
  "restoreDrainSeconds": 120, "restoreAbandonSeconds": 900,
  "storageMode": "local",
  "remoteReplicationEnabled": false
}

Response

200, same shape as 8.3, reflecting the new version.

Side Effects

• Filesystem write: settings.json (atomic temp-file + rename).
• If scheduleCron/scheduleTimezone changed: BackupScheduleScheduler.reload() re-registers the cron job immediately, without an API restart.
• Audit: activityRecordService.recordActivity with a full before/after changes[] diff.

Error Cases

Example Requests

curl -X PUT "$API_URL/api/admin/system/backups/settings" \
  -H "Authorization: Bearer TOKEN" -H "Content-Type: application/json" \
  -d '{"version":3,"backupEnabled":true,"scheduleCron":"0 3 * * *", "...": "..."}'

8.5 GET /api/admin/system/backups/{publicId}

Purpose

Fetches one backup's detail for the catalog detail view.

Source Evidence

Auth and Permissions

• Permission: Backup_READ; Rate limit: ADMIN_READ

Request

Response

Same shape as one item from 8.1.

Error Cases

8.6 PATCH /api/admin/system/backups/{publicId}

Purpose

Pins or unpins a backup, protecting (or exposing) it to count-based retention pruning.

Source Evidence

Auth and Permissions

• Permission: Backup_UPDATE; Rate limit: ADMIN_WRITE

Request

{ "pinned": true }

Response

200, updated BackupResponseDto.

Side Effects

• UPDATE backup_run SET pinned = ...
• Audit: backup.pin / backup.unpin with a changes: [{field:"pinned", from, to}] entry.

Error Cases

8.7 DELETE /api/admin/system/backups/{publicId}

Purpose

Requests that a backup be pruned. Validates the three retention floors synchronously (so an ineligible delete gets an immediate 409, never a silently-skipped async job), then enqueues the actual deletion through the outbox.

Source Evidence

Auth and Permissions

• Permission: Backup_DELETE; Rate limit: ADMIN_WRITE

Response

{ "success": true, "message": "Prune requested", "data": { "publicId": "018f..." } }

Side Effects

• Outbox enqueue: backup.prune {reason:"manual", backupRunPublicId}, dedupeKey: "manual-prune-<publicId>".
• Audit: backup.delete_requested.
• The actual filesystem rm -rf and row update to pruned happen later, in BackupPruneHandler, which re-checks assertPrunable — the floors can change between the click and the worker picking up the job.

Error Cases

8.8 GET /api/admin/system/backups/{publicId}/download

Purpose

Streams the raw pg_dump artifact for a completed backup. The highest-consequence read this module offers — every check runs before the first byte, and the audit write is the final gate and fails closed.

Source Evidence

Auth and Permissions

• Permission: BackupDownload_READ — a separate permission module from Backup_*, deliberately: seeing that a backup succeeded and walking out with the entire customer database are different acts with different audiences.
• Rate limit: ADMIN_BACKUP_DOWNLOAD — 3/hour, user-keyed (not IP), because an IP-keyed cap on the endpoint that streams every customer record and password hash is bypassed by rotating source IPs.

Request

Response

Raw application/octet-stream, streamed via createReadStream(...).pipe(res) — never buffered. Headers: Content-Type: application/octet-stream, Content-Disposition: attachment; filename="backup-<publicId>.dump" (filename derived from publicId alone, never the operator-editable note, to avoid CRLF header injection), Content-Length, Cache-Control: no-store.

Side Effects

• stat() and streaming sha256 read of the dump file (integrity checks, size first then checksum).
• MongoDB write: AuditLog.create({action:"backup.download", ...}) — must succeed, or the download itself is refused with 500 BACKUP_DOWNLOAD_AUDIT_FAILED. This is the one write in the whole module that is not fire-and-forget.

Error Cases

Example Requests

curl -X GET "$API_URL/api/admin/system/backups/018f.../download" \
  -H "Authorization: Bearer TOKEN" -o backup.dump

8.9 POST /api/admin/system/backups/{publicId}/restore

Purpose

Accepts a request to replace the entire live database from a completed backup. Irreversible in intent. Runs every safety check synchronously, then engages maintenance, writes a live-state file, and enqueues backup.restore, which BackupRestoreProcessor consumes: it takes and links an inline pre-restore safety dump, then spawns and supervises restore-runner.ts, which fences the database, drains connections, runs pg_restore, and verifies row counts against the manifest before reporting success — see the note below for exactly what still is not covered.

Source Evidence

Auth and Permissions

• Permission: Backup_RESTORE — superadmin-only.
• Additionally armed by BACKUP_RESTORE_ENABLED=true (deploy-time env; preflight refuses otherwise).
• Rate limit: ADMIN_RESTORE_SUBMIT — 2/day, user-keyed, matching AUTH_DAILY_IRREVERSIBLE.

Request

{
  "confirmDatabaseName": "skoolsewa_production",
  "acknowledgeDataLoss": true,
  "acknowledgeMongoNotRestored": true
}

Response

202, RestoreLiveStateDto:

{
  "restorePublicId": "018f...",
  "stage": "queued",
  "message": "Accepted. Waiting for the restore runner to start.",
  "startedAt": "2026-08-30T10:00:00.000Z",
  "heartbeatAt": "2026-08-30T10:00:00.000Z"
}

Side Effects

• Database: INSERT backup_restore (status: 'queued', maintenanceEngaged: true) + outbox enqueue, one transaction.
• Maintenance is engaged immediately (MaintenanceService.engageIf) — not deferred to a worker, because the outbox dispatcher polls every ~5s and any gap would let customers keep writing to a database about to be replaced. It is refused while the lock still owes a previous restore's reconciliation (see the 409 below).
• Filesystem: restore-state.json written with stage: "queued", after the lock is engaged.
• Audit: backup.restore.requested.
• Server log: warn-level "RESTORE ACCEPTED" line.

8.9a Known gap: verification is row-count only

A 202 from this endpoint means preflight passed, maintenance is engaged, backup.restore reached QueueName.BACKUP_RESTORE, and BackupRestoreProcessor genuinely: (1) takes an inline kind:'pre_restore_safety' backup, links it to the restore via backup_restore.safety_backup_run_id, and surfaces its id as RestoreLiveStateDto.safetyBackupPublicId — aborting the whole restore if the dump does not reach completed or the link cannot be recorded; (2) spawns workers/restore-runner.ts as a detached process, which fences the database, drains existing connections (up to the admin-configured restoreDrainSeconds) before terminating them, runs pg_restore --clean --single-transaction (up to the admin-configured restoreTimeoutMinutes), and — at the new verifying stage — compares every manifest-listed table's row count against the freshly-restored database (excluding RESTORE_VERIFICATION_EXCLUDED_TABLES: session tables, backup_run, backup_restore, outbox_events, drizzle_migrations — tables the restore and the safety dump themselves are expected to change), reporting BACKUP_RESTORE_VERIFICATION_FAILED specifically if any mismatch; and (3), on full success, disengages maintenance. The restore genuinely happens, is genuinely safeguarded by an automatic, linked, discoverable backup, and is genuinely checked afterward.

What still is not covered: verification is row-count only, not content-level — identical counts with different row contents, or --no-owner/--no-privileges permission side effects, would not be caught. BackupRestoreVerification's structured shape ({tablesChecked, tablesMatched, mismatches[], excluded[]}) is never written to backup_restore.verification, even though the comparison itself now happens — only pass/throw is observed. There is also still no one-click "undo": restoring the safety dump back requires an operator to submit it (now discoverable via safetyBackupPublicId) as the source of a second, ordinary restore request. See the backend doc's 16.8 Risk Register. If a runner dies or never spawns, force-clear marks the row failed and disengages maintenance without confirming whether pg_restore actually ran to completion.

Error Cases

Example Requests

curl -X POST "$API_URL/api/admin/system/backups/018f.../restore" \
  -H "Authorization: Bearer TOKEN" -H "Content-Type: application/json" \
  -d '{"confirmDatabaseName":"skoolsewa_production","acknowledgeDataLoss":true,"acknowledgeMongoNotRestored":true}'

8.10 GET /api/admin/system/restores/{publicId}/live

Purpose

The one endpoint that stays answerable while a restore is running — it reads only restore-state.json, never a database table, so it does not block behind pg_restore --clean's table locks or behind JwtStrategy reading admin_sessions. The admin UI is expected to poll this with backoff, to show progress through safety_dump → draining → restoring → verifying → completed/failed, and to treat a network failure as expected, not as "failed".

Source Evidence

Auth and Permissions

• Auth: @Public() to JwtAuthGuard, because the session lookup reads admin_sessions, which the restore locks. RestorePollTokenGuard requires the x-restore-poll-token header to match the token stored in restore-state.json.
• Permission: none. Rate limit: ADMIN_RESTORE_POLL.

Response

Same shape as 8.9's response.

Error Cases

8.11 POST /api/admin/system/restores/{publicId}/force-clear

Purpose

Clears a restore that will not finish — for a runner that was killed, a host that rebooted mid-operation, or an operator who wants out regardless of a still-running pg_restore. Without this, a stuck row blocks every future restore via uq_backup_restore_single_active, with no code path out other than hand-written SQL. It does not stop a genuinely running subprocess — see 8.9a.

Source Evidence

Auth and Permissions

• Permission: Backup_RESTORE; Rate limit: ADMIN_RESTORE_SUBMIT

Response

{ "cleared": true }

Side Effects

• UPDATE backup_restore SET status='failed', errorCode='BACKUP_RESTORE_ABANDONED', ...
• The traffic lock lifts (MaintenanceService.disengageIf) when restore-state.json shows this restore never reached pg_restore (stage queued, draining or safety_dump, or failed with BACKUP_RESTORE_FAILED), or when the lock's reconciliation is neither pending nor reconcile_failed. Otherwise it stays engaged with its marker, for the supervisor — or a restarted API — to settle; { "cleared": true } is answered either way. Read 8.12 to see which.
• Server log: warn-level.

Error Cases

There is no age threshold, so force-clearing a fresh restore whose runner has not yet written a pid is indistinguishable from force-clearing a stuck one. Because force-clear refuses a terminal row, it cannot lift a lock that a restore which already failed left engaged — that is what 8.13 is for.

8.12 GET /api/admin/system/restores/lock

Purpose

Reads the traffic lock a restore engages, so the Backups page can show that the API is refusing non-operator traffic, which restore engaged it, where its post-restore reconciliation stands, what blocks a release, and whether it can be released now. A restore engages the lock when it is accepted (reconciliation: "pending") and lifts it when it completes cleanly and has reconciled the restored database; a failed restore leaves it engaged on purpose, because a half-known database is not put back in front of users until an operator has looked.

Source Evidence

Auth and Permissions

• Guard chain: JwtAuthGuard, RoleGuard, IpThrottlerGuard
• Permission: Backup_READ — superadmin-only.
• Rate limit: ADMIN_READ
• Reachable while the lock holds: the path is under the /api/admin/system/restores/ prefix MaintenanceGuard exempts.

Request

No params, query, or body.

GET /api/admin/system/restores/lock HTTP/1.1
Authorization: Bearer TOKEN

Response

200, RestoreLockDto. Not engaged:

{
  "active": false,
  "restorePublicId": null,
  "reason": null,
  "engagedAt": null,
  "engagedBy": null,
  "reconciliation": null,
  "blockingRestorePublicId": null,
  "releasable": false
}

Left engaged by a restore that failed after pg_restore committed, and reconciled:

{
  "active": true,
  "restorePublicId": "0199a3c2-7f10-7c4e-9b1a-2d5e8f3a6b01",
  "reason": "Database restore in progress",
  "engagedAt": "2026-10-05T09:12:44.118Z",
  "engagedBy": "Super Admin",
  "reconciliation": "reconciled",
  "blockingRestorePublicId": null,
  "releasable": true
}

A restore that has not settled yet answers "reconciliation": "pending" with "releasable": false.

Side Effects

None — read-only. Reads backup_restore for a non-terminal row and restore-state.json for the runner pid (blockingRestorePublicId), then the lock through MaintenanceService.getState() (in-process 1s cache → Redis → maintenance.json).

Error Cases

8.13 POST /api/admin/system/restores/lock/release

Purpose

Releases a traffic lock that a finished or failed restore left engaged. This is the operator's way back after a failed restore: check the database, then release the lock from the Backups page. It refuses while any restore can still be running, because lifting the lock while pg_restore --clean holds every table puts users in front of a database mid-replacement; and while the restore that engaged the lock has not reconciled the restored database, because the restored rows hold every integration key revoked after the dump as active.

Source Evidence

Auth and Permissions

• Guard chain: JwtAuthGuard, RoleGuard, IpThrottlerGuard
• Permission: Backup_RESTORE — superadmin-only, the same permission that starts a restore.
• Rate limit: ADMIN_WRITE — 10/min, IP-keyed, the ordinary admin write budget. A release is refused until the restore settles, so an operator retries it; it does not share the restore's 2/day budget.
• Idempotent: releasing a lock that is not engaged answers 200 { "released": false } and writes nothing.
• Reachable while the lock holds: the path is under the /api/admin/system/restores/ prefix MaintenanceGuard exempts.

Request

No params, query, or body.

POST /api/admin/system/restores/lock/release HTTP/1.1
Authorization: Bearer TOKEN

Response

200, RestoreLockReleaseDto:

{ "released": true }

Side Effects

The checks run in this order, and the first that fails answers:

1. Reads maintenance.json directly, past every cache — the snapshot the write is compared against.
2. Any backup_restore row in queued, safety_dump, restoring or verifying → 409 BACKUP_RESTORE_ALREADY_ACTIVE.
3. restore-state.json names a runnerPid that is a live process, whichever restore it belongs to → 409 BACKUP_RESTORE_ALREADY_ACTIVE.
4. Inside MaintenanceService's critical section, re-reads maintenance.json:
   • not engaged → 200 { "released": false };
   • engaged by a different restorePublicId than step 1 read (a restore accepted meanwhile) → 409 BACKUP_RESTORE_ALREADY_ACTIVE;
   • reconciliation: "pending" → 409 BACKUP_RESTORE_RECONCILE_PENDING;
   • reconciliation: "reconcile_failed" → 409 BACKUP_RESTORE_RECONCILE_FAILED;
   • otherwise writes active: false to maintenance.json first, then mirrors it to Redis, and clears this process's in-process cache. An unreadable file, unreadable at both reads, is overwritten. Every API process stops refusing traffic once its own 1-second in-process cache expires.

• Audit: a Mongo audit entry with action backup.restore.lock_released, module system, resource backup_restore/the restorePublicId that engaged the lock, metadata restorePublicId and unreadableState — beside the ActivityAuditInterceptor entry for the request.
• Server log: warn-level [backup] restore lock released (<restorePublicId>).
• The backup_restore row is not touched.

Error Cases

Example 409 body:

{
  "statusCode": 409,
  "errorCode": "BACKUP_RESTORE_ALREADY_ACTIVE",
  "message": "A restore is still in progress (0199a3c2-7f10-7c4e-9b1a-2d5e8f3a6b01). Wait for it to finish, or force-clear it if its runner has died.",
  "details": { "blockingRestorePublicId": "0199a3c2-7f10-7c4e-9b1a-2d5e8f3a6b01" }
}

Example Requests

curl -X POST "$API_URL/api/admin/system/restores/lock/release" \
  -H "Authorization: Bearer TOKEN"

8.14 POST /api/admin/system/backups/upload

Purpose

Catalogues an operator-supplied backup archive (SE-4) — a pg_dump produced by this deployment, or by another one whose key this deployment generated (an imported key does not authenticate an upload; see §8.18) — so the existing, unmodified restore path can act on it. Used when a database needs to be seeded from a backup taken outside the normal scheduled/manual flow, or moved between deployments. Does not restore anything itself.

Source Evidence

Auth and Permissions

• Auth: JwtAuthGuard + RoleGuard
• Guard chain: JwtAuthGuard, RoleGuard, IpThrottlerGuard
• Permission: BackupUpload_CREATE — a separate permission module from Backup_*, deliberately: introducing a database from outside is not the same act as running a dump of this one.
• Guest support: none
• Rate limit: ADMIN_BACKUP_UPLOAD — 3/hour, user-keyed (matching ADMIN_BACKUP_DOWNLOAD's reasoning — both move a whole database dump over HTTP)
• Idempotency: none declared; a duplicate upload of the same archive inserts a second uploaded row, eventually rejected by BACKUP_UPLOAD_RETAINED_LIMIT_REACHED

Request

POST /api/admin/system/backups/upload HTTP/1.1
Authorization: Bearer TOKEN
Content-Type: multipart/form-data; boundary=----WebKitFormBoundary

Response

201, same shape as 8.1's list item, with kind: "uploaded", pinned: true, status: "completed" — immediately, since there is no async job.

Side Effects

• Filesystem: moves the streamed .partial dump into a real artifact directory; writes a fresh, server-generated manifest.json and artifact.sig (never the operator's uploaded manifest — see the backend doc's 6.9).
• Subprocess: pg_restore --list (format check) and pg_restore -f - --schema-only (renders SQL for the content scan) — read-only, no database connection.
• Database write: INSERT backup_run (kind:'uploaded').
• Audit: ActivityRecordService.recordActivity (action: "backup.upload"), fire-and-forget.

Error Cases

Edge Cases

• All three parts required — there is no partial upload.
• The operator's own manifest.json is discarded after two fields are read from it; a fresh one is always written, so re-downloading and re-uploading an already-catalogued row works (it is re-signed over its own new manifest).
• A truncated dump passes pg_restore --list (the format check only reads the table of contents at the front of the archive) — the sha256-vs-manifest comparison is what actually catches truncation.
• Uploaded rows are pinned by default and excluded from GFS retention entirely; the only way to remove one is DELETE /admin/system/backups/{publicId} (8.7).

Example Requests

curl -X POST "$API_URL/api/admin/system/backups/upload" \
  -H "Authorization: Bearer TOKEN" \
  -F "file=@database.dump" \
  -F "manifest=@manifest.json" \
  -F "signature=@artifact.sig"

8.15 While the traffic lock holds — what answers and what refuses

Not a route: the behaviour every consumer sees while a restore's traffic lock is engaged, enforced by MaintenanceGuard (shared/maintenance.guard.ts), which app.module.ts registers as an APP_GUARD, so it runs in front of every HTTP route in the application.

A request passes when any of these holds, checked in this order:

Everything else is refused with 503:

{
  "statusCode": 503,
  "errorCode": "SYSTEM_MAINTENANCE_ACTIVE",
  "message": "The store is briefly unavailable while we carry out maintenance. Please try again shortly."
}

Outside production, AllExceptionsFilter adds timestamp and path. That includes every other @Public() route and every signed-in route that carries no @Permissions(...) — the /api/mobile audience surfaces among them. The direction is deny by default: a route added later is refused unless it is named here.

An unreadable maintenance.json counts as engaged, so a corrupt lock file refuses traffic rather than letting it through; 8.13 overwrites it.

Consumer handling. Treat 503 SYSTEM_MAINTENANCE_ACTIVE as "a restore holds the API", not as an outage: show a holding message and retry with backoff. On the Backups page, read 8.12 to tell an operator the lock is still engaged and offer 8.13 when releasable is true.

8.16 GET /api/admin/system/backup-key

Never returns key material. There is no endpoint anywhere in this module that returns an existing key: material appears in exactly two responses, both at the moment it is created. A standing "download the key" route would be a permanently wider target that buys nothing, since the key is only needed when it is made.

{
  "keyPathConfigured": true,
  "storeState": "ready",
  "current": {
    "fingerprint": "1289db31",
    "provenance": "generated",
    "createdAt": "2026-09-01T08:30:59.185Z",
    "retiredAt": null
  },
  "retired": [
    {
      "fingerprint": "c035c8a9",
      "provenance": "generated",
      "createdAt": "2026-09-01T08:53:28.278Z",
      "retiredAt": "2026-09-01T08:56:46.128Z",
      "retainedBackupCount": 1
    }
  ],
  "unresolvedCount": 0
}

storeState is one of no_path_configured · legacy_env · no_key · ready · unreadable. unreadable is not no_key, and the distinction is load-bearing: "no key configured" makes the restore preflight SKIP signature verification, which is correct for a deployment that never had a key and catastrophic for one whose key file was truncated by a crash. A corrupt file therefore degrades to unreadable rather than failing the boot, and the preflight refuses on it.

retainedBackupCount counts completed backups on this host carrying that fingerprint. An archive kept off-box has no row here and cannot be counted — and that is exactly the population import exists for, so a 0 never means "nothing depends on this key".

unresolvedCount is why the prune refusal can be trusted. It counts completed runs whose signature_state is NULL or signed_unknown_key, and it is deliberately not filtered on local_artifact_present: a run reclaimed to a remote copy is invisible to retainedBackupCount (its fingerprint is NULL) and would be invisible here too, so the operator would read 0, 0, prune, and destroy the only key able to verify archives that still exist.

8.17 POST /api/admin/system/backup-key/generate

{ "fingerprint": "60bb212b", "key": "6d4e7c6f26f34ea3…142cd75f" }

This is the only time the value is returned. Nothing can retrieve it later. If the response is lost, rotate again.

No @IdempotentCreate on this route, or on rotate — deliberately. IdempotencyInterceptor persists responseJson to Redis for 24 hours, which would park a plaintext signing key in a store an operator may snapshot. generate is already idempotent by refusal: it 409s once a current key exists, so a retry cannot mint a second key.

8.18 POST /api/admin/system/backup-key/import

An imported key joins the retired set. It never signs and never becomes current.

provenance is load-bearing, not metadata. An imported key is a legitimate verification anchor on RESTORE, where backup_run.database_dump_sha256 is an independent server-computed value the backup directory cannot forge. It is not an anchor on UPLOAD: there the caller supplies the archive, the signature and — via this endpoint — the key that validates it. Three operands from one actor prove nothing, so BackupSignatureService filters verification candidates on this field. It lives in the key file's format rather than in a branch somebody can forget to write.

8.19 POST /api/admin/system/backup-key/rotate

confirmFingerprint must equal the fingerprint of the key being retired — 8 lowercase hex characters. It stops a rotation aimed at a key the operator was not looking at.

The retired key is kept, and that is the entire point. A retired key is not a dead key: it is what still verifies every artifact signed before the rotation, which is the difference between "rotate the signing key" and "make every existing backup unrestorable". Rotating and then restoring a pre-rotation backup is verified end to end.

8.20 POST /api/admin/system/backup-key/resolve-signatures

Batched: call again while remaining is above zero.

It reveals no key material and mutates no key — it recomputes derived state on backup_run by reading each artifact's artifact.sig — so it takes ADMIN_HEAVY_OP rather than the step-up budget, whose shape assumes every attempt is a password guess. It requires no password for the same reason.

It re-examines runs previously resolved as signed_unknown_key, because importing a key changes that answer, and that is precisely what makes the retired-key prune count trustworthy.

remaining reports work this host can do — rows whose artifacts are readable here. That is deliberately a different set from unresolvedCount in §8.16, which is "reasons not to prune" and includes rows this host can never resolve.

8.21 DELETE /api/admin/system/backup-key/retired/{fingerprint}

Irreversible. The only way back is to import the same key material again — if the operator still has it.

Two independent refusals, and force overrides both:

The server-side confirmFingerprint check is not evidence of intent. It is compared against the path segment, and both come from the same client, so it catches a malformed request and nothing more. It is not the analogue of the restore dialog's confirmDatabaseName, which is checked against SELECT current_database() — a fact only the server holds and a client that has not been told the answer cannot satisfy. There is no server-side fact about a key fingerprint that the client does not already supply, so for an accidental prune the admin panel's typed-confirmation dialog is the control, and calling it a speed bump in front of one would misdescribe what protects an irreversible delete.

8.23 GET /api/admin/system/backups/{publicId}/download/uploads

Why this is a second route rather than a bigger tar. §8.8 hands out exactly database.dump, manifest.json and artifact.sig, because those three are what §8.14's upload endpoint consumes. Adding uploads to that response would break that round trip and turn every routine dump download into a multi-gigabyte transfer. The operator picks.

Before this route existed, an operator who ticked "Also archive uploaded files" got a download containing none of them, and nothing said so.

Two source formats, one response shape. row.uploadsFormat === "tar_gz" (older runs) streams the run's pre-built uploads.tar.gz straight from disk with a server-computed sha256 and a size check against the recorded byte count. row.uploadsFormat === "snapshot" (current runs) has no single pre-built archive to stream — the run stored a hard-linked file tree instead — so the handler tars uploads/, uploads.index.json, manifest.json and artifact.sig (when present) fresh, on the fly, from the artifact directory. Both paths answer the same headers and the same backup-<publicId>-uploads.tar.gz filename, so a caller never needs to know which format a given run used.

The same constant does not mean a shared budget. IpThrottlerGuard keys the bucket as `throttle:ip:${ControllerClass}:${handlerName}:${principal}` — the RATE_LIMITS constant supplies only limit, windowSeconds and keyStrategy, and contributes nothing to the key. So each handler has its own 3/hour and the two downloads never compete. Reusing the constant is therefore a naming decision, not a capacity one; per-operator capacity is 3+3 either way.

Refusals consume the budget. The guard runs before the handler, so a 409 or 410 costs a slot exactly like a delivered file does.

Refusals

BACKUP_UPLOADS_ARCHIVE_NOT_INCLUDED is 409, not 404: the backup exists and its database half downloads fine. A 404 would send the operator hunting a wrong id instead of a missing option chosen at backup time.

No server-side hash, and why that is not a weakening

§8.8 computes the dump's sha256 before streaming it. This route does not, and the reason is specific rather than general:

• The dump's hash is justified by RESTORE — "a silently corrupted dump is worse than a missing one, because it restores and produces a wrong database rather than an error." Nothing restores uploads.tar.gz; the only consumer is the operator.
• Hashing reads the whole file before the first response header, and the admin BFF abandons a download whose headers have not arrived in time. On the artifact that is large by design, that turns every download into a timeout — and since refusals consume the rate budget, a few deterministic retries lock the operator out during the incident the feature exists for.

Integrity moves to where it can be acted on: uploadsArchiveSha256 is on the backup detail response and rendered in the panel, so the operator verifies the file they actually received —

sha256sum backup-<publicId>-uploads.tar.gz

Content-Length is set (unlike §8.8, whose tar is built on the fly), so a truncated transfer is detectable without any hashing.

The open happens before any header is set

A failure opening the archive is an ordinary thrown exception rendered as a normal envelope. That ordering is load-bearing and was arrived at by measurement:

This matters because the failure is reachable: deleteHeavyArtifacts removes uploads.tar.gz before it flips local_artifact_present, so a completed, locally-present run can lose its file at any moment, and the window between the size check and the open spans the audit write.

Audit

backup.download_uploads — distinct from §8.8's backup.download. After an incident the question is which artifact left this host, and customer files leaving is a different disclosure from a database dump leaving. One shared action string cannot answer it.

Like §8.8, the record is written before the first byte and the download is refused if it fails. Note the corollary: the row says success before delivery is attempted, so an aborted transfer leaves an optimistic record.

8.22 Forwarding the operator's address

Not an endpoint — a header pair every admin route accepts, added because the admin panel is a BFF and so every admin action landed in the activity log as 127.0.0.1: literally accurate, and useless for a log whose purpose is "who did this, from where".

Both halves are required and the address is discarded outright unless the token matches. Trust is bound to a shared secret rather than to Express's trust proxy because a numeric hop count does not bind to the peer at all — measured against Express 5.2.1, with the socket peer on loopback and a single-entry x-forwarded-for, req.ip becomes whatever the caller sent. The token authenticates the caller, which is the thing a hop count cannot express.

Verified by observation, three ways: a forged address with a wrong token records 127.0.0.1; a forged address with no token records 127.0.0.1; the address with the correct token is recorded.

Unset is a no-op that preserves today's behaviour exactly, which is what lets this ship without a coordinated deploy. The forwarded address reaches the activity log and nothing else — in particular it is not wired into IpThrottlerGuard, because doing so would hand an unauthenticated attacker a rate-limit bypass on admin credential stuffing.

8.24 GET /api/admin/system/backups/failures

Dead letters from QueueName.BACKUP and QueueName.BACKUP_REPLICATE — backup.run, backup.prune and backup.replicate jobs that exhausted their BullMQ attempts — newest first. A failed backup.run is listed here for the record (retry: "none") but cannot be retried through either route below; the operator starts a new backup instead. Declared BEFORE GET :publicId in the controller so route matching does not take the literal "failures" as a publicId and answer 400.

8.25 POST /api/admin/system/backups/failures/{publicId}/retry-prune

Replays the failed backup.prune once, under a fresh job id. The worker re-checks the run against the current retention settings and floors before deleting anything — a stale approval from before the original attempt is never trusted, since settings or the retention floors may have changed since.

Refusals

8.26 POST /api/admin/system/backups/failures/{publicId}/retry-replication

Replays the failed backup.replicate once, under a fresh job id — an ordinary re-run of the same off-host copy BackupReplicationHandler performs after any run completion.

Refusals

Same three codes as §8.25, with BACKUP_FAILURE_NOT_RETRYABLE firing when the named failure is not a backup.replicate job.

9. Flow Diagrams

9.1 Route Ownership

9.2 Request Sequence — manual backup

9.3 Error Branch — restore request

10. Pagination, Sorting, Filtering, and Search

sort is accepted by the inherited QueryDto but not read by BackupAdminService.findAll — only order (asc/desc on createdAt) is applied. search is likewise accepted but not used. pagination=false returns every matching row with meta omitted from the response envelope.

No caching, no relevance scoring — this is a straightforward filtered/ordered/offset-paginated list over backup_run.

11. Caching, Jobs, and External Integrations

13. Mandatory Deep API Documentation Pack

13.1 Route-by-Route Completeness Matrix

13.2 Request/Response Exhaustiveness

Covered per-endpoint in Section 8 — minimal and full request bodies, success responses, and every declared error code are shown for each mutating route. The public/guest row of the standard checklist applies to no route: every route requires superadmin-level permissions, except the live restore read, which requires the restore's poll token.

13.3 API Diagram Pack

See Section 9 for route ownership, request sequence, and error-branch diagrams. A dedicated auth/permission-flow diagram is omitted as redundant: every route in this module except GET .../restores/{publicId}/live uses the identical three-guard chain (JwtAuthGuard, RoleGuard, IpThrottlerGuard) with only the @Permissions(...) argument varying, fully enumerated in Section 4.

13.4 Consumer Integration Notes

13.5 API Tradeoffs and Rationale

13.6 API Change Impact

14. Zero-Omission API Checklist

• Every controller route is documented (24 route templates across 5 controllers, cross-checked against structure.baseline.json).
• Every parent route prefix and runtime URL is documented.
• Every DTO field, enum, default, and validator is documented.
• Every response field, nullable field, and server-computed field is documented.
• Every auth, guard, and permission is documented.
• Every success and failure branch is documented, including what the restore path still does not do (safety dump, verification).
• Every database read/write, queue job, and external call is documented.
• Every mutating route has example requests and responses.
• Route-ownership, sequence, and error-branch diagrams are included.
• Every tradeoff and compatibility risk is documented, including the remaining restore-safety gap.
• Links to backend and features/flows docs are present.

15. Integration Checklist

• Every route from controllers is documented.
• Every DTO field is documented.
• Every enum value is documented.
• Every response envelope is documented.
• Every error code is documented.
• Every auth guard and permission is documented.
• Every queue job and external call is documented.
• Every diagram matches the current code, including the restore queue's real consumer.
• The API doc links to backend and features/flows docs.

See Also

• Backend doc: /docs/developer/backup/backend
• Features and flows doc: /docs/developer/backup/feature
• TDD: not present for this module at time of writing.