Backup API Reference
Complete API contracts for backup, including routes, auth, DTOs, responses, errors, examples, and integration notes.
Backup - API Reference
Audience: Admin panel engineers, QA, storefront engineers (for the one public route), and API consumers with superadmin-level access.
Scope: Almost entirely admin-only. Backup catalog, backup download, upload/catalogue, restore request/live-state, artifact signing-key custody, and site-wide maintenance — all under admin/system/*. The single exception is GET /api/system/maintenance, a @Public() route with no permission and no admin auth, which exists so the storefront can discover a maintenance window even when its own caches would otherwise hide it.
1. Documentation Evidence
| Area | Files Inspected | What Was Verified |
|---|---|---|
| Controllers | apps/api/src/modules/backup/admin/catalog/backup-admin.controller.ts, admin/download/backup-download-admin.controller.ts, admin/restore/backup-restore-admin.controller.ts, admin/maintenance/maintenance-admin.controller.ts, admin/upload/backup-upload-admin.controller.ts | Routes, methods, guards, decorators, status codes. |
| DTOs | admin/catalog/dto/*.ts, admin/restore/dto/restore.dto.ts, admin/maintenance/dto/maintenance.dto.ts, admin/download/dto/backup-download-params.dto.ts, admin/upload/dto/backup-upload.dto.ts | Request, query, response, validation. |
| Services | admin/catalog/backup-admin.service.ts, admin/download/backup-download-admin.service.ts, admin/restore/backup-restore-admin.service.ts, admin/maintenance/maintenance-admin.service.ts, admin/upload/backup-upload-admin.service.ts, shared/*.ts | Behavior, side effects, response mapping, errors. |
| Schema | packages/db/src/schema/backup/*.ts | IDs, enums, persisted fields, constraints. |
| Restore executor | workers/backup-restore.processor.ts, workers/restore-runner.ts | The queue consumer and the detached process it spawns — restore genuinely executes, including a pre-restore safety dump and post-restore row-count verification, not just acceptance. |
| Public surface | customer/maintenance-customer.controller.ts, customer/maintenance-customer.service.ts, customer/dto/maintenance-status.dto.ts, shared/maintenance.guard.ts (MAINTENANCE_EXEMPT_PATHS) | The one @Public() route in the module, its narrower response DTO, and its exemption from MaintenanceGuard. |
| Route ground truth | apps/api/test/structure/structure.baseline.json | Grepped for system/backups, system/restores, system/maintenance — 15 distinct route templates confirmed: 14 under admin/system/* (including POST admin/system/backups/upload) plus the public GET /api/system/maintenance; none invented. |
| Rate limits | apps/api/src/common/guards/ip-throttler.config.ts | ADMIN_READ, ADMIN_WRITE, ADMIN_ASYNC_JOB_SUBMIT, ADMIN_BACKUP_DOWNLOAD, ADMIN_BACKUP_UPLOAD, ADMIN_RESTORE_SUBMIT, PUBLIC_HIGH_FREQUENCY. |
| Permissions | packages/db/src/authorization/permission-catalog.ts | Backup, BackupDownload, BackupConfigure, BackupUpload, System — all superadmin-only except System. |
| Env | apps/api/src/config/env.validation.ts | Every BACKUP_*/PG_*/TAR_PATH variable, and the admin-editable vs. environment-only split, including BACKUP_ARTIFACT_HMAC_KEY_PATH and the deprecated-but-still-declared BACKUP_ARTIFACT_HMAC_KEY. |
| Error registry | apps/api/src/common/types/error-codes.ts | Every BACKUP_* and SYSTEM_MAINTENANCE_ACTIVE code, including the BACKUP_UPLOAD_* group. |
| Key custody | admin/key/backup-key-admin.controller.ts, admin/key/backup-key-admin.service.ts, admin/key/dto/*.ts, shared/backup-key-store.service.ts | The six admin/system/backup-key routes, their DTOs, the on-disk key file format, and the reveal-once contract. |
| Signature state | packages/db/src/schema/backup/enums.ts, packages/db/src/schema/backup/backup-run.ts, packages/db/src/migrations/0053_backup_signature_state.sql | The backup_signature_state enum, the two new columns, six CHECK constraints and one partial index. |
| Trusted client IP | apps/api/src/common/interceptors/trusted-client-ip.interceptor.ts | The x-internal-client-ip / x-internal-client-ip-token pair, and that the address is discarded unless the token matches. |
| Route reachability | apps/api/test/structure/route-shadowing.spec.ts | That no literal route is shadowed by a parameter route from another controller — the defect that made GET admin/system/backups/key answer 400 publicId must be a UUID and moved these routes to their own prefix. |
2. Module Summary
| Field | Value |
|---|---|
| Module name | backup |
| Module slug | backup |
| Primary actors | admin (regular), superadmin — every admin route in this module requires superadmin in practice, since Backup, BackupDownload, BackupConfigure and BackupUpload are withheld from the admin role; MaintenanceAdminController uses System_*, also superadmin-only. guest/customer (unauthenticated) is the actor for the one public route. |
| API surfaces | admin (14 routes), public (1 route: GET /api/system/maintenance) |
| Base route prefixes | /api/admin/system/backups, /api/admin/system/restores, /api/admin/system/maintenance, /api/system/maintenance (public) |
| Auth model | Admin routes: JwtAuthGuard + RoleGuard + IpThrottlerGuard, @Permissions("Module_ACTION") per route. The public route: IpThrottlerGuard only, @Public(), no permission. |
| Persistence | PostgreSQL (backup_run, backup_restore), local filesystem (backup root: manifest.json, settings.json, restore-state.json, maintenance.json), Redis (maintenance flag mirror), MongoDB (AuditLog, download only), BullMQ (3 queues) |
| Runtime source of truth | backup_run/backup_restore tables for the catalog; the backup-root files for anything a restore itself would destroy — see the backend doc's Source of Truth table |
| Sibling docs | Backend, Features and flows |
3. Concepts and Terminology
| Term | Meaning | Source File | Used By |
|---|---|---|---|
| Backup run | One dump attempt — the unit tracked by backup_run | backup-run.ts | All catalog/download endpoints |
| Kind | Why a backup exists: scheduled, manual, pre_restore_safety, uploaded | enums.ts | FetchBackupsDto, BackupResponseDto |
| Uploaded | An operator-supplied archive catalogued via POST .../backups/upload (SE-4), rather than dumped by this deployment | enums.ts, backup-upload-admin.service.ts | FetchBackupsDto, BackupResponseDto, BackupUploadDto |
| Tier | The GFS bucket a scheduled run belongs to: daily, weekly, monthly | enums.ts | Same |
| Pinned | Operator-set protection from count-based retention pruning | backup-run.ts | UpdateBackupDto |
| Includes uploads | Whether the run also archived the local upload root | backup-run.ts | CreateBackupDto, BackupResponseDto |
| Storage mode | Where a run's artifacts are kept: local, both, remote_only | enums.ts | BackupSettingsResponseDto |
| Replication status | Whether a run's artifacts reached the configured remote host: not_requested, pending, replicated, failed | enums.ts | Settings health fields |
| Restore | An in-progress or completed request to replace the live database from a backup — tracked by backup_restore and, while in flight, by restore-state.json | backup-restore.ts | Restore endpoints |
| Live restore stage | The value read from restore-state.json — accepted|queued|draining|safety_dump|restoring|verifying|reconciling|completed|failed. In practice the runner only ever assigns draining, restoring, completed and failed — safety_dump, verifying and reconciling are declared but never reached, since no safety dump or post-restore verification is implemented. | backup-state-file.service.ts, restore.dto.ts, workers/restore-runner.ts | GET .../live |
| Artifact signature | Keyed HMAC-SHA256(key, "<dumpSha256>:<manifestSha256>"), the key coming from the key file at BACKUP_ARTIFACT_HMAC_KEY_PATH — the control an uploaded archive is authenticated by, and the control a restore checks | backup-signature.service.ts, backup-key-store.service.ts | POST .../backups/upload, POST .../backups/{publicId}/restore |
| Retired key | A key that no longer signs but still verifies. Kept so a rotation does not make every existing backup unrestorable | backup-key-store.service.ts | POST .../backup-key/rotate, POST .../backup-key/import |
| Key fingerprint | The first 8 hex characters of sha256("hs-backup-artifact-key-fp-v1\n" + key). The only thing about a key that ever appears in a response after creation | backup-key-store.service.ts | GET .../backup-key |
| Maintenance | The site-wide flag that refuses customer traffic | maintenance.service.ts | Maintenance endpoints, MaintenanceGuard |
| Version (settings) | Optimistic-concurrency token for settings.json | backup-settings.service.ts | BackupSettingsResponseDto, UpdateBackupSettingsDto |
4. API Surface Map
Every route below is confirmed against structure.baseline.json.
| Surface | Method | Path | Actor | Auth/Guard | Permission | Controller | Purpose |
|---|---|---|---|---|---|---|---|
| Admin | GET | /api/admin/system/backups | Superadmin | JWT+Role+IP | Backup_READ | BackupAdminController | Paginated list of backup runs. |
| Admin | POST | /api/admin/system/backups | Superadmin | JWT+Role+IP | Backup_CREATE | BackupAdminController | Run a manual backup now. |
| Admin | GET | /api/admin/system/backups/settings | Superadmin | JWT+Role+IP | Backup_READ | BackupAdminController | Read effective backup/retention config. Registered before :publicId. |
| Admin | PUT | /api/admin/system/backups/settings | Superadmin | JWT+Role+IP | BackupConfigure_UPDATE | BackupAdminController | Update backup/retention config. |
| Admin | GET | /api/admin/system/backups/{publicId} | Superadmin | JWT+Role+IP | Backup_READ | BackupAdminController | Backup detail. |
| Admin | PATCH | /api/admin/system/backups/{publicId} | Superadmin | JWT+Role+IP | Backup_UPDATE | BackupAdminController | Pin/unpin a backup. |
| Admin | DELETE | /api/admin/system/backups/{publicId} | Superadmin | JWT+Role+IP | Backup_DELETE | BackupAdminController | Prune a backup now (async, via outbox). |
| Admin | GET | /api/admin/system/backups/{publicId}/download | Superadmin | JWT+Role+IP | BackupDownload_READ | BackupDownloadAdminController | Stream the raw pg_dump artifact. |
| Admin | GET | /api/admin/system/backups/{publicId}/download/uploads | Superadmin | JWT+Role+IP | BackupDownload_READ | BackupDownloadAdminController | Stream uploads.tar.gz alone. Separate from the dump download on purpose — see §8.23. |
| Admin | POST | /api/admin/system/backups/upload | Superadmin | JWT+Role+IP | BackupUpload_CREATE | BackupUploadAdminController | Catalogue an operator-supplied, HMAC-verified archive (SE-4). |
| Admin | POST | /api/admin/system/backups/{publicId}/restore | Superadmin | JWT+Role+IP | Backup_RESTORE | BackupRestoreAdminController | Accept a restore request (202) — genuinely executes via BackupRestoreProcessor/restore-runner.ts, including an inline pre-restore safety dump and post-restore row-count verification. |
| Admin | GET | /api/admin/system/restores/{publicId}/live | Superadmin | JWT+Role+IP | Backup_READ | BackupRestoreAdminController | Poll live restore stage from the state file. |
| Admin | POST | /api/admin/system/restores/{publicId}/force-clear | Superadmin | JWT+Role+IP | Backup_RESTORE | BackupRestoreAdminController | Clear a restore that will not finish. |
| Admin | GET | /api/admin/system/maintenance | Admin (System) | JWT+Role+IP | System_READ | MaintenanceAdminController | Read maintenance state. |
| Admin | PUT | /api/admin/system/maintenance | Admin (System) | JWT+Role+IP | System_UPDATE | MaintenanceAdminController | Engage/disengage maintenance. |
| Admin | GET | /api/admin/system/backup-key | Superadmin | JWT+Role+IP | BackupKey_READ | BackupKeyAdminController | Fingerprints of the keys this deployment holds. Never key material. |
| Admin | POST | /api/admin/system/backup-key/generate | Superadmin | JWT+Role+IP | BackupKey_CREATE | BackupKeyAdminController | Create the signing key and return it once. 409 if one already exists. |
| Admin | POST | /api/admin/system/backup-key/import | Superadmin | JWT+Role+IP | BackupKey_CREATE | BackupKeyAdminController | Add another deployment's key to the RETIRED set. Verifies a restore, never an upload. |
| Admin | POST | /api/admin/system/backup-key/rotate | Superadmin | JWT+Role+IP | BackupKey_UPDATE | BackupKeyAdminController | Retire the current key, create a new one, return it once. |
| Admin | POST | /api/admin/system/backup-key/resolve-signatures | Superadmin | JWT+Role+IP | BackupKey_UPDATE | BackupKeyAdminController | Batched: resolve signature_state for runs that have none. |
| Admin | DELETE | /api/admin/system/backup-key/retired/{fingerprint} | Superadmin | JWT+Role+IP | BackupKey_DELETE | BackupKeyAdminController | Permanently remove a retired key. Refuses while anything may still need it. |
| Public | GET | /api/system/maintenance | Guest/customer (unauthenticated) | IpThrottlerGuard only, @Public() | N/A | MaintenanceCustomerController | Whether the store is currently refusing traffic — the storefront's uncached way to discover a maintenance window even when its own page cache would otherwise hide it. |
Why the key routes are backup-key, not backups/key
They were originally admin/system/backups/key*, on the same prefix as
BackupAdminController's @Get(":publicId"). Express matches in registration order and
module-import order decided the winner, so GET /api/admin/system/backups/key was matched by
:publicId and answered 400 publicId must be a UUID — every read on the key custody page was
dead.
Nothing caught it. Unit tests call the controller method directly and never build a route table;
structure.baseline.json records that a route is REGISTERED, which it was; and an unauthenticated
probe gets 401 from JwtAuthGuard before the parameter pipe runs, so the collision is invisible
from outside a session. It was found by opening the page in a browser, and
apps/api/test/structure/route-shadowing.spec.ts now fails the build on the whole class.
21 distinct route templates, matching the unique entries found in structure.baseline.json for these prefixes (the file lists each twice — once per Swagger document it appears in): 10 under admin/system/backups (including POST .../upload), 2 under admin/system/restores, 2 under admin/system/maintenance, and 1 under the public system/maintenance.
5. Auth, Identity, and Permissions
| Surface | Guard/Decorator | Identity Shape | Permission | Guest Allowed | Notes |
|---|---|---|---|---|---|
| Backup catalog/download/restore/upload | JwtAuthGuard, RoleGuard, IpThrottlerGuard | req.user.id (admin) | Backup_*, BackupDownload_READ, BackupConfigure_UPDATE, BackupUpload_CREATE — all in SUPERADMIN_ONLY_MODULES | No | The admin role never receives these permissions; only superadmin does, via buildAdminPermissionCatalog's exclusion list. |
| Maintenance | JwtAuthGuard, RoleGuard, IpThrottlerGuard | req.user.id (admin) | System_READ/System_UPDATE — also superadmin-only | No | Gated on System_* rather than a Settings_* module deliberately, so no ordinary content administrator can take the storefront offline or disengage maintenance mid-restore. |
| Restore | Two independent gates | — | Backup_RESTORE (grantable, superadmin-only) and BACKUP_RESTORE_ENABLED (deploy-time env, never grantable) | No | A permission can be mis-granted; an environment variable cannot be granted from inside the app. |
| Signing-key custody | JwtAuthGuard, RoleGuard, IpThrottlerGuard, plus a password step-up in the service | req.user.id (admin) | BackupKey_READ/CREATE/UPDATE/DELETE — its own superadmin-only module | No | Split from BackupConfigure deliberately: that permission changes dump frequency, this one replaces the credential every artifact is authenticated against and hands the caller plaintext key material. Every mutating route re-checks the operator's password, because a session token proves the session and not the person. The step-up limiter is keyed by USER, never by IP. |
| Public maintenance status | IpThrottlerGuard only | None — no req.user | N/A, @Public() | Yes | The only guest/@Public()-reachable route in this module. Response is deliberately narrower than the admin DTO (active/reason only — never engagedBy/engagedAt). |
One endpoint in this module — GET /api/system/maintenance — supports guest access via @Public(). Nothing in the module has a dedicated mobile surface.
6. DTO and Model Reference
6.1 CreateBackupDto (request body — POST /admin/system/backups)
| Field | Type | Required | Default | Validation | Example | Source |
|---|---|---|---|---|---|---|
includeUploads | boolean | No | Server default (BACKUP_INCLUDE_UPLOADS, ships false) | @IsOptional @IsBoolean | true | create-backup.dto.ts |
note | string | No | — | @IsOptional @IsString @MaxLength(200) | "pre-migration" | same |
An explicit false for includeUploads is honoured — only absence falls back to the configured default. Under a remote (non-local) storage driver, the effective value is forced to false regardless of what was requested; the run records includesUploads: false.
6.2 FetchBackupsDto (query — GET /admin/system/backups)
Extends the shared QueryDto (pagination, page, size, sort, order, search — search is accepted by the base class but not applied by BackupAdminService.findAll, which filters only on the three fields below).
| Field | Type | Required | Default | Validation | Example | Source |
|---|---|---|---|---|---|---|
kind | enum | No | — | @IsIn(["scheduled","manual","pre_restore_safety"]) | "manual" | fetch-backups.dto.ts |
status | enum | No | — | @IsIn(["queued","running","completed","failed","pruned"]) | "completed" | same |
tier | enum | No | — | @IsIn(["daily","weekly","monthly"]) | "daily" | same |
page (inherited) | number | No | 1 | @IsInt @Min(1) | 1 | query.dto.ts |
size (inherited) | number | No | 20 | @IsInt @Min(1) @Max(100) | 20 | same |
order (inherited) | "asc"|"desc" | No | "desc" | @IsEnum | "desc" | same — sorts by createdAt |
sort (inherited) | string | No | "updatedAt" | accepted but not applied (service always orders by createdAt) | — | same |
6.3 UpdateBackupDto (request body — PATCH /admin/system/backups/{publicId})
| Field | Type | Required | Validation | Example |
|---|---|---|---|---|
pinned | boolean | Yes | @IsBoolean | true |
6.4 BackupParamsDto / BackupDownloadParamsDto / RestoreParamsDto
| Field | Type | Required | Validation | Example |
|---|---|---|---|---|
publicId | string (UUID) | Yes | @IsUUID("7") | 018f4e2a-7b3c-7c1e-9b2a-3d4e5f6a7b8c |
Identical shape, three separate classes (one per controller's route params).
6.5 BackupResponseDto
| Field | Type | Nullable | Notes |
|---|---|---|---|
publicId | string | No | Only identifier ever exposed. |
kind | enum | No | scheduled|manual|pre_restore_safety |
tier | enum | Yes | daily|weekly|monthly, or null |
status | enum | No | queued|running|completed|failed|pruned |
pinned | boolean | No | — |
includesUploads | boolean | No | — |
databaseDumpBytes / uploadsArchiveBytes | number | Yes | — |
totalBytes | number | No | Server-computed sum of the two above (each defaulting to 0). |
postgresVersion | string | Yes | — |
schemaMigrationTag | string | Yes | — |
tableCount | number | Yes | Server-computed: Object.keys(manifest.tables).length, or null if no manifest. |
totalRows | number | Yes | Server-computed: sum of every table's row count in the manifest. |
requestedByEmail | string | Yes | null for scheduled. |
note | string | Yes | ≤200 chars. |
startedAt / finishedAt | Date | Yes | — |
durationMs | number | Yes | Server-computed: finishedAt - startedAt, or null if either is missing. |
errorCode / errorMessage | string | Yes | errorMessage is always the sanitised excerpt, never raw subprocess output. |
isRestorable | boolean | No | Server-computed: status === "completed". Does not account for whether local artifacts are still present — see 12.5 in the features doc. |
createdAt | Date | No | — |
Deliberately absent, by design (matching RestoreLiveStateDto's frozen contract): the integer PK, artifactDir (a filesystem path), and either checksum.
6.6 BackupSettingsResponseDto / UpdateBackupSettingsDto
| Field | Type | Editable? | Validation (on PUT) | Notes |
|---|---|---|---|---|
version | number | Echoed back, required on PUT | @IsInt @Min(0) | Optimistic concurrency; mismatch is 409 BACKUP_SETTINGS_CONFLICT. |
backupEnabled | boolean | Admin | @IsBoolean | — |
scheduleCron | string | Admin | @IsString, service re-validates as a 5-field cron expression | 6-field (seconds) rejected explicitly. |
scheduleTimezone | string | Admin | @IsString, service re-validates as a recognised IANA zone | — |
includeUploadsByDefault | boolean | Admin | @IsBoolean | — |
retainDaily/Weekly/Monthly/Manual/Safety | number | Admin | @IsInt @Min(1) @Max(365) each | — |
minFreeDiskMb | number | Admin | @IsInt @Min(256) @Max(1048576) | — |
dumpTimeoutMinutes / restoreTimeoutMinutes / restoreDrainSeconds / restoreAbandonSeconds | number | Admin | @IsInt @Min(1) each | dumpTimeoutMinutes is read fresh per dump, not cached — a saved change takes effect on the next dump. restoreTimeoutMinutes and restoreDrainSeconds are read once per restore and injected into the detached runner's own process environment by BackupRestoreProcessor.spawnRunner (the runner is a standalone process with no DI, so this is what makes them editable at all); restoreAbandonSeconds is passed directly to the in-process supervisor. All three are genuinely effective on the next restore submitted after a save. |
storageMode | enum | Admin | @IsIn(["local","both","remote_only"]) + cross-check | Rejected with BACKUP_REMOTE_NOT_CONFIGURED unless a remote target is configured and remoteReplicationEnabled is true. |
remoteReplicationEnabled | boolean | Admin | @IsBoolean | The runtime toggle only — never the destination. |
restoreEnabled | boolean | Environment-only (BACKUP_RESTORE_ENABLED) | — | Read-only in the response; not accepted by UpdateBackupSettingsDto at all. |
restoreAdminDatabaseConfigured | boolean | Environment-only (presence of BACKUP_ADMIN_DATABASE_URL) | — | Never the connection string itself. |
pgDumpPath / pgRestorePath | string | Environment-only (PG_DUMP_PATH, PG_RESTORE_PATH) | — | — |
remoteConfigured | boolean | Environment-only (derived from BACKUP_REMOTE_ENABLED + BACKUP_REMOTE_TARGET presence) | — | Never the target string, which carries a username and a path. |
lastReplicationFailureAt | Date | Server-computed (read-only) | — | Most recent updated_at among rows currently replication_status='failed'. |
replicationFailureStreak | number | Server-computed (read-only) | — | Consecutive most-recent replication attempts ending in failed, reset by any replicated. |
Why the environment-only fields can never be form fields: pgDumpPath/pgRestorePath/TAR_PATH (not shown in the response, but the same class) are absolute binary paths passed directly to child_process.spawn — an admin-editable path is arbitrary code execution on every scheduled backup. BACKUP_DIR (not in the response at all) is validated at boot against every statically-served root; an admin-editable value could not receive that boot-time safety check and would risk publishing every database dump over HTTP with no guard able to intervene (ServeStaticModule sits outside the guard pipeline). The remote replication destination and both replication credentials (BACKUP_REMOTE_TARGET, BACKUP_REMOTE_SSH_KEY_PATH) are a lower bar than BackupDownload_READ itself — an admin who could edit the destination could redirect every future backup to a host of their choosing, and one who could edit the key path could substitute their own key. BACKUP_RESTORE_ENABLED is a deploy-time kill switch specifically because an admin-editable kill switch is not a kill switch — its entire value is that arming the single most destructive endpoint in the product requires shell access to the box.
6.7 RestoreBackupDto (request body — POST .../restore)
| Field | Type | Required | Validation | Notes |
|---|---|---|---|---|
confirmDatabaseName | string | Yes | @IsString @MaxLength(128) | Compared server-side against current_database() — the dialog is a speed bump, this is the control. |
acknowledgeDataLoss | true | Yes | @IsBoolean @Equals(true) | Must be literally true. |
acknowledgeMongoNotRestored | true | Yes | @IsBoolean @Equals(true) | Must be literally true. MongoDB is never restored by this feature. |
6.8 RestoreLiveStateDto (response — POST .../restore, GET .../live)
| Field | Type | Nullable | Notes |
|---|---|---|---|
restorePublicId | string | No | — |
stage | enum | No | accepted|queued|draining|safety_dump|restoring|verifying|reconciling|completed|failed. queued is written by the request handler; safety_dump is written by BackupRestoreProcessor.takeSafetyDump while the inline pre-restore dump runs; draining, restoring, verifying, completed and failed are written by restore-runner.ts as it fences, runs pg_restore, and checks row counts against the manifest. Only reconciling remains declared but never assigned by any current code path. |
message | string | Yes (optional) | — |
errorCode | string | Yes (optional) | — |
startedAt / heartbeatAt | string (ISO) | No | heartbeatAt is written by restore-runner.ts every 5 seconds while it runs; a gap past BACKUP_RESTORE_ABANDON_SECONDS means the runner died, not that the restore failed, per the controller's own doc comment — BackupRestoreProcessor treats that gap as BACKUP_RESTORE_ABANDONED. |
finishedAt | string | Yes (optional) | — |
safetyBackupPublicId | string | Yes (optional) | Written by BackupRestoreProcessor.takeSafetyDump into restore-state.json before the safety dump even runs (so a crash mid-dump still names the half-written artifact), and surfaced on every RestoreLiveStateDto response from stage:"safety_dump" onward. This is the id an operator uses to find the way back after a failed or wrong restore. |
6.9 UpdateMaintenanceDto / MaintenanceResponseDto
| Field | Type | Required | Validation | Notes |
|---|---|---|---|---|
active | boolean | Yes (request) | @IsBoolean | — |
reason | string | No (request) | @IsOptional @IsString @MaxLength(200) | Internal only — never shown to customers. |
active | boolean | — (response) | — | — |
reason | string | — (response, optional) | — | — |
restorePublicId | string | — (response, optional) | — | Present only when maintenance was engaged by a restore, not an operator. |
engagedAt | string (ISO) | — (response) | — | — |
engagedBy | string | — (response, optional) | — | — |
6.10 BackupUploadDto (multipart request body — POST .../backups/upload)
| Field | Type | Required | Validation | Notes |
|---|---|---|---|---|
file | binary (multipart/form-data) | Yes | Enforced by BackupUploadAdminController.upload checking presence, not class-validator (Swagger shape only) | The pg_dump custom-format archive (database.dump). Streamed straight to a .partial file by BackupUploadStorageEngine, hashed as it writes. |
manifest | binary | Yes | Same | manifest.json exactly as downloaded from the source deployment. Never written to disk — parsed in memory for schemaMigrationTag and manifest.tables only, then discarded. |
signature | binary | Yes | Same | artifact.sig exactly as downloaded — HMAC-SHA256(key, "<dumpSha256>:<manifestSha256>"). The control this endpoint rests on. |
All three parts are required — there is no partial-upload path. Response is a 201 BackupResponseDto (the same shape as 6.5), with kind: "uploaded", pinned: true and status: "completed" immediately (the upload is synchronous; there is no async job).
6.11 MaintenanceStatusDto (response — GET /system/maintenance, public)
| Field | Type | Nullable | Notes |
|---|---|---|---|
active | boolean | No | true while the store is refusing customer traffic. |
reason | string | Yes | Operator-supplied explanation, shown to customers on the maintenance page. null whenever active is false, even if a reason happens to be stored from a previous engagement — MaintenanceCustomerService.getStatus clears it explicitly rather than passing through whatever the shared state holds. |
Deliberately narrower than MaintenanceResponseDto (6.9): engagedBy (an operator's display name) and engagedAt (which would tell an unauthenticated caller exactly how long the database has been offline — the one fact that distinguishes a routine restore from an incident) are never included. MaintenanceCustomerService is a separate mapper from the admin one specifically so that boundary is a mapping decision, not a field an admin-table change could accidentally leak.
7. Enum Reference
| Enum | Value | Meaning | Runtime Effect | Source |
|---|---|---|---|---|
backup_kind | scheduled | Nightly cron-created run | Carries a non-null tier; no requester | enums.ts |
manual | Admin-triggered "Run now" | Requires a requestedByEmail; no tier | ||
pre_restore_safety | Would be taken automatically before a restore | No tier; not pinned by default; currently never created — no code path in the module takes a safety dump before restoring, despite the retention floor built for it | ||
uploaded | Catalogued via POST .../backups/upload (SE-4) rather than dumped by this deployment | No tier; requires requestedByEmail like manual; pinned: true always; excluded from count-based GFS retention entirely | ||
backup_tier | daily|weekly|monthly | GFS retention bucket for a scheduled run | Governs which retain* setting applies | enums.ts |
backup_status | queued|running|completed|failed|pruned | Run lifecycle | pruned is terminal and distinct from a delete — the row survives with artifactDir=null | enums.ts |
backup_restore_status | queued|safety_dump|restoring|verifying|completed|failed | Restore lifecycle | Only queued and the two terminal values (completed, failed) are ever persisted to the backup_restore.status column — safety_dump, draining (not in this enum, file-only), restoring and verifying are all genuinely reached, but only in restore-state.json, since pg_restore --clean replaces the table holding its own row mid-restore | enums.ts |
backup_signature_state | unsigned | No key was held when the artifact was written. PERMANENT for that artifact | Renders "Unsigned"; the restore preflight proceeds loudly rather than refusing | enums.ts |
signed | An HMAC over "<dumpSha256>:<manifestSha256>" exists and signing_key_fingerprint names the key | A missing artifact.sig beside a row in this state now REFUSES the restore — the signature having been removed is exactly the attack it guards | ||
signed_unknown_key | A signature exists and matched nothing this deployment currently holds | A cache, not a fact: importing a key changes the answer, so every resolution predicate reads signature_state IS NULL OR signature_state = 'signed_unknown_key' — never IS NULL alone | ||
| (NULL) | Not yet resolved, or the key store was unreadable when the run completed | Counts as unresolved, which is what blocks a retired-key prune. Writing unsigned here instead would be a permanent lie about an artifact that becomes signable the moment the key file is repaired | ||
backup_storage_mode | local|both|remote_only | Where a run's artifacts are kept | Recorded per run at completion time | enums.ts |
backup_replication_status | not_requested|pending|replicated|failed | Whether a run's artifacts reached the remote host | failed here does not fail the backup itself | enums.ts |
8. Endpoint Reference
8.1 GET /api/admin/system/backups
Purpose
Lists backup runs for the admin catalog view, with optional filtering by kind, status and tier and standard offset pagination. Used to populate the backup history table and to check whether a scheduled run completed.
Source Evidence
| Evidence | Path |
|---|---|
| Controller | backup-admin.controller.ts (findAll) |
| DTO | fetch-backups.dto.ts |
| Service | backup-admin.service.ts (findAll) |
| Schema | packages/db/src/schema/backup/backup-run.ts |
| Tests | backup-admin.service.spec.ts |
Auth and Permissions
- Auth:
JwtAuthGuard+RoleGuard - Guard chain:
JwtAuthGuard, RoleGuard, IpThrottlerGuard - Permission:
Backup_READ - Guest support: none
- Rate limit:
ADMIN_READ— 30/min, IP-keyed - Idempotency: N/A (read)
Request
| Part | Required | Details |
|---|---|---|
| Query | No | kind?, status?, tier?, pagination?, page?, size?, order?, sort? |
GET /api/admin/system/backups?status=completed&page=1&size=20 HTTP/1.1
Authorization: Bearer TOKENResponse
{
"success": true,
"message": "Backups fetched",
"data": [
{
"publicId": "018f4e2a-7b3c-7c1e-9b2a-3d4e5f6a7b8c",
"kind": "scheduled",
"tier": "daily",
"status": "completed",
"pinned": false,
"includesUploads": false,
"databaseDumpBytes": 23012582,
"uploadsArchiveBytes": null,
"totalBytes": 23012582,
"postgresVersion": "PostgreSQL 16.4",
"schemaMigrationTag": "a1b2c3",
"tableCount": 42,
"totalRows": 18342,
"requestedByEmail": null,
"note": null,
"startedAt": "2026-08-30T02:00:03.000Z",
"finishedAt": "2026-08-30T02:00:41.000Z",
"durationMs": 38000,
"errorCode": null,
"errorMessage": null,
"isRestorable": true,
"createdAt": "2026-08-30T02:00:01.000Z"
}
],
"meta": { "count": 42, "page": 1, "size": 20 }
}Side Effects
None — this is a read-only endpoint.
Error Cases
| HTTP Status | Error Code | Condition | User-Facing Meaning | Source |
|---|---|---|---|---|
| 400 | (class-validator) | Invalid kind/status/tier/pagination value | Fix the query | DTO validation |
| 401/403 | — | Missing/invalid JWT, or not superadmin | Not authorized | JwtAuthGuard/RoleGuard |
| 429 | — | Rate limit exceeded | Retry later | IpThrottlerGuard |
Edge Cases
- Empty result:
data: [],meta.count: 0. pagination=false:metaisundefinedand every matching row is returned unpaged.- No filters: returns every run, newest first.
Example Requests
curl -X GET "$API_URL/api/admin/system/backups?status=completed" \
-H "Authorization: Bearer TOKEN"8.2 POST /api/admin/system/backups
Purpose
Triggers a manual backup ("Run now"). Validates that backups are enabled, no run is currently in progress, and there is enough free disk space, then inserts a queued row and returns immediately — the actual dump runs asynchronously via the outbox and QueueName.BACKUP.
Source Evidence
| Evidence | Path |
|---|---|
| Controller | backup-admin.controller.ts (create) |
| DTO | create-backup.dto.ts |
| Service | backup-admin.service.ts (create) → backup-run-create.service.ts (createRun) |
| Schema | backup-run.ts |
| Tests | backup-run-create.service.spec.ts |
Auth and Permissions
- Permission:
Backup_CREATE - Rate limit:
ADMIN_ASYNC_JOB_SUBMIT— 10/hour, IP-keyed - Idempotency: none declared (
@IdempotentCreatenot used) — a duplicate click is refused synchronously byBACKUP_ALREADY_RUNNINGas soon as the first row isqueued, not only once a worker has claimed it intorunning.
Request
{ "includeUploads": true, "note": "before catalog migration" }Minimal valid request: {} (both fields optional).
Response
201, same shape as 8.1's list item, status: "queued".
Side Effects
- Database write:
INSERT backup_run+outbox_events(same transaction). - Queue:
backup.runreachesQueueName.BACKUPvia the outbox dispatcher (~5s poll interval). - Audit:
ActivityRecordService.recordActivity(action: "backup.create"), fire-and-forget.
Error Cases
| HTTP Status | Error Code | Condition | Source |
|---|---|---|---|
| 503 | BACKUP_DISABLED | backupEnabled=false in settings | backup-run-create.service.ts |
| 409 | BACKUP_ALREADY_RUNNING | A row is already queued or running | same |
| 503 | BACKUP_INSUFFICIENT_DISK_SPACE | Free disk below minFreeDiskMb + 1.2x last dump | same |
| 400 | (class-validator) | note over 200 chars, or wrong type | DTO |
Edge Cases
- First-ever backup: the disk-space check uses a stated floor (no prior dump to multiply against), never dividing by/multiplying
NULL. includeUploads: falseexplicit: honoured even though the configured default is alsofalse—??, not||.- Under a remote storage driver:
includeUploads: trueis silently overridden tofalseat dump time; the response later reportsincludesUploads: false.
Example Requests
curl -X POST "$API_URL/api/admin/system/backups" \
-H "Authorization: Bearer TOKEN" -H "Content-Type: application/json" \
-d '{"includeUploads": true, "note": "before catalog migration"}'8.3 GET /api/admin/system/backups/settings
Purpose
Reads the full effective backup/retention configuration for the settings screen — the merge of admin-edited settings.json (or env-seeded defaults if never written) plus read-only environment fields plus computed replication health. Registered ahead of :publicId so "settings" is never parsed as a UUID.
Source Evidence
| Evidence | Path |
|---|---|
| Controller | backup-admin.controller.ts (getSettings) |
| Service | backup-admin.service.ts (getSettings), backup-settings.service.ts |
| DTO | backup-settings-response.dto.ts |
Auth and Permissions
- Permission:
Backup_READ - Rate limit:
ADMIN_READ
Response
See 6.6 for every field.
{
"version": 3,
"backupEnabled": true,
"scheduleCron": "0 2 * * *",
"scheduleTimezone": "Asia/Kathmandu",
"includeUploadsByDefault": false,
"retainDaily": 7, "retainWeekly": 4, "retainMonthly": 6, "retainManual": 10, "retainSafety": 3,
"minFreeDiskMb": 2048,
"dumpTimeoutMinutes": 30, "restoreTimeoutMinutes": 60,
"restoreDrainSeconds": 120, "restoreAbandonSeconds": 900,
"storageMode": "local",
"remoteReplicationEnabled": false,
"restoreEnabled": false,
"restoreAdminDatabaseConfigured": false,
"pgDumpPath": "/usr/bin/pg_dump",
"pgRestorePath": "/usr/bin/pg_restore",
"remoteConfigured": false,
"lastReplicationFailureAt": null,
"replicationFailureStreak": 0
}Side Effects
None.
Error Cases
None beyond auth/rate-limit — getEffectiveSettings never throws.
8.4 PUT /api/admin/system/backups/settings
Purpose
Updates the admin-editable backup/retention policy in one full-replace write. The admin panel reads 8.3, edits it, and sends the whole shape back with the version it read.
Source Evidence
| Evidence | Path |
|---|---|
| Controller | backup-admin.controller.ts (updateSettings) |
| DTO | update-backup-settings.dto.ts |
| Service | backup-admin.service.ts (updateSettings) → backup-settings.service.ts (updateSettings) |
Auth and Permissions
- Permission:
BackupConfigure_UPDATE— notBackup_UPDATE, which today gates only pin/unpin on one row. - Rate limit:
ADMIN_WRITE
Request
Full valid request — every field in 6.6 except the read-only ones, plus version:
{
"version": 3,
"backupEnabled": true,
"scheduleCron": "0 3 * * *",
"scheduleTimezone": "Asia/Kathmandu",
"includeUploadsByDefault": false,
"retainDaily": 7, "retainWeekly": 4, "retainMonthly": 6, "retainManual": 10, "retainSafety": 3,
"minFreeDiskMb": 2048,
"dumpTimeoutMinutes": 30, "restoreTimeoutMinutes": 60,
"restoreDrainSeconds": 120, "restoreAbandonSeconds": 900,
"storageMode": "local",
"remoteReplicationEnabled": false
}Response
200, same shape as 8.3, reflecting the new version.
Side Effects
- Filesystem write:
settings.json(atomic temp-file + rename). - If
scheduleCron/scheduleTimezonechanged:BackupScheduleScheduler.reload()re-registers the cron job immediately, without an API restart. - Audit:
activityRecordService.recordActivitywith a full before/afterchanges[]diff.
Error Cases
| HTTP Status | Error Code | Condition |
|---|---|---|
| 409 | BACKUP_SETTINGS_CONFLICT | version does not match the current value |
| 400 | BACKUP_INVALID_SCHEDULE | scheduleCron not a valid 5-field expression, or scheduleTimezone not a recognised IANA zone |
| 400 | BACKUP_REMOTE_NOT_CONFIGURED | storageMode is both/remote_only without a configured remote target, or without remoteReplicationEnabled: true |
| 400 | BACKUP_SETTINGS_INVALID | Any other field fails its bound |
Example Requests
curl -X PUT "$API_URL/api/admin/system/backups/settings" \
-H "Authorization: Bearer TOKEN" -H "Content-Type: application/json" \
-d '{"version":3,"backupEnabled":true,"scheduleCron":"0 3 * * *", "...": "..."}'8.5 GET /api/admin/system/backups/{publicId}
Purpose
Fetches one backup's detail for the catalog detail view.
Source Evidence
| Evidence | Path |
|---|---|
| Controller | backup-admin.controller.ts (findByPublicId) |
| Service | backup-admin.service.ts → backup-retention.service.ts (getRowByPublicId) |
Auth and Permissions
- Permission:
Backup_READ; Rate limit:ADMIN_READ
Request
| Part | Required | Details |
|---|---|---|
| Params | Yes | publicId (UUID) |
Response
Same shape as one item from 8.1.
Error Cases
| HTTP Status | Error Code | Condition |
|---|---|---|
| 404 | BACKUP_NOT_FOUND | No row for that publicId |
8.6 PATCH /api/admin/system/backups/{publicId}
Purpose
Pins or unpins a backup, protecting (or exposing) it to count-based retention pruning.
Source Evidence
| Evidence | Path |
|---|---|
| Controller | backup-admin.controller.ts (updatePinned) |
| DTO | update-backup.dto.ts |
| Service | backup-admin.service.ts (updatePinned) |
Auth and Permissions
- Permission:
Backup_UPDATE; Rate limit:ADMIN_WRITE
Request
{ "pinned": true }Response
200, updated BackupResponseDto.
Side Effects
UPDATE backup_run SET pinned = ...- Audit:
backup.pin/backup.unpinwith achanges: [{field:"pinned", from, to}]entry.
Error Cases
| HTTP Status | Error Code | Condition |
|---|---|---|
| 404 | BACKUP_NOT_FOUND | No row for that publicId |
8.7 DELETE /api/admin/system/backups/{publicId}
Purpose
Requests that a backup be pruned. Validates the three retention floors synchronously (so an ineligible delete gets an immediate 409, never a silently-skipped async job), then enqueues the actual deletion through the outbox.
Source Evidence
| Evidence | Path |
|---|---|
| Controller | backup-admin.controller.ts (remove) |
| Service | backup-admin.service.ts (remove) → backup-retention.service.ts (assertPrunable) |
Auth and Permissions
- Permission:
Backup_DELETE; Rate limit:ADMIN_WRITE
Response
{ "success": true, "message": "Prune requested", "data": { "publicId": "018f..." } }Side Effects
- Outbox enqueue:
backup.prune {reason:"manual", backupRunPublicId},dedupeKey: "manual-prune-<publicId>". - Audit:
backup.delete_requested. - The actual filesystem
rm -rfand row update toprunedhappen later, inBackupPruneHandler, which re-checksassertPrunable— the floors can change between the click and the worker picking up the job.
Error Cases
| HTTP Status | Error Code | Condition |
|---|---|---|
| 404 | BACKUP_NOT_FOUND | No row for that publicId |
| 409 | BACKUP_PINNED_CANNOT_DELETE | Row is pinned |
| 409 | BACKUP_LAST_REMAINING_CANNOT_DELETE | Newest overall, newest in its bucket, or the safety dump of the most recent restore |
8.8 GET /api/admin/system/backups/{publicId}/download
Purpose
Streams the raw pg_dump artifact for a completed backup. The highest-consequence read this module offers — every check runs before the first byte, and the audit write is the final gate and fails closed.
Source Evidence
| Evidence | Path |
|---|---|
| Controller | backup-download-admin.controller.ts |
| Service | backup-download-admin.service.ts (prepareDownload) |
Auth and Permissions
- Permission:
BackupDownload_READ— a separate permission module fromBackup_*, deliberately: seeing that a backup succeeded and walking out with the entire customer database are different acts with different audiences. - Rate limit:
ADMIN_BACKUP_DOWNLOAD— 3/hour, user-keyed (not IP), because an IP-keyed cap on the endpoint that streams every customer record and password hash is bypassed by rotating source IPs.
Request
| Part | Required | Details |
|---|---|---|
| Params | Yes | publicId (UUID) |
Response
Raw application/octet-stream, streamed via createReadStream(...).pipe(res) — never buffered. Headers: Content-Type: application/octet-stream, Content-Disposition: attachment; filename="backup-<publicId>.dump" (filename derived from publicId alone, never the operator-editable note, to avoid CRLF header injection), Content-Length, Cache-Control: no-store.
Side Effects
stat()and streamingsha256read of the dump file (integrity checks, size first then checksum).- MongoDB write:
AuditLog.create({action:"backup.download", ...})— must succeed, or the download itself is refused with500 BACKUP_DOWNLOAD_AUDIT_FAILED. This is the one write in the whole module that is not fire-and-forget.
Error Cases
| HTTP Status | Error Code | Condition |
|---|---|---|
| 409 | BACKUP_NOT_RESTORABLE | status !== "completed" |
| 410 | BACKUP_ARTIFACT_MISSING | !localArtifactPresent, or the file is absent from disk |
| 409 | BACKUP_SIZE_MISMATCH | Actual file size ≠ recorded databaseDumpBytes |
| 409 | BACKUP_CHECKSUM_MISMATCH | Actual sha256 ≠ recorded databaseDumpSha256 |
| 500 | BACKUP_DOWNLOAD_AUDIT_FAILED | The AuditLog write itself failed |
Example Requests
curl -X GET "$API_URL/api/admin/system/backups/018f.../download" \
-H "Authorization: Bearer TOKEN" -o backup.dump8.9 POST /api/admin/system/backups/{publicId}/restore
Purpose
Accepts a request to replace the entire live database from a completed backup. Irreversible in intent. Runs every safety check synchronously, then engages maintenance, writes a live-state file, and enqueues backup.restore, which BackupRestoreProcessor consumes: it takes and links an inline pre-restore safety dump, then spawns and supervises restore-runner.ts, which fences the database, drains connections, runs pg_restore, and verifies row counts against the manifest before reporting success — see the note below for exactly what still is not covered.
Source Evidence
| Evidence | Path |
|---|---|
| Controller | backup-restore-admin.controller.ts (restore) |
| DTO | restore.dto.ts |
| Service | backup-restore-admin.service.ts (requestRestore) |
| Preflight | backup-restore-preflight.service.ts |
Auth and Permissions
- Permission:
Backup_RESTORE— superadmin-only. - Additionally armed by
BACKUP_RESTORE_ENABLED=true(deploy-time env; preflight refuses otherwise). - Rate limit:
ADMIN_RESTORE_SUBMIT— 2/day, user-keyed, matchingAUTH_DAILY_IRREVERSIBLE.
Request
{
"confirmDatabaseName": "skoolsewa_production",
"acknowledgeDataLoss": true,
"acknowledgeMongoNotRestored": true
}Response
202, RestoreLiveStateDto:
{
"restorePublicId": "018f...",
"stage": "queued",
"message": "Accepted. Waiting for the restore runner to start.",
"startedAt": "2026-08-30T10:00:00.000Z",
"heartbeatAt": "2026-08-30T10:00:00.000Z"
}Side Effects
- Database:
INSERT backup_restore (status: 'queued', maintenanceEngaged: true)+ outbox enqueue, one transaction. - Filesystem:
restore-state.jsonwritten withstage: "queued". - Maintenance is engaged immediately (
MaintenanceService.engage) — not deferred to a worker, because the outbox dispatcher polls every ~5s and any gap would let customers keep writing to a database about to be replaced. - Audit:
backup.restore.requested. - Server log:
warn-level"RESTORE ACCEPTED"line.
8.9a Known gap: verification is row-count only
A 202 from this endpoint means preflight passed, maintenance is engaged, backup.restore reached QueueName.BACKUP_RESTORE, and BackupRestoreProcessor genuinely: (1) takes an inline kind:'pre_restore_safety' backup, links it to the restore via backup_restore.safety_backup_run_id, and surfaces its id as RestoreLiveStateDto.safetyBackupPublicId — aborting the whole restore if the dump does not reach completed or the link cannot be recorded; (2) spawns workers/restore-runner.ts as a detached process, which fences the database, drains existing connections (up to the admin-configured restoreDrainSeconds) before terminating them, runs pg_restore --clean --single-transaction (up to the admin-configured restoreTimeoutMinutes), and — at the new verifying stage — compares every manifest-listed table's row count against the freshly-restored database (excluding RESTORE_VERIFICATION_EXCLUDED_TABLES: session tables, backup_run, backup_restore, outbox_events, drizzle_migrations — tables the restore and the safety dump themselves are expected to change), reporting BACKUP_RESTORE_VERIFICATION_FAILED specifically if any mismatch; and (3), on full success, disengages maintenance. The restore genuinely happens, is genuinely safeguarded by an automatic, linked, discoverable backup, and is genuinely checked afterward.
What still is not covered: verification is row-count only, not content-level — identical counts with different row contents, or --no-owner/--no-privileges permission side effects, would not be caught. BackupRestoreVerification's structured shape ({tablesChecked, tablesMatched, mismatches[], excluded[]}) is never written to backup_restore.verification, even though the comparison itself now happens — only pass/throw is observed. There is also still no one-click "undo": restoring the safety dump back requires an operator to submit it (now discoverable via safetyBackupPublicId) as the source of a second, ordinary restore request. See the backend doc's 16.8 Risk Register. If a runner dies or never spawns, force-clear marks the row failed and disengages maintenance without confirming whether pg_restore actually ran to completion.
Error Cases
| HTTP Status | Error Code | Condition |
|---|---|---|
| 503 | BACKUP_RESTORE_DISABLED | BACKUP_RESTORE_ENABLED=false, or BACKUP_ADMIN_DATABASE_URL unset |
| 409 | BACKUP_RESTORE_APP_ROLE_IS_SUPERUSER | The application's own DB role has rolsuper |
| 400 | BACKUP_RESTORE_CONFIRMATION_MISMATCH | Either acknowledgement is not literally true, or the typed database name does not match current_database() |
| 404→ mapped | BACKUP_NOT_FOUND | Source backup does not exist |
| 409 | BACKUP_NOT_RESTORABLE | Source status !== "completed" |
| 410 | BACKUP_ARTIFACT_MISSING | Source has no local artifact |
| 409 | BACKUP_RESTORE_SCHEMA_VERSION_MISMATCH | Backup's schema tag differs from (or is unknown vs.) the running app's — no override |
| 409 | BACKUP_SIZE_MISMATCH / BACKUP_CHECKSUM_MISMATCH | Artifact integrity check failed |
| 409 | BACKUP_RESTORE_ALREADY_ACTIVE | Another restore is non-terminal |
| 409 | BACKUP_RESTORE_RUN_IN_PROGRESS | A backup dump is queued/running |
Example Requests
curl -X POST "$API_URL/api/admin/system/backups/018f.../restore" \
-H "Authorization: Bearer TOKEN" -H "Content-Type: application/json" \
-d '{"confirmDatabaseName":"skoolsewa_production","acknowledgeDataLoss":true,"acknowledgeMongoNotRestored":true}'8.10 GET /api/admin/system/restores/{publicId}/live
Purpose
The one endpoint that stays answerable while a restore is running — it reads only restore-state.json, never a database table, so it does not block behind pg_restore --clean's table locks or behind JwtStrategy reading admin_sessions. The admin UI is expected to poll this with backoff, to show progress through safety_dump → draining → restoring → verifying → completed/failed, and to treat a network failure as expected, not as "failed".
Source Evidence
| Evidence | Path |
|---|---|
| Controller | backup-restore-admin.controller.ts (live) |
| Service | backup-restore-admin.service.ts (getLiveState) |
Auth and Permissions
- Permission:
Backup_READ; Rate limit:ADMIN_READ
Response
Same shape as 8.9's response.
Error Cases
| HTTP Status | Error Code | Condition |
|---|---|---|
| 410 | BACKUP_RESTORE_NOT_FOUND | No state file, or it names a different publicId |
8.11 POST /api/admin/system/restores/{publicId}/force-clear
Purpose
Clears a restore that will not finish — for a runner that was killed, a host that rebooted mid-operation, or an operator who wants out regardless of a still-running pg_restore. Without this, a stuck row blocks every future restore via uq_backup_restore_single_active, with no code path out other than hand-written SQL. It does not stop a genuinely running subprocess — see 8.9a.
Source Evidence
| Evidence | Path |
|---|---|
| Controller | backup-restore-admin.controller.ts (forceClear) |
| Service | backup-restore-admin.service.ts (forceClear) |
Auth and Permissions
- Permission:
Backup_RESTORE; Rate limit:ADMIN_RESTORE_SUBMIT
Response
{ "cleared": true }Side Effects
UPDATE backup_restore SET status='failed', errorCode='BACKUP_RESTORE_ABANDONED', ...MaintenanceService.disengage()— customer traffic resumes.- Server log:
warn-level.
Error Cases
| HTTP Status | Error Code | Condition |
|---|---|---|
| 410 | BACKUP_RESTORE_NOT_FOUND | No row with that publicId |
No check that the restore is actually stuck (no age threshold), so force-clearing a fresh restore is indistinguishable from force-clearing a genuinely stuck one — an operator must independently confirm pg_restore is not still running before treating the site as safely back online.
8.12 GET /api/admin/system/maintenance
Purpose
Reads the current maintenance state, for the admin panel's status banner.
Source Evidence
| Evidence | Path |
|---|---|
| Controller | maintenance-admin.controller.ts (get) |
| Service | maintenance-admin.service.ts (get) |
Auth and Permissions
- Permission:
System_READ; Rate limit:ADMIN_READ
Response
{ "active": false, "engagedAt": "1970-01-01T00:00:00.000Z" }engagedAt of the Unix epoch is the sentinel for "never engaged, no state file exists yet."
8.13 PUT /api/admin/system/maintenance
Purpose
Engages or disengages the site-wide kill switch. While engaged, all customer traffic receives 503 SYSTEM_MAINTENANCE_ACTIVE; operator routes, health probes, and the operator's own login stay reachable.
Source Evidence
| Evidence | Path |
|---|---|
| Controller | maintenance-admin.controller.ts (set) |
| DTO | maintenance.dto.ts |
| Service | maintenance-admin.service.ts (set) |
Auth and Permissions
- Permission:
System_UPDATE— deliberately not aSettings_*module, so no ordinary content administrator can take the storefront offline (or, worse, disengage maintenance mid-restore). - Rate limit:
ADMIN_WRITE
Request
{ "active": true, "reason": "scheduled migration" }Response
{ "active": true, "reason": "scheduled migration", "engagedAt": "2026-08-30T10:00:00.000Z", "engagedBy": "ops@skoolsewa.example" }Side Effects
- File-first-then-Redis write of the maintenance state (both
MaintenanceService.engage/disengage). - Audit:
maintenance.engage/maintenance.disengage, fire-and-forget (the flag write must succeed on its own even if the audit write fails, or an operator could be stuck unable to disengage).
Error Cases
None beyond auth/validation.
8.14 POST /api/admin/system/backups/upload
Purpose
Catalogues an operator-supplied backup archive (SE-4) — a pg_dump produced by this deployment, or by another one whose key this deployment generated (an imported key does not authenticate an upload; see §8.18) — so the existing, unmodified restore path can act on it. Used when a database needs to be seeded from a backup taken outside the normal scheduled/manual flow, or moved between deployments. Does not restore anything itself.
Source Evidence
| Evidence | Path |
|---|---|
| Controller | admin/upload/backup-upload-admin.controller.ts (upload) |
| DTO | admin/upload/dto/backup-upload.dto.ts |
| Storage engine | admin/upload/backup-upload-storage.engine.ts |
| Service | admin/upload/backup-upload-admin.service.ts (upload) |
| Signature | shared/backup-signature.service.ts |
| Content scan | shared/backup-archive-content.util.ts (scanArchiveSql) |
| Tests | admin/upload/backup-upload-admin.service.spec.ts |
Auth and Permissions
- Auth:
JwtAuthGuard+RoleGuard - Guard chain:
JwtAuthGuard, RoleGuard, IpThrottlerGuard - Permission:
BackupUpload_CREATE— a separate permission module fromBackup_*, deliberately: introducing a database from outside is not the same act as running a dump of this one. - Guest support: none
- Rate limit:
ADMIN_BACKUP_UPLOAD— 3/hour, user-keyed (matchingADMIN_BACKUP_DOWNLOAD's reasoning — both move a whole database dump over HTTP) - Idempotency: none declared; a duplicate upload of the same archive inserts a second
uploadedrow, eventually rejected byBACKUP_UPLOAD_RETAINED_LIMIT_REACHED
Request
| Part | Required | Details |
|---|---|---|
| Body | Yes | multipart/form-data with exactly three parts: file (the dump), manifest (manifest.json), signature (artifact.sig) — see 6.10 |
POST /api/admin/system/backups/upload HTTP/1.1
Authorization: Bearer TOKEN
Content-Type: multipart/form-data; boundary=----WebKitFormBoundaryResponse
201, same shape as 8.1's list item, with kind: "uploaded", pinned: true, status: "completed" — immediately, since there is no async job.
Side Effects
- Filesystem: moves the streamed
.partialdump into a real artifact directory; writes a fresh, server-generatedmanifest.jsonandartifact.sig(never the operator's uploaded manifest — see the backend doc's 6.9). - Subprocess:
pg_restore --list(format check) andpg_restore -f - --schema-only(renders SQL for the content scan) — read-only, no database connection. - Database write:
INSERT backup_run(kind:'uploaded'). - Audit:
ActivityRecordService.recordActivity(action: "backup.upload"), fire-and-forget.
Error Cases
| HTTP Status | Error Code | Condition | Source |
|---|---|---|---|
| 400 | BACKUP_ARCHIVE_UNREADABLE | The file part is missing | Controller |
| 400 | BACKUP_UPLOAD_MANIFEST_INVALID | The manifest part is missing, not valid JSON, not an object, or missing schemaMigrationTag/databaseDumpSha256 | Controller/Service |
| 400 | BACKUP_UPLOAD_SIGNATURE_INVALID (missing-part case) | The signature part is missing | Controller |
| 409 | BACKUP_UPLOAD_RETAINED_LIMIT_REACHED | BACKUP_UPLOAD_MAX_RETAINED completed uploaded backups already exist | Service |
| 409 | BACKUP_CHECKSUM_MISMATCH | The dump's sha256 does not match the value the manifest claims (corruption check) | Service |
| 503 | BACKUP_UPLOAD_SIGNING_NOT_CONFIGURED | This deployment holds no current signing key | Service |
| 409 | BACKUP_UPLOAD_SIGNATURE_INVALID (mismatch case) | The HMAC does not match this archive+manifest — not produced by a deployment holding this system's key | Service |
| 409 | BACKUP_ARCHIVE_UNREADABLE | pg_restore --list cannot read the file as a PostgreSQL archive | Service |
| 409 | BACKUP_UPLOAD_TOC_REJECTED | The archive's schema SQL contains a construct a restore must not execute | Service |
Edge Cases
- All three parts required — there is no partial upload.
- The operator's own
manifest.jsonis discarded after two fields are read from it; a fresh one is always written, so re-downloading and re-uploading an already-catalogued row works (it is re-signed over its own new manifest). - A truncated dump passes
pg_restore --list(the format check only reads the table of contents at the front of the archive) — the sha256-vs-manifest comparison is what actually catches truncation. - Uploaded rows are pinned by default and excluded from GFS retention entirely; the only way to remove one is
DELETE /admin/system/backups/{publicId}(8.7).
Example Requests
curl -X POST "$API_URL/api/admin/system/backups/upload" \
-H "Authorization: Bearer TOKEN" \
-F "file=@database.dump" \
-F "manifest=@manifest.json" \
-F "signature=@artifact.sig"8.15 GET /api/system/maintenance
Purpose
The one public route in this module. Answers "is the store currently refusing customer traffic" without authentication, uncached, on every call. Exists because the storefront's other reads go through Next's data cache (the header/content page has revalidate: 3600) — during a restore that cache keeps serving 200 responses with full content and never calls the API at all, so no fetch-level 503 handler could ever fire. Verified by engaging maintenance and requesting /: 200, full page, zero maintenance markers, per the controller's own docblock. The storefront is expected to call this directly, uncached, on every request it serves.
Source Evidence
| Evidence | Path |
|---|---|
| Controller | customer/maintenance-customer.controller.ts (getStatus) |
| DTO | customer/dto/maintenance-status.dto.ts |
| Service | customer/maintenance-customer.service.ts (getStatus) |
| Exemption | shared/maintenance.guard.ts (MAINTENANCE_EXEMPT_PATHS includes /api/system/maintenance by literal path) |
Auth and Permissions
- Auth:
@Public()— noJwtAuthGuard, noRoleGuard. - Guard chain:
IpThrottlerGuardonly. - Permission: N/A.
- Guest support: yes — this is the guest/customer surface.
- Rate limit:
PUBLIC_HIGH_FREQUENCY— 300/min, IP-keyed. Chosen because it is read on effectively every storefront page render, through the middleware. - Idempotency: N/A (read).
Request
No params, query, or body.
GET /api/system/maintenance HTTP/1.1Response
{
"active": false,
"reason": null
}While engaged:
{
"active": true,
"reason": "Database restore in progress"
}Side Effects
None — read-only. Reads the same 3-layer maintenance state (in-process cache → Redis → file) as the admin endpoint, via the shared MaintenanceService, but maps it onto the narrower MaintenanceStatusDto.
Error Cases
| HTTP Status | Error Code | Condition | Source |
|---|---|---|---|
| 429 | — | Rate limit exceeded | IpThrottlerGuard |
MaintenanceService.getState() does not throw for this caller in the way isActive() fails closed elsewhere in the module — getStatus() reads whatever state is available and maps a missing/falsy state to {active:false, reason:null}.
Edge Cases
- This route is itself exempt from
MaintenanceGuard(MAINTENANCE_EXEMPT_PATHS), so it stays answerable precisely during the window it exists to report on. - It returns no data that is not already observable from every other route in the API answering
503while maintenance is engaged — the exemption discloses nothing new. engagedBy/engagedAtare never in this response, even though the underlying shared state carries them — see 6.11.
Example Requests
curl -X GET "$API_URL/api/system/maintenance"8.16 GET /api/admin/system/backup-key
| Field | Value |
|---|---|
| Auth | JwtAuthGuard + RoleGuard + IpThrottlerGuard |
| Permission | BackupKey_READ |
| Rate limit | ADMIN_KEY_MATERIAL_READ — 30/min, IP-keyed |
| Request | None |
| Success | 200 BackupKeyStateResponseDto |
Never returns key material. There is no endpoint anywhere in this module that returns an existing key: material appears in exactly two responses, both at the moment it is created. A standing "download the key" route would be a permanently wider target that buys nothing, since the key is only needed when it is made.
{
"keyPathConfigured": true,
"storeState": "ready",
"current": {
"fingerprint": "1289db31",
"provenance": "generated",
"createdAt": "2026-09-01T08:30:59.185Z",
"retiredAt": null
},
"retired": [
{
"fingerprint": "c035c8a9",
"provenance": "generated",
"createdAt": "2026-09-01T08:53:28.278Z",
"retiredAt": "2026-09-01T08:56:46.128Z",
"retainedBackupCount": 1
}
],
"unresolvedCount": 0
}storeState is one of no_path_configured · legacy_env · no_key · ready · unreadable.
unreadable is not no_key, and the distinction is load-bearing: "no key configured" makes
the restore preflight SKIP signature verification, which is correct for a deployment that never had
a key and catastrophic for one whose key file was truncated by a crash. A corrupt file therefore
degrades to unreadable rather than failing the boot, and the preflight refuses on it.
retainedBackupCount counts completed backups on this host carrying that fingerprint. An
archive kept off-box has no row here and cannot be counted — and that is exactly the population
import exists for, so a 0 never means "nothing depends on this key".
unresolvedCount is why the prune refusal can be trusted. It counts completed runs whose
signature_state is NULL or signed_unknown_key, and it is deliberately not filtered on
local_artifact_present: a run reclaimed to a remote copy is invisible to retainedBackupCount
(its fingerprint is NULL) and would be invisible here too, so the operator would read 0, 0, prune,
and destroy the only key able to verify archives that still exist.
8.17 POST /api/admin/system/backup-key/generate
| Field | Value |
|---|---|
| Permission | BackupKey_CREATE |
| Rate limit | ADMIN_KEY_MATERIAL_WRITE — 5 per 15 min, USER-keyed |
| Request | { "currentPassword": "..." } |
| Success | 201 RevealedBackupKeyResponseDto — { fingerprint, key } |
{ "fingerprint": "60bb212b", "key": "6d4e7c6f26f34ea3…142cd75f" }This is the only time the value is returned. Nothing can retrieve it later. If the response is lost, rotate again.
No @IdempotentCreate on this route, or on rotate — deliberately. IdempotencyInterceptor
persists responseJson to Redis for 24 hours, which would park a plaintext signing key in a store
an operator may snapshot. generate is already idempotent by refusal: it 409s once a current key
exists, so a retry cannot mint a second key.
| Status | errorCode | Meaning |
|---|---|---|
| 400 | BACKUP_KEY_REAUTH_REQUIRED | currentPassword was absent. Distinct from a wrong one, so the panel can say "you left it blank". |
| 401 | BACKUP_KEY_REAUTH_FAILED | The password did not match. |
| 409 | BACKUP_KEY_ALREADY_EXISTS | A current key is already held — replacing one silently is a rotation nobody audited. Use rotate. |
| 409 | BACKUP_KEY_NO_PASSWORD_SET | The admin account has no password (OAuth-only), so the step-up cannot be performed. |
| 422 | BACKUP_KEY_PATH_NOT_CONFIGURED | BACKUP_ARTIFACT_HMAC_KEY_PATH is unset; there is nowhere to write the key. |
| 422 | BACKUP_KEY_STORE_UNREADABLE | The key file exists and cannot be parsed. Repair it before mutating it. |
8.18 POST /api/admin/system/backup-key/import
| Field | Value |
|---|---|
| Permission | BackupKey_CREATE |
| Rate limit | ADMIN_KEY_MATERIAL_WRITE |
| Request | { "currentPassword": "...", "key": "<the material>" } |
| Success | 201 ImportedBackupKeyResponseDto — { fingerprint, retainedBackupCount } |
An imported key joins the retired set. It never signs and never becomes current.
provenance is load-bearing, not metadata. An imported key is a legitimate verification anchor
on RESTORE, where backup_run.database_dump_sha256 is an independent server-computed value the
backup directory cannot forge. It is not an anchor on UPLOAD: there the caller supplies the
archive, the signature and — via this endpoint — the key that validates it. Three operands from
one actor prove nothing, so BackupSignatureService filters verification candidates on this field.
It lives in the key file's format rather than in a branch somebody can forget to write.
| Status | errorCode | Meaning |
|---|---|---|
| 409 | BACKUP_KEY_ALREADY_EXISTS | That fingerprint is already held. |
| 422 | BACKUP_KEY_INVALID_MATERIAL | Shorter than BACKUP_KEY_MIN_IMPORT_LENGTH. |
8.19 POST /api/admin/system/backup-key/rotate
| Field | Value |
|---|---|
| Permission | BackupKey_UPDATE |
| Rate limit | ADMIN_KEY_MATERIAL_WRITE |
| Request | { "currentPassword": "...", "confirmFingerprint": "1289db31" } |
| Success | 201 RevealedBackupKeyResponseDto |
confirmFingerprint must equal the fingerprint of the key being retired — 8 lowercase hex
characters. It stops a rotation aimed at a key the operator was not looking at.
The retired key is kept, and that is the entire point. A retired key is not a dead key: it is what still verifies every artifact signed before the rotation, which is the difference between "rotate the signing key" and "make every existing backup unrestorable". Rotating and then restoring a pre-rotation backup is verified end to end.
| Status | errorCode | Meaning |
|---|---|---|
| 409 | BACKUP_KEY_FINGERPRINT_UNKNOWN | confirmFingerprint does not match the current key. |
| 422 | BACKUP_KEY_NOT_FOUND | There is no current key to rotate. Use generate. |
8.20 POST /api/admin/system/backup-key/resolve-signatures
| Field | Value |
|---|---|
| Permission | BackupKey_UPDATE |
| Rate limit | ADMIN_HEAVY_OP — 20/min |
| Request | None |
| Success | 200 ResolveSignaturesResponseDto — { resolved, remaining } |
Batched: call again while remaining is above zero.
It reveals no key material and mutates no key — it recomputes derived state on backup_run by
reading each artifact's artifact.sig — so it takes ADMIN_HEAVY_OP rather than the step-up
budget, whose shape assumes every attempt is a password guess. It requires no password for the
same reason.
It re-examines runs previously resolved as signed_unknown_key, because importing a key changes
that answer, and that is precisely what makes the retired-key prune count trustworthy.
remaining reports work this host can do — rows whose artifacts are readable here. That is
deliberately a different set from unresolvedCount in §8.16, which is "reasons not to prune" and
includes rows this host can never resolve.
8.21 DELETE /api/admin/system/backup-key/retired/{fingerprint}
| Field | Value |
|---|---|
| Permission | BackupKey_DELETE |
| Rate limit | ADMIN_KEY_MATERIAL_WRITE |
| Path param | fingerprint — 8 lowercase hex characters |
| Request | { "currentPassword": "...", "confirmFingerprint": "…", "force": false } |
| Success | 200 ImportedBackupKeyResponseDto |
Irreversible. The only way back is to import the same key material again — if the operator still has it.
Two independent refusals, and force overrides both:
| Status | errorCode | Refuses because |
|---|---|---|
| 409 | BACKUP_KEY_STILL_IN_USE | Completed backups still carry this fingerprint, and would become unverifiable. |
| 409 | BACKUP_KEY_STILL_IN_USE | Some completed run is still unresolved, so it is not yet known whether this key is the only thing able to verify it. Run resolve-signatures first. |
| 404 | BACKUP_KEY_NOT_FOUND | No retired key has that fingerprint. |
The server-side confirmFingerprint check is not evidence of intent. It is compared against the
path segment, and both come from the same client, so it catches a malformed request and nothing
more. It is not the analogue of the restore dialog's confirmDatabaseName, which is checked
against SELECT current_database() — a fact only the server holds and a client that has not been
told the answer cannot satisfy. There is no server-side fact about a key fingerprint that the client
does not already supply, so for an accidental prune the admin panel's typed-confirmation dialog
is the control, and calling it a speed bump in front of one would misdescribe what protects an
irreversible delete.
8.23 GET /api/admin/system/backups/{publicId}/download/uploads
| Field | Value |
|---|---|
| Permission | BackupDownload_READ — the same as the dump download |
| Rate limit | ADMIN_BACKUP_DOWNLOAD — the same constant, but a separate bucket |
| Params | BackupDownloadParamsDto — @IsUUID("7") |
| Success | 200 · application/gzip · Content-Length · Cache-Control: no-store · attachment; filename="backup-<publicId>-uploads.tar.gz" |
Why this is a second route rather than a bigger tar. §8.8 hands out exactly database.dump,
manifest.json and artifact.sig, because those three are what §8.14's upload endpoint consumes.
Adding uploads.tar.gz would break that round trip and turn every routine dump download into a
multi-gigabyte transfer. The operator picks.
Before this route existed, an operator who ticked "Also archive uploaded files" got a download containing none of them, and nothing said so.
The same constant does not mean a shared budget. IpThrottlerGuard keys the bucket as
`throttle:ip:${ControllerClass}:${handlerName}:${principal}` — the RATE_LIMITS constant
supplies only limit, windowSeconds and keyStrategy, and contributes nothing to the key. So each
handler has its own 3/hour and the two downloads never compete. Reusing the constant is therefore a
naming decision, not a capacity one; per-operator capacity is 3+3 either way.
Refusals consume the budget. The guard runs before the handler, so a 409 or 410 costs a slot exactly like a delivered file does.
Refusals
| Status | errorCode | Condition |
|---|---|---|
| 400 | VALIDATION_FAILED | publicId is not a UUIDv7 |
| 404 | BACKUP_NOT_FOUND | No such run |
| 409 | BACKUP_NOT_RESTORABLE | status !== 'completed' |
| 409 | BACKUP_UPLOADS_ARCHIVE_NOT_INCLUDED | The run archived no uploaded files |
| 409 | BACKUP_SIZE_MISMATCH | Recorded byte count ≠ the file on disk |
| 410 | BACKUP_ARTIFACT_MISSING | Reclaimed to a remote copy, or not openable |
| 500 | BACKUP_DOWNLOAD_AUDIT_FAILED | The audit write failed — the download is refused, never left unaudited |
BACKUP_UPLOADS_ARCHIVE_NOT_INCLUDED is 409, not 404: the backup exists and its database half
downloads fine. A 404 would send the operator hunting a wrong id instead of a missing option chosen
at backup time.
No server-side hash, and why that is not a weakening
§8.8 computes the dump's sha256 before streaming it. This route does not, and the reason is specific rather than general:
- The dump's hash is justified by RESTORE — "a silently corrupted dump is worse than a missing one,
because it restores and produces a wrong database rather than an error." Nothing restores
uploads.tar.gz; the only consumer is the operator. - Hashing reads the whole file before the first response header, and the admin BFF abandons a download whose headers have not arrived in time. On the artifact that is large by design, that turns every download into a timeout — and since refusals consume the rate budget, a few deterministic retries lock the operator out during the incident the feature exists for.
Integrity moves to where it can be acted on: uploadsArchiveSha256 is on the backup detail response
and rendered in the panel, so the operator verifies the file they actually received —
sha256sum backup-<publicId>-uploads.tar.gzContent-Length is set (unlike §8.8, whose tar is built on the fly), so a truncated transfer is
detectable without any hashing.
The open happens before any header is set
A failure opening the archive is an ordinary thrown exception rendered as a normal envelope. That ordering is load-bearing and was arrived at by measurement:
| Shape | Behaviour on a failed open |
|---|---|
stream.pipe(res) + an error handler that logs | Response hangs with no status — pipe does not forward a source error to the destination |
pipeline(stream, res) + a catch answering 410 | Connection reset, no status — pipeline destroys the response before the catch runs |
Open first, then set headers, then pipeline | 410 BACKUP_ARTIFACT_MISSING |
This matters because the failure is reachable: deleteHeavyArtifacts removes uploads.tar.gz
before it flips local_artifact_present, so a completed, locally-present run can lose its file at
any moment, and the window between the size check and the open spans the audit write.
Audit
backup.download_uploads — distinct from §8.8's backup.download. After an incident the
question is which artifact left this host, and customer files leaving is a different disclosure
from a database dump leaving. One shared action string cannot answer it.
Like §8.8, the record is written before the first byte and the download is refused if it fails.
Note the corollary: the row says success before delivery is attempted, so an aborted transfer
leaves an optimistic record.
8.22 Forwarding the operator's address
Not an endpoint — a header pair every admin route accepts, added because the admin panel is a BFF
and so every admin action landed in the activity log as 127.0.0.1: literally accurate, and useless
for a log whose purpose is "who did this, from where".
| Header | Value |
|---|---|
x-internal-client-ip | The address to record |
x-internal-client-ip-token | Must equal INTERNAL_CLIENT_IP_TOKEN |
Both halves are required and the address is discarded outright unless the token matches. Trust
is bound to a shared secret rather than to Express's trust proxy because a numeric hop count does
not bind to the peer at all — measured against Express 5.2.1, with the socket peer on loopback and a
single-entry x-forwarded-for, req.ip becomes whatever the caller sent. The token authenticates
the caller, which is the thing a hop count cannot express.
Verified by observation, three ways: a forged address with a wrong token records 127.0.0.1; a
forged address with no token records 127.0.0.1; the address with the correct token is recorded.
Unset is a no-op that preserves today's behaviour exactly, which is what lets this ship without a
coordinated deploy. The forwarded address reaches the activity log and nothing else — in particular
it is not wired into IpThrottlerGuard, because doing so would hand an unauthenticated attacker
a rate-limit bypass on admin credential stuffing.
9. Flow Diagrams
9.1 Route Ownership
9.2 Request Sequence — manual backup
9.3 Error Branch — restore request
10. Pagination, Sorting, Filtering, and Search
| Endpoint | Pagination Type | Default Size | Max Size | Sort Fields | Filters | Result Cap |
|---|---|---|---|---|---|---|
GET /admin/system/backups | offset (PaginationUtil) | 20 | 100 | createdAt (only field actually applied, via order) | kind, status, tier | None beyond size |
sort is accepted by the inherited QueryDto but not read by BackupAdminService.findAll — only order (asc/desc on createdAt) is applied. search is likewise accepted but not used. pagination=false returns every matching row with meta omitted from the response envelope.
No caching, no relevance scoring — this is a straightforward filtered/ordered/offset-paginated list over backup_run.
11. Caching, Jobs, and External Integrations
| Integration | Used? | Details | Source |
|---|---|---|---|
| Redis cache | Yes | system:flag=maintenance, mirrored on every engage/disengage, no explicit TTL, in-process 1s cache on top | maintenance.service.ts |
| BullMQ | Yes | backup.run/backup.prune on QueueName.BACKUP; backup.replicate on QueueName.BACKUP_REPLICATE; backup.restore on QueueName.BACKUP_RESTORE, consumed by BackupRestoreProcessor, which takes and links an inline safety dump then spawns restore-runner.ts — see 8.9a | see the backend doc's Section 9 |
| External process (not a network API) | Yes | pg_dump (dumps), pg_restore (restores live via restore-runner.ts; also --list/-f - format-check and schema-render an uploaded archive), tar, rsync, ssh — all spawned as local subprocesses, not HTTP calls | backup-artifact.service.ts, backup-replication.service.ts, workers/restore-runner.ts, backup-upload-admin.service.ts |
| Direct database queries (not a subprocess) | Yes | restore-runner.ts's verifyRowCounts() opens its own pg Client against BACKUP_ADMIN_DATABASE_URL and runs SELECT count(*) per manifest table, separate from pg_restore itself | workers/restore-runner.ts |
| MongoDB | Yes (download only) | AuditLog.create() — the one non-fire-and-forget write in the module | backup-download-admin.service.ts |
13. Mandatory Deep API Documentation Pack
13.1 Route-by-Route Completeness Matrix
| Route | Controller Method | DTOs | Service Method | Guards | Permissions | Cache | Jobs | DB Touches | Errors | Tests | Documented? |
|---|---|---|---|---|---|---|---|---|---|---|---|
GET /admin/system/backups | findAll | FetchBackupsDto | BackupAdminService.findAll | JWT+Role+IP | Backup_READ | N/A | N/A | backup_run (read) | 400 | backup-admin.service.spec.ts | Yes |
POST /admin/system/backups | create | CreateBackupDto | .create → BackupRunCreateService.createRun | JWT+Role+IP | Backup_CREATE | N/A | backup.run (outbox) | backup_run (write), outbox_events | 503/409/400 | backup-run-create.service.spec.ts | Yes |
GET /admin/system/backups/settings | getSettings | N/A | .getSettings | JWT+Role+IP | Backup_READ | N/A | N/A | backup_run (read, health) | none | backup-admin.service.spec.ts | Yes |
PUT /admin/system/backups/settings | updateSettings | UpdateBackupSettingsDto | .updateSettings | JWT+Role+IP | BackupConfigure_UPDATE | N/A | N/A (may reload() cron in-process) | none (writes settings.json) | 409/400 | backup-settings.service.spec.ts | Yes |
GET /admin/system/backups/{publicId} | findByPublicId | BackupParamsDto | .findByPublicId | JWT+Role+IP | Backup_READ | N/A | N/A | backup_run (read) | 404 | — | Yes |
PATCH /admin/system/backups/{publicId} | updatePinned | BackupParamsDto, UpdateBackupDto | .updatePinned | JWT+Role+IP | Backup_UPDATE | N/A | N/A | backup_run (write) | 404 | — | Yes |
DELETE /admin/system/backups/{publicId} | remove | BackupParamsDto | .remove | JWT+Role+IP | Backup_DELETE | N/A | backup.prune (outbox) | outbox_events | 404/409 | backup-retention.service.spec.ts | Yes |
GET /admin/system/backups/{publicId}/download | download | BackupDownloadParamsDto | .prepareDownload | JWT+Role+IP | BackupDownload_READ | N/A | N/A | backup_run (read) | 409/410/500 | backup-download-admin.service.spec.ts | Yes |
POST /admin/system/backups/upload | upload | BackupUploadDto | .upload | JWT+Role+IP | BackupUpload_CREATE | N/A | N/A (synchronous) | backup_run (write) | 400/409/503 | backup-upload-admin.service.spec.ts | Yes |
POST /admin/system/backups/{publicId}/restore | restore | RestoreParamsDto, RestoreBackupDto | .requestRestore | JWT+Role+IP | Backup_RESTORE | N/A | backup.restore (outbox) → BackupRestoreProcessor takes an inline safety dump, then spawns restore-runner.ts | backup_restore (write), outbox_events, backup_run (inline pre_restore_safety insert+update via BackupRunHandler) | 503/409/400/410 | workers/backup-restore-wiring.spec.ts | Yes |
GET /admin/system/restores/{publicId}/live | live | RestoreParamsDto | .getLiveState | JWT+Role+IP | Backup_READ | N/A | N/A | none (file only) | 410 | — | Yes |
POST /admin/system/restores/{publicId}/force-clear | forceClear | RestoreParamsDto | .forceClear | JWT+Role+IP | Backup_RESTORE | N/A | N/A | backup_restore (write) | 410 | — | Yes |
GET /admin/system/maintenance | get | N/A | .get | JWT+Role+IP | System_READ | Redis mirror | N/A | none | none | — | Yes |
PUT /admin/system/maintenance | set | UpdateMaintenanceDto | .set | JWT+Role+IP | System_UPDATE | Redis mirror | N/A | none | none | maintenance.service.spec.ts | Yes |
GET /system/maintenance | getStatus | N/A | MaintenanceCustomerService.getStatus | IpThrottlerGuard only, @Public() | N/A | Redis mirror (read) | N/A | none | none | — | Yes |
13.2 Request/Response Exhaustiveness
Covered per-endpoint in Section 8 — minimal and full request bodies, success responses, and every declared error code are shown for each mutating route. The public/guest row of the standard checklist applies to exactly one route, 8.15, whose request/response examples are shown there; every other route requires superadmin-level permissions.
13.3 API Diagram Pack
See Section 9 for route ownership, request sequence, and error-branch diagrams. A dedicated auth/permission-flow diagram is omitted as redundant: every route in this module uses the identical three-guard chain (JwtAuthGuard, RoleGuard, IpThrottlerGuard) with only the @Permissions(...) argument varying, fully enumerated in Section 4.
13.4 Consumer Integration Notes
| Consumer | Required Knowledge | Failure Handling | Contract Stability |
|---|---|---|---|
| Admin panel | Every route requires superadmin; regular admin accounts will see 403 on all of them, including maintenance | Show the error code, not just the HTTP status — BACKUP_* codes are specific | Stable |
| Admin panel — restore screen | POST .../restore returns 202, not a completed restore. Poll GET .../live with backoff and treat network failures as expected, never as "failed" | stage genuinely advances (queued → safety_dump → draining → restoring → verifying → completed/failed) as BackupRestoreProcessor and restore-runner.ts run. Only reconciling remains declared but never assigned. safetyBackupPublicId is populated from stage:"safety_dump" onward — surface it in the UI as the operator's way back. | Stable for accept/poll/force-clear, and for the safety-dump/verification behavior itself; verification is row-count only, not content-level — see 8.9a. |
| Admin panel — upload screen | POST .../backups/upload is synchronous — 201 means the archive is already catalogued, not queued | All three multipart parts are required; surface BACKUP_UPLOAD_SIGNATURE_INVALID distinctly from BACKUP_CHECKSUM_MISMATCH (wrong deployment's key vs. a corrupted transfer) | Stable |
| Admin panel — settings page | GET/PUT .../backups/settings back a dedicated settings screen; PUT is a full-replace with the version read from the last GET | Stale version on save is 409 BACKUP_SETTINGS_CONFLICT — re-fetch and retry, do not silently overwrite | Stable |
| Storefront (public) | GET /api/system/maintenance is the uncached, unauthenticated signal the storefront's edge middleware asks on every request it serves — Next's data cache would otherwise keep serving stale 200 pages during a restore | No auth to fail; only 429 from PUBLIC_HIGH_FREQUENCY is possible | Stable |
| QA | The four retention floors (pinned / newest-overall / newest-in-bucket / most-recent-restore's-safety-dump) make certain deletes always fail with 409 regardless of permission — this is intended, not a bug | — | Stable |
| Internal service | None of these endpoints are intended for service-to-service calls; there is no service-token surface in this module | — | N/A |
13.5 API Tradeoffs and Rationale
| Decision | Chosen Behavior | Alternatives Considered | Why This Tradeoff | Risk | Mitigation |
|---|---|---|---|---|---|
| Restore accept vs. synchronous execution | 202 immediately, execution genuinely async via a detached subprocess | Synchronous restore in the request handler | A restore can take minutes; holding an HTTP connection open that long is fragile, and pg_restore --clean needs to survive the API process itself being redeployed mid-restore | The client must poll GET .../live rather than trust the accept response; no built-in push notification on completion | force-clear is the escape hatch when polling reveals a stuck restore |
| Download rate limit keyed by user, not IP | keyStrategy: "user", 3/hour | Default IP-keyed | An IP-keyed cap on a full-database-exfiltration endpoint is bypassed by rotating source IPs with a stolen token | — | — |
| Upload authenticity via keyed HMAC | An archive signed by a key this deployment holds and generated | Accept any pg_dump with a client-supplied checksum | A checksum authenticates nothing when the uploader supplies both the file and the hash in the same request | Operators must provision and protect one more secret, shared across the deployments that need to move backups between each other | 503 BACKUP_UPLOAD_SIGNING_NOT_CONFIGURED if the key is unset — fails closed rather than silently accepting unauthenticated archives |
Settings full-replace PUT with version | Whole-object write with optimistic concurrency | Partial PATCH per field | Backup policy fields interact (storageMode needs remoteReplicationEnabled); a partial write could leave an inconsistent combination mid-edit | Client must always read-then-write the whole object | 409 BACKUP_SETTINGS_CONFLICT on stale version |
| Settings response includes read-only environment fields | restoreEnabled, pgDumpPath, etc. surfaced but not editable via this DTO | Omit them entirely | Visibility without editability — an operator should be able to see why restore is armed without being able to arm it from the UI | None — fields are documented as read-only in the DTO's own comments | — |
| Pre-restore safety dump taken inline, synchronously, before the runner spawns | BackupRestoreProcessor awaits BackupRunHandler.run() directly rather than enqueuing it | Enqueue the safety dump like any other backup.run job | Enqueuing returns immediately; the runner could start draining connections and running pg_restore --clean while the safety dump was still mid-snapshot, letting the two race | The restore job is blocked on the safety dump's own duration before pg_restore even starts | If the safety dump is ever slow enough to threaten BULL_QUEUE_BACKUP_RESTORE_ATTEMPTS/timeout budgets |
| Public maintenance-status route exempted from the guard it reports on | Literal-path exemption in MAINTENANCE_EXEMPT_PATHS | Block it like every other route during maintenance | The storefront's only way to discover a maintenance window is this endpoint; blocking it during the exact window it exists to report on would defeat its purpose | Discloses that the store is offline to anyone — judged not sensitive, since every other route already answers 503 | — |
13.6 API Change Impact
| Change | Affected Consumers | Backend Impact | Data Impact | Migration Needed? | Compatibility Plan |
|---|---|---|---|---|---|
backup_restore.verification (structured, not just row-count) is wired in | Admin panel restore screen | No RestoreLiveStateDto field currently exposes it; a new field would be additive | backup_restore.verification jsonb will begin being written | No schema migration — the column already exists | Additive only if/when exposed through the DTO |
BACKUP_DIR becomes admin-editable | Deployment runbook | Would remove a boot-time safety check | — | N/A (deliberately not planned — see 6.6) | Not recommended |
14. Zero-Omission API Checklist
- Every controller route is documented (15 methods across 6 controllers, all 15 route templates cross-checked against
structure.baseline.json, including the one public route). - Every parent route prefix and runtime URL is documented.
- Every DTO field, enum, default, and validator is documented.
- Every response field, nullable field, and server-computed field is documented.
- Every auth, guard, and permission is documented.
- Every success and failure branch is documented, including what the restore path still does not do (safety dump, verification).
- Every database read/write, queue job, and external call is documented.
- Every mutating route has example requests and responses.
- Route-ownership, sequence, and error-branch diagrams are included.
- Every tradeoff and compatibility risk is documented, including the remaining restore-safety gap.
- Links to backend and features/flows docs are present.
15. Integration Checklist
- Every route from controllers is documented.
- Every DTO field is documented.
- Every enum value is documented.
- Every response envelope is documented.
- Every error code is documented.
- Every auth guard and permission is documented.
- Every queue job and external call is documented.
- Every diagram matches the current code, including the restore queue's real consumer.
- The API doc links to backend and features/flows docs.
See Also
- Backend doc:
/docs/developer/backup/backend - Features and flows doc:
/docs/developer/backup/feature - TDD: not present for this module at time of writing.