Notification Backend Documentation

The platform carries two delivery models over one word, and almost every mistake made in this area comes from conflating them.

Anything addressed to a person fans out on write: notification_event records the business fact, notification_recipient records each individual it concerns, and notification_delivery records one attempt per channel per provider target. That chain is what this module owns.

Operational events for administrators fan out on read: one row per event in the notification table carrying the permission code a viewer must hold, filtered at query time against the caller's active role. That is the notification-feed module, and it is a different feature that happens to share a noun. Neither is a special case of the other, and one table cannot carry both without a discriminator every query then has to remember.

1. Documentation Evidence

2. Backend Scope and Boundaries

Owns

• The public send interface — NotificationService.send(tx, input) — that every other module calls to tell somebody something.
• The notification vocabulary: channels, categories, priorities, delivery statuses, audience kinds, push platforms and token invalidation reasons.
• The template registry in code, and the operator override table that layers on top of it.
• Audience resolution: turning an audience specification into people, one bounded keyset page at a time.
• Preference resolution: the four-step order that decides whether a given channel is delivered for a given person.
• The delivery state machine, its lease, its retry budget, its reaper and its backoff.
• The four channel implementations — email, sms, push, in_app — and their provider adapters.
• Push token registration, invalidation and per-user capping.
• The consumer notification centre, the consumer preference matrix and consumer device registration.
• The admin surfaces: template overrides, delivery history, the channel dead-letter queue with replay, and cancellation of a scheduled event.
• Retention for both delivery models plus invalidated push tokens.
• Operational alerts to a fixed support mailbox on QueueName.NOTIFICATION_OPERATIONAL.
• The per-user realtime publish that lets an open session update its badge without polling.

Does Not Own

• The transactional outbox. OutboxService.enqueue and OutboxDispatcherProcessor live in apps/api/src/modules/outbox/. This module writes outbox rows and never calls queue.add() on a send path.
• The admin operational feed. notification + notification_read and their routes belong to apps/api/src/modules/notification-feed/. This module's only contact with that table is the retention sweep, which had no other owner.
• Identity and liveness. users, students, guardians, staff, user_role and role are read by the audience resolvers and never written.
• Email transport. packages/email resolves the from-address, renders preview text and talks to Resend; EMAIL_CLIENT is provided by apps/api/src/modules/notifications/. EmailChannelProvider adapts it rather than reimplementing it.
• SMS and push transport. packages/sms and packages/firebase own the HTTP calls, the response parsing and the fixed error tables.
• Authentication. JwtAuthGuard and RoleGuard are applied here but implemented in apps/api/src/modules/auth/ and apps/api/src/common/authorization/.
• The dead-letter writer. JobFailureRecorderService is a shared common/jobs service; this module supplies the payload_ref and nothing else.
• The secrets themselves. Auth mints a reset token or an OTP and stashes it; this module resolves a reference at send time and never learns how the value was produced.

Source of Truth

3. Module Composition

The module is split into a service half and a worker half, and the split is load-bearing. Importing a module that declares a @Processor starts a BullMQ worker. Every module that wants to send a notification imports NotificationModule; if the processors lived there, every such import would start a duplicate set of workers, and the fast runtime profile — which deliberately runs none — could not opt out.

Each consumer leaf is registered in MOBILE_CHILDREN as a concrete leaf, never an aggregate: RouterModule.register() does not recurse, so listing an aggregate mounts its controllers at /api/<thing>/... instead of /api/mobile/<thing>/..., silently and with no error.

EmailChannelProvider, SecretReferenceService, DeliveryRecorderService, TemplateRendererService, PreferenceResolverService, AudienceResolverService, ChannelSendService and JobFailureRecorderService are all exported as well as provided. A provider in providers but not exports resolves fine inside its own module and dies at InstanceLoader in the consumer — naming the consumer rather than the module that failed to export it — and pnpm build exits 0 on it, because it is a runtime graph error and not a type error. NotificationProvidersModule is re-exported so the worker module receives the same provider instances: an FCM app initialised twice under one name throws, and a second mock would record into an array nobody reads.

4. File and Directory Map

apps/api/src/modules/notification/
  notification.module.ts
  notification-worker.module.ts
  shared/
    notification.service.ts
    notification.types.ts
    notification.constants.ts
    audience-resolver.service.ts
    preference-resolver.service.ts
    template-renderer.service.ts
    delivery-recorder.service.ts
    secret-reference.service.ts
  templates/
    template-registry.ts
  channels/
    channel-send.service.ts
    email.provider.ts
    notification-providers.module.ts
  workers/
    notification-fanout-queue.processor.ts
    notification-fanout.processor.ts
    notification-channel.processors.ts
    notification-reaper.processor.ts
    notification-backfill.processor.ts
    notification-retention.processor.ts
    notification-credit.processor.ts
    notification-operational.processor.ts
    notification.scheduler.ts
  admin/
    notification-admin-aggregate.module.ts
    template/   { controller, dto, module, service }
    history/    { controller, dto, module, service }
    dead-letter/{ controller, dto, module, service }
    event/      { controller, dto, module, service }
  customer/
    notification-centre/      { controller, module, service, dto/ }
    notification-devices/     { controller, module, service, dto/ }
    notification-preferences/ { controller, module, service, dto/ }
    realtime/
      notification-realtime-publisher.service.ts
  __tests__/
    notification-fanout.int.spec.ts
    channel-send.int.spec.ts
    notification-retention.int.spec.ts
    notification-operational.int.spec.ts

apps/api/src/modules/notification-feed/
  notification-feed.module.ts
  notification-feed.controller.ts
  notification-feed.service.ts
  dto/notification.dto.ts

5. Data Model

5.1 Schema Source

packages/db/src/notification/
  notification-contract.ts        the vocabulary every consumer shares

packages/db/src/schema/notification/
  index.ts
  notification-event.ts
  notification-recipient.ts
  notification-delivery.ts
  notification-preference.ts      three tables: set, channel, category
  notification-push-token.ts
  notification-template.ts

packages/db/src/schema/
  notifications.ts                the ADMIN OPERATIONAL FEED: notification + notification_read

packages/db/src/schema/jobs/
  job-failures.ts                 shared dead-letter table, read by the replay screen

The vocabulary lives in packages/db/src/notification/notification-contract.ts rather than beside the tables because three things need the same lists across package boundaries: packages/db generates the CHECK constraints from them, apps/api builds compile-time unions from them, and the queue payload types reference them. A package must never import from an app, so the lists live in the one package both already depend on. Generating each constraint from its constant makes drift impossible — a status that does not appear in the list cannot be written by code that type-checks, and a status that does appear is in the constraint by construction.

Vocabulary constants

There is deliberately no created delivery status. A delivery row does not exist before it is queued: the fan-out worker writes queued and queued_at in the same INSERT. A created default with no writer for created -> queued strands every delivery in the system, because the channel worker's claim is WHERE status = 'queued', which matches nothing — and a zero-row claim is indistinguishable from "another worker holds it", so every job reports success having sent nothing.

5.2 Tables

notification_event

One row per business event that somebody should be told about. Written by NotificationService.send() inside the caller's transaction.

Why the claim and the completion are two columns. One column cannot be both. Used as a claim it must be set before the work, when the recipient count is unknown; used as a completion marker it is set after, leaving no claim and letting two workers fan the same event out concurrently. With one column, a crash after batch three of six left the event looking complete with half its recipients missing, matched by no sweep, permanently under-delivered.

Why variables must never carry a secret. A rendered password-reset email embeds a live single-use token. This column is retained and readable by anyone holding NotificationHistory_READ, which is not a superadmin-only permission — so a token here turns an operational read permission into account takeover: list the history filtered to auth.password_reset, copy the token, complete the reset as that user. Security kinds pass { secretRef: "<type>:<publicId>" } instead. The CHECK is a backstop, not the control.

notification_recipient

One row per (event, person). The unit a notification centre lists and a person marks read.

Why user_id is SET NULL and not CASCADE. A cascade would erase, along with the person, every record that a security notification was ever sent to them — at exactly the moment that record is most likely to be needed, because production deletion is soft and a hard delete fires only on a genuine erasure request. So the person goes and the record stays, identified only by a snapshot of an id that no longer resolves. What survives is: that a notification of a given kind was sent, to a subject now identified only by an orphaned uuid, and when. What does not survive is anything personal — an erasure hook nulls destination_hint, rendered_title and rendered_body on every delivery beneath the row in the same transaction, because those carry a masked address and, for kinds that persist their rendering, the pupil's name. This is a deliberate trade; if total erasure is ever required, this becomes CASCADE and the audit evidence goes with it.

Why audience_role_id exists at all. The notification centre filters on it against the caller's active role, so a teacher who is also a parent, viewing as Guardian, does not see staff-audience notifications in the same list. Without it the centre's only predicate is user_id = actor.id, and a dismissed teacher who is still a parent keeps a live users row — so they would keep reading disciplinary and roster notifications after dismissal. ON DELETE SET NULL rather than the default NO ACTION, because a NO ACTION key makes role deletion impossible once any recipient references it.

The uniqueness on (event_id, user_public_id_snapshot) is declared as a table unique() and not a uniqueIndex. drizzle-kit emits every ADD CONSTRAINT ... FOREIGN KEY before every CREATE INDEX, so a composite key whose target is only an index fails on a clean database with 42830 — while a hand-ordered probe passes, because the probe uses dependency order and the migration uses generator order.

notification_delivery

One row per (recipient, channel, provider target). The delivery state machine, and the highest-growth table in the design.

Two partial uniques, exact complements, both excluding replays.

notification_delivery_single_target_key
  UNIQUE (recipient_id, channel)
  WHERE provider_target_id IS NULL AND replay_of_delivery_id IS NULL

notification_delivery_multi_target_key
  UNIQUE (recipient_id, channel, provider_target_id)
  WHERE provider_target_id IS NOT NULL AND replay_of_delivery_id IS NULL

provider_target_id IS NULL and IS NOT NULL partition the space, so every non-replay row is covered by exactly one and none by both. Replay rows fall outside both on purpose — that is what lets a replay of a dead push delivery insert at all, since it shares (recipient, channel, token) with the row it replaces. A double replay is prevented one level up, by NotificationFailureService compare-and-setting job_failures.replayed_at in its own UPDATE before ever reaching this insert.

Why push fans out per token. One row cannot hold two devices with different outcomes: token A returning UNREGISTERED and token B delivering is two statuses and two provider message ids, and collapsing them discards exactly the per-target information the SMS adapter is written to preserve.

Why the sent_at CHECK is channel-scoped. in_app is written inline by the fan-out worker. There is no send step, so it legitimately reaches delivered with sent_at NULL. The unscoped form of that implication rejects the row — and because the fan-out writes recipient, delivery and outbox rows in one transaction, the rejection aborts the entire batch.

Why the skipped CHECK is an explicit IN list and not LIKE 'skipped_%'. A backslash inside a drizzle sql template is a JavaScript NonEscapeCharacter, so the cooked string Postgres receives contains a bare _ — which is LIKE's single-character wildcard, not a literal underscore. The constraint would still have looked right under \d+ and would still have passed a probe written from the same assumption.

Why notification_delivery_recipient_idx is not redundant. Both uniques are partial, and Postgres can use a partial index only where the query implies its predicate. The notification centre's EXISTS, the unread count, the retention join and the ON DELETE CASCADE referential scan all filter on recipient_id alone and mention neither provider_target_id nor replay_of_delivery_id — so without this index none of them has one, on the largest table here.

notification_preference_set

The concurrency token for a person's whole preference set.

A "turn off email everywhere" write touches one row per category, so two devices saving at once interleave into a mixed state with no error. A version on the set is the smallest thing that makes the write atomic from the client's point of view. A person who has never saved has no row here: GET reports version: 0 for that state and the first PUT sends 0 and upserts at 1. Absence is a real state and gets a real token, rather than a row lazily created on read — which would make a read a write.

notification_channel_preference

The global per-channel switch — "no email at all".

Implementing the global switch as one row per category would materialise a list, which is the same snapshot defect the audience specification exists to avoid one table over: a user mutes email in March, a transport category ships in June, that category has no row, absence means the code default, and they start receiving transport email having explicitly turned email off. Two levels resolved in order has no such state, and needs no sentinel category — a magic '__all__' value would have to be permitted by the CHECK, special-cased by the resolver, and remembered by every category added later.

notification_preference

The per-category override.

Absence means the category's code default, so the table stays proportional to real choices rather than to categories times channels times users. Enforcing the category vocabulary at one end and not the other would let an event persist in a category no user is able to express a preference for — the event delivers to nobody through an undefined default, and the one control that could have surfaced it is rejected by this very CHECK. Neither this table nor the channel table can override an unsuppressible kind: that property lives on the registry entry, not on any column here and not on anything a caller passes.

notification_push_token

A push credential, one row per token per device per user.

Why not a column on user_device. Three reasons, each independently sufficient. sessions.user_device_id foreign-keys user_device, so a push credential on that table sits directly in the session join path. A token rotates independently of the device — FCM invalidates asynchronously and the app re-registers — so one device produces many tokens over its life, and a single column would overwrite the invalidation history that answers "why did this parent stop receiving notifications". And web push has no user_device row at all, because a browser has no installation.

Why a token arriving for a second user is invalidated, not reassigned. Reassigning mutates the one row and destroys the record that the first user ever held it. Worse, reassignment on its own is a denial of service: anyone who learns a victim's token — a shared school tablet, a resold device, a sibling — can POST it and silently stop the victim's push, including the security notifications that are otherwise unsuppressible. So registration invalidates the existing row with reason replaced and inserts a fresh one, and the caller must present the token on their own session.

The unique on token is partial on is_active, so an invalidated token's history survives alongside the live row that replaced it. A full unique would force reassignment-in-place, which is the behaviour the table exists to avoid. The invalidation CHECK is biconditional, because a one-directional form permits (is_active = false, invalidated_at = NULL) — "deactivated, cause and time unknown" — which is precisely the state the invalidation columns exist to prevent, and which the partial index would then hide from every read.

notification_template

An operator override of a template. Not the template itself.

This table is not seeded, and empty is the correct state. Seeding the registry into it would create the rename-then-duplicate hole a bare onConflictDoNothing produces: an operator renames a seeded row, the next deploy's insert collides with nothing, and a duplicate appears silently. The table holds only rows somebody deliberately created.

Why version is not a trigger and not a timestamp. Not a timestamp, because updated_at is millisecond-resolution and two commits inside one millisecond produce the identical token, so a stale write passes its own check. Not a trigger, despite a trigger being the usual answer, because drizzle-kit does not generate them: a hand-written trigger appears in neither the snapshot nor the .down.sql, DROP TABLE removes the trigger and not its function, and the second up of an up -> down -> up then dies on 42723 creating a function the down never dropped. So the application increments it, and zero rows returned is the conflict.

The body CHECK uses btrim, which strips spaces only — a body of just a tab or a newline still passes. That is deliberate: the constraint exists to catch the empty string a form submits, not to be a whitespace validator, and a POSIX class here would invite the backslash-in-a-template trap for no gain.

notification and notification_read — the admin operational feed

A different delivery model, owned by notification-feed. Documented here because this module's retention worker is the only thing that deletes from it.

notification_read is (notification_id, user_id) primary-keyed with a read_at default of now(), plus notification_read_user_idx on user_id; both foreign keys cascade. It is a join table rather than a read_at column, because one notification row is seen by everyone holding its permission and each of them reads it — or does not — separately. A column would make "read" a property of the event, so the first admin to open the bell would clear it for the whole office. Absence means unread.

Why fan-out on read here. One row per event, not per recipient. Fan-out on write would mean resolving every holder of a permission at publish time and inserting a row each: a query per event over user_role and role_permission, rows that multiply by staff count, and — the reason it is actually wrong — a snapshot, so someone granted the permission an hour later would never see the event and someone who lost it would keep seeing it.

job_failures — the shared dead-letter table

Not owned here, but read and written by the channel workers and the replay screen. The columns that matter to this module: public_id, queue_name, job_name, job_id, payload_ref (jsonb, Record<string, string>), actor_id, attempts, last_error, replayed_at, replayed_by, replay_job_id, failed_at. A notification channel failure writes payload_ref as { notificationDeliveryPublicId, channel }; an operational-email failure writes { aggregateType, aggregateId, eventType }, omitting absent keys rather than writing nulls, because payload_ref is jsonb an operator reads by eye and a row of nulls is indistinguishable from a row nobody populated.

payload_ref, never the payload. A rendered notification payload carries the message body, and a password-reset body embeds a live single-use token — storing it behind a read permission turns that permission into account takeover. The reference answers every operational question without carrying a secret.

5.3 Relationship Diagram

The two clusters do not touch. NOTIFICATION and NOTIFICATION_READ are the read-time fan-out; everything else is the write-time one.

6. Services and Responsibilities

6.1 NotificationService

The public interface. Every module that needs to tell somebody something calls this and nothing else.

It writes exactly two rows and returns. It resolves no audience, renders no template, reads no preference and contacts no provider — which is what makes it safe to call from inside a business transaction of any size. A three-thousand-parent announcement resolved here would hold the caller's transaction open for the duration and put a notification concern on the critical path of the thing being notified about.

executor must be a transaction, not the root handle. Passing the root handle compiles and appears to work, and silently gives up the atomicity the method exists to provide: the event row would commit independently of the business write it describes. There is no way to detect that from inside, which is why it is stated loudly rather than checked.

Validation order, all before any write:

1. assertKnownKind — the kind exists in the registry, or 400 NOTIFICATION_TEMPLATE_NOT_FOUND.
2. assertChannelsAreBuildable — every requested channel has a builder for that kind, or 400 NOTIFICATION_CHANNEL_UNKNOWN. The alternative is a delivery row that terminates skipped_no_template — correct, but invisible to the caller, who asked for a channel and got silence. A caller's mistake belongs in the caller's 400.
3. assertAudienceIsWithinCaps — an explicit list at most NOTIFICATION_MAX_EXPLICIT_RECIPIENTS (500), a compound between 1 and NOTIFICATION_MAX_COMPOUND_MEMBERS (10), or 400 NOTIFICATION_AUDIENCE_TOO_LARGE. compound is depth-1 by type and unbounded in breadth without this: a caller could send ten thousand all_users members, depth-1 and type-legal, expanding to ten thousand keyset walks over every person in the school.
4. assertActionUrlIsSafe — https: or an app-relative path beginning with a single /, or 400 NOTIFICATION_ACTION_URL_SCHEME_FORBIDDEN.
5. buildDedupeKey — enforces the kind's dedupe policy.

The insert uses onConflictDoUpdate with a no-op SET, not onConflictDoNothing. A bare onConflictDoNothing().returning() yields zero rows on the dedupe path, leaving the method with no event id for the outbox row it must write in the same transaction, and no id to return. The no-op update makes RETURNING always produce the row, whether it was inserted now or already existed. created is then derived as row.fanoutClaimedAt === null.

The dedupe key is hashed and bounded. A key built from a title or a path overflows varchar(64) and raises 22001 inside the caller's own business transaction, turning a notification concern into a failed write of the thing being notified about. Hashing also normalises, so "foo " and "foo" cannot become two events. The material is NUL-separated so ("a","bc") and ("ab","c") cannot hash alike, and the separator is written as an escape rather than a literal byte because a raw NUL survives an editor round-trip only by luck.

The policy is the kind's, not the caller's. collapse folds only domain facts, so a repeat is a duplicate. repeatable folds the caller's requestId, so a second password reset for one user is a second event. Getting this backwards in either direction is a real defect: collapse on a reset denies account recovery permanently, repeatable on a reminder sends it twice.

6.2 AudienceResolverService

The spec is re-evaluated for every page rather than materialised once, which is the whole reason the audience is stored as a spec. A pupil enrolled between page two and page five is included; one who left is not.

Keyset, never offset. Ordered by users.id and filtered u.id > cursor. An offset walk over an audience being written to re-reads and skips rows as the underlying set shifts.

Liveness lifts as a pair. users.deleted_at IS NULL is applied in the outer query, and each profile table's own deleted_at inside its fragment. A liveness filter over a join lifts together or none does — the split version has shipped twice in this repository, and the symptom was a screen permanently empty with no error anywhere. That is still only the first pass: the channel workers re-check liveness at send time, because an event scheduled twenty-four hours out fans out before a deletion an hour later.

can_login is deliberately absent. It means "may authenticate" and nothing more. A pupil with no portal account still has a name on a register and a guardian who must be told things, so gating delivery on it made every such person invisible to every audience — and most primary-age pupils are exactly that person. Deletion decides who is reachable; sign-in access decides who can log in.

A compound audience is a union deduplicated by users.id, unioned in SQL rather than merged in memory — resolving each member separately would break the keyset, because each member would have its own cursor.

fragmentFor is a Record over the audience kind, so a new kind with no resolver is a compile error. The runtime symptom of a missing branch is a fan-out that completes having notified nobody. The CHECK constraint on audience->>'kind' guards the data path; this guards the code path, and neither alone is sufficient.

The guardian audiences resolve through student_guardian, never from a caller-supplied user list. A guardian audience that accepted arbitrary user ids would let a caller address one family's notification to another.

roleForScope is a correlated subquery rather than a join, so a person holding no matching role yields NULL — "not role-scoped" — instead of dropping out of the audience entirely. A join would silently shrink a class announcement to the pupils who happen to have been granted the student role.

uuidArray builds ARRAY[$1, $2]::uuid[] rather than interpolating the array straight into a sql template. Drizzle renders a JS array in a template as a record — ($1, $2) — and cannot cast type record to uuid[] is what comes back. Every element is still a bound parameter, so an id list can never be concatenated into SQL. An empty list yields ARRAY[]::uuid[], which is valid and matches nothing — the correct answer for an audience naming nobody.

The per-recipient overlay is { name: full_name ?? "there" }. full_name is a generated column so it is never absent for a real row, but it is nullable in the type, and a null here would render the literal text {{name}} to a parent.

6.3 PreferenceResolverService

loadForUsers runs two queries per batch, not two per person. Five hundred recipients times four channels resolved one at a time is two thousand round trips per batch.

Resolution order, and the order is the design:

1. Unsuppressible — the kind says so. Nothing below can turn it off.
2. Per-category override — the most specific thing the person said.
3. Global channel switch — "no email at all".
4. Category default, from code.

Step 1 reads a property of the kind, not a column any caller sets. A caller-settable bypass with a CHECK constraint is not a control, because the same caller supplies both operands — it would let any module send a marketing blast as system past every opt-out in the school, at real SMS cost, with the constraint satisfied. Steps 2 and 3 are in that order because a person who turned email off globally and then turned assignment email back on meant the second thing.

Category defaults:

SMS is off by default across the board except security, because it is metered and a school-wide default of "on" is a bill nobody chose. The final fallback returns false rather than asserting: if a category ever reaches the resolver without an entry, silently sending is the wrong failure and silently not sending is the safe one.

Defaults live in code and not as seeded rows, because seeding one row per person per category per channel is nine categories times four channels times every user in the school, all of it saying "the default" — and absence meaning the default lets a default change take effect for everyone who never expressed an opinion, which is the point of having one.

6.4 TemplateRendererService

loadOverrides runs once per fan-out batch. Rendering three thousand recipients across three channels with a per-render lookup is nine thousand identical SELECTs for one event.

Render order: an active override for (kind, channel, locale); if it throws or leaves a variable unresolved, log once and fall through; then the registry builder, which always exists and always compiles. The fall-through is what makes an operator's bad edit a cosmetic problem rather than a silenced password reset, and it is also why the registry is never seeded into the table — the code default has to be a real fallback, not a row somebody can delete.

Interpolation is {{name}} against a flat record. No expressions, no property paths, no function calls. An admin-editable template that can evaluate expressions is a code execution surface behind a CRUD permission. Three further properties matter:

• An unresolved placeholder suppresses the send, returning null, rather than delivering the literal text {{name}} to a parent — which would be metered, paid for, and invisible to every gate.
• The replacement is a callback, not a replacement string. String.prototype.replace with a string second argument treats $&, $', $` and $1 as substitution patterns, so a variable value containing $' would splice in the remainder of the template — an information leak driven by data.
• Email bodies are HTML, so every interpolated value is passed through escapeHtml, because a variable is the one part of a template a caller controls. Email subjects additionally pass through escapeHeader, which collapses CR and LF: escapeHtml is an HTML-body escape and does not touch them, and a newline in an interpolated subject splits the header.

A builder that throws is a code defect, not an operator's — it is logged with the stack and the channel is skipped, rather than failing the whole batch. warnOnce keys on kind:channel:locale, so one bad template is one log line and not three thousand identical ERRORs.

6.5 DeliveryRecorderService

The only writer of notification_delivery.status outside the fan-out worker's initial insert.

Every transition writes every column it owes. The table carries paired CHECKs — (claimed_at IS NULL) = (lease_expires_at IS NULL), and skipped-status implies skipped_at — so a transition that clears one half of a pair and not the other is rejected by Postgres inside a queue worker, and surfaces as a job failure rather than as anything traceable.

Every claim carries a lease. UPDATE ... SET status='processing' WHERE status='queued' is safe against a second worker and useless against a dead one. Without lease_expires_at and a reaper, a worker that dies after the provider call and before the result write leaves the row processing forever: no sweep looks at that status, the redelivered job's claim matches zero rows, and zero rows is indistinguishable from "another worker holds it" — so the job reports success. The message was sent and the record says it never was.

The reclaim increments attempts, and that is load-bearing. A reclaim that incremented nothing re-queues a persistently crashing send forever, because attempts >= max never fires. On SMS that is unbounded duplicate messages at metered cost. lease_expiry_count is kept separately from failure_count so "how often did workers die here" stays answerable apart from "how often did the provider refuse".

A non-retryable failure goes straight to dead with attempts incremented but no backoff: retrying "Not enough balance." or a dead FCM token is a guaranteed failure per attempt, and on SMS each one is a paid HTTP call.

Requeueing a backoff clears failed_at and last_error. Leaving them set would make any query written failed_at IS NOT NULL — the obvious form, given the column exists — permanently over-count failures, because a delivery that failed once and then succeeded would still carry both.

Backoff is min(NOTIFICATION_BACKOFF_MAX_SECONDS, base * 2^(failureCount - 1)), with the shift guarded above exponent 30: 2 ** 1024 is Infinity, and Infinity * base is NaN, which would write a NULL next_attempt_at and strand the row. Without the ceiling, a large attempt count means "never" — the row would be alive, never dead, and never retried.

6.6 SecretReferenceService

A rendered password-reset email embeds a live single-use token. Put it in notification_event.variables and it is stored in plaintext, retained, and readable behind NotificationHistory_READ — an operational read permission, not a superadmin one. The caller stashes the secret and stores only { secretRef }; the channel worker resolves it immediately before rendering, and the resolved values live in memory for the duration of one send.

Why Redis and not a row. The obvious design references a database row and cannot work here: the verification table stores token_hash and otp_hash, not the token — deliberately — so the raw value exists only in memory at the moment it is created and can never be re-derived from Postgres. What the vault buys: the secret is never in Postgres, so it is not in the retained history; it is never in a dead-letter row, a log line, or rendered_body; it expires on its own, so an abandoned notification does not leave a live token lying about; and it is deleted on read, so a redelivered job cannot re-render it. The cost is stated rather than hidden: if Redis is flushed between the write and the send, that one email cannot be rendered — the delivery records skipped_no_destination and the user asks again.

GETDEL, not GET then DEL. At-least-once delivery means the same job can arrive twice, and a secret that survives its first read is a live token available to a redelivery long after the first send.

Two resolution paths exist: the vault (the default, and what auth uses), and a registered resolver for a secret that can be re-derived from a row its owning module holds. A module registers one at boot; a switch here would make this service import the modules that already import it. An unregistered prefix falls through to the vault, and an unresolvable reference returns null — never a partial render. secretRef itself is dropped from the merged set so a template cannot interpolate the reference into a message. The reference is logged on failure; the resolved value never is.

6.7 ChannelSendService

Sends one delivery, for any channel.

Order: claim, liveness, secret, render, address, send, record — for email, SMS and push alike. Three near-identical workers would be three places for the liveness re-check or the secret rule to be forgotten. The claim comes first so nothing else can be doing this work; liveness comes before rendering so a deleted person's name is never rendered; the secret is resolved last before sending and never persisted.

Branches, in order:

The context join is a LEFT JOIN on users, not an inner one: user_id is nullable after an erasure, and an inner join would silently drop the row instead of letting the liveness branch record why nothing was sent.

Per channel:

• Email. A vault-supplied toEmail wins over the address on the user row, because change-email verification must reach the new address, which is not on the row yet — that is what is being verified. The override arrives through the vault, so an unverified address is never written to retained history either.
• SMS. The same override applies as toPhone, then normalizePhone. An unnormalisable number is a missing destination, not a failed send: the provider would refuse it and bill nothing, and retrying cannot change the number. The body is truncated by truncateToSegments(body, 3), which is encoding-aware — Devanagari is UCS-2 at 70 characters a segment, and a plain character count over-charges roughly twofold on Nepali text against a provider that bills per segment.
• Push. Requires a provider_target_id; the token row is re-read and must still be is_active. platform is read because FCM applies a different block per platform. actionUrl is re-derived through safeActionUrlOrNull rather than passed down, so a second caller cannot supply an unchecked URL by forgetting an argument. When the provider answers with invalidateToken, the token row is deactivated in the same call — which is what stops every future send to that device failing forever, in a school that accumulates stale tokens continuously.
• In-app. Never reaches a worker; the fan-out completes it inline.

Push reports no destinationHint: its target is a token whose public id is already on the row, and masking a token would say nothing an operator could use.

6.8 EmailChannelProvider

An adapter rather than a rewrite: packages/email already resolves the from-address, renders preview text and talks to Resend, and it is used by paths this module has not absorbed. Wrapping it keeps one implementation of "how an email is sent" while giving this module the same ProviderOutcome shape the SMS and push ports return, so the channel worker has one code path for three channels.

A mocked result is reported skipped, never sent — marking a mocked send as sent is the "looks healthy, delivers nothing" failure, and skipped_unconfigured is the state the backfill worker re-drives. A provider rejection is non-retryable, because it is a decision about this message — a bad address, a blocked domain — and retrying produces the same answer at the same cost. A throw is transport (DNS, TLS, a timeout, a 5xx) and is retryable; it is logged, because errorCode collapses every transport fault into one value and without the log a misconfigured DNS entry and a provider outage are indistinguishable on the record. The log carries the message only — never the body, and the address is masked, because a rendered password-reset email carries a live token.

EMAIL_ERROR_CODE is a fixed table: EMAIL_NO_ADDRESS, EMAIL_REJECTED (suffixed with the provider's own error name), EMAIL_TRANSPORT, EMAIL_UNKNOWN. maskEmail renders j●●●@example.com; the full set of addresses behind an operational permission is a contact-details export of every family in the school.

6.9 Provider construction — NotificationProvidersModule

Every provider is constructed whether or not its credentials exist, and isConfigured() answers the question. An unconfigured provider returns { outcome: "skipped", reason: "unconfigured" } rather than throwing, and the delivery goes to skipped_unconfigured — no attempt consumed, no dead letter, excluded from every failure metric. The path from send() down to the provider is identical in both states, which is what makes "correct the moment credentials arrive" a property of the design rather than a hope, and why the backfill worker can simply re-queue the backlog with no other change.

NOTIFICATION_REQUIRED_CHANNELS is the guard against silent degradation: a channel named there and not configured refuses to boot. It is per channel, deliberately — a single global "require every provider" flag set true in production with credentials not yet issued fails the first deploy outright, the whole API down rather than one channel degraded. A mock provider reports itself configured, so requiring a channel and mocking it in production is also a hard throw: that is the one combination that looks healthy and delivers nothing.

Outside that, a mock logs one WARN at boot saying messages are recorded and not sent, and an unconfigured provider logs one WARN saying deliveries will be recorded skipped_unconfigured and re-driven automatically. Once at boot, not once per send — a per-send warning on an unconfigured machine is a log nobody reads, which is exactly how a real failure gets buried. The admin delivery-history screen surfaces the skipped_unconfigured count as a first-class state, so the condition is visible without reading a log at all.

6.10 NotificationRealtimePublisherService

Postgres is the authoritative unread store — notification_recipient.read_at IS NULL. This stream is an enhancement that lets an open session update itself without polling, and every screen must still be correct on a plain refresh with Redis unreachable. It needs no durable record of its own, unlike the admin stream's publisher, because the fan-out worker's batch transaction has already written the recipient row before this runs.

It is called after the batch transaction commits, never inside it: a rolled-back batch must never notify anyone in real time about something that, per the durable record, never happened. And it never throws, because a throw would fail the whole fan-out batch and re-deliver notifications already recorded and, for remote channels, already queued.

The payload carries recipientPublicId, kind, category, priority and occurredAt — no title and no body. A payload rich enough to render a screen from would become a second, unversioned read API that drifts from the real one; the centre's own GET /api/mobile/notifications is that real one.

6.11 Consumer services

The centre's predicate is the only access control. These handlers declare no @Permissions() — guardians and students hold no admin-catalogue permission at all, so a permissioned handler would refuse them before any scoping ran — and RoleGuard returns true for an allowlisted handler before it ever reads request.user. So every method builds its own predicate:

1. r.user_id = actor.id — ownership.
2. r.audience_role_id IS NULL OR r.audience_role_id = activeRoleId — active-role scoping. activeRoleId is null when the session has no active role, and x = NULL is never true in SQL, so that state correctly narrows to not-role-scoped rows only, with no extra branch.
3. A staff-audience row (its role's scope_kind is neither guardian nor student) additionally requires a live staff row for that user. A dismissed teacher who is also a parent keeps a live users row, so (1) and (2) alone would keep serving them disciplinary and roster notifications after dismissal.
4. EXISTS an in_app delivery whose status is not skipped_preference or cancelled. Without it the centre lists rows whose only channels were email or SMS, and rows the user suppressed.

title and body come from the in-app delivery's rendered_title/rendered_body, correlated with ORDER BY d.id DESC LIMIT 1 so a replay's row wins. They are typed nullable because the columns are, but a row reaching this query always has them: the registry refuses at boot any kind declaring in_app without persistRendered, and a kind whose template fails to render records skipped_no_template, which predicate 4 excludes.

Order is fixed at e.occurred_at DESC, r.id DESC so two events fanned out in one transaction cannot be dropped or duplicated under offset paging. pagination=false is refused outright. markRead uses coalesce(read_at, now()) so re-reading does not move the timestamp, and zero rows is a 404, never a 403 — a 403 would confirm the row exists for someone else, or under a role the caller is not currently acting as.

Device registration never takes user_id from the body. The DTO deliberately carries no userId or deviceId field; every write is keyed on actor.id from the verified session. The three cases: same user and same token is a touch (updates last_used_at and platform), because invalidating and reinserting on every app launch would defeat the per-user cap for no reason; a token held by another user is invalidated with reason replaced and a fresh row inserted; and the cap is checked against active rows and refused rather than silently evicting the oldest, so the caller learns the limit rather than losing a device it never asked to remove. Removal is a soft invalidation with reason user_logout, not a hard delete — the table exists to keep the history that answers "why did this parent stop receiving notifications".

The preference matrix computes locked categories from the registry, not from UNSUPPRESSIBLE_CATEGORY. That constant names categories a kind is allowed to lock, not categories that are fully locked today: system permits an unsuppressible kind but currently holds none, so locking it would show the UI a switch that works for a category with nothing to switch. A category is locked only when every registered kind in it is unsuppressible. Locked entries are dropped from a PUT before persisting, because isEnabled always returns true for an unsuppressible kind regardless of any stored row, so persisting one would be a switch with no effect on delivery. Overrides are de-duplicated last-write-wins on (category, channel).

The version guard has two shapes in one transaction. version === 0 means "I read no row", so the write is an INSERT ... onConflictDoNothing that must produce a row; any other value is an UPDATE ... WHERE version = $expected that must match. Either returning zero rows is 409 NOTIFICATION_PREFERENCE_VERSION_CONFLICT.

6.12 Admin services

NotificationTemplateService.update is the one caller that computes changes by hand. The global ActivityAuditInterceptor records every mutation automatically from the handler's own @Permissions(), but it has no before-value — it never reads the row a PATCH changed. So update also calls ActivityRecordService directly with per-field changes. create and remove need no such call; the interceptor covers them.

Every admin list refuses pagination=false. These tables only grow, and there is no ceiling small enough to make an unbounded read safe to buffer at once. Every list sorts with id as the tie-break, because two rows saved in the same millisecond share a timestamp and an unstable sort under offset pagination silently drops and duplicates rows across pages.

The history service enforces its own security, not only its DTO. The SELECTs never touch notification_event.variables, never touch rendered_title/rendered_body, and never join a people table — so there is nothing to accidentally spread, because nothing carrying a name, email or secret is read into memory in the first place. failedDeliveryCount counts statuses that are neither the happy path nor any skipped_* value, which leaves exactly failed and dead.

Cancellation is a compare-and-set, not a read-then-write. The fan-out worker can be racing the same row, and a read-then-write would let a cancel appear to succeed a heartbeat after the worker already committed. On zero rows the service re-reads to report precisely why: already cancelled, or the fan-out won.

The dead-letter screen is deny-by-default. job_failures is global across every queue in the platform, including QueueName.BACKUP_RESTORE — an unfiltered read would show a clerk holding NotificationFailure_READ a database restore's failures, and an unfiltered replay would let them re-run one. REPLAYABLE_QUEUES is a hardcoded allowlist of the three channel queues, ANDed unconditionally; the channel query parameter can only narrow it. A failure from another queue is reported as a 409 refusal rather than a 404: the row exists and another screen may legitimately show it, so pretending it is absent would be a lie to an operator who can see it elsewhere.

Replay inserts a new delivery rather than re-enqueueing the original job. A channel job's payload is { deliveryPublicId }, so re-queueing the same delivery would leave two in-flight claims on one row — and the original's failed_at, last_error and provider_message_id are the evidence an operator opened the screen to read. The new row points replay_of_delivery_id at the original, which is exactly why both delivery uniques carry AND replay_of_delivery_id IS NULL. The job_failures compare-and-set happens inside the same transaction and last, so a concurrent claim rolls back the delivery and outbox inserts with it.

6.13 NotificationFeedService — the admin operational feed

The permission filter is the access control. There is no single permission that gates "notifications": each row carries the permission of the screen its event belongs to, and a caller sees exactly the rows their active role could already have read by opening that screen. One blanket permission would either gate the whole feed behind the widest of them or show every operator everything. It is the same rule the live admin stream applies at connect time, resolved the same way — from the active role, never the union of roles held — so the bell's history and its live events cannot disagree about who may see what.

visiblePermissions returns null for a superadmin active role, which the callers turn into no filter at all rather than materialising the whole catalogue into an IN (...). It returns [] for a session with no active role, and an empty list becomes the SQL literal false — inArray with an empty list is not portable and reads as "no filter" in some drivers, which would show a role with no permissions everything.

The read-state join is LEFT JOIN notification_read ON notification_id = ... AND user_id = actor.id. Joining without the user predicate would mark a row read because somebody read it. unreadCount counts across everything visible, not just the page returned, because a badge that counted the page would cap at its size. markAllRead is scoped to what the caller can see, so clearing the bell never creates a receipt for a notification they were not shown — which would otherwise silently hide it if they gained the permission later. markRead checks visibility first and answers 404 for an invisible row, exactly as it does for an absent one: the id is a uuid, and a 403 would confirm it exists.

7. Runtime Flows

7.1 Raising a notification

The outbox row goes in the same transaction. A direct queue.add() here would be exactly the defect the outbox exists to prevent: the event commits, the enqueue throws, the caller returns 200, and nothing ever fans it out. The outbox dedupeKey is the event's own public id, so a retried business transaction cannot schedule two fan-outs for one event.

Every caller in AuthEmailService swallows a failure and logs it: a notification that cannot be scheduled must not roll back the password change that prompted it. The account holder is worse off with a failed change than with a missing email, and that log line is the only channel the failure has.

7.2 Fan-out

The cursor advances inside the batch transaction. Advancing it after would mean a crash between commit and cursor-write replays the batch — idempotent for recipients, but the channel outbox rows for that batch were already committed and dispatched, so a restart writes a second set.

recipient_count is a COUNT(*), never an accumulator, because a resumed fan-out would double it. Both completion columns move in one UPDATE because a CHECK is evaluated at the end of the statement and cannot be deferred, so "write the marker now, count on the next line" is rejected and takes the whole transaction with it.

The recipient insert uses a targeted onConflictDoNothing naming (event_id, user_public_id_snapshot). A targetless call is idempotent only when the inserted column is the conflict target, which is not the case here — a redelivered batch would otherwise raise a unique violation and fail the whole job.

Per recipient, per requested channel:

A suppressed channel still gets a row. That is what makes the status reachable and what answers "why did this parent not get it" from the record rather than from a preference set that has since changed.

In-app is complete the moment it is written, rendered content and all. Inserting it queued and flipping it to delivered by a bulk update that wrote no content is the shape to avoid: every notification in the centre would carry a null title and body, so it would arrive, count toward the unread badge, and display as blank. Nothing would fail — not the fan-out, not the delivery, not a test that only counts rows.

Template overrides and locales are loaded only when in-app is actually requested: an email-only announcement should not pay for a template lookup and a locale scan on every batch.

A fan-out that resolves to zero recipients is legal — a class with no enrolments — and is logged at warn rather than passed over silently, because an operator who sent to nobody should be able to see that they did.

7.3 Channel send

The failure is thrown rather than returned quietly, so BullMQ records it and the onFailed handler can write the dead letter. Returning quietly is how a failed send becomes invisible.

The dead letter is the gap the outbox does not cover. outbox_events.status = 'dead' records a job that could not be enqueued; the moment queue.add returns, the row is marked dispatched and the outbox never hears about that job again. A job enqueued successfully and then failing while running reaches the outbox never, and would otherwise live and die in BullMQ's own failed set, which is age- and count-trimmed and is not queryable as a business record — so "the invitation never arrived" becomes unanswerable.

7.4 The recovery tick

NotificationJob.REAP runs four bounded sweeps in order.

There is deliberately no watermark. The general rule is that a bounded sweep with no cursor re-reads the same prefix forever and never audits the rest — but that rule applies to a sweep whose candidates are unchanged by examining them. These are not: each mutates the rows it finds, so every row it handles leaves the candidate set immediately. A watermark here would skip rows that legitimately reappear, and a watermark ahead of a backdated row hides that row permanently.

Sweep 3 exists because sweep 4 cannot see the gap it leaves. Sweep 4 is self-clearing because the fan-out worker's claim removes the event from its predicate — but that same claim means an event whose fan-out claimed it and then died is in neither predicate: not unclaimed, not complete, matched by nothing, half-fanned-out forever. Sweep 3 is a lease on the fan-out claim, the same shape as the lease on a delivery claim; without it the delivery lease would protect the small unit while the large one had no protection at all. The threshold is a generous ten times the delivery lease, because reclaiming a fan-out that is still working would run two workers over one event — survivable for the idempotent recipient insert, but the channel outbox rows would be written twice. On release, the fan-out resumes from fanout_cursor.

Sweep 4 keys on created_at, not scheduled_for. scheduled_for IS NULL means immediate, which is the majority of all events, and NULL never satisfies <= now() — so a scheduled_for predicate makes the sweep structurally unreachable for the common case and permanently hides any backdated row. The grace period is what stops it racing the normal path: the outbox relays within seconds, and sweeping immediately would double-dispatch every event in the system. Re-dispatch is one transaction, and the outbox's own dedupe index makes a repeat a no-op.

7.5 Backfill

Without this, skipped_unconfigured is terminal and "works the moment keys are added" holds only for future sends: every notification produced before the key arrived is permanently unsent, with no replay path, because job_failures never sees them — a skip is deliberately not a failure. A write path with a fallback and no matching read path is a 200 that delivers nothing; this is the read path.

Only channels configured now are considered — re-queueing a delivery whose provider is still absent just walks it back to skipped_unconfigured, burning a claim and a queue round trip each tick. The status change is a compare-and-set so two ticks cannot both re-queue one row, and skipped_at is cleared in the same statement or the biconditional CHECK rejects the write and takes the transaction with it. The outbox dedupeKey is <publicId>:backfill, distinct from the original enqueue's, or the outbox's own unique index would treat this as the already-dispatched first attempt and drop it silently.

The window is bounded by NOTIFICATION_BACKFILL_MAX_AGE_HOURS so adding a provider a month late does not resurrect weeks of stale announcements at somebody's expense.

7.6 Retention

Two predicates make it safe:

1. No in-flight delivery. A queued, processing or failed row means a worker may still touch it. Deleting it makes the worker's claim return zero rows — which the worker reads as "another worker holds it" and reports as success. The message is then neither sent nor recorded.
2. No unreplayed dead letter. job_failures.payload_ref is a jsonb reference and deliberately not a foreign key, so nothing cascades and nothing else would stop this deleting the delivery an operator is about to replay.

Bottom-up, not a cascade. Deleting one all_users event cascades roughly three thousand recipients and twelve thousand deliveries in a single statement, holding a long lock on the table the fan-out is inserting into and the notification centre is reading. Each level is deleted in its own bounded statement instead, and because Postgres has no DELETE ... LIMIT, each bound is a subquery. Only events whose children are all gone are deleted; a partially-deleted event survives to the next tick.

Both sweeps key on occurred_at — the column an operator reasons about, and the one indexed for it. created_at would delete a long-scheduled event before it ever fired.

Token pruning and the admin feed sweep run even when no aged event is found, and that early-return path is the interesting one. These are independent retentions that happen to share a tick: a school can go a whole holiday without an expiring notification while its pupils reinstall the app steadily, and every reinstall invalidates a token. Returning early without pruning meant UNIQUE (token) WHERE is_active grew monotonically for as long as the notification tables happened to stay inside the window — and nothing reported it, because the job kept completing successfully.

The admin feed had no sweeper at all. notification grew for the lifetime of the deployment, one row per operational event forever, and no gate could see that because an ever-growing table is not an error. notification_read cascades from it, so the read receipts go with their notification and need no statement of their own.

7.7 SMS credit check

Aakash answers "Not enough balance." with HTTP 200 and error: true, and that failure is non-retryable — the delivery goes straight to dead. Without this check, the first symptom of an exhausted balance is a school-wide SMS outage discovered by a parent who never got their OTP, with a pile of dead deliveries behind it. Reading the balance costs one HTTP call a day and turns that into a warning with days of lead time.

It returns early and quietly when the provider is unconfigured (saying so once a day would be noise on every development machine) or when the provider exposes no credit API at all (a mock has no balance, and absence is not a failure). A throw is swallowed and logged at error: a credit check that throws must not fail a maintenance tick and take the other sweeps with it, so that log line is the only channel the failure has.

7.8 Operational email

Everything on the channel queues resolves a notification_delivery row: a recipient, their preferences, their locale, their in-app copy, a lease the reaper can reclaim. An internal notice has none of that. It is addressed to a support mailbox, not to a person — nobody can opt out of it, it does not belong in anybody's notification centre, and there is no user whose row could carry it. Forcing it through the recipient machinery would mean inventing a fake user to hang it on. So it shares the one thing that matters — EmailChannelProvider, and with it a single Resend configuration, a single isConfigured() rule and a single place a provider is ever swapped — and skips the rest.

The payload is rendered at enqueue, the opposite of every other notification: an internal notice is three label/value rows about a row written in the same transaction, so rendering it at the producer keeps the notification workers from having to learn about feedback and reviews to build one. The trade is that editing a template does not change a notice already enqueued — for something with a lifetime measured in minutes, that is the right side of it.

An unconfigured provider completes the job with a warning rather than throwing. Throwing would dead-letter every internal notice raised before the key arrived, which is noise about a known state rather than information. The cost is stated rather than hidden: unlike a delivery row there is nothing for the backfill worker to re-drive, so a notice skipped this way is not sent later. The admin toast still fires — it is relayed at dispatch, independently of this send — so the operator does learn the thing happened; only the durable email copy is lost.

An unknown job name on this queue throws rather than returning quietly, even though the queue carries exactly one name: the silent alternative is a job that completes having done nothing.

7.9 Consumer and admin request flows

The 404-not-403 rule is applied consistently on every id-addressed consumer route and on the admin feed: public ids are uuids, and a 403 confirms that a row exists.

8. Caching

This module has no read-through cache, and that is deliberate. Every read surface is either per-caller (the centre, the preference matrix, the feed) or an operator screen whose value is being current (history, dead letters). A cache over a per-caller predicate is a cache with a cardinality of one entry per person, and a cache over an operator screen trades correctness for a saving nobody asked for. The CacheKeyUtil / Redis-cache pattern used elsewhere in the platform is therefore absent here on purpose.

Redis is used for exactly two things, neither of them a cache:

Two in-memory caches exist inside a process and are not shared:

• TemplateRendererService.warnedOverrides — a Set of kind:channel:locale keys already logged. Bounded by the number of templates; exists so one bad override is one log line rather than three thousand identical ERRORs.
• SecretReferenceService.resolvers — the prefix-to-resolver map registered at boot. A duplicate registration throws, because two resolvers for one prefix means one of them silently never runs.

Template overrides and preference snapshots are loaded once per fan-out batch and held for the duration of that batch only. That is batching, not caching: nothing survives the transaction, so a template edit takes effect on the next batch.

9. BullMQ, Schedulers, and Async Work

Concurrency is set per channel for a reason. SMS is 2 because every message costs real money and the provider rate-limits — a burst of workers against a metered gateway spends faster and gets throttled sooner. Push is 8 because it is high-volume and cheap. Email sits between at 4.

Why exactly one decorated class per queue. @nestjs/bullmq builds one BullMQ Worker per decorated class, and WorkerOptions.name is a monitoring label, never a filter. Two classes each carrying @Processor(queue, { name: "x" }) therefore produce two workers racing for every job on that queue, each receiving the other's work, so a job won by the wrong class completes having done nothing. So NotificationFanoutQueueProcessor is the only @Processor on NOTIFICATION_FANOUT and routes by job.name.

Its handler map is a Record over a six-member union. Adding a job to that union without registering a handler is a compile error, because the runtime symptom otherwise is a job that completes having done nothing — silent, and indistinguishable from success. The union is deliberately a subset of NotificationJob: the legacy queue's own job names are excluded, because including them would force a stub handler or make the map non-exhaustive. An unknown name at runtime throws rather than logging and returning, because BullMQ records a throw and records nothing at all about a job that returns successfully having done nothing.

Why the channel queues pin attempts to 1. The notification_delivery row owns retry — its attempts column, its backoff and the reaper. Two independent budgets mean nobody owns the terminate decision: if BullMQ's is larger, the row reaches dead while the job keeps retrying against a claim that now fails, and job_failures is written on the wrong attempt; if smaller, the job stops while attempts < max and the row sits in processing with nothing to reclaim it. The pin is per queue in bull.module.ts's defaultJobOptions, driven by BULL_QUEUE_NOTIFICATION_EMAIL_ATTEMPTS, BULL_QUEUE_NOTIFICATION_SMS_ATTEMPTS and BULL_QUEUE_NOTIFICATION_PUSH_ATTEMPTS, all 1 in .env.example. Without those three lines each queue silently falls back to BULL_DEFAULT_ATTEMPTS.

NOTIFICATION_OPERATIONAL is registered beside them for the opposite reason: it must take the ordinary default, because an operational alert has no delivery row and nothing else would ever retry it, so a transient SMTP refusal would lose the notice outright.

Scheduler job ids. jobId deduplicates — BullMQ's add() returns the existing job for a duplicate id rather than throwing — so the bucket must match the interval. A per-minute key on a thirty-second cron collapses both ticks of each minute into the first: the schedule would look applied and run at half the stated rate. Each bucket is derived from the same interval its cron uses, so two ticks in one window (a restart, two replicas with no distributed lock) still collapse to one job, which is what the id is for.

And the id must not contain a colon. BullMQ rejects a jobId containing : and the rejection does not surface — the job is simply never scheduled, while type-check, lint, build, the full suite and every rule gate stay green. buildJobId turns that into a throw, and the rules:job-ids gate catches it statically.

Every cron handler catches its own enqueue failure, because a cron handler that throws can take the scheduler's next tick with it. That absorption is exactly why each catch logs at error — the log line is the only channel the failure has.

Provider timeouts (NOTIFICATION_PROVIDER_CONNECT_TIMEOUT_MS 3000, NOTIFICATION_PROVIDER_READ_TIMEOUT_MS 10000) are both below BullMQ's default stalledInterval of 30s, so a slow provider cannot cause a stalled-job redelivery while the first call is still open — which would send the message twice and bill for both.

10. Realtime and Events

Three different outbox dedupe keys can legitimately exist for one delivery public id — one per code path that schedules it. Reusing a key across paths would make the outbox's unique index silently drop the second enqueue.

11. Security, Auth, and Abuse Controls

Guards. Every controller in this module applies JwtAuthGuard then RoleGuard. Nothing here is @Public().

Permissions. Three catalogue modules, each producing _CREATE, _READ, _UPDATE, _DELETE and _RESTORE codes:

None is in SUPERADMIN_ONLY_MODULES, which is precisely why notification_event.variables may never carry a credential and why the history DTO excludes rendered content.

Handlers with no permission. Eleven handlers appear on RoleGuard's NO_PERMISSION_ADMIN_HANDLERS allowlist: the four NotificationCentreController methods, the two NotificationDevicesController methods, the two NotificationPreferencesController methods, and the three NotificationFeedController methods. An admin route that declares no permission is otherwise refused at runtime, precisely so one cannot ship open by accident. For each of them the service predicate is the only control, because RoleGuard returns true for an allowlisted handler before it ever reads request.user:

• The centre scopes by ownership, active role and staff liveness.
• Devices key every write on actor.id from the verified session; the DTO carries no userId field at all.
• Preferences key every write on actor.id.
• The feed filters every row by the caller's active-role permission set.

Input normalisation. Phone numbers pass normalizePhone before a send and an unnormalisable one is a skip, not a failure. escapeHtml is applied to every interpolated value in an email body; escapeHeader collapses CR and LF in a subject, because a newline in an interpolated header splits it and everything after becomes a header of the attacker's choosing.

Action URLs are allowlisted, and validated twice. buildEmailLayout emits href="${escapeHtml(ctaUrl)}", and escapeHtml replaces exactly five characters — & < > " ' — none of which appear in javascript:alert(1) or data:text/html;base64,..., so both would pass through intact and become working links in the recipient's mail client and in the parent and student app. isSafeActionUrl allows https: absolute URLs, or an app-relative path beginning with exactly one slash: //evil.example is a protocol-relative URL a browser resolves to an absolute one, so a naive startsWith("/") is an open redirect wearing a relative path. The check is on url.protocol, not on whether parsing threw, because URL parses javascript:alert(1) happily. http: is refused even in development — a link in an email leaves the machine that generated it. Validation happens at write so a bad value never reaches the database, and again at render, because a row can arrive by migration, by a fixture, or from a future writer that forgets; the render-time check is the one that actually protects the recipient, and it degrades to "no button" rather than failing the send.

Sensitive data redaction.

The Aakash failure body echoes back the text it was asked to send, which for an OTP message is the OTP. That single fact is why last_error is redacted rather than merely truncated, and why every provider maps failures to a code from a fixed table plus a bounded allowlisted detail.

Abuse controls.

Audit. NotificationTemplateService.update writes an explicit ActivityRecordService entry with per-field changes, module NotificationTemplate, resource type notification_template. create and remove are covered by the global ActivityAuditInterceptor. Replay records replayed_by on the job_failures row.

Fail-closed decisions. An unknown audience kind throws rather than resolving to nobody. An empty permission list becomes SQL false rather than "no filter". A category with no default entry resolves to false rather than sending. An unresolved template variable suppresses the send. A required channel with no credentials refuses to boot.

Fail-open decisions, each stated as a trade: a bad template override falls through to the shipped wording rather than silencing the notification; an unsafe action URL is dropped rather than failing the send; a realtime publish failure is logged rather than failing the batch; a scheduling failure inside AuthEmailService is swallowed rather than rolling back the password change.

13. Error Handling

Internal, non-HTTP failure states are recorded on the delivery row rather than raised: skipped_unconfigured, skipped_preference, skipped_no_template, skipped_no_destination, failed and dead. Provider-level codes reaching last_error come from three fixed tables — EMAIL_ERROR_CODE, SMS_ERROR_CODE and PUSH_ERROR_CODE — and never from a provider's own response text.

14. Observability

failure_count and lease_expiry_count are separate columns specifically so that "the provider is failing" and "workers are dying" remain distinguishable in a query, and skipped_* statuses are excluded from every failure count so an unconfigured environment never looks like an outage.

15. Testing and Validation

Validation commands:

pnpm check-types
pnpm lint
pnpm --filter @skoolsewa/api test
pnpm --filter @skoolsewa/api test:int
pnpm rules
pnpm build

Three of the invariants above are enforced at import or boot time rather than by a test, because the failure they prevent is invisible to every gate: a legal-but-wrong backfill/retention pair silently loses the backlog, an in_app kind without persistRendered delivers a blank notification that counts toward the badge, and a required channel with no credentials returns 200 and sends nothing.

16. Backend Deep-Dive Pack

16.1 Submodule Coverage Matrix

Imported from elsewhere, and why. OutboxService because no send path may call queue.add directly. EMAIL_CLIENT because "how an email is sent" must have one implementation. RoleService because the feed's filter is the caller's active-role permission set. ActivityRecordService because a template edit needs a before-value the global interceptor cannot supply. JobFailureRecorderService because the dead-letter shape is platform-wide.

Deliberately local rather than shared. NotificationPreferencesService constructs its own PreferenceResolverService rather than injecting one, because it needs only defaultsFor — a pure function over a constant — and injecting would couple a read screen to the fan-out module's provider graph. computeLockedCategories is a module-level function in the preferences service rather than a registry export, because it answers a UI question ("would this switch do anything?") that the registry has no reason to know about.

16.2 Architecture Diagram Pack

Component view:

Class view:

Delivery state machine:

Event fan-out state machine:

Runtime topology:

Data lineage for one recipient:

16.3 Code Flow Narrative — ChannelSendService.send

16.4 Data Layer Deep Dive

Field-level meaning, nullability, validation source, delete behaviour, timestamps and JSON shapes are given per table in section 5.2. What follows is the index rationale and the JSON contracts.

JSON contracts:

// notification_event.audience — a specification, never a list
{ "kind": "class", "classPublicIds": ["6b1f...", "9c22..."] }
{ "kind": "compound", "any": [
    { "kind": "role", "rolePublicIds": ["..."] },
    { "kind": "guardians_of_class", "classPublicIds": ["..."] }
] }
{ "kind": "all_users" }

// notification_event.variables — shared template variables, never a credential
{ "title": "Sports day moved", "message": "It is now on Friday." }
{ "secretRef": "password_reset:0f2b1c9e-..." }

// notification_recipient.variables — the per-recipient overlay
{ "name": "Aarati Shrestha" }

// job_failures.payload_ref — a reference, never the payload
{ "notificationDeliveryPublicId": "018f...", "channel": "sms" }
{ "aggregateType": "feedback", "aggregateId": "018f...", "eventType": "feedback.submitted" }

Money units, timezones and seeds. No column here carries money. Every timestamp is timestamptz, and the only local-time reasoning in the module is the credit check's day key, computed in Asia/Kathmandu because a UTC key rolls over at 05:45 local. No table in this module is seeded — notification_template deliberately so, and the preference tables because absence is the code default.

16.5 Business Logic and Invariant Catalog

16.6 Tradeoffs, Alternatives, and ADR Notes

16.7 Operational Runbook

16.8 Backend Risk Register

17. Zero-Omission Backend Checklist

• Every file in the module directory is represented in section 4 and 16.1.
• Every controller, service, provider, processor, scheduler, helper, DTO, enum and schema is documented.
• Every method with business behaviour has a code-flow narrative or a per-method table.
• Every table, cache object and job payload has field-level detail.
• Every index, constraint, relation and delete behaviour has a rationale.
• Both lifecycles — the delivery state machine and the event fan-out — have a transition diagram and a table.
• Every read, write, action and job flow has a diagram and branch notes.
• Every business invariant is catalogued.
• Every Redis key, queue job, realtime event and external call is documented.
• Every architectural tradeoff is documented with alternatives and a revisit trigger.
• Every operational failure mode has a runbook entry.

18. Backend Completion Checklist

• Module boundaries are documented, including what this module explicitly does not own.
• Every controller, service, DTO, schema file, job, Redis key and event is covered.
• Every table has a field table, and the two clusters have a relationship diagram.
• Every runtime flow has a diagram and branch notes.
• The API and features/flows docs are linked.
• No claim is made without a source file behind it.

See Also

• API doc: /docs/developer/notification/api
• Features and flows doc: /docs/developer/notification/feature