feat(backup): run a restore as a background job with progress

Restoring a large account was one request: 94 s for 1.25 million readings.
A reverse proxy that cuts a request at 60 s (nginx's default) showed the
operator an error for a restore that went on to finish on the server.

The restore request now checks the typed confirmation, that the backup
exists and that its stored copy opens, admits one job for the account and
answers 202 with its id. The `backup-restore` queue runs the restore and
records its phase, counts and outcome on a `backup_restore_jobs` row
(migration 0347), which `GET /api/admin/backups/restores` and
`GET /api/admin/backups/restores/{id}` read back.

What a restore checks and writes is unchanged. The body moved from the
route into `restoreBackup` (src/lib/export/restore-backup.ts): every
refusal still happens before the first delete and the replacement is still
one transaction, so a failure part-way leaves the account as it was. While
it runs the account keeps showing its current data, and other writes are
neither refused nor held, as before.

- One restore per account: a partial unique index admits one queued or
  running job, and a second request gets 409 `backup.restore.active` naming
  the running job. A per-account advisory lock inside the transaction holds
  even if a worker thought gone is still writing.
- Time budget: the job's expiry is two hours and a file whose transaction
  limit would run past three quarters of it is refused before anything is
  deleted (`jobDeadline`, beside `jobBudget`).
- Resuming: the job writes a heartbeat every two seconds. A boot sweep
  queues a job whose worker stopped again from the start, at most twice,
  then fails it as interrupted. The admission check releases an abandoned
  job so it never holds the account's slot.
- The staged file is the stored `DataBackup` row, already encrypted at
  rest. It stays whatever the outcome, so a retry needs no new upload. The
  job records a digest of the copy and refuses one that was replaced while
  it waited.
- Failures carry a stable code and a sentence that says what to do.
- The console follows the job: phase and counts while it runs, the outcome
  after, and after a reload it finds the job again from the server. A poll
  that fails on the network is never shown as a failed restore. A web
  process separate from the worker evicts the account's caches when it
  first reports the restore finished. The success line goes through the
  outcome module, so an empty file is not drawn as a tick.

The model is registered in the backup plan (not carried) and the wipe plan
(wiped with the account); the queue in the maintenance registrar, the
subsystem surface and the job-fact allowlist; the new code in the error
catalogue; the routes in the OpenAPI contract. The tests that pin restore
semantics now drive the real route and the real job through one helper, and
`backup-restore-job.test.ts` pins the asynchronous contract itself.

docs/api/openapi.yaml

@@ -20489,12 +20489,14 @@ paths:
     post:
       tags:
         - Admin
-      summary: Restore one stored backup over its owner's record
-      description: "Replaces the snapshot owner's data tables from the stored copy, in one transaction, under the ids the file
-        carries. Replacing is not merging: every row the account gained after the snapshot was taken is deleted with the
-        rest of its class. The target is the account the snapshot was taken for, never the admin running it, and a
-        payload declaring a different owner than the stored row is refused. Admin session cookie required; Bearer tokens
-        cannot reach admin endpoints."
+      summary: Queue the restore of one stored backup over its owner's record
+      description: "Queues a restore that replaces the snapshot owner's data tables from the stored copy, in one transaction,
+        under the ids the file carries, and answers 202 with the job's id. Replacing is not merging: every row the
+        account gained after the snapshot was taken is deleted with the rest of its class. The target is the account the
+        snapshot was taken for, never the admin running it. The request refuses a missing confirmation, an unknown
+        backup and a copy that cannot be decrypted; the file's own checks (schema, declared owner, manifest, documents)
+        run in the job before anything is deleted, and a refusal there ends the job as `failed` with the same reason.
+        One restore per account at a time. Admin session cookie required; Bearer tokens cannot reach admin endpoints."
       parameters:
         - in: path
           name: id
@@ -20511,12 +20513,12 @@ paths:
             schema:
               $ref: "#/components/schemas/AdminBackupRestoreRequest"
       responses:
-        "200":
-          description: The restore ran.
+        "202":
+          description: The restore is queued. Its progress and outcome are at `statusUrl`.
           content:
             application/json:
               schema:
-                $ref: "#/components/schemas/AdminBackupRestoreResponse"
+                $ref: "#/components/schemas/AdminBackupRestoreQueuedResponse"
         "400":
           description: Body was not JSON.
           content:
@@ -20526,8 +20528,9 @@ paths:
         "403": *a50
         "404": *a51
         "409":
-          description: Either the payload declares a different owner than the stored row, or a request under the same
-            `Idempotency-Key` is still in flight. Nothing was changed.
+          description: A restore of the same account is already queued or running (`meta.errorCode` = `backup.restore.active`,
+            `meta.jobId` names it), or a request under the same `Idempotency-Key` is still in flight. Nothing was
+            changed.
           content:
             application/json:
               schema:
@@ -20539,10 +20542,59 @@ paths:
               schema:
                 $ref: "#/components/schemas/ErrorEnvelope"
         "422":
-          description: "The request or the file was refused before anything was written: `confirm` missing, the copy undecryptable
-            (`meta.errorCode` = `backup.payload.undecryptable`), the payload failing schema validation, an unsupported
-            `schemaVersion`, a section the file's own manifest says it carries and does not (`meta.errorCode` =
-            `backup.section.missing`), a metadata-only document entry, or an owner who no longer exists here."
+          description: "The request was refused before anything was queued: `confirm` missing, or the copy undecryptable
+            (`meta.errorCode` = `backup.payload.undecryptable`)."
+          content:
+            application/json:
+              schema:
+                $ref: "#/components/schemas/ErrorEnvelope"
+        "503":
+          description: The restore could not be handed to the background worker. Nothing was changed.
+          content:
+            application/json:
+              schema:
+                $ref: "#/components/schemas/ErrorEnvelope"
+  /api/admin/backups/restores:
+    get:
+      tags:
+        - Admin
+      summary: List the restores the backups console shows
+      description: Every queued or running restore, and those created in the last day, newest first, at most 20. What the
+        console reads when it opens, so a reload during a restore picks its progress up again. Admin session cookie
+        required; Bearer tokens cannot reach admin endpoints.
+      responses:
+        "200":
+          description: The restore jobs.
+          content:
+            application/json:
+              schema:
+                $ref: "#/components/schemas/AdminBackupRestoreJobListResponse"
+        "403": *a50
+  /api/admin/backups/restores/{jobId}:
+    get:
+      tags:
+        - Admin
+      summary: Read one restore job
+      description: The status, phase and counts of one restore, and its report or its failure once it finished. Admin session
+        cookie required; Bearer tokens cannot reach admin endpoints.
+      parameters:
+        - in: path
+          name: jobId
+          schema:
+            type: string
+            description: The `jobId` the restore request answered with.
+          required: true
+          description: The `jobId` the restore request answered with.
+      responses:
+        "200":
+          description: The restore job.
+          content:
+            application/json:
+              schema:
+                $ref: "#/components/schemas/AdminBackupRestoreJobResponse"
+        "403": *a50
+        "404":
+          description: No restore job with this id.
           content:
             application/json:
               schema:
@@ -30247,39 +30299,40 @@ components:
                 anamnesis.fact.invalidValue; auth — auth.admin.required, auth.already_authenticated,
                 auth.mfa.code_invalid, auth.missing, auth.refresh.invalid, auth.refresh.reuse, auth.refresh.revoked,
                 auth.scope.insufficient, auth.stepup.required, auth.token.expired, auth.token.invalid; backup —
-                backup.payload.undecryptable, backup.section.missing; coach — coach.budget.exceeded,
-                coach.conversation.invalidTitle, coach.conversation.notFound, coach.fenced.attachmentConflict,
-                coach.fenced.attachmentLimit, coach.fenced.attachmentNotFound, coach.fenced.attachmentNotIndexed,
-                coach.fenced.attachmentRequired, coach.fenced.attachmentUnavailable, coach.fenced.invalid,
-                coach.request.invalid; coach-prefs — coach-prefs.body.invalid_json, coach-prefs.body.invalid_shape;
-                consent — consent.ai.required; cycle — cycle.bulk.invalid, cycle.bulk.too_large, cycle.calendar.invalid,
-                cycle.calendar.range, cycle.cycles.invalid, cycle.day-log.conflict, cycle.day-log.invalid,
-                cycle.disabled, cycle.period.invalid, cycle.symptom.custom.invalid, cycle.symptom.custom.limit;
-                cycle-prefs — cycle-prefs.body.invalid_json, cycle-prefs.invalid; disable-coach —
-                disable-coach.body.invalid_json; documents — documents.chat.conversationNotFound,
-                documents.chat.invalid, documents.inbound.alreadyConfirmed, documents.inbound.alreadyPartlyConfirmed,
-                documents.inbound.budgetExceeded, documents.inbound.decryptFailed, documents.inbound.encounterNotFound,
-                documents.inbound.episodeNotFound, documents.inbound.extractFailed, documents.inbound.factNotFound,
-                documents.inbound.factNotPending, documents.inbound.factTypeMismatch, documents.inbound.fileTooLarge,
-                documents.inbound.fileType, documents.inbound.invalidMetadata, documents.inbound.invalidQuery,
-                documents.inbound.invalidUpdate, documents.inbound.localOcrDisabled, documents.inbound.notFound,
-                documents.inbound.notIndexed, documents.inbound.pdfNeedsAnthropic,
-                documents.inbound.providerUnsupported, documents.inbound.quotaExceeded, documents.inbound.rateLimited,
-                documents.inbound.restoreDuplicate, documents.inbound.restoreGone, documents.inbound.uploadBusy,
-                documents.inbound.uploadTimeout, documents.inbound.vaccinationNotFound; encounter — encounter.invalid,
-                encounter.link.invalid, encounter.practitioner-not-found, encounter.reminder-conflict,
-                encounter.reminder-not-found; environment — environment.invalid, environment.no_home,
-                environment.range_too_large, environment.travel.not_found; export — export.selection.empty,
-                export.selection.unknown_leaf; familyHistory — familyHistory.invalid; feedback — feedback.body.invalid;
-                health_score_config — health_score_config.invalid, health_score_config.too_narrow; illness —
-                illness.day-log.invalid, illness.disabled, illness.episode.chronic-no-resolve, illness.episode.invalid,
-                illness.episode.invalid-window, illness.episode.parent-not-found, illness.insights.invalid; import —
-                import.write_failed; ingest — ingest.rate_limited; insights — insights.generate.budgetExceeded,
-                insights.generate.outboundScreened, insights.generate.profileScopeChanged,
-                insights.generate.scopeChanged; labs — labs.biomarker.create.invalid, labs.biomarker.duplicate,
-                labs.biomarker.notFound, labs.biomarker.update.invalid, labs.biomarker.update.referenceRangeInvalid,
-                labs.create.invalid, labs.ocr.budgetExceeded, labs.ocr.extractFailed, labs.ocr.fileTooLarge,
-                labs.ocr.fileType, labs.ocr.localOcrDisabled, labs.ocr.pdfNeedsAnthropic, labs.ocr.providerUnsupported,
+                backup.payload.undecryptable, backup.restore.active, backup.section.missing; coach —
+                coach.budget.exceeded, coach.conversation.invalidTitle, coach.conversation.notFound,
+                coach.fenced.attachmentConflict, coach.fenced.attachmentLimit, coach.fenced.attachmentNotFound,
+                coach.fenced.attachmentNotIndexed, coach.fenced.attachmentRequired, coach.fenced.attachmentUnavailable,
+                coach.fenced.invalid, coach.request.invalid; coach-prefs — coach-prefs.body.invalid_json,
+                coach-prefs.body.invalid_shape; consent — consent.ai.required; cycle — cycle.bulk.invalid,
+                cycle.bulk.too_large, cycle.calendar.invalid, cycle.calendar.range, cycle.cycles.invalid,
+                cycle.day-log.conflict, cycle.day-log.invalid, cycle.disabled, cycle.period.invalid,
+                cycle.symptom.custom.invalid, cycle.symptom.custom.limit; cycle-prefs — cycle-prefs.body.invalid_json,
+                cycle-prefs.invalid; disable-coach — disable-coach.body.invalid_json; documents —
+                documents.chat.conversationNotFound, documents.chat.invalid, documents.inbound.alreadyConfirmed,
+                documents.inbound.alreadyPartlyConfirmed, documents.inbound.budgetExceeded,
+                documents.inbound.decryptFailed, documents.inbound.encounterNotFound, documents.inbound.episodeNotFound,
+                documents.inbound.extractFailed, documents.inbound.factNotFound, documents.inbound.factNotPending,
+                documents.inbound.factTypeMismatch, documents.inbound.fileTooLarge, documents.inbound.fileType,
+                documents.inbound.invalidMetadata, documents.inbound.invalidQuery, documents.inbound.invalidUpdate,
+                documents.inbound.localOcrDisabled, documents.inbound.notFound, documents.inbound.notIndexed,
+                documents.inbound.pdfNeedsAnthropic, documents.inbound.providerUnsupported,
+                documents.inbound.quotaExceeded, documents.inbound.rateLimited, documents.inbound.restoreDuplicate,
+                documents.inbound.restoreGone, documents.inbound.uploadBusy, documents.inbound.uploadTimeout,
+                documents.inbound.vaccinationNotFound; encounter — encounter.invalid, encounter.link.invalid,
+                encounter.practitioner-not-found, encounter.reminder-conflict, encounter.reminder-not-found; environment
+                — environment.invalid, environment.no_home, environment.range_too_large, environment.travel.not_found;
+                export — export.selection.empty, export.selection.unknown_leaf; familyHistory — familyHistory.invalid;
+                feedback — feedback.body.invalid; health_score_config — health_score_config.invalid,
+                health_score_config.too_narrow; illness — illness.day-log.invalid, illness.disabled,
+                illness.episode.chronic-no-resolve, illness.episode.invalid, illness.episode.invalid-window,
+                illness.episode.parent-not-found, illness.insights.invalid; import — import.write_failed; ingest —
+                ingest.rate_limited; insights — insights.generate.budgetExceeded, insights.generate.outboundScreened,
+                insights.generate.profileScopeChanged, insights.generate.scopeChanged; labs —
+                labs.biomarker.create.invalid, labs.biomarker.duplicate, labs.biomarker.notFound,
+                labs.biomarker.update.invalid, labs.biomarker.update.referenceRangeInvalid, labs.create.invalid,
+                labs.ocr.budgetExceeded, labs.ocr.extractFailed, labs.ocr.fileTooLarge, labs.ocr.fileType,
+                labs.ocr.localOcrDisabled, labs.ocr.pdfNeedsAnthropic, labs.ocr.providerUnsupported,
                 labs.ocr.rateLimited, labs.restore.invalid, labs.result.notFound, labs.update.invalid,
                 labs.update.linkedFieldsImmutable, labs.update.numericExpected, labs.update.qualitativeExpected,
                 labs.update.referenceRangeInvalid, labs.update.sourceReferenceRangeInvalid; managed_profile —
@@ -70802,11 +70855,11 @@ components:
       additionalProperties: {}
       description: "The backup payload as the writer serialised it: `schemaVersion`, `exportedAt`, `userId`, and one array or
         object per serialised class."
-    AdminBackupRestoreResponse:
+    AdminBackupRestoreQueuedResponse:
       type: object
       properties:
         data:
-          $ref: "#/components/schemas/AdminBackupRestoreResult"
+          $ref: "#/components/schemas/AdminBackupRestoreQueued"
         error:
           type: "null"
         meta:
@@ -70819,49 +70872,196 @@ components:
         - data
         - error
       additionalProperties: false
-    AdminBackupRestoreResult:
+    AdminBackupRestoreQueued:
       type: object
       properties:
-        restored:
-          type: boolean
-          const: true
-        summary:
-          type: object
-          propertyNames:
-            type: string
-          additionalProperties: {}
-        skipped:
+        jobId:
+          type: string
+        status:
+          type: string
+          const: queued
+        statusUrl:
+          type: string
+      required:
+        - jobId
+        - status
+        - statusUrl
+      additionalProperties: false
+      description: The restore was accepted and queued. Poll `statusUrl` (`GET /api/admin/backups/restores/{jobId}`) for its
+        progress and outcome.
+    AdminBackupRestoreJobListResponse:
+      type: object
+      properties:
+        data:
           type: object
           properties:
-            catalogueKeys:
+            jobs:
               type: array
               items:
-                $ref: "#/components/schemas/AdminBackupRestoreSkipEntry"
-            links:
-              type: integer
+                $ref: "#/components/schemas/AdminBackupRestoreJob"
+          required:
+            - jobs
+          additionalProperties: false
+        error:
+          type: "null"
+        meta:
+          type: object
+          properties:
+            requestId:
+              type: string
+          additionalProperties: false
+      required:
+        - data
+        - error
+      additionalProperties: false
+    AdminBackupRestoreJob:
+      type: object
+      properties:
+        id:
+          type: string
+        userId:
+          type: string
+        username:
+          anyOf:
+            - type: string
+            - type: "null"
+        backupId:
+          type: string
+        restoreInstanceSettings:
+          type: boolean
+        status:
+          type: string
+          enum:
+            - queued
+            - running
+            - succeeded
+            - failed
+        phase:
+          anyOf:
+            - $ref: "#/components/schemas/AdminBackupRestorePhase"
+            - type: "null"
+        progress:
+          anyOf:
+            - $ref: "#/components/schemas/AdminBackupRestoreProgress"
+            - type: "null"
+        result:
+          anyOf:
+            - type: object
+              properties:
+                summary:
+                  type: object
+                  propertyNames:
+                    type: string
+                  additionalProperties: {}
+                skipped:
+                  type: object
+                  properties:
+                    catalogueKeys:
+                      type: array
+                      items:
+                        $ref: "#/components/schemas/AdminBackupRestoreSkipEntry"
+                    links:
+                      type: integer
+                      minimum: -9007199254740991
+                      maximum: 9007199254740991
+                  required:
+                    - catalogueKeys
+                    - links
+                  additionalProperties: false
+                cleared:
+                  type: object
+                  propertyNames:
+                    type: string
+                  additionalProperties:
+                    type: integer
+                    minimum: -9007199254740991
+                    maximum: 9007199254740991
+              required:
+                - summary
+                - skipped
+                - cleared
+              additionalProperties: false
+            - type: "null"
+        failure:
+          anyOf:
+            - $ref: "#/components/schemas/AdminBackupRestoreFailure"
+            - type: "null"
+        attempts:
+          type: integer
+          minimum: -9007199254740991
+          maximum: 9007199254740991
+        createdAt:
+          type: string
+        startedAt:
+          anyOf:
+            - type: string
+            - type: "null"
+        completedAt:
+          anyOf:
+            - type: string
+            - type: "null"
+      required:
+        - id
+        - userId
+        - username
+        - backupId
+        - restoreInstanceSettings
+        - status
+        - phase
+        - progress
+        - result
+        - failure
+        - attempts
+        - createdAt
+        - startedAt
+        - completedAt
+      additionalProperties: false
+      description: "One restore job. `phase` and `progress` describe a running job; `result` is set once it `succeeded` (the
+        same report the synchronous restore used to answer with, less `restored`); `failure` once it `failed`.
+        `attempts` counts the starts: a job whose worker stopped is started again, at most twice."
+    AdminBackupRestorePhase:
+      type: string
+      enum:
+        - validating
+        - clearing
+        - measurements
+        - sections
+        - rebuilding
+      description: "Where a running restore is: reading and checking the file, clearing the account's current data, writing
+        the readings, writing the other sections, rebuilding the chart tiers after the commit."
+    AdminBackupRestoreProgress:
+      type: object
+      properties:
+        measurementsChecked:
+          type: integer
+          minimum: -9007199254740991
+          maximum: 9007199254740991
+        measurementsTotal:
+          anyOf:
+            - type: integer
               minimum: -9007199254740991
               maximum: 9007199254740991
-          required:
-            - catalogueKeys
-            - links
-          additionalProperties: false
-        cleared:
-          type: object
-          propertyNames:
-            type: string
-          additionalProperties:
-            type: integer
-            minimum: -9007199254740991
-            maximum: 9007199254740991
+            - type: "null"
+        measurementsWritten:
+          type: integer
+          minimum: -9007199254740991
+          maximum: 9007199254740991
+        sectionsDone:
+          type: integer
+          minimum: -9007199254740991
+          maximum: 9007199254740991
+        sectionsTotal:
+          type: integer
+          minimum: -9007199254740991
+          maximum: 9007199254740991
       required:
-        - restored
-        - summary
-        - skipped
-        - cleared
+        - measurementsChecked
+        - measurementsTotal
+        - measurementsWritten
+        - sectionsDone
+        - sectionsTotal
       additionalProperties: false
-      description: What the transaction wrote. `cleared` counts the rows each class held before it was rebuilt from the file,
-        `summary` counts what the file carried, and `skipped.links` is the number of links dropped because a catalogue
-        key in the file does not exist here — zero is the normal answer.
+      description: Counts only. `measurementsTotal` is null until the first read of the file has counted the readings.
     AdminBackupRestoreSkipEntry:
       type: object
       properties:
@@ -70879,6 +71079,45 @@ components:
         - links
       additionalProperties: false
       description: One catalogue key the file referenced and this instance could not resolve, with the number of links it cost.
+    AdminBackupRestoreFailure:
+      type: object
+      properties:
+        code:
+          type: string
+          description: "Stable reason: `backup_not_found`, `backup_changed`, `backup.payload.undecryptable`, `schema_invalid`,
+            `incompatible_schema_version`, `owner_mismatch`, `owner_not_found`, `backup.section.missing`,
+            `document_ciphertext_missing`, `time_budget`, `transaction_failed`, `interrupted`, `not_started`,
+            `enqueue_failed`, `failed_after_commit`, `unexpected`. Every code but `failed_after_commit` means the
+            account was not changed."
+        message:
+          type: string
+        sections:
+          type: array
+          items:
+            type: string
+      required:
+        - code
+        - message
+      additionalProperties: false
+      description: Why the restore did not happen, in a sentence an operator can act on. `sections` names the missing sections
+        for `backup.section.missing`.
+    AdminBackupRestoreJobResponse:
+      type: object
+      properties:
+        data:
+          $ref: "#/components/schemas/AdminBackupRestoreJob"
+        error:
+          type: "null"
+        meta:
+          type: object
+          properties:
+            requestId:
+              type: string
+          additionalProperties: false
+      required:
+        - data
+        - error
+      additionalProperties: false
     __schema0:
       anyOf:
         - type: string