mirror of
https://github.com/MBombeck/HealthLog.git
synced 2026-11-18 08:02:12 +00:00
b299904552c00e7614e097a745633159c2f89098
5
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
|
|
900a5c4f1f |
perf: stop keeping folded samples and re-reading duplicates
The nightly folds soft-deleted every raw Apple Health sample they folded into an hourly or daily stats row and kept it for 75 days. Nothing ever read those rows as deletions; they only held a re-upload of the sample off, and they were most of the measurements table and its indexes. Ingest now refuses such a re-upload directly: a raw sample of a dense or mean type, old enough to have been folded, whose hour or day a live stats row covers, is a duplicate with reason folded_window. The batch route and the Apple Health export import apply the same rule. With that in place the dense retention and the daily-mean consolidation delete the raw rows outright, and a new compaction-tombstone-purge job removes the backlog: 5,000 rows a batch, one account at a time under the restore lock, a short pause between batches, resumable because the predicate is idempotent, queued at every boot and continuing itself while it makes progress. A person's own deletions are never in its predicate. The sync feed leaves compaction tombstones out and a restore does not write them back; all three classify through the same function. The batch route answers an exact re-sent sample without a transaction: one read over the batch's sample ids, live or deleted, decides what the reconciler would have decided, at the same point in the loop, so every status, reason, counter and the cross-source merge stay as they were. A batch of duplicates only no longer queues PR detection, a reminder sweep or the detection audit row. Caches are evicted only for a foreground or manual sync or a batch that wrote a hand-entered reading, and marked stale otherwise; the targets cache follows the same split. Also: - POST /api/admin/maintenance/measurements queues VACUUM (ANALYZE) and a REINDEX INDEX CONCURRENTLY per index, largest first, on a connection of its own; it refuses while the purge runs, drops invalid leftover copies and stops at the first failed rebuild. Runbook in docs/ops. - The pure volume queues keep their finished jobs for 96 hours instead of seven days, never less than the failure window plus a day. - work_mem is set per app connection from DATABASE_WORK_MEM (16MB), also when the statement timeout is disabled. - The PR-detection fallback tick re-scans only accounts whose readings or workouts changed since the previous tick. - Type discovery for the status refill walks the live-rows index as a loose index scan instead of grouping every live row. - The host metrics chart polls every five minutes, like its sampler. - The batch route records per-type HealthKit arrivals. |
||
|
|
13ea849188 |
perf(backup): load the backup payload schema on first use
backupPayloadSchema is built when validations/backup.ts is evaluated and holds about 12 MB of heap. The restore job, the stored-backup readers and the export writer import that module, and they sit on the boot path, so every server process built the schema at start whether or not it ever touched a backup. The schema version, the compatibility check and the per-section summary move to validations/backup-summary.ts, which never builds the schema. The call sites that validate a file (upload, restore, the summary and download routes, and the streamed reader's element check) load backup.ts with a dynamic import at that moment; everything else takes only types. The OpenAPI registry keeps its static import, since only the spec generator loads it. After warm-up on a production build under --max-old-space-size=524, the used heap drops by about 29 MB, and the schema no longer appears in a heap snapshot. A structural test refuses a new static value import of the module outside the registry. |
||
|
|
405c996b6a |
fix(backup): refuse a restore that needs a dropped key or reaches into another account
A backup carries every encrypted column as the stored ciphertext, and a restore writes it back verbatim. Key rotation re-seals the stored copy's envelope but cannot reach inside it, so a copy taken before a rotation still needs the retired key. Without it the restore went through and left rows no reader can open. The restore, the restore preview, the admin upload and the monthly restore drill now collect the key ids of every inner ciphertext value, hold them against ENCRYPTION_KEYS and decrypt one value per key. A missing key, or an id configured with different key material, is refused before anything is deleted, with the key id named. The restore status and the preview say which key to put back. The restore also checks, inside its transaction, that it added no reference from the account's rows to rows another account owns. The check is derived from the schema's foreign keys rather than written per section, so the tag and category upserts keyed on a global key are covered along with every link table. The illness parent update is scoped to the owner as well. The off-host listing now follows every page: the first page held only the oldest dates, so the drill could read an old object and report the chain as stale. |
||
|
|
47e9f7e81c |
feat(backup): run a restore as a background job with progress
Restoring a large account was one request: 94 s for 1.25 million readings.
A reverse proxy that cuts a request at 60 s (nginx's default) showed the
operator an error for a restore that went on to finish on the server.
The restore request now checks the typed confirmation, that the backup
exists and that its stored copy opens, admits one job for the account and
answers 202 with its id. The `backup-restore` queue runs the restore and
records its phase, counts and outcome on a `backup_restore_jobs` row
(migration 0347), which `GET /api/admin/backups/restores` and
`GET /api/admin/backups/restores/{id}` read back.
What a restore checks and writes is unchanged. The body moved from the
route into `restoreBackup` (src/lib/export/restore-backup.ts): every
refusal still happens before the first delete and the replacement is still
one transaction, so a failure part-way leaves the account as it was. While
it runs the account keeps showing its current data, and other writes are
neither refused nor held, as before.
- One restore per account: a partial unique index admits one queued or
running job, and a second request gets 409 `backup.restore.active` naming
the running job. A per-account advisory lock inside the transaction holds
even if a worker thought gone is still writing.
- Time budget: the job's expiry is two hours and a file whose transaction
limit would run past three quarters of it is refused before anything is
deleted (`jobDeadline`, beside `jobBudget`).
- Resuming: the job writes a heartbeat every two seconds. A boot sweep
queues a job whose worker stopped again from the start, at most twice,
then fails it as interrupted. The admission check releases an abandoned
job so it never holds the account's slot.
- The staged file is the stored `DataBackup` row, already encrypted at
rest. It stays whatever the outcome, so a retry needs no new upload. The
job records a digest of the copy and refuses one that was replaced while
it waited.
- Failures carry a stable code and a sentence that says what to do.
- The console follows the job: phase and counts while it runs, the outcome
after, and after a reload it finds the job again from the server. A poll
that fails on the network is never shown as a failed restore. A web
process separate from the worker evicts the account's caches when it
first reports the restore finished. The success line goes through the
outcome module, so an empty file is not drawn as a tick.
The model is registered in the backup plan (not carried) and the wipe plan
(wiped with the account); the queue in the maintenance registrar, the
subsystem surface and the job-fact allowlist; the new code in the error
catalogue; the routes in the OpenAPI contract. The tests that pin restore
semantics now drive the real route and the real job through one helper, and
`backup-restore-job.test.ts` pins the asynchronous contract itself.
|
||
|
|
237acc0002 |
fix(backup): export, upload and restore a large account as streams
An account of 1.25 million measurements could not export its data or be restored from any backup (#1031). Measured in a 1 GB container: - The full export and the passphrase export built the whole payload and its JSON string in memory; the app ran out of heap and restarted. - The weekly backup's JSON is 662 MB, longer than any string V8 can hold, so `unpackBackupBlob` threw on every read. Restore, download and the restore preview all answered "could not be decrypted". - Upload accepted 10 MB, a fraction of a large export. Every reader and writer of a backup file now streams it: - `scanBackupJson` reads a backup's bytes once, keeps every section but the named bulk arrays, and hands their elements out one at a time. `readStreamedBackup` uses it twice: a first pass that validates each measurement against the element schema and keeps the rest of the file, and a second pass that hands the measurements to the writer in batches. Stored copies and off-host objects are authenticated whole on their compressed ciphertext before any plaintext is read; only the decompression is streamed. - Restore writes the measurements with one `INSERT … FROM unnest` per batch. Prisma caches a query plan per `createMany` row count, and one 1 000-row plan kept about 145 MB alive, which alone put the restore past the heap limit. The transaction budget grows with the measurement count. The awards and ECG sections are handed the ids they reference that were written, not every written id. - The export routes write the file into the response as it is produced. The passphrase export spools its ciphertext to a temporary file because the HLX1 format puts the tag first; the format is unchanged, so existing archives open as before. - Upload accepts the raw body, JSON or gzip, up to the 512 MB request ceiling, checks it as it arrives and stores it in the same pass; the admin page compresses the file in the browser before sending it. Multipart stays for small files. - Download streams the stored JSON; the preview counts measurements from the stream. The monthly restore drill and `scripts/restore-backup.ts` read the off-host object the same way; the script can write `.json.gz`. On the seeded account in a 1 GB container: export 25 s at 389 MB peak, passphrase export 25 s, both uploads 19 s, restore of the weekly backup 94 s at 554 MB peak, and export, upload and restore round-trip to exactly the same rows. The encrypted-column guard now also recognises the raw SQL statement that writes `DataBackup.data`. |