On a host whose CPU cannot run the page renderer (x64 without AVX2, arm64
musl without LSE/RCpc), every AI read of a PDF for a provider that only
takes images failed with "Scanning PDFs needs a Claude vision provider",
which says nothing about the cause. Most medical PDFs are produced
digitally and carry every word in their text layer, which pdf.js reads
without rendering.
`prepareVisionInput` now answers a text read (`mode: "text"`) when the
renderer is unavailable on this host, the provider cannot take the PDF
whole and the PDF has a usable text layer. Summary, suggest and the
background summary send that text through their existing text mode;
"Read with AI" and the index backfill write it as the local text index it
is, with no provider call; the automatic index leaves it to its local path.
`planVisionReads` does the same for the extraction and the lab scan: text
reads by page group, with anchored provenance, and the pages without a text
layer reported as `extractionIncomplete` on a document and as `unreadPages`
on a lab scan. A supplement label PDF is read from its text layer too.
A scan on such a host now answers `documents.inbound.pdfRendererUnavailable`
and `labs.ocr.pdfRendererUnavailable` (422): this server's processor cannot
run the page renderer, so a scanned PDF can only be read by a provider that
takes PDF files directly, or as photos. The `pdfNeedsAnthropic` codes are no
longer emitted, since with a working renderer every provider reads a PDF;
the recorded index outcome keeps its `pdf-needs-anthropic` value with
corrected copy.
The extraction path no longer folds a render failure of one document into
the capability error: like `prepareVisionInput` since #776 it reports
"The PDF pages couldn't be rendered for scanning." (`extractFailed`), and
the lab scan answers `labs.ocr.extractFailed`.
Hosts that render, and providers that take PDFs whole, are unchanged.
Refs #1311
The image runs the glibc canvas build now, which needs no ARMv8.2
features, so the Pi 4 gets thumbnails and scanned-PDF rendering back
without a code change: the gate only ever closed for musl on arm64. The
gate's comment, the env example and the getting-started prerequisites
say so, and keep NATIVE_CANVAS as the override. The x64 AVX2 gate stays:
the glibc x64 build rendered a PDF page under an emulated core without
AVX2, but it still carries AVX2 code and that one run does not cover
every path.
The arm64 musl build of the native canvas uses LSE atomics and RCpc
loads throughout, so loading it on an ARMv8.0 core such as the
Raspberry Pi 4 ends the process with SIGILL. The CPU gate only knew
the x64 AVX2 case, and pdf.js loads the canvas as soon as it starts,
so on such a host every PDF read died in the child: no text layer, no
index, and the manual read reported a busy reader.
The gate now also closes on linux/arm64 under musl when the CPU lacks
the atomics or lrcpc feature. A closed gate travels to the PDF child,
which refuses to resolve the canvas module, so pdf.js carries on
without it and the text layer reads; rendering and thumbnails keep
their existing fail-soft path. A child that ends without an answer
now logs its exit code and signal.
Refs #1311
The protocol spec now documents the /models request, the fields read
from it, the order the walk is built in, the per-sign-in cache and the
codex.models.* annotations (the codex.slug.* keys it named never existed).
The operator env comments say that CODEX_MODEL still goes first and that
CODEX_MODEL_FALLBACK_CHAIN replaces the order and skips the list.
Refs #1287
DEPLOY_WEBHOOK_REQUIRE_TIMESTAMP had been changed to mean signed requests
only, so an operator who set it for a sender that adds a timestamp but cannot
sign would have had every deploy notification refused after the upgrade. It
means the secret header plus a fresh timestamp again. Signature-only, the
setting that stops a replay, is DEPLOY_WEBHOOK_REQUIRE_SIGNATURE, on the
compose whitelist, in the env manifest and examples, and in the deploy
guide.
Forwarding headers were believed from any connection on a private address.
Behind Docker Desktop, OrbStack and docker-proxy every connection to a
published port arrives from the gateway, a private address, so a caller
reaching the port directly could again choose the client address that rate
limits, the audit log and sign-in notices record. TRUSTED_PROXY_CIDRS
(comma-separated CIDRs) now names the proxies: when set, only those peers may
report a client address. The private-range default stays when it is unset,
the boot log states the rule in force in one line, and the reverse-proxy
guide says what to set for a proxy on the same host, on a Docker network, or
on a public address (which, unlisted, puts all clients in one bucket).
An invalid TRUST_PROXY_HOPS threw on every request that resolved the client
address, so every route answered 500, the health probe included. It now
resolves no address, warns once, and is named at boot.
The flag makes the app believe CF-Connecting-IP from every connection,
before the peer check and whatever TRUST_PROXY_HOPS says, which is safe
only when nothing but Cloudflare can reach the origin. The reverse-proxy
page now says so with the ways to get there, the compose comment and both
example files carry the same condition, and the variable is in the env
manifest. The page's list of peers whose forwarding headers are read now
matches the code: loopback, 0.0.0.0/8, private, CGNAT, link-local,
unique-local and site-local, the reserved blocks and multicast. Those last
two stay trusted, since no internet host can connect from them; a test
pins every listed range and the Cloudflare precedence.
The image creates DB_BACKUP_DIR owned by the app user, so an install that
runs it without the backups volume could write there, and every nightly and
pre-migration dump landed in the container's own layer. The next update
threw them away while the admin page, the boot line and the readiness card
called them kept on this server.
A dump directory now counts as kept only when it survives the container:
inside a container (Docker, Podman, Kubernetes, a container cgroup or an
overlay root) the directory must sit on a mount of its own that is not a
tmpfs, read from /proc/self/mountinfo with a device-number fallback;
outside one, a writable directory is a disk. Otherwise no dump is written
there: the nightly run uploads off-host only and records not_persistent,
the pre-migration dump is skipped with a warning, the boot line says the
directory is inside the container, the backups page stops promising local
dumps and ignores local-only ledger rows, and readiness fails with no
bucket and notes the bucket-only setup otherwise.
The Unraid template gains a path for the dumps, the Unraid and TrueNAS
pages say why it is needed, and the restore drill first runs the stack
without the volume and checks that nothing is kept or claimed.
The translation files merge key by key (no key changed on both sides), the
backups list carries both the database dump and the pre-wipe copy window,
and the backup-restore doc keeps the shorter rewrite plus the dump in the
bucket and a lifecycle rule that covers database/.
The deploy-status webhook authenticated with a static secret in a header,
and the timestamp the docs called replay protection was covered by nothing:
whoever captures one request holds the secret and can send a fresh request
with any content. The strict setting, documented as closing the replay
window completely, only made that timestamp mandatory.
A sender can now sign instead: X-Deploy-Webhook-Signature is the
HMAC-SHA256 of the timestamp and the exact body under the shared secret,
with the timestamp within five minutes, and the secret never travels. The
strict setting accepts only signed requests. The secret header still works
by default for senders that cannot sign, and the docs describe its
timestamp as the staleness hint it is.
The client-IP resolver trusted one forwarding hop by default and read the
rightmost X-Forwarded-For entry. Next fills that header with the socket's
address only when the request carries none, so a caller reaching the
published port directly could send its own and choose a fresh address per
request: every per-IP limit (sign-in, second factor, passkey, share unlock,
webhooks) became per-request, and sessions, the audit log and sign-in
notices recorded the chosen address.
The server now writes each connection's own peer address into an internal
header from Node's http.server.request.start diagnostics channel, installed
at boot and overwriting anything a caller sent under that name. The resolver
reads forwarding headers only when that peer is loopback, private,
unique-local or CGNAT, where a reverse proxy sits; a public peer is its own
client address, and a log line says once that its forwarding headers were
ignored. With TRUST_PROXY_HOPS=0 the peer is used instead of no address, so
a directly exposed host no longer shares one bucket across all anonymous
callers. Proxies on the same host, Docker network or tailnet need no change.
Deleting an account, a guardian deleting a managed profile, and "Delete
all data" were final the moment they were confirmed: the account row
cascaded into every table and every stored copy, and the off-host objects
were purged in the same breath. Nothing short of rolling the whole
instance back could return one account.
ACCOUNT_DELETION_GRACE_DAYS (14 by default, at most 30, 0 erases at once
as before) now sets a window:
- Account and managed-profile deletion lock the account (migration 0488:
deletion_requested_at, deletion_purge_at, deletion_requested_by_id).
Every session, refresh token, API token, connector, share link and
step-up elevation is revoked in the same transaction, and the record,
its stored copies and its off-host objects are left as they are.
- Sign-in is refused with 403 auth.account.deletion_pending and the
erasure date, after the credential was proved and before a second
factor is asked for. The session and Bearer resolvers and the acting-
account resolver refuse a pending account too, so a credential minted
in the moment before the lock is useless.
- POST /api/admin/users/{id}/undo-deletion (cookie-only, audit-logged)
clears the lock. The admin users list reports the pending state.
- The hourly account-deletion-purge job erases accounts whose window has
ended, exactly as the immediate deletion did, with the off-host purge
request committed alongside. The Guardian invariant is decided again;
a refused erasure stays pending and is reported.
- A wipe first stores the whole record as a PRE_WIPE_<ms> copy through
the writer the weekly copy uses, keeps it out of the wipe, and holds
its off-host purge back with a new not_before on the request. The
ordinary restore brings the record back; restoring that copy withdraws
the held purge. The purge job deletes the copy when its window ends.
Background work skips accounts pending deletion through one shared
predicate (src/lib/account-deletion/active-account.ts): every reminder,
digest, Coach, sync, backfill, rollup, score and backup enumeration
filters with it, and account-deletion-enumeration-guard.test.ts fails on
a query in background work that neither filters nor says why it need
not. Notification delivery and the AI capability loader refuse a pending
account as chokepoints behind it, and the Withings, WHOOP and Telegram
webhooks and the notification quick actions ignore one.
The privacy summary publishes the window. Two pieces of the reminder tick
and the off-host pass moved into modules of their own to keep the files
inside their size ratchet.
The image now carries postgresql18-client (pg_dump and pg_restore 18.6)
and a /var/lib/healthlog/backups directory owned by the app user, which
the bundled compose file backs with a new `backups` volume for `app` and
the commented `app-worker`; a fresh named volume inherits that owner.
Before `migrate deploy`, when migrations are pending on a database that
has applied some, the entrypoint writes healthlog-<time>-pre-migration.dump
with its manifest and keeps the last two. A failed dump on a writable
directory stops the boot and names DB_BACKUP_BEFORE_MIGRATE=false; without
a writable directory it warns and continues.
`docker compose run --rm db-restore <file|latest|list>` runs
scripts/db-restore.mjs in the app image with plain node. It opens an
off-host .dump.enc with BACKUP_ENCRYPTION_KEY (or a previous key), checks
the archive against the manifest's SHA-256, compares the keys in the
environment with the manifest's fingerprints and refuses on a different
active encryption key unless --ignore-key-mismatch, refuses a database
that holds any table, restores in one transaction, runs ANALYZE, and
compares every table's row count with the manifest, emptying the database
again on a difference.
DB_BACKUP_ENABLED, DB_BACKUP_DIR, DB_BACKUP_SCHEDULE, DB_BACKUP_KEEP and
DB_BACKUP_BEFORE_MIGRATE join the compose whitelist, both example files
and the env manifest.
The getting-started requirements now say a host needs 3 GB of RAM, 4 GB
recommended, and drop the advice to lower APP_MEMORY_LIMIT to 1g on a
2 GB host. The compose and .env example comments say the same.
At boot the app reads its cgroup memory limit (v2, then v1) and logs one
line when it is below MEMORY_FLOOR_MB. The support report adds the
container limit and a note below the floor, and Admin > System status
shows "below the supported floor" next to the existing low-heap flag.
The app keeps running either way.
Resilience sits behind Oura's `stress` scope, not `heart_health`, so a
connection that granted every scope HealthLog asked for still answered 401
on `daily_resilience`. `src/lib/oura/scope.ts` now maps every collection the
sync reads to the scope the live API enforces and is the one source for the
authorize request, the per-collection permission detection, the reconnect
hint and the settings card. The request asks for `stress` as well.
Oura names a missing scope in its 401 body ("Token is not authorized access
stress scope."), while a dead token reads differently. A collection refusal
now carries that scope: it spends no token refresh, never counts toward the
reconnect threshold and marks only the collections it names. A 403 (a lapsed
membership) is a persistent failure rather than a reconnect. The callback
also records what the token response says was granted, so an unticked
permission shows on the card before the first sync.
A browser can deliver an OAuth callback twice with the same code. Both
requests exchanged it, and Oura answers a reused code by revoking the token
the first exchange issued. Every connect callback (Oura, Polar, Strava,
WHOOP, Withings, Fitbit, Google Health) now claims its state in
`oauth_callback_claims` before the exchange: one request exchanges, the
duplicate waits for it and redirects to the same page without touching any
token. The ledger flows claim inside the transaction that consumes their
state row. A failed exchange settles with its error page, and a new connect
mints a new state.
Oura token refreshes are serialised per user with an advisory lock, so
overlapping syncs no longer spend the same single-use refresh token, and the
write stays conditional so a reconnect's new grant is never overwritten by a
refresh of the old one.
A resumable collection counted the pages it had written across every
backfill attempt and stopped at its ceiling (10,000 for heart rate). A
watch that records heart rate every few seconds reaches that count two to
three years back, and since it never reset, every later slice stopped at
the same place, failed as stuck, and the backfill was never stamped
complete.
A walk is now stopped only when it stops moving: 20 pages in a row that
reach no further back than the pages before them end it. A history
backfill reads back at most GOOGLE_HEALTH_BACKFILL_HORIZON_YEARS (20 when
unset). A data type that ends either way counts as finished, and where and
why it ended is kept in backfillProgress past the completion stamp. An
oldest-first list is read whole, as before. The slice chain's ceiling is
derived from sixty days of back-to-back slices instead of the old page
count. An account left at the old ceiling continues from its edge with
the next slice.
Refs #1220
runtime.heap.sampled records heap, old space, RSS, external memory and
the work in flight every five minutes, every minute at or above 70 % of
the heap limit. At 85 % runtime.heap.pressure lists the oldest requests
and jobs in flight and the routes most of them share, once, re-armed
only after the heap falls below 70 % and thirty minutes pass. A
heap-limit crash leaves no other record of what the process was doing.
HEALTHLOG_HEAP_SNAPSHOT=1 adds one heap snapshot at that moment into
HEALTHLOG_DIAG_DIR (0700 directory, 0600 file), off by default.
docs/ops/heap-diagnostics.md warns that a snapshot holds decrypted
health data and keys and says how to delete it.
Connect, status, places, settings, test, sync and disconnect replace their
stubs. A nightly job (03:20) and the connect backfill read the timeline,
visits and residency endpoints through safeFetch and store one row of daily
values per local day: distances, places visited, minutes away from the home
place and the sealed country with its travel-day verdict. A private instance
needs its exact origin in DAWARICH_PRIVATE_ORIGINS.
The connection joins the shared ledger and the connection health model,
watched for errors only. The day carries a movement block, the model
projection drops the travel country, the two movement channels join the
correlation matrix under the environment module, travel days become trip
suggestions, and the days travel in the environment backup section.
A sync from a watch or an integration now waits the instance's sync gap
(AppSettings.aiSyncRegenGapHours, else AI_SYNC_REGEN_GAP_HOURS, else 6 h)
before it may regenerate a status card again; a hand entry keeps the hour.
Every caller of afterMeasurementMutation and invalidateStatusInsightsForTypes
names its source, so a new ingest path cannot skip the choice.
Refs #1217
The statement timeout and, since v1.42, work_mem reach Postgres as the
libpq options startup parameter, which PgBouncer refuses unless it is in
ignore_startup_parameters. A host behind it could drop the string by
setting the timeout to 0; work_mem is now always sent, so that no longer
worked. DATABASE_SESSION_OPTIONS_DISABLED sends no session settings,
documented in the example env, the env manifest, the compose whitelist
and the scaling guide.
Land every file the v1.42 work shares before any of it is built, so the
parts built side by side never edit the same file.
Schema and migrations 0379-0386: the HEALTH_CONNECT measurement source,
ImportJob.kind, the air-quality columns and the per-user switch on the
environment tables, sealed environment locations with nullable plaintext,
managed-profile handover links, the per-type HealthKit arrival ledger, lab
unit-conversion provenance, and user-defined vaccines. Every new column is
annotated for the column-reader sweep.
HEALTH_CONNECT reaches every exhaustive list: the source labels, the
read enum (not the write allowlist), the source-priority ladder cap, now
the number of sources rather than a fixed eight, and the write-path smoke
test. RMSSD gets its own priority class and a 300 ms ceiling. The batch
syncTrigger gains `manual`, with its label on the Apple Health card.
Queues for the Health Connect import, the compaction-tombstone purge and
the measurement-table maintenance are registered with stub handlers;
host-metric sampling moves to every five minutes. Environment and work_mem
variables reach the compose whitelist, the production example and the
manifest. The proxy admits the handover link and redirects it at the edge,
and the Health Connect upload bypasses the proxy body limit.
The new routes answer 501 behind their final authentication and are
published in the contract; backup, wipe and key rotation classify the new
models and columns; the seven message bundles carry the base copy. Guards
that list release-pending keys and columns fail once an entry is wired, so
those lists empty themselves before the tag.
Also re-anchor a login-throttle fixture to the wall clock, where its
expiry is compared, and re-base the aggregate bundle cap on a measured
12 KB of new locale strings.
Node sizes its heap from the container's memory cap. The 1g default gave
524 MB, which a long-used account fills in normal operation: on a live
instance 490 to 507 MB stayed live after a full GC, garbage collection
stalled requests for up to 8.5 s, and the process ran out of heap 26 times
in a week, taking the weekly backup with it. 1536m gives 792 MB. The
example env and the backup runbook say how to lower it on a small host.
At a first start there is no recorded key yet, so an operator facing a
refusal they are certain is false has no row to delete. With
ENCRYPTION_KEY_CHECK=warn (default enforce) the check still runs and logs
its full outcome, /api/health names a mismatch as a warning field without
failing, requests are served, and no key is recorded unless every key id
passes.
The variable is on the compose whitelist, in both example files, the env
manifest, the Unraid and Portainer templates and the published-env guard.
The log block, the mismatch page and the self-hosting pages offer it as the
last resort after the three real fixes, and say that it disables a safety
check.
Every configured key id now has a canary in encryption_key_canaries,
checked at boot. A key that cannot open its canary, or existing data
written before the canary existed, puts the process in a refusal state:
every API route answers 503 encryption.key_mismatch, /api/health reports
reason encryption_key_mismatch, the pages render one explanation, and
neither the queue producer nor the worker starts. The process stays up
so the reason is visible instead of hiding in a crash loop.
The admin Encryption page gains a "Back up your encryption key" step:
key id and fingerprint (never the key), where the key lives per
platform, a check of a pasted copy that is compared and discarded, and
a confirmation bound to the active key id and fingerprint. The overview
gets a tile and admins get a dashboard banner while it is due.
HEALTHLOG_PLATFORM picks the platform the step opens on. Adds the
TrueNAS self-hosting page and key backup sections in the docs.
With TRUST_PROXY_HOPS=0 behind a proxy that sends only X-Real-IP, every
client address resolves to nothing and all anonymous sign-ins share one
rate-limit bucket, with nothing in the log to say why. The first request
that carries a forwarding header the setting does not let HealthLog read
now logs one warning naming the header and the setting to change.
X-Real-IP stood in for a missing X-Forwarded-For at any hop count. It is a
single address, so it is now read only with TRUST_PROXY_HOPS=1; with two
or more hops the chain the count describes is X-Forwarded-For.
The upload, import and native batch routes that bypass the proxy now carry
Cross-Origin-Resource-Policy and Cross-Origin-Embedder-Policy like every
other response, and so do the proxy's early exits.
The env examples described an unset TRUST_PROXY_HOPS as right for a
directly exposed host; unset means one hop, and that host wants 0.
Making `=true` grant nothing would stop a single-user self-host's local
model on a patch update. It is deprecated for one release cycle instead,
in a narrowed and safe form: it still grants a private host, but only to
the operator's own configurations (the instance-wide admin provider and
AI settings saved on an admin account), and only through the pinned
operator-approved dispatcher, so redirects stay forbidden and metadata,
link-local and unspecified addresses stay refused. A non-admin account's
own base URL needs an exact origin in AI_PRIVATE_ORIGINS.
Provider resolution passes the owning account's role to the client
(`operatorTrusted`); the save route and the connection test apply the
same rule to the account being configured. The boot summary now says the
setting is deprecated and names the replacement, and the admin AI
settings page shows the exact AI_PRIVATE_ORIGINS line for the private
origins saved on the instance (every account's Local and gateway URL and
the admin base URL, scheme, host and port only), so the operator can copy
it and drop the old setting.
Off-host objects are now written as envelope version 4: the header names
the off-host key by fingerprint, and the object's key in the bucket
(<date>/user-<id>.json.enc) is authenticated as GCM associated data, so
a copy moved to another account or another night no longer opens.
Objects of versions 1 to 3 still read. BACKUP_ENCRYPTION_PREVIOUS_KEYS
(comma-separated, on the compose whitelist and in the env manifest)
keeps retired off-host keys readable, which is the rotation path the
off-host key did not have.
The admin backup upload counted the bytes that arrived and none after
the gunzip, so a small gzip could expand without limit into the JSON
scanner and take the app down. The inflated bytes are now held to 100
times the compressed bytes read (at least 256 MB, at most 4 GB) and a
file past that is refused with 413 before anything is stored.
ALLOW_LOCAL_AI_PRIVATE_HOSTS=true let any user of the instance point the
Local provider or the OpenAI-compatible gateway at any private address
the server could reach, the metadata range included, and a granted host
was dialled with no connect-time pin at all.
AI_PRIVATE_ORIGINS grants a private AI endpoint with the exact-origin
grammar NOTIFICATION_PRIVATE_ORIGINS and NIGHTSCOUT_PRIVATE_ORIGINS use.
A granted call goes through the pinned operator-approved dispatcher:
redirects forbidden, and metadata, link-local and unspecified answers
dropped at dial time even for a granted name. Everything else stays on
the public pin. The Local client (buffered and streaming) and the
gateway and admin-key tags of the OpenAI client all take their policy
from one function.
The host-list form of ALLOW_LOCAL_AI_PRIVATE_HOSTS keeps working with the
same pinning; a never-grantable literal or a host:port entry is refused
with a one-time warning. Its "true" form grants nothing any more and is
reported in the boot readiness summary with the replacement.
A failed lab-report scan no longer puts the endpoint's response body into
the wide event; status and model stay. The response to the caller never
carried it.
Two requests left the server without anything being connected: the
HaveIBeenPwned range query on every password set (registration, change,
admin reset) and the daily GitHub releases request behind the About page
and the admin version tile. Neither could be turned off, which made
"nothing leaves" untrue for an instance that wanted it to be.
PASSWORD_BREACH_CHECK_DISABLED and UPDATE_CHECK_DISABLED (1, true, yes or
on) stop each request. Both stay on by default. The breach check keeps
its fail-open contract: switched off, a password is simply not looked up.
The update route answers status "unknown" with reason "disabled" and
the admin tile says the check is off. The password routes reach the
check only through the switch, and a test holds that import line.
Both are on the compose whitelist, in the env manifest under a new
"Outbound requests" group with IP_GEO_LOOKUP_DISABLED, and documented in
both env examples and the getting-started guide.
docker-compose.yml forwards its whitelist as "${NAME:-}", so a variable
the operator never set reaches the process as the empty string. Reads
written as a nullish fallback kept that string: the weather client built
an empty base URL and every geocode failed with ERR_INVALID_URL, and the
IP geo lookup requested a relative path.
Reads with a fallback now go through src/lib/env.ts (envValue, envOr,
envFlag), which treats empty and whitespace as unset. IP_GEO_LOOKUP_DISABLED
accepts 1, true, yes and on instead of only "1". A structural test refuses
a ?? fallback written directly against process.env anywhere in src/, and
the compose whitelist guard counts the helper's call shape as a read.
A person stores one connection per system (base address, Papra
organization id, API token encrypted at rest) and can search it by
document name with tag and date filters, then import picked documents
one per request through the same ingest path as an upload: source-key
answer before any download, content dedup, deleted stays deleted, quota,
provenance, and the person's own AI reading setting. An import can link
the document to the visit, condition or vaccination it was picked for.
The operator lists the instances in DOCUMENT_SOURCE_ORIGINS; unset, the
feature is off. Only listed origins are contacted, through safeFetch with
the origin pinned, redirects refused, timeouts and byte caps. The routes
are cookie-only and owner-only. The token is registered for rotation,
removed by Delete all data, and not carried in backups.
Every upload that sends a file counts towards the token's hourly limit,
workflow re-sends included, and a spent lookup allowance makes the
script wait rather than skip the wait. Note the 20 extra source ids a
document may collect, and name the settings section in the environment
example as the app shows it.
A stored backup was one value in data_backups.data. v1.39.1 wrote it a
piece at a time but still joined the pieces into that one value, and one
value has to pass through the app whole, so a copy was capped at a fifth of
the heap: 105 MB in the default 1 GB container. An account with 1.75
million readings is past that and could not be backed up at all (#1031).
A copy is now kept as ordered pieces of about a megabyte in the new
data_backup_chunks table (migration 0352). Each piece is sealed on its own
with AES-256-GCM, and its sealed header names the copy (a random stream id
kept on the row), its position and whether it is the last. A reader checks
every piece before it releases a byte, so a piece that is missing, moved,
altered, taken from another copy, or a copy cut short is refused with
backup.payload.undecryptable before a restore deletes anything, and the
second read during the restore checks each piece again.
The write is one transaction, so the previous copy stays whole and readable
until the new one commits. Pieces are written through the model API: a raw
query with a large parameter is kept alive by Prisma's query plan cache,
which left 160 MB of heap behind per hundred one-megabyte inserts.
Restore, preview, download, the restore job and the admin list read both
forms; copies written before this change keep their single value and still
restore. The heap-derived limit is gone. The one limit left is on storage,
BACKUP_MAX_STORED_MB (default 2048), and its message names the setting. Key
rotation covers the pieces, the wipe and account deletion take them with
their backup, and the backup plan lists the table as not carried.
Measured on an account of 2.6 million readings in a 1 GB container with web
and worker in one process: backup 87 s, 97 MiB in 96 pieces, preview 20 s,
download 25 s, restore 264 s with every reading back exactly, peak 567 MiB,
no restart.
A narrow `documents:write` token, minted cookie-only at
`POST /api/tokens/documents` and accepted by the vault upload route alone.
A scoped caller draws on its own hourly bucket per token
(`DOCUMENT_UPLOAD_LIMIT_PER_HOUR`, default 120) and gets a receipt
`{ id, duplicate }` instead of the stored row.
Uploads take optional `sourceSystem` / `sourceId` fields. The source key is
unique per account across live and deleted documents, and the tombstone
purge carries it to a small ledger, so re-sending a document the owner
deleted answers 200 `deleted: true` and stores nothing. `aiRead=defer`
indexes locally and skips the summary and lab staging.
The detail sheet shows where an imported document came from; backup and
restore carry both fields; Delete All Data clears the ledger.
Refs #1038, #938
A self-hoster scanning a stack of lab reports ran into a fixed six scans
an hour with no way to raise it. LABS_OCR_LIMIT_PER_HOUR now sets the
per-user ceiling (default 6, clamped to 1-1000, falling back to the
default when unset or not a number), the same shape as
DOCUMENT_AI_LIMIT_PER_HOUR. It is on the compose whitelist, in the
production example, the env manifest and the AI provider guide.
The route also charged the slot before it had looked at the request, so
a malformed body, an unsupported or oversized file, a PDF that could not
be rendered or a scan the daily budget refused still used up the hour.
Those failures hand the slot back now; a scan that reached the provider
keeps it. The 429 carries meta.retryAt. Refs #985
LOKI_ENDPOINT is read only from the server environment, so the push now
dials its exact origin through the pinned operator-approved resolver
instead of the public-host pin that refused a Loki on the operator's
own network. Link-local, metadata and unspecified addresses stay
refused.
A thrown push or a non-2xx answer used to vanish in an empty catch. It
now writes one line to stderr per reason at most every five minutes,
naming the Loki origin, the reason and the dropped event count, without
credentials. The endpoint accepts both the base URL and the full push
URL; the second form used to double the path into a silent 404.
Refs #947
A Nightscout self-host on host networking with
NIGHTSCOUT_PRIVATE_ORIGINS=http://localhost:1337 worked before and must
keep working unchanged; an operator who lists an exact loopback origin
with its port has made the same deliberate decision as one who lists a
LAN origin. The never-grantable floor is now the unspecified address,
link-local and the metadata endpoint only. Loopback (127/8, ::1,
localhost) is grantable when listed exactly, still resolved and pinned
at connect time with redirects forbidden, and still refused when not
listed.
::1 is answered before the embedded-IPv4 unwrap, which read it as the
IPv4-compatible spelling of 0.0.0.1 and refused it as unspecified. The
pinned Nightscout transport test runs the real floor again. Each refused
range keeps its own test; loopback moves to the grantable-when-listed
set with a test that an unlisted loopback is still refused, and a
Nightscout grant of http://localhost:1337 is asserted to parse.
Refs #947
Put the variable on the compose environment whitelist (without it a
value in .env never reaches the process), on both example files, and in
the env manifest so check-env reports it. The self-hosting notifications
page gains a section on notification targets on a private network: the
exact-origin grammar, what stays refused even when listed, and a table
of which target is refused where so an operator knows which lever to
pull.
Refs #947
Marc-André Bombeck