From 0d223fdd9c9e4577f9fa72b19f9a3cd3ec8f98e7 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Marc-Andr=C3=A9=20Bombeck?= Date: Fri, 25 Sep 2026 21:48:12 +0200 Subject: [PATCH] docs: add the document import guide Document tokens, the Paperless-ngx workflow recipe, the import script for Paperless-ngx and Papra, scheduling for Papra, storage, and DOCUMENT_UPLOAD_LIMIT_PER_HOUR. Refs #1038 --- docs/integrations/data-import.md | 4 + docs/integrations/document-import.md | 203 +++++++++++++++++++++++++++ 2 files changed, 207 insertions(+) create mode 100644 docs/integrations/document-import.md diff --git a/docs/integrations/data-import.md b/docs/integrations/data-import.md index 775093405..5e3d43edd 100644 --- a/docs/integrations/data-import.md +++ b/docs/integrations/data-import.md @@ -19,6 +19,10 @@ HealthLog has four ways to bring data in, all reachable from All paths skip rows that already exist, so re-running an import is safe — it merges rather than duplicating. +Documents (letters, lab reports, scans) from Paperless-ngx or Papra come in +through a document token and an import script instead; see +[`document-import.md`](./document-import.md). + Every import reports what it actually wrote. A file where nothing was written is shown as a failure, a file where some rows landed and some were refused is shown as a partial result, and the refused rows are diff --git a/docs/integrations/document-import.md b/docs/integrations/document-import.md new file mode 100644 index 000000000..51d74e95e --- /dev/null +++ b/docs/integrations/document-import.md @@ -0,0 +1,203 @@ +# Importing documents from Paperless-ngx or Papra + +HealthLog's document vault can take documents from another document system +without you uploading them one by one. There are two ways in, and both use +the same kind of credential: a **document token**. + +- **Paperless-ngx** can send each newly tagged document to HealthLog by + itself, through a workflow. Good for new documents as they arrive. +- The **import script** copies documents from Paperless-ngx or Papra in one + go. Good for the archive you already have, and, run on a schedule, for + keeping Papra in step. + +HealthLog never holds a password or token for Paperless-ngx or Papra. They +push into HealthLog; HealthLog does not reach into them. + +## 1. Turn on Documents and create a document token + +1. In HealthLog, turn on **Documents** under **Settings → Modules** if you + have not already. Uploads are refused while the vault is off. +2. Open **Settings → API & Tokens → Document tokens**, give the token a name (for + example "Paperless workflow"), and select **Create**. +3. Copy the token (it starts with `hlk_`). It is shown once. + +A document token can add documents to your own vault and do nothing else. It +cannot list, open, download, change or delete a document, cannot use the AI +features, and cannot create another token. When it uploads a file it gets +back only whether the file was stored, not the title or anything else about +your vault. You can revoke it at any time under **Settings → API & Tokens → +API Tokens**. It expires after a year unless you revoke it sooner. + +## 2. New documents from Paperless-ngx (workflow) + +In Paperless-ngx, open **Workflows** and add one: + +- **Triggers:** add _Document Added_ and _Document Updated_. On each, set + the filter to "has any of these tags" and pick the tag you use for health + documents, for example `HealthLog`. +- **Action:** _Webhook_. + - URL: `https:///api/documents/inbound` + - **Use parameters** (not JSON), with these parameters: + + | Name | Value | + | -------------- | ---------------------------------------------------- | + | `title` | `{{doc_title}}` | + | `documentDate` | `{{created_year}}-{{created_month}}-{{created_day}}` | + | `sourceSystem` | `PAPERLESS` | + | `sourceId` | `{{doc_id}}` | + + - **Include document:** on. Paperless then sends the original file along + with the parameters, which is exactly what HealthLog expects. + - **Headers:** `Authorization` = `Bearer hlk_…` (your document token). + +Optionally add a parameter `kind` with one of `DOCTOR_REPORT`, +`DISCHARGE_LETTER`, `LAB_RESULT`, `IMAGING`, `PRESCRIPTION`, `REFERRAL`, +`INSURANCE`, `VACCINATION` or `OTHER` if the workflow only ever sends one +kind of document. Without it, documents are filed as _Other_ and you can +change the type in HealthLog. + +Tagging a document again, or editing it in Paperless, fires the workflow +again. That is harmless: HealthLog recognises the Paperless document id and +answers "already stored". If you delete the document in HealthLog, a later +re-send does not bring it back. + +**What the workflow is not for.** Paperless gives up on a webhook after +three quick retries and does not retry at all when a request takes longer +than five seconds. If you tag hundreds of documents at once, most of those +webhooks arrive while HealthLog is still asking them to slow down, and +Paperless drops them. Use the import script below for the existing archive +and let the workflow handle what comes after. + +## 3. The archive: the import script + +The script is a single file, `scripts/import-documents.mjs` in the +HealthLog repository. It needs Node 22 and nothing else, and it runs on any +machine that can reach both systems. Download it: + +```sh +curl -O https://raw.githubusercontent.com/MBombeck/HealthLog/main/scripts/import-documents.mjs +``` + +Tokens go in environment variables, never on the command line: + +```sh +export HEALTHLOG_TOKEN=hlk_… # the document token +export PAPERLESS_TOKEN=… # Paperless: My Profile → API auth token +export PAPRA_TOKEN=… # Papra: an API key with documents:read and tags:read +``` + +**Paperless-ngx:** + +```sh +node import-documents.mjs paperless \ + --paperless-url https://paperless.example \ + --healthlog-url https://health.example \ + --tag HealthLog +``` + +**Papra** (the organization id is in the Papra address bar, +`/organizations//…`): + +```sh +node import-documents.mjs papra \ + --papra-url https://papra.example \ + --papra-org \ + --healthlog-url https://health.example \ + --tag Health +``` + +No Node on the machine? Run it in a throwaway container: + +```sh +docker run --rm -it -v "$PWD:/w" -w /w \ + -e HEALTHLOG_TOKEN -e PAPERLESS_TOKEN \ + node:22-alpine node import-documents.mjs paperless … +``` + +Options: + +- `--tag ` copies only documents with that tag. Recommended; without + it the script copies everything the source token can see. +- `--since YYYY-MM-DD` copies only documents dated on or after that day. +- `--kind LAB_RESULT` files everything as one type. `--kind-map +"Befund=DOCTOR_REPORT,Labor=LAB_RESULT"` picks the type by Paperless + document type or by Papra tag; anything unmatched gets `--kind`, or + _Other_. +- `--dry-run` lists what would be copied, with sizes and a total, and sends + nothing. Worth running first. +- `--ai-read` lets HealthLog read the documents with AI as they arrive (see + below). Off by default. + +What it does: + +- It copies one document at a time and keeps only one file in memory. +- It sends the original file (for Paperless, the original rather than the + archived PDF), the title, the document date, and the source id. +- The document date is the date the source has for the document. When the + source has none, the script uses the day the document was added there, and + lists those documents in its summary so you can correct the date. +- Running it again is safe. HealthLog skips documents it already has, and + documents you deleted in HealthLog stay deleted. An interrupted run simply + picks up where it stopped. +- When HealthLog asks it to slow down, it waits as long as HealthLog says + and carries on. By default a document token may upload 120 documents an + hour, so an archive of a thousand documents takes an evening. Uploads you + make yourself in the web app or on your phone are counted separately and + are not held up. +- At the end it prints how many documents were imported, how many were + already there, how many you had deleted, and every document it skipped + with the reason. It exits with `0` when everything went through, `3` when + some documents were skipped, `1` when it had to stop (for example a + refused token, the vault switched off, or storage full), and `2` for a + mistake in the command. + +### AI reading of imported documents + +If you have turned on automatic AI reading of documents, HealthLog normally +reads each new document with your AI provider as it arrives. For an import +the script holds that back, so a whole archive does not turn into a +thousand AI requests at once. Imported documents still get a preview and +are searchable by their text where the file has a text layer. + +To have documents read later, open one and choose **Read with AI** or +**Generate summary**, or use **Index all for search** in Documents for the +ones that had no text to search. Pass `--ai-read` if you do want every +imported document read as it arrives. + +The Paperless workflow does not hold AI reading back: new documents are +treated like any other upload. + +### Keeping Papra in step + +Papra's webhooks only say that something changed, not what the file is, and +cannot carry a token, so Papra cannot post into HealthLog directly. Run the +script on a schedule instead, for example nightly with cron: + +```cron +15 3 * * * cd /opt/healthlog-import && HEALTHLOG_TOKEN=hlk_… PAPRA_TOKEN=… node import-documents.mjs papra --papra-url https://papra.example --papra-org --healthlog-url https://health.example --tag Health --since 2026-01-01 >> import.log 2>&1 +``` + +Documents already copied are recognised and skipped, so a nightly run only +adds what is new. `--since` keeps each run short. + +## Storage + +HealthLog keeps its own encrypted copy of each document, so an imported +archive takes space in both systems. The preview, text search, lab import, +the doctor report and sharing all need the file itself, which is why +HealthLog does not link to documents stored elsewhere. + +The default storage limit is 1 GB per person, counting deleted documents +until they are removed for good 30 days later. Run the script with +`--dry-run` to see how much an import needs. An admin can raise the limit in +the admin area, for everyone or for one account. If storage runs out during +an import, the script stops and says so; raise the limit and run it again. + +## For operators + +`DOCUMENT_UPLOAD_LIMIT_PER_HOUR` sets how many documents one document token +may upload per hour (default `120`, clamped to 1–1000). It applies to +document tokens only; uploads from the web app and the phone keep their own +limit of 60 per hour per person. Raise it to let a large import finish +faster; the storage limit still applies. The variable is on the +`docker-compose.yml` whitelist, so setting it in `.env` is enough.