Merge branch 'feat/document-import-token' into release/v1.39.2

This commit is contained in:
Marc-André Bombeck committed 2026-09-26 00:22:43 +02:00
commit bec79879b5
33 files changed
+1302 -155

No files matched your search

+115 -14
View File
@@ -2927,11 +2927,13 @@ paths:
once; it is stored as an HMAC and no path re-reveals it.
**What it can do.** `POST /api/documents/inbound` on its owner's own record, and nothing else. It is the door
another document system pushes through: a Paperless-ngx workflow posting a newly tagged document, or an import
script copying an archive. Uploads through it draw on a bucket of their own, keyed on the token (default 120 an
hour, operator-tunable via `DOCUMENT_UPLOAD_LIMIT_PER_HOUR`), and get back a receipt (`id`, `duplicate`, and
`deleted` when the source key belongs to a document the owner deleted) rather than the stored row.
**What it can do.** `POST /api/documents/inbound` on its owner's own record, and `GET
/api/documents/inbound/source`, which answers whether a source key is already held so an importer can skip a
download. Nothing else. It is the door another document system pushes through: a Paperless-ngx workflow posting
a newly tagged document, or an import script copying an archive. Uploads through it draw on a bucket of their
own, keyed on the token (default 120 an hour, operator-tunable via `DOCUMENT_UPLOAD_LIMIT_PER_HOUR`), and get
back a receipt (`id`, `duplicate`, and `deleted` when the source key belongs to a document the owner deleted)
rather than the stored row.
**What it cannot do.** List, read, download, preview, edit, delete or bulk-act on a document, reach the
@@ -15714,19 +15716,45 @@ paths:
`sourceId` (its id there, printable, up to 128 characters; needs `sourceSystem`) key the upload. A re-send with
a known key is a duplicate: 200 with the live row (`meta.duplicate: true`), or, when the owner DELETED that
document, 200 with `data.deleted: true` and `meta.deleted: true` and nothing stored; this holds after the 30-day
purge too. `aiRead=defer` holds back automatic AI reading for this upload: the thumbnail and a local text index
still run, the summary and lab staging do not, so the document waits for the person to read it deliberately.
Absent, behaviour is unchanged.
purge too. The key may instead ride the query string (`?sourceSystem=…&sourceId=…`): it is then answered before
the body is read and before the hourly allowance is charged, so a re-send of a document already held costs
neither; a key in both places must agree (else 422). A key that is answered with an existing document by its
bytes is remembered for that document, so deleting it keeps that key deleted too. `OTHER` is one shared
namespace per account: two different systems both sent as `OTHER` must not reuse each other's ids. An upload
answered as a duplicate or as deleted hands its slot of the hourly allowance back. `aiRead=defer` holds back
automatic AI reading for this upload: the thumbnail and a local text index still run, the summary and lab
staging do not, and the document stays out of the summary catch-up until the person reads it with AI or
generates its summary. Absent, behaviour is unchanged.
**Narrow token.** Also reachable with a `documents:write` Bearer (minted at `POST /api/tokens/documents`), and
it is the only route that scope reaches. Such a caller draws on a bucket of its own, keyed on the token (default
120 an hour, `DOCUMENT_UPLOAD_LIMIT_PER_HOUR`, clamped 1-1000; the 429 carries `Retry-After` and
`X-RateLimit-*`), and gets a `DocumentUploadReceipt` instead of the stored row. A cookie or wildcard caller
keeps the 60-an-hour per-user bucket and the full row. With the vault module off the answer is 403
`module.disabled` for every caller.
**Narrow token.** Also reachable with a `documents:write` Bearer (minted at `POST /api/tokens/documents`), which
reaches only this route and the source-key lookup `GET /api/documents/inbound/source`. Such a caller draws on a
bucket of its own, keyed on the token (default 120 an hour, `DOCUMENT_UPLOAD_LIMIT_PER_HOUR`, clamped 1-1000;
the 429 carries `Retry-After` and `X-RateLimit-*`), and gets a `DocumentUploadReceipt` instead of the stored
row; its 413 `quotaExceeded` carries no `quotaBytes` / `usedBytes`. The response is an untagged union (row or
receipt) on purpose: a discriminator would need a new field on the row every existing client decodes. A cookie
or wildcard caller keeps the 60-an-hour per-user bucket and the full row. With the vault module off the answer
is 403 `module.disabled` for every caller.
parameters:
- *a6
- name: sourceSystem
in: query
required: false
schema:
type: string
enum:
- PAPERLESS
- PAPRA
- OTHER
description: Import source key, first half. Sent here instead of (or agreeing with) the form field, it is checked before
the body is read or the hourly allowance is charged.
- name: sourceId
in: query
required: false
schema:
type: string
maxLength: 128
description: Import source key, second half. Both halves or neither; half a key is 422.
requestBody:
required: true
content:
@@ -15822,6 +15850,44 @@ paths:
$ref: "#/components/schemas/ErrorEnvelope"
"422": *a4
"429": *a5
/api/documents/inbound/source:
get:
tags:
- Documents
summary: Look up an import source key
description: "Whether the vault already holds a document under this import source key, so an importer can skip a
document before downloading it from the source system. Answers exactly what an upload with the same key would:
`known`, the document `id` (null once a deleted document was purged), and whether the owner `deleted` it. Checks
the document's own key, keys remembered for it when an import sent the same bytes under another key, and the
keys of purged documents. Reachable with a `documents:write` Bearer (the scope's second and last route) or a
cookie session; stores nothing. 5000 lookups an hour per token (per person for a session); gated on the
`inboundDocuments` module."
parameters:
- name: sourceSystem
in: query
required: true
schema:
type: string
enum:
- PAPERLESS
- PAPRA
- OTHER
- name: sourceId
in: query
required: true
schema:
type: string
maxLength: 128
responses:
"200":
description: The answer for this key.
content:
application/json:
schema:
$ref: "#/components/schemas/DocumentSourceLookupEnvelope"
"401": *a1
"422": *a4
"429": *a5
/api/documents/inbound/capability:
get:
tags:
@@ -35305,6 +35371,8 @@ components:
- type: string
maxLength: 128
- type: "null"
aiReadDeferred:
type: boolean
summary:
anyOf:
- type: string
@@ -63920,6 +63988,39 @@ components:
stored, not its title, filename or links. Every caller gets it when the upload's source key belongs to a
document the owner DELETED (`deleted: true`, 200, nothing stored); `id` is the tombstoned row, or null once the
30-day purge removed it."
DocumentSourceLookupEnvelope:
type: object
properties:
data:
$ref: "#/components/schemas/DocumentSourceLookup"
error:
type: "null"
meta:
type: object
properties:
requestId:
type: string
additionalProperties: false
required:
- data
- error
additionalProperties: false
DocumentSourceLookup:
type: object
properties:
known:
type: boolean
id:
anyOf:
- type: string
- type: "null"
deleted:
type: boolean
required:
- known
- id
- deleted
additionalProperties: false
DocumentAiCapabilityEnvelope:
type: object
properties:
+29 -9
View File
@@ -30,7 +30,9 @@ API Tokens**. It expires after a year unless you revoke it sooner.
## 2. New documents from Paperless-ngx (workflow)
In Paperless-ngx, open **Workflows** and add one:
This needs Paperless-ngx 3.0 or later (the `{{doc_id}}` placeholder and the
double-brace placeholder syntax below arrived there). In Paperless-ngx, open
**Workflows** and add one:
- **Triggers:** add _Document Added_ and _Document Updated_. On each, set
the filter to "has any of these tags" and pick the tag you use for health
@@ -71,7 +73,8 @@ and let the workflow handle what comes after.
## 3. The archive: the import script
The script is a single file, `scripts/import-documents.mjs` in the
HealthLog repository. It needs Node 22 and nothing else, and it runs on any
HealthLog repository. It works with Paperless-ngx 2.16 or later (API
version 9) and current Papra releases. It needs Node 22 and nothing else, and it runs on any
machine that can reach both systems. Download it:
```sh
@@ -136,14 +139,20 @@ What it does:
- The document date is the date the source has for the document. When the
source has none, the script uses the day the document was added there, and
lists those documents in its summary so you can correct the date.
- Running it again is safe. HealthLog skips documents it already has, and
documents you deleted in HealthLog stay deleted. An interrupted run simply
picks up where it stopped.
- Running it again is safe. Before downloading a document the script asks
HealthLog whether it already has it, so documents already there are
neither downloaded nor sent again, and a re-run over a large archive is
quick. Documents you deleted in HealthLog stay deleted, also after they
are removed for good. The same goes for a file you had already uploaded
by hand: the import recognises it by its content, and if you delete it
later, the import leaves it deleted. An interrupted run simply picks up
where it stopped.
- When HealthLog asks it to slow down, it waits as long as HealthLog says
and carries on. By default a document token may upload 120 documents an
hour, so an archive of a thousand documents takes an evening. Uploads you
make yourself in the web app or on your phone are counted separately and
are not held up.
hour, so an archive of a thousand documents takes an evening. Documents
HealthLog already has do not count towards that. Uploads you make yourself
in the web app or on your phone are counted separately and are not held
up.
- At the end it prints how many documents were imported, how many were
already there, how many you had deleted, and every document it skipped
with the reason. It exits with `0` when everything went through, `3` when
@@ -159,7 +168,9 @@ the script holds that back, so a whole archive does not turn into a
thousand AI requests at once. Imported documents still get a preview and
are searchable by their text where the file has a text layer.
To have documents read later, open one and choose **Read with AI** or
HealthLog remembers that these documents were held back: turning on
automatic AI reading later does not send them to your AI provider either.
To have documents read, open one and choose **Read with AI** or
**Generate summary**, or use **Index all for search** in Documents for the
ones that had no text to search. Pass `--ai-read` if you do want every
imported document read as it arrives.
@@ -193,6 +204,15 @@ until they are removed for good 30 days later. Run the script with
the admin area, for everyone or for one account. If storage runs out during
an import, the script stops and says so; raise the limit and run it again.
## Source ids
Documents are recognised by the pair of source system and document id:
`PAPERLESS` plus the Paperless document id, `PAPRA` plus the Papra document
id. Anything else is sent as `OTHER`, and `OTHER` is one shared set of ids
per account: if you push documents from two other systems, make sure their
ids cannot collide, for example by prefixing them (`nextcloud-123`,
`scanner-123`).
## For operators
`DOCUMENT_UPLOAD_LIMIT_PER_HOUR` sets how many documents one document token