mirror of
https://github.com/MBombeck/HealthLog.git
synced 2026-11-09 21:10:59 +00:00
Merge branch 'feat/document-import-token' into release/v1.39.2
This commit is contained in:
Marc-André Bombeck committed
commit
bec79879b5
33 files changed
+1302
-155
No files matched your search
+115
-14
@@ -2927,11 +2927,13 @@ paths:
|
||||
once; it is stored as an HMAC and no path re-reveals it.
|
||||
|
||||
|
||||
**What it can do.** `POST /api/documents/inbound` on its owner's own record, and nothing else. It is the door
|
||||
another document system pushes through: a Paperless-ngx workflow posting a newly tagged document, or an import
|
||||
script copying an archive. Uploads through it draw on a bucket of their own, keyed on the token (default 120 an
|
||||
hour, operator-tunable via `DOCUMENT_UPLOAD_LIMIT_PER_HOUR`), and get back a receipt (`id`, `duplicate`, and
|
||||
`deleted` when the source key belongs to a document the owner deleted) rather than the stored row.
|
||||
**What it can do.** `POST /api/documents/inbound` on its owner's own record, and `GET
|
||||
/api/documents/inbound/source`, which answers whether a source key is already held so an importer can skip a
|
||||
download. Nothing else. It is the door another document system pushes through: a Paperless-ngx workflow posting
|
||||
a newly tagged document, or an import script copying an archive. Uploads through it draw on a bucket of their
|
||||
own, keyed on the token (default 120 an hour, operator-tunable via `DOCUMENT_UPLOAD_LIMIT_PER_HOUR`), and get
|
||||
back a receipt (`id`, `duplicate`, and `deleted` when the source key belongs to a document the owner deleted)
|
||||
rather than the stored row.
|
||||
|
||||
|
||||
**What it cannot do.** List, read, download, preview, edit, delete or bulk-act on a document, reach the
|
||||
@@ -15714,19 +15716,45 @@ paths:
|
||||
`sourceId` (its id there, printable, up to 128 characters; needs `sourceSystem`) key the upload. A re-send with
|
||||
a known key is a duplicate: 200 with the live row (`meta.duplicate: true`), or, when the owner DELETED that
|
||||
document, 200 with `data.deleted: true` and `meta.deleted: true` and nothing stored; this holds after the 30-day
|
||||
purge too. `aiRead=defer` holds back automatic AI reading for this upload: the thumbnail and a local text index
|
||||
still run, the summary and lab staging do not, so the document waits for the person to read it deliberately.
|
||||
Absent, behaviour is unchanged.
|
||||
purge too. The key may instead ride the query string (`?sourceSystem=…&sourceId=…`): it is then answered before
|
||||
the body is read and before the hourly allowance is charged, so a re-send of a document already held costs
|
||||
neither; a key in both places must agree (else 422). A key that is answered with an existing document by its
|
||||
bytes is remembered for that document, so deleting it keeps that key deleted too. `OTHER` is one shared
|
||||
namespace per account: two different systems both sent as `OTHER` must not reuse each other's ids. An upload
|
||||
answered as a duplicate or as deleted hands its slot of the hourly allowance back. `aiRead=defer` holds back
|
||||
automatic AI reading for this upload: the thumbnail and a local text index still run, the summary and lab
|
||||
staging do not, and the document stays out of the summary catch-up until the person reads it with AI or
|
||||
generates its summary. Absent, behaviour is unchanged.
|
||||
|
||||
|
||||
**Narrow token.** Also reachable with a `documents:write` Bearer (minted at `POST /api/tokens/documents`), and
|
||||
it is the only route that scope reaches. Such a caller draws on a bucket of its own, keyed on the token (default
|
||||
120 an hour, `DOCUMENT_UPLOAD_LIMIT_PER_HOUR`, clamped 1-1000; the 429 carries `Retry-After` and
|
||||
`X-RateLimit-*`), and gets a `DocumentUploadReceipt` instead of the stored row. A cookie or wildcard caller
|
||||
keeps the 60-an-hour per-user bucket and the full row. With the vault module off the answer is 403
|
||||
`module.disabled` for every caller.
|
||||
**Narrow token.** Also reachable with a `documents:write` Bearer (minted at `POST /api/tokens/documents`), which
|
||||
reaches only this route and the source-key lookup `GET /api/documents/inbound/source`. Such a caller draws on a
|
||||
bucket of its own, keyed on the token (default 120 an hour, `DOCUMENT_UPLOAD_LIMIT_PER_HOUR`, clamped 1-1000;
|
||||
the 429 carries `Retry-After` and `X-RateLimit-*`), and gets a `DocumentUploadReceipt` instead of the stored
|
||||
row; its 413 `quotaExceeded` carries no `quotaBytes` / `usedBytes`. The response is an untagged union (row or
|
||||
receipt) on purpose: a discriminator would need a new field on the row every existing client decodes. A cookie
|
||||
or wildcard caller keeps the 60-an-hour per-user bucket and the full row. With the vault module off the answer
|
||||
is 403 `module.disabled` for every caller.
|
||||
parameters:
|
||||
- *a6
|
||||
- name: sourceSystem
|
||||
in: query
|
||||
required: false
|
||||
schema:
|
||||
type: string
|
||||
enum:
|
||||
- PAPERLESS
|
||||
- PAPRA
|
||||
- OTHER
|
||||
description: Import source key, first half. Sent here instead of (or agreeing with) the form field, it is checked before
|
||||
the body is read or the hourly allowance is charged.
|
||||
- name: sourceId
|
||||
in: query
|
||||
required: false
|
||||
schema:
|
||||
type: string
|
||||
maxLength: 128
|
||||
description: Import source key, second half. Both halves or neither; half a key is 422.
|
||||
requestBody:
|
||||
required: true
|
||||
content:
|
||||
@@ -15822,6 +15850,44 @@ paths:
|
||||
$ref: "#/components/schemas/ErrorEnvelope"
|
||||
"422": *a4
|
||||
"429": *a5
|
||||
/api/documents/inbound/source:
|
||||
get:
|
||||
tags:
|
||||
- Documents
|
||||
summary: Look up an import source key
|
||||
description: "Whether the vault already holds a document under this import source key, so an importer can skip a
|
||||
document before downloading it from the source system. Answers exactly what an upload with the same key would:
|
||||
`known`, the document `id` (null once a deleted document was purged), and whether the owner `deleted` it. Checks
|
||||
the document's own key, keys remembered for it when an import sent the same bytes under another key, and the
|
||||
keys of purged documents. Reachable with a `documents:write` Bearer (the scope's second and last route) or a
|
||||
cookie session; stores nothing. 5000 lookups an hour per token (per person for a session); gated on the
|
||||
`inboundDocuments` module."
|
||||
parameters:
|
||||
- name: sourceSystem
|
||||
in: query
|
||||
required: true
|
||||
schema:
|
||||
type: string
|
||||
enum:
|
||||
- PAPERLESS
|
||||
- PAPRA
|
||||
- OTHER
|
||||
- name: sourceId
|
||||
in: query
|
||||
required: true
|
||||
schema:
|
||||
type: string
|
||||
maxLength: 128
|
||||
responses:
|
||||
"200":
|
||||
description: The answer for this key.
|
||||
content:
|
||||
application/json:
|
||||
schema:
|
||||
$ref: "#/components/schemas/DocumentSourceLookupEnvelope"
|
||||
"401": *a1
|
||||
"422": *a4
|
||||
"429": *a5
|
||||
/api/documents/inbound/capability:
|
||||
get:
|
||||
tags:
|
||||
@@ -35305,6 +35371,8 @@ components:
|
||||
- type: string
|
||||
maxLength: 128
|
||||
- type: "null"
|
||||
aiReadDeferred:
|
||||
type: boolean
|
||||
summary:
|
||||
anyOf:
|
||||
- type: string
|
||||
@@ -63920,6 +63988,39 @@ components:
|
||||
stored, not its title, filename or links. Every caller gets it when the upload's source key belongs to a
|
||||
document the owner DELETED (`deleted: true`, 200, nothing stored); `id` is the tombstoned row, or null once the
|
||||
30-day purge removed it."
|
||||
DocumentSourceLookupEnvelope:
|
||||
type: object
|
||||
properties:
|
||||
data:
|
||||
$ref: "#/components/schemas/DocumentSourceLookup"
|
||||
error:
|
||||
type: "null"
|
||||
meta:
|
||||
type: object
|
||||
properties:
|
||||
requestId:
|
||||
type: string
|
||||
additionalProperties: false
|
||||
required:
|
||||
- data
|
||||
- error
|
||||
additionalProperties: false
|
||||
DocumentSourceLookup:
|
||||
type: object
|
||||
properties:
|
||||
known:
|
||||
type: boolean
|
||||
id:
|
||||
anyOf:
|
||||
- type: string
|
||||
- type: "null"
|
||||
deleted:
|
||||
type: boolean
|
||||
required:
|
||||
- known
|
||||
- id
|
||||
- deleted
|
||||
additionalProperties: false
|
||||
DocumentAiCapabilityEnvelope:
|
||||
type: object
|
||||
properties:
|
||||
|
||||
@@ -30,7 +30,9 @@ API Tokens**. It expires after a year unless you revoke it sooner.
|
||||
|
||||
## 2. New documents from Paperless-ngx (workflow)
|
||||
|
||||
In Paperless-ngx, open **Workflows** and add one:
|
||||
This needs Paperless-ngx 3.0 or later (the `{{doc_id}}` placeholder and the
|
||||
double-brace placeholder syntax below arrived there). In Paperless-ngx, open
|
||||
**Workflows** and add one:
|
||||
|
||||
- **Triggers:** add _Document Added_ and _Document Updated_. On each, set
|
||||
the filter to "has any of these tags" and pick the tag you use for health
|
||||
@@ -71,7 +73,8 @@ and let the workflow handle what comes after.
|
||||
## 3. The archive: the import script
|
||||
|
||||
The script is a single file, `scripts/import-documents.mjs` in the
|
||||
HealthLog repository. It needs Node 22 and nothing else, and it runs on any
|
||||
HealthLog repository. It works with Paperless-ngx 2.16 or later (API
|
||||
version 9) and current Papra releases. It needs Node 22 and nothing else, and it runs on any
|
||||
machine that can reach both systems. Download it:
|
||||
|
||||
```sh
|
||||
@@ -136,14 +139,20 @@ What it does:
|
||||
- The document date is the date the source has for the document. When the
|
||||
source has none, the script uses the day the document was added there, and
|
||||
lists those documents in its summary so you can correct the date.
|
||||
- Running it again is safe. HealthLog skips documents it already has, and
|
||||
documents you deleted in HealthLog stay deleted. An interrupted run simply
|
||||
picks up where it stopped.
|
||||
- Running it again is safe. Before downloading a document the script asks
|
||||
HealthLog whether it already has it, so documents already there are
|
||||
neither downloaded nor sent again, and a re-run over a large archive is
|
||||
quick. Documents you deleted in HealthLog stay deleted, also after they
|
||||
are removed for good. The same goes for a file you had already uploaded
|
||||
by hand: the import recognises it by its content, and if you delete it
|
||||
later, the import leaves it deleted. An interrupted run simply picks up
|
||||
where it stopped.
|
||||
- When HealthLog asks it to slow down, it waits as long as HealthLog says
|
||||
and carries on. By default a document token may upload 120 documents an
|
||||
hour, so an archive of a thousand documents takes an evening. Uploads you
|
||||
make yourself in the web app or on your phone are counted separately and
|
||||
are not held up.
|
||||
hour, so an archive of a thousand documents takes an evening. Documents
|
||||
HealthLog already has do not count towards that. Uploads you make yourself
|
||||
in the web app or on your phone are counted separately and are not held
|
||||
up.
|
||||
- At the end it prints how many documents were imported, how many were
|
||||
already there, how many you had deleted, and every document it skipped
|
||||
with the reason. It exits with `0` when everything went through, `3` when
|
||||
@@ -159,7 +168,9 @@ the script holds that back, so a whole archive does not turn into a
|
||||
thousand AI requests at once. Imported documents still get a preview and
|
||||
are searchable by their text where the file has a text layer.
|
||||
|
||||
To have documents read later, open one and choose **Read with AI** or
|
||||
HealthLog remembers that these documents were held back: turning on
|
||||
automatic AI reading later does not send them to your AI provider either.
|
||||
To have documents read, open one and choose **Read with AI** or
|
||||
**Generate summary**, or use **Index all for search** in Documents for the
|
||||
ones that had no text to search. Pass `--ai-read` if you do want every
|
||||
imported document read as it arrives.
|
||||
@@ -193,6 +204,15 @@ until they are removed for good 30 days later. Run the script with
|
||||
the admin area, for everyone or for one account. If storage runs out during
|
||||
an import, the script stops and says so; raise the limit and run it again.
|
||||
|
||||
## Source ids
|
||||
|
||||
Documents are recognised by the pair of source system and document id:
|
||||
`PAPERLESS` plus the Paperless document id, `PAPRA` plus the Papra document
|
||||
id. Anything else is sent as `OTHER`, and `OTHER` is one shared set of ids
|
||||
per account: if you push documents from two other systems, make sure their
|
||||
ids cannot collide, for example by prefixing them (`nextcloud-123`,
|
||||
`scanner-123`).
|
||||
|
||||
## For operators
|
||||
|
||||
`DOCUMENT_UPLOAD_LIMIT_PER_HOUR` sets how many documents one document token
|
||||
|
||||
@@ -29,3 +29,34 @@ CREATE UNIQUE INDEX "document_import_keys_user_id_source_system_source_id_key"
|
||||
ALTER TABLE "document_import_keys"
|
||||
ADD CONSTRAINT "document_import_keys_user_id_fkey"
|
||||
FOREIGN KEY ("user_id") REFERENCES "users"("id") ON DELETE CASCADE ON UPDATE CASCADE;
|
||||
|
||||
-- Imports held back from automatic AI reading (`aiRead=defer`). The summary
|
||||
-- catch-up, the summary job and lab auto-staging skip a row that carries it.
|
||||
ALTER TABLE "inbound_documents"
|
||||
ADD COLUMN "ai_read_deferred" BOOLEAN NOT NULL DEFAULT false;
|
||||
|
||||
-- Further source keys an import answered with an existing document (same
|
||||
-- bytes under another key). The purge copies them into the ledger above.
|
||||
CREATE TABLE "document_source_aliases" (
|
||||
"id" TEXT NOT NULL,
|
||||
"user_id" TEXT NOT NULL,
|
||||
"document_id" TEXT NOT NULL,
|
||||
"source_system" VARCHAR(16) NOT NULL,
|
||||
"source_id" VARCHAR(128) NOT NULL,
|
||||
"created_at" TIMESTAMP(3) NOT NULL DEFAULT CURRENT_TIMESTAMP,
|
||||
CONSTRAINT "document_source_aliases_pkey" PRIMARY KEY ("id")
|
||||
);
|
||||
|
||||
CREATE UNIQUE INDEX "document_source_aliases_user_id_source_system_source_id_key"
|
||||
ON "document_source_aliases" ("user_id", "source_system", "source_id");
|
||||
|
||||
CREATE INDEX "document_source_aliases_document_id_idx"
|
||||
ON "document_source_aliases" ("document_id");
|
||||
|
||||
ALTER TABLE "document_source_aliases"
|
||||
ADD CONSTRAINT "document_source_aliases_user_id_fkey"
|
||||
FOREIGN KEY ("user_id") REFERENCES "users"("id") ON DELETE CASCADE ON UPDATE CASCADE;
|
||||
|
||||
ALTER TABLE "document_source_aliases"
|
||||
ADD CONSTRAINT "document_source_aliases_document_id_fkey"
|
||||
FOREIGN KEY ("document_id") REFERENCES "inbound_documents"("id") ON DELETE CASCADE ON UPDATE CASCADE;
|
||||
@@ -925,6 +925,9 @@ model User {
|
||||
// v1.39.2 — source keys of imported documents that the tombstone purge
|
||||
// removed, so a re-run of an importer cannot bring a deleted document back.
|
||||
documentImportKeys DocumentImportKey[]
|
||||
// v1.39.2 — further source keys an import answered with an existing
|
||||
// document (same bytes under another key). See DocumentSourceAlias.
|
||||
documentSourceAliases DocumentSourceAlias[]
|
||||
extractedFacts ExtractedFact[]
|
||||
documentConditionLinks DocumentConditionLink[]
|
||||
documentContentIndexes DocumentContentIndex[]
|
||||
@@ -7975,6 +7978,11 @@ model InboundDocument {
|
||||
/// back a document the person deleted. The purge carries the key over to
|
||||
/// `DocumentImportKey` when it removes the row.
|
||||
sourceId String? @map("source_id") @db.VarChar(128)
|
||||
/// v1.39.2 (#1038) — the upload asked for `aiRead=defer`: nothing may send
|
||||
/// this document to an AI provider on its own (the summary catch-up, the
|
||||
/// summary job, lab auto-staging all skip it). Cleared when the person
|
||||
/// explicitly reads it with AI or generates its summary.
|
||||
aiReadDeferred Boolean @default(false) @map("ai_read_deferred")
|
||||
|
||||
facts ExtractedFact[]
|
||||
conditionLinks DocumentConditionLink[]
|
||||
@@ -7994,6 +8002,8 @@ model InboundDocument {
|
||||
/// share-link memberships (m:n via `ClinicianShareLinkDocument`).
|
||||
/// Cascades: deleting this document removes it from every share it was on.
|
||||
shareLinks ClinicianShareLinkDocument[]
|
||||
/// Further import source keys that resolved to this document by content.
|
||||
sourceAliases DocumentSourceAlias[]
|
||||
|
||||
createdAt DateTime @default(now()) @map("created_at")
|
||||
updatedAt DateTime @updatedAt @map("updated_at")
|
||||
@@ -8021,6 +8031,30 @@ model InboundDocument {
|
||||
@@map("inbound_documents")
|
||||
}
|
||||
|
||||
/// v1.39.2 (#1038) — a source key an import sent for bytes that were already
|
||||
/// stored under another document (a manual upload, or the same file under a
|
||||
/// second id in the source). The upload answered "duplicate" with that
|
||||
/// document, so the key belongs to it: a re-send resolves to it, and when the
|
||||
/// purge removes the document the key moves to `DocumentImportKey` with its
|
||||
/// own, so deleting the document keeps every key that ever reached it deleted.
|
||||
/// Cascades with the document; the purge copies it first.
|
||||
model DocumentSourceAlias {
|
||||
id String @id @default(cuid())
|
||||
userId String @map("user_id")
|
||||
documentId String @map("document_id")
|
||||
/// Same closed vocabulary as `InboundDocument.sourceSystem`.
|
||||
sourceSystem String @map("source_system") @db.VarChar(16)
|
||||
sourceId String @map("source_id") @db.VarChar(128)
|
||||
createdAt DateTime @default(now()) @map("created_at")
|
||||
|
||||
user User @relation(fields: [userId], references: [id], onDelete: Cascade)
|
||||
document InboundDocument @relation(fields: [documentId], references: [id], onDelete: Cascade)
|
||||
|
||||
@@unique([userId, sourceSystem, sourceId])
|
||||
@@index([documentId])
|
||||
@@map("document_source_aliases")
|
||||
}
|
||||
|
||||
/// v1.39.2 (#1038) — the source key of an imported document that the daily
|
||||
/// tombstone purge has physically removed. While the tombstone exists the key
|
||||
/// lives on the document row itself; once the purge takes the row, this is the
|
||||
|
||||
@@ -86,12 +86,40 @@ async function healthlog(
|
||||
deleted?: string[];
|
||||
tooLarge?: string[];
|
||||
moduleOff?: boolean;
|
||||
/** An older HealthLog with no lookup route. */
|
||||
noLookup?: boolean;
|
||||
/** The address redirects elsewhere (http → https, a moved host). */
|
||||
redirect?: string;
|
||||
} = {},
|
||||
) {
|
||||
const stored = new Set<string>();
|
||||
const received: Received[] = [];
|
||||
const lookups: string[] = [];
|
||||
let throttled = false;
|
||||
const url = await serve(async (req, url, body) => {
|
||||
if (opts.redirect) {
|
||||
return {
|
||||
status: 301,
|
||||
headers: { location: `${opts.redirect}${url.pathname}` },
|
||||
};
|
||||
}
|
||||
const queryKey = `${url.searchParams.get("sourceSystem")}:${url.searchParams.get("sourceId")}`;
|
||||
if (
|
||||
url.pathname === "/api/documents/inbound/source" &&
|
||||
req.method === "GET" &&
|
||||
!opts.noLookup
|
||||
) {
|
||||
lookups.push(queryKey);
|
||||
const deleted = opts.deleted?.includes(queryKey) ?? false;
|
||||
const known = deleted || stored.has(queryKey);
|
||||
return {
|
||||
status: 200,
|
||||
json: {
|
||||
data: { known, id: known ? queryKey : null, deleted },
|
||||
error: null,
|
||||
},
|
||||
};
|
||||
}
|
||||
if (url.pathname !== "/api/documents/inbound" || req.method !== "POST") {
|
||||
return { status: 404, json: { data: null, error: "Not found" } };
|
||||
}
|
||||
@@ -130,8 +158,11 @@ async function healthlog(
|
||||
size = v.size;
|
||||
}
|
||||
}
|
||||
// The key rides the query string, where HealthLog reads it first.
|
||||
fields.sourceSystem = url.searchParams.get("sourceSystem") ?? "";
|
||||
fields.sourceId = url.searchParams.get("sourceId") ?? "";
|
||||
received.push({ auth: req.headers.authorization, fields, filename, size });
|
||||
const key = `${fields.sourceSystem}:${fields.sourceId}`;
|
||||
const key = queryKey;
|
||||
if (opts.tooLarge?.includes(key)) {
|
||||
return {
|
||||
status: 413,
|
||||
@@ -168,7 +199,7 @@ async function healthlog(
|
||||
json: { data: { id: key, duplicate: false }, error: null },
|
||||
};
|
||||
});
|
||||
return { url, received };
|
||||
return { url, received, lookups };
|
||||
}
|
||||
|
||||
/** Paperless-ngx v10 stand-in: two pages of documents, one outside the tag. */
|
||||
@@ -238,7 +269,8 @@ async function paperless() {
|
||||
},
|
||||
{
|
||||
id: 2,
|
||||
title: "Arztbrief",
|
||||
// Blank in the source: the file name stands in.
|
||||
title: " ",
|
||||
created: null,
|
||||
added: "2024-04-01T09:30:00Z",
|
||||
tags: [7],
|
||||
@@ -379,6 +411,7 @@ describe("import-documents.mjs — Paperless-ngx", () => {
|
||||
});
|
||||
// No created date: filed under the day it was added, and said so.
|
||||
expect(hl.received[1].fields.documentDate).toBe("2024-04-01");
|
||||
expect(hl.received[1].fields.title).toBe("brief");
|
||||
expect(hl.received[1].fields.kind).toBeUndefined();
|
||||
expect(first.stdout).toMatch(/Imported: 2/);
|
||||
expect(first.stdout).toMatch(/filed under the day it was added/);
|
||||
@@ -387,7 +420,8 @@ describe("import-documents.mjs — Paperless-ngx", () => {
|
||||
// The API contract the script leans on.
|
||||
const docs = pl.seen.filter((s) => s.path.startsWith("/api/documents/?"));
|
||||
expect(docs[0].path).toContain("tags__id__all=7");
|
||||
expect(docs[0].accept).toBe("application/json; version=10");
|
||||
// API v9: the lowest the current Paperless-ngx releases accept.
|
||||
expect(docs[0].accept).toBe("application/json; version=9");
|
||||
expect(docs[0].auth).toBe("Token pl_test");
|
||||
// `next` was followed on the source's own origin.
|
||||
expect(docs[1].path).toContain("page=2");
|
||||
@@ -398,11 +432,20 @@ describe("import-documents.mjs — Paperless-ngx", () => {
|
||||
"/api/documents/2/download/?original=true",
|
||||
]);
|
||||
|
||||
// A second run changes nothing: HealthLog answers duplicate for both.
|
||||
// A second run changes nothing: HealthLog already holds both, and the
|
||||
// lookup says so before anything is downloaded or uploaded again.
|
||||
const downloadsBefore = pl.seen.filter((s) =>
|
||||
s.path.includes("/download/"),
|
||||
).length;
|
||||
const second = await runScript(args, env);
|
||||
expect(second.code).toBe(0);
|
||||
expect(second.stdout).toMatch(/Imported: 0/);
|
||||
expect(second.stdout).toMatch(/Already in HealthLog: 2/);
|
||||
expect(pl.seen.filter((s) => s.path.includes("/download/")).length).toBe(
|
||||
downloadsBefore,
|
||||
);
|
||||
expect(hl.received).toHaveLength(2);
|
||||
expect(hl.lookups).toContain("PAPERLESS:1");
|
||||
});
|
||||
|
||||
it("leaves a document deleted in HealthLog alone", async () => {
|
||||
@@ -481,6 +524,73 @@ describe("import-documents.mjs — Paperless-ngx", () => {
|
||||
});
|
||||
});
|
||||
|
||||
describe("import-documents.mjs — edges", () => {
|
||||
const env = { HEALTHLOG_TOKEN: "hlk_test", PAPERLESS_TOKEN: "pl_test" };
|
||||
|
||||
it("still imports against a HealthLog without the lookup", async () => {
|
||||
const hl = await healthlog({ noLookup: true });
|
||||
const pl = await paperless();
|
||||
const res = await runScript(
|
||||
[
|
||||
"paperless",
|
||||
"--paperless-url",
|
||||
pl.url,
|
||||
"--healthlog-url",
|
||||
hl.url,
|
||||
"--tag",
|
||||
"HealthLog",
|
||||
],
|
||||
env,
|
||||
);
|
||||
expect(res.code).toBe(0);
|
||||
expect(res.stdout).toMatch(/Imported: 2/);
|
||||
});
|
||||
|
||||
it("stops on a redirecting HealthLog address and names where it points", async () => {
|
||||
const hl = await healthlog({ redirect: "https://health.example" });
|
||||
const pl = await paperless();
|
||||
const res = await runScript(
|
||||
[
|
||||
"paperless",
|
||||
"--paperless-url",
|
||||
pl.url,
|
||||
"--healthlog-url",
|
||||
hl.url,
|
||||
"--tag",
|
||||
"HealthLog",
|
||||
],
|
||||
env,
|
||||
);
|
||||
expect(res.code).toBe(1);
|
||||
expect(res.stderr).toMatch(/redirects to https:\/\/health\.example/);
|
||||
});
|
||||
|
||||
it("applies --since itself when the server ignores the filter", async () => {
|
||||
const hl = await healthlog();
|
||||
const pl = await paperless();
|
||||
const res = await runScript(
|
||||
[
|
||||
"paperless",
|
||||
"--paperless-url",
|
||||
pl.url,
|
||||
"--healthlog-url",
|
||||
hl.url,
|
||||
"--tag",
|
||||
"HealthLog",
|
||||
"--since",
|
||||
"2024-04-01",
|
||||
],
|
||||
env,
|
||||
);
|
||||
expect(res.code).toBe(0);
|
||||
// Document 1 (created 2024-03-05) is before the cut and never fetched.
|
||||
expect(hl.received.map((r) => r.fields.sourceId)).toEqual(["2"]);
|
||||
expect(pl.seen.some((s) => s.path.startsWith("/api/documents/1/"))).toBe(
|
||||
false,
|
||||
);
|
||||
});
|
||||
});
|
||||
|
||||
describe("import-documents.mjs — Papra", () => {
|
||||
it("imports by tag since a date, and skips what HealthLog refuses as too large", async () => {
|
||||
const hl = await healthlog({ tooLarge: ["PAPRA:doc_b"] });
|
||||
|
||||
+108
-28
@@ -283,7 +283,9 @@ function filenameFrom(response, fallback) {
|
||||
function paperless(options) {
|
||||
const headers = {
|
||||
Authorization: `Token ${options.sourceToken}`,
|
||||
Accept: "application/json; version=10",
|
||||
// API version 9 (Paperless-ngx 2.16 and later): the lowest the current
|
||||
// servers still accept, and the one where `created` became a plain date.
|
||||
Accept: "application/json; version=9",
|
||||
};
|
||||
const base = options.baseUrl;
|
||||
const label = "Paperless-ngx";
|
||||
@@ -345,10 +347,17 @@ function paperless(options) {
|
||||
}
|
||||
const created = dayOf(doc.created);
|
||||
const added = dayOf(doc.added);
|
||||
const date = created ?? added;
|
||||
// Checked here as well: an older server may ignore the date filter.
|
||||
if (options.since && date && date < options.since) continue;
|
||||
yield {
|
||||
sourceId: String(doc.id),
|
||||
title: doc.title || doc.original_file_name || `Paperless ${doc.id}`,
|
||||
date: created ?? added,
|
||||
title: titleOf(
|
||||
doc.title,
|
||||
doc.original_file_name,
|
||||
`Paperless ${doc.id}`,
|
||||
),
|
||||
date,
|
||||
dateFallback: created ? null : added ? "added" : "none",
|
||||
filename: doc.original_file_name ?? null,
|
||||
kindKeys:
|
||||
@@ -429,7 +438,11 @@ function papra(options) {
|
||||
if (options.since && date && date < options.since) continue;
|
||||
yield {
|
||||
sourceId: String(doc.id),
|
||||
title: stripExtension(doc.name) || `Papra ${doc.id}`,
|
||||
title: titleOf(
|
||||
stripExtension(doc.name),
|
||||
doc.originalName,
|
||||
`Papra ${doc.id}`,
|
||||
),
|
||||
date,
|
||||
dateFallback: documentDate ? null : created ? "added" : "none",
|
||||
filename: doc.originalName ?? doc.name ?? null,
|
||||
@@ -480,6 +493,19 @@ function dayOf(value) {
|
||||
return /^\d{4}-\d{2}-\d{2}$/.test(day) ? day : null;
|
||||
}
|
||||
|
||||
/**
|
||||
* A title that is not blank: the source's title, else its file name without
|
||||
* the extension, else a placeholder naming the source id. HealthLog refuses a
|
||||
* blank title.
|
||||
*/
|
||||
function titleOf(title, filename, fallback) {
|
||||
return (
|
||||
(typeof title === "string" ? title.trim() : "") ||
|
||||
stripExtension(filename).trim() ||
|
||||
fallback
|
||||
);
|
||||
}
|
||||
|
||||
function stripExtension(name) {
|
||||
if (typeof name !== "string") return "";
|
||||
return name.replace(/\.[A-Za-z0-9]{1,5}$/, "");
|
||||
@@ -495,8 +521,68 @@ function kindFor(options, doc) {
|
||||
return options.kind;
|
||||
}
|
||||
|
||||
/** The source key as query parameters, which HealthLog reads before the body. */
|
||||
function keyQuery(source, doc) {
|
||||
return new URLSearchParams({
|
||||
sourceSystem: source.system,
|
||||
sourceId: doc.sourceId.slice(0, SOURCE_ID_MAX),
|
||||
});
|
||||
}
|
||||
|
||||
async function readBody(response) {
|
||||
try {
|
||||
return await response.json();
|
||||
} catch {
|
||||
return null;
|
||||
}
|
||||
}
|
||||
|
||||
/** Refusals that end the run, whichever HealthLog request met them. */
|
||||
function stopOnRefusal(response, body, url) {
|
||||
if (response.status >= 300 && response.status < 400) {
|
||||
const location = response.headers.get("location");
|
||||
throw new StopError(
|
||||
`The HealthLog address redirects${location ? ` to ${new URL(location, url).origin}` : ""}. Use that address with --healthlog-url.`,
|
||||
);
|
||||
}
|
||||
if (response.status === 401) {
|
||||
throw new StopError(
|
||||
"HealthLog refused the token (401). It may be expired or revoked; create a new document token.",
|
||||
);
|
||||
}
|
||||
if (response.status === 403) {
|
||||
if (body?.meta?.errorCode === "module.disabled") {
|
||||
throw new StopError(
|
||||
"The document vault is switched off in HealthLog. Turn on Documents in Settings, then run the script again.",
|
||||
);
|
||||
}
|
||||
throw new StopError(
|
||||
"HealthLog refused the request (403). Use a document token from Settings > API & Tokens; other tokens cannot upload documents.",
|
||||
);
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Ask HealthLog whether it already holds this document, before downloading
|
||||
* it from the source. Null when the answer is "no", or when the HealthLog
|
||||
* version has no lookup yet (the upload then decides, as before).
|
||||
*/
|
||||
async function lookup(options, source, doc) {
|
||||
const url = `${options.healthlogUrl}/api/documents/inbound/source?${keyQuery(source, doc)}`;
|
||||
const response = await request("HealthLog", url, {
|
||||
headers: {
|
||||
Authorization: `Bearer ${options.healthlogToken}`,
|
||||
Accept: "application/json",
|
||||
},
|
||||
});
|
||||
const body = await readBody(response);
|
||||
stopOnRefusal(response, body, url);
|
||||
if (!response.ok || !body?.data?.known) return null;
|
||||
return body.data.deleted ? { outcome: "deleted" } : { outcome: "duplicate" };
|
||||
}
|
||||
|
||||
async function upload(options, source, doc, file) {
|
||||
const url = `${options.healthlogUrl}/api/documents/inbound`;
|
||||
const url = `${options.healthlogUrl}/api/documents/inbound?${keyQuery(source, doc)}`;
|
||||
const kind = kindFor(options, doc);
|
||||
const response = await request(
|
||||
"HealthLog",
|
||||
@@ -513,22 +599,16 @@ async function upload(options, source, doc, file) {
|
||||
new Blob([file.bytes], { type: file.type }),
|
||||
file.filename,
|
||||
);
|
||||
form.append("title", doc.title.trim().slice(0, TITLE_MAX));
|
||||
form.append("title", doc.title.slice(0, TITLE_MAX));
|
||||
if (doc.date) form.append("documentDate", doc.date);
|
||||
if (kind) form.append("kind", kind);
|
||||
form.append("sourceSystem", source.system);
|
||||
form.append("sourceId", doc.sourceId.slice(0, SOURCE_ID_MAX));
|
||||
if (!options.aiRead) form.append("aiRead", "defer");
|
||||
return form;
|
||||
},
|
||||
);
|
||||
|
||||
let body = null;
|
||||
try {
|
||||
body = await response.json();
|
||||
} catch {
|
||||
body = null;
|
||||
}
|
||||
const body = await readBody(response);
|
||||
stopOnRefusal(response, body, url);
|
||||
const meta = body?.meta ?? {};
|
||||
switch (response.status) {
|
||||
case 201:
|
||||
@@ -537,23 +617,10 @@ async function upload(options, source, doc, file) {
|
||||
return body?.data?.deleted
|
||||
? { outcome: "deleted" }
|
||||
: { outcome: "duplicate" };
|
||||
case 401:
|
||||
throw new StopError(
|
||||
"HealthLog refused the token (401). It may be expired or revoked; create a new document token.",
|
||||
);
|
||||
case 403:
|
||||
if (meta.errorCode === "module.disabled") {
|
||||
throw new StopError(
|
||||
"The document vault is switched off in HealthLog. Turn on Documents in Settings, then run the script again.",
|
||||
);
|
||||
}
|
||||
throw new StopError(
|
||||
"HealthLog refused the upload (403). Use a document token from Settings > API & Tokens; other tokens cannot upload documents.",
|
||||
);
|
||||
case 413:
|
||||
if (meta.reason === "quotaExceeded") {
|
||||
throw new StopError(
|
||||
`Your HealthLog storage is full (${mib(meta.usedBytes)} of ${mib(meta.quotaBytes)} used). An admin can raise the limit in the admin area, for everyone or just for your account.`,
|
||||
"Your HealthLog storage is full. An admin can raise the limit in the admin area, for everyone or just for your account.",
|
||||
);
|
||||
}
|
||||
return {
|
||||
@@ -642,6 +709,19 @@ async function run(argv, env = process.env) {
|
||||
continue;
|
||||
}
|
||||
|
||||
// Asked first so a document HealthLog already holds is not downloaded.
|
||||
const known = await lookup(options, source, doc);
|
||||
if (known) {
|
||||
if (known.outcome === "duplicate") {
|
||||
tally.duplicate++;
|
||||
out(` present ${where}`);
|
||||
} else {
|
||||
tally.deleted++;
|
||||
out(` deleted ${where} (you deleted it in HealthLog; left alone)`);
|
||||
}
|
||||
continue;
|
||||
}
|
||||
|
||||
const file = await source.download(doc);
|
||||
if (file.skip) {
|
||||
tally.skipped.push({ where, reason: file.skip });
|
||||
|
||||
@@ -253,21 +253,34 @@ describe("T4 — declared scopes are exported constants, never string literals",
|
||||
});
|
||||
});
|
||||
|
||||
describe("T4b — `documents:write` opens exactly one door", () => {
|
||||
it("only the vault upload route declares the scope", () => {
|
||||
describe("T4b — `documents:write` opens exactly two doors", () => {
|
||||
it("only the vault upload and the source-key lookup declare the scope", () => {
|
||||
// The scope admits a narrow token wherever it is named, so the set of
|
||||
// files naming it IS its reach. The upload, and nothing else: not the
|
||||
// list or detail beside it, not the original or thumbnail, not bulk, not
|
||||
// the AI legs. A second route here is a widening and belongs in review.
|
||||
// files naming it IS its reach. The upload, and the lookup that answers
|
||||
// what an upload of the same key would answer (known / id / deleted) so
|
||||
// an importer can skip a download. Nothing else: not the list or detail,
|
||||
// not the original or thumbnail, not bulk, not the AI legs. A third route
|
||||
// here is a widening and belongs in review.
|
||||
const declaring = filesMatching(
|
||||
/requireAuth\(\s*DOCUMENTS_WRITE_SCOPE\s*\)|requireRecordAuth\([^)]*\bscope:\s*DOCUMENTS_WRITE_SCOPE\b/,
|
||||
);
|
||||
expect(declaring).toEqual(["app/api/documents/inbound/route.ts"]);
|
||||
// And within that file, once: the GET beside the POST must not name it.
|
||||
const src = read("app/api/documents/inbound/route.ts");
|
||||
expect(declaring).toEqual([
|
||||
"app/api/documents/inbound/route.ts",
|
||||
"app/api/documents/inbound/source/route.ts",
|
||||
]);
|
||||
// Within the upload file, once: the GET list beside the POST must not
|
||||
// name it.
|
||||
const upload = read("app/api/documents/inbound/route.ts");
|
||||
expect(
|
||||
src.match(/requireAuth\(\s*DOCUMENTS_WRITE_SCOPE\s*\)/g),
|
||||
upload.match(/requireAuth\(\s*DOCUMENTS_WRITE_SCOPE\s*\)/g),
|
||||
).toHaveLength(1);
|
||||
// And the lookup is a read of one key and nothing else: a single GET.
|
||||
const lookup = read("app/api/documents/inbound/source/route.ts");
|
||||
expect(
|
||||
[...lookup.matchAll(/export const (GET|POST|PUT|PATCH|DELETE)\b/g)].map(
|
||||
(m) => m[1],
|
||||
),
|
||||
).toEqual(["GET"]);
|
||||
});
|
||||
});
|
||||
|
||||
|
||||
@@ -15,7 +15,11 @@ import { NextRequest } from "next/server";
|
||||
|
||||
vi.mock("@/lib/db", () => ({
|
||||
prisma: {
|
||||
inboundDocument: { findFirst: vi.fn(), update: vi.fn() },
|
||||
inboundDocument: {
|
||||
findFirst: vi.fn(),
|
||||
update: vi.fn(),
|
||||
updateMany: vi.fn().mockResolvedValue({ count: 0 }),
|
||||
},
|
||||
extractedFact: { create: vi.fn() },
|
||||
documentContentIndex: { upsert: vi.fn() },
|
||||
user: { findUnique: vi.fn() },
|
||||
@@ -290,6 +294,17 @@ describe("POST /api/documents/inbound/[id]/index — staging continuation", () =
|
||||
expect(maybeAutoStageLabFacts).toHaveBeenCalledWith("user-1", "doc-1");
|
||||
const body = await res.json();
|
||||
expect(body.data.labFactsStaged).toBe(4);
|
||||
// Read with AI ends an import's hold on automatic AI reading (#1038),
|
||||
// before the staging that would otherwise refuse it.
|
||||
expect(prisma.inboundDocument.updateMany).toHaveBeenCalledWith({
|
||||
where: { id: "doc-1", userId: "user-1", aiReadDeferred: true },
|
||||
data: { aiReadDeferred: false },
|
||||
});
|
||||
expect(
|
||||
vi.mocked(prisma.inboundDocument.updateMany).mock.invocationCallOrder[0],
|
||||
).toBeLessThan(
|
||||
vi.mocked(maybeAutoStageLabFacts).mock.invocationCallOrder[0],
|
||||
);
|
||||
});
|
||||
|
||||
it("reports zero staged facts as an honest zero, never an error", async () => {
|
||||
|
||||
@@ -106,6 +106,13 @@ async function finishIndex(
|
||||
source,
|
||||
tokenCount,
|
||||
});
|
||||
// Read with AI is the person reading the document on purpose, so an
|
||||
// import's hold on automatic AI reading (`aiRead=defer`) ends here, before
|
||||
// the lab staging below that would otherwise refuse it.
|
||||
await prisma.inboundDocument.updateMany({
|
||||
where: { id: documentId, userId, aiReadDeferred: true },
|
||||
data: { aiReadDeferred: false },
|
||||
});
|
||||
await auditLog("documents.inbound.index", {
|
||||
userId,
|
||||
ipAddress: getClientIp(request),
|
||||
|
||||
@@ -192,7 +192,10 @@ describe("POST /api/documents/inbound/[id]/summary", () => {
|
||||
deletedAt: null,
|
||||
summaryEncrypted: null,
|
||||
},
|
||||
data: expect.objectContaining({ summaryState: "READY" }),
|
||||
data: expect.objectContaining({
|
||||
summaryState: "READY",
|
||||
aiReadDeferred: false,
|
||||
}),
|
||||
});
|
||||
assertNoPersistence();
|
||||
});
|
||||
@@ -266,7 +269,7 @@ describe("POST /api/documents/inbound/[id]/summary", () => {
|
||||
deletedAt: null,
|
||||
summaryState: { not: "READY" },
|
||||
},
|
||||
data: { summaryState: "WITHHELD" },
|
||||
data: { summaryState: "WITHHELD", aiReadDeferred: false },
|
||||
});
|
||||
const calls = vi.mocked(prisma.inboundDocument.updateMany).mock.calls;
|
||||
for (const [arg] of calls) {
|
||||
|
||||
@@ -155,7 +155,9 @@ async function persistSummary(
|
||||
deletedAt: null,
|
||||
summaryState: { not: "READY" },
|
||||
},
|
||||
data: { summaryState: "WITHHELD" },
|
||||
// An explicit request is the person reading it on purpose, so an
|
||||
// import's hold on automatic AI reading ends here.
|
||||
data: { summaryState: "WITHHELD", aiReadDeferred: false },
|
||||
});
|
||||
return "withheld";
|
||||
}
|
||||
@@ -173,6 +175,7 @@ async function persistSummary(
|
||||
summaryEncrypted: encryptDocumentSummary(summary),
|
||||
summaryGeneratedAt: new Date(),
|
||||
summaryState: "READY",
|
||||
aiReadDeferred: false,
|
||||
},
|
||||
});
|
||||
annotate({
|
||||
|
||||
@@ -53,6 +53,10 @@ vi.mock("@/lib/db", () => {
|
||||
documentImportKey: {
|
||||
findUnique: vi.fn(),
|
||||
},
|
||||
documentSourceAlias: {
|
||||
findUnique: vi.fn(),
|
||||
createMany: vi.fn(),
|
||||
},
|
||||
extractedFact: {
|
||||
groupBy: vi.fn(),
|
||||
},
|
||||
@@ -114,6 +118,7 @@ vi.mock("@/lib/rate-limit", () => ({
|
||||
resetAt: Date.now() + 3_600_000,
|
||||
}),
|
||||
rateLimitHeaders: () => ({}),
|
||||
refundRateLimit: vi.fn().mockResolvedValue(undefined),
|
||||
}));
|
||||
|
||||
vi.mock("@/lib/auth/session", () => ({ getSession: vi.fn() }));
|
||||
@@ -146,7 +151,7 @@ import { prisma } from "@/lib/db";
|
||||
import { getSession } from "@/lib/auth/session";
|
||||
import { headers } from "next/headers";
|
||||
import { resolveBearerToken, BearerAuthError } from "@/lib/auth/bearer";
|
||||
import { checkRateLimit } from "@/lib/rate-limit";
|
||||
import { checkRateLimit, refundRateLimit } from "@/lib/rate-limit";
|
||||
import { enqueueDocumentThumbnail } from "@/lib/jobs/document-thumbnail";
|
||||
import { DOCUMENTS_WRITE_SCOPE } from "@/lib/documents/scopes";
|
||||
import { requireModuleEnabled } from "@/lib/modules/gate";
|
||||
@@ -236,6 +241,12 @@ beforeEach(() => {
|
||||
vi.mocked(prisma.documentImportKey.findUnique).mockResolvedValue(
|
||||
null as never,
|
||||
);
|
||||
vi.mocked(prisma.documentSourceAlias.findUnique).mockResolvedValue(
|
||||
null as never,
|
||||
);
|
||||
vi.mocked(prisma.documentSourceAlias.createMany).mockResolvedValue({
|
||||
count: 1,
|
||||
} as never);
|
||||
// No settings row / no user override → policy defaults (25 MiB / 1 GiB).
|
||||
vi.mocked(prisma.appSettings.findUnique).mockResolvedValue(null as never);
|
||||
vi.mocked(prisma.user.findUnique).mockResolvedValue(null as never);
|
||||
@@ -810,4 +821,129 @@ describe("POST /api/documents/inbound — document import", () => {
|
||||
expect(enqueueDocumentIndex).toHaveBeenCalledWith("user-1", "doc-1");
|
||||
expect(enqueueDocumentSummary).toHaveBeenCalledWith("user-1", "doc-1");
|
||||
});
|
||||
|
||||
it("answers a query-string key before charging the bucket or reading the body", async () => {
|
||||
armToken([DOCUMENTS_WRITE_SCOPE]);
|
||||
vi.mocked(prisma.inboundDocument.findFirst).mockResolvedValueOnce(
|
||||
docRow({
|
||||
id: "doc-src",
|
||||
sourceSystem: "PAPRA",
|
||||
sourceId: "doc_1",
|
||||
}) as never,
|
||||
);
|
||||
// Not multipart at all: reading it would be a 400.
|
||||
const res = await post(
|
||||
new Request(
|
||||
"http://localhost/api/documents/inbound?sourceSystem=PAPRA&sourceId=doc_1",
|
||||
{
|
||||
method: "POST",
|
||||
body: "not a form",
|
||||
headers: { "content-type": "text/plain" },
|
||||
},
|
||||
),
|
||||
);
|
||||
expect(res.status).toBe(200);
|
||||
expect((await res.json()).data).toEqual({ id: "doc-src", duplicate: true });
|
||||
expect(checkRateLimit).not.toHaveBeenCalled();
|
||||
});
|
||||
|
||||
it("422s half a query-string key", async () => {
|
||||
const res = await post(
|
||||
new Request("http://localhost/api/documents/inbound?sourceSystem=PAPRA", {
|
||||
method: "POST",
|
||||
body: "x",
|
||||
}),
|
||||
);
|
||||
expect(res.status).toBe(422);
|
||||
expect(checkRateLimit).not.toHaveBeenCalled();
|
||||
});
|
||||
|
||||
it("422s a query key and a form key that disagree", async () => {
|
||||
const r = mkUpload({ sourceSystem: "PAPRA", sourceId: "other" });
|
||||
const res = await post(
|
||||
new Request(`${r.url}?sourceSystem=PAPRA&sourceId=doc_1`, {
|
||||
method: "POST",
|
||||
body: await r.arrayBuffer(),
|
||||
headers: { "content-type": r.headers.get("content-type")! },
|
||||
}),
|
||||
);
|
||||
expect(res.status).toBe(422);
|
||||
expect(txCreate).not.toHaveBeenCalled();
|
||||
});
|
||||
|
||||
it("remembers the key when keyed bytes are already stored, and refunds the slot", async () => {
|
||||
// Own-key lookup misses; the content lookup finds a live row.
|
||||
vi.mocked(prisma.inboundDocument.findFirst)
|
||||
.mockResolvedValueOnce(null as never)
|
||||
.mockResolvedValueOnce(docRow({ id: "doc-manual" }) as never);
|
||||
const res = await post(
|
||||
mkUpload({ sourceSystem: "PAPRA", sourceId: "doc_9" }),
|
||||
);
|
||||
expect(res.status).toBe(200);
|
||||
expect(prisma.documentSourceAlias.createMany).toHaveBeenCalledWith({
|
||||
data: [
|
||||
{
|
||||
userId: "user-1",
|
||||
documentId: "doc-manual",
|
||||
sourceSystem: "PAPRA",
|
||||
sourceId: "doc_9",
|
||||
},
|
||||
],
|
||||
skipDuplicates: true,
|
||||
});
|
||||
expect(refundRateLimit).toHaveBeenCalledWith("documents-upload:user-1");
|
||||
expect(txCreate).not.toHaveBeenCalled();
|
||||
});
|
||||
|
||||
it("resolves a key held as an alias of a deleted document to deleted", async () => {
|
||||
vi.mocked(prisma.documentSourceAlias.findUnique).mockResolvedValue({
|
||||
document: docRow({ id: "doc-gone", deletedAt: new Date() }),
|
||||
} as never);
|
||||
const res = await post(
|
||||
mkUpload({ sourceSystem: "PAPRA", sourceId: "doc_9" }),
|
||||
);
|
||||
expect(res.status).toBe(200);
|
||||
expect((await res.json()).data).toEqual({
|
||||
id: "doc-gone",
|
||||
duplicate: true,
|
||||
deleted: true,
|
||||
});
|
||||
expect(refundRateLimit).toHaveBeenCalled();
|
||||
expect(txCreate).not.toHaveBeenCalled();
|
||||
});
|
||||
|
||||
it("refunds the slot on a plain content duplicate", async () => {
|
||||
vi.mocked(prisma.inboundDocument.findFirst).mockResolvedValue(
|
||||
docRow({ id: "doc-existing" }) as never,
|
||||
);
|
||||
await post(mkUpload());
|
||||
expect(refundRateLimit).toHaveBeenCalledWith("documents-upload:user-1");
|
||||
expect(prisma.documentSourceAlias.createMany).not.toHaveBeenCalled();
|
||||
});
|
||||
|
||||
it("does not refund a stored upload", async () => {
|
||||
await post(mkUpload());
|
||||
expect(refundRateLimit).not.toHaveBeenCalled();
|
||||
});
|
||||
|
||||
it("persists the deferral on the row", async () => {
|
||||
await post(mkUpload({ aiRead: "defer" }));
|
||||
expect(txCreate.mock.calls[0]![0]!.data.aiReadDeferred).toBe(true);
|
||||
await post(mkUpload());
|
||||
expect(txCreate.mock.calls[1]![0]!.data.aiReadDeferred).toBe(false);
|
||||
});
|
||||
|
||||
it("gives a scoped caller no quota figures on a full vault", async () => {
|
||||
armToken([DOCUMENTS_WRITE_SCOPE]);
|
||||
vi.mocked(prisma.user.findUnique).mockResolvedValue({
|
||||
documentQuotaBytes: BigInt(100),
|
||||
} as never);
|
||||
txQueryRaw.mockResolvedValue([{ used: BigInt(60) }]);
|
||||
const res = await post(mkUpload());
|
||||
expect(res.status).toBe(413);
|
||||
const meta = (await res.json()).meta;
|
||||
expect(meta.reason).toBe("quotaExceeded");
|
||||
expect(meta).not.toHaveProperty("usedBytes");
|
||||
expect(meta).not.toHaveProperty("quotaBytes");
|
||||
});
|
||||
});
|
||||
@@ -69,6 +69,11 @@ import { enqueueDocumentIndex } from "@/lib/jobs/document-index";
|
||||
import { enqueueDocumentSummary } from "@/lib/jobs/document-summary";
|
||||
import { enqueueDocumentThumbnail } from "@/lib/jobs/document-thumbnail";
|
||||
import { DOCUMENTS_WRITE_SCOPE } from "@/lib/documents/scopes";
|
||||
import {
|
||||
findSourceKey,
|
||||
rememberSourceAlias,
|
||||
type SourceKeyMatch,
|
||||
} from "@/lib/documents/source-key";
|
||||
import {
|
||||
acquireDocumentUploadSlot,
|
||||
detectDocumentType,
|
||||
@@ -81,10 +86,15 @@ import { linkTargets } from "@/lib/links";
|
||||
import { annotate } from "@/lib/logging/context";
|
||||
import { requireModuleEnabled } from "@/lib/modules/gate";
|
||||
import { isP2002 } from "@/lib/prisma-errors";
|
||||
import { checkRateLimit, rateLimitHeaders } from "@/lib/rate-limit";
|
||||
import {
|
||||
checkRateLimit,
|
||||
rateLimitHeaders,
|
||||
refundRateLimit,
|
||||
} from "@/lib/rate-limit";
|
||||
import {
|
||||
documentCreateSchema,
|
||||
documentListQuerySchema,
|
||||
documentSourceKeySchema,
|
||||
toContentIndexSource,
|
||||
type DocumentSourceSystemValue,
|
||||
} from "@/lib/validations/inbound-documents";
|
||||
@@ -193,6 +203,16 @@ function deletedResponse(ctx: UploadContext, id: string | null): NextResponse {
|
||||
return receiptResponse(id, 200, { duplicate: true, deleted: true });
|
||||
}
|
||||
|
||||
/** Answer a source key that is already held: duplicate, or deleted. */
|
||||
function answerSourceKey(
|
||||
ctx: UploadContext,
|
||||
match: SourceKeyMatch,
|
||||
): Promise<NextResponse> | NextResponse {
|
||||
return match.state === "live"
|
||||
? duplicateResponse(ctx, match.document)
|
||||
: deletedResponse(ctx, match.id);
|
||||
}
|
||||
|
||||
/**
|
||||
* §3.2 — a duplicate upload is NOT an error: return the existing live row
|
||||
* with `meta.duplicate: true` at the envelope level (the UI toasts "already
|
||||
@@ -262,21 +282,66 @@ async function processUpload(
|
||||
const gate = await requireModuleEnabled(user.id, "inboundDocuments");
|
||||
if (!gate.enabled) return gate.response;
|
||||
|
||||
// A source key may ride the query string. It is answered here, before the
|
||||
// bucket is charged and before a byte of the body is read: a nightly
|
||||
// re-sync of an archive HealthLog already holds costs neither. Both halves
|
||||
// or neither; half a key is a mistake worth saying so about.
|
||||
const url = new URL(request.url);
|
||||
const querySystem = url.searchParams.get("sourceSystem");
|
||||
const queryId = url.searchParams.get("sourceId");
|
||||
let queryKey: {
|
||||
sourceSystem: DocumentSourceSystemValue;
|
||||
sourceId: string;
|
||||
} | null = null;
|
||||
if (querySystem !== null || queryId !== null) {
|
||||
const parsedKey = documentSourceKeySchema.safeParse({
|
||||
sourceSystem: querySystem ?? undefined,
|
||||
sourceId: queryId ?? undefined,
|
||||
});
|
||||
if (!parsedKey.success) {
|
||||
return apiValidationError(
|
||||
"Invalid source key",
|
||||
sanitiseZodIssues(parsedKey.error.issues),
|
||||
422,
|
||||
{ errorCode: "documents.inbound.invalidMetadata" },
|
||||
);
|
||||
}
|
||||
queryKey = parsedKey.data;
|
||||
const match = await findSourceKey(
|
||||
user.id,
|
||||
queryKey.sourceSystem,
|
||||
queryKey.sourceId,
|
||||
);
|
||||
if (match) {
|
||||
return answerSourceKey(
|
||||
{
|
||||
userId: user.id,
|
||||
scoped,
|
||||
sourceSystem: queryKey.sourceSystem,
|
||||
aiDeferred: false,
|
||||
},
|
||||
match,
|
||||
);
|
||||
}
|
||||
}
|
||||
|
||||
// A scoped token draws on its own bucket, keyed on the token (its id rides
|
||||
// `session.id` on the Bearer path), so an import running overnight cannot
|
||||
// use up the allowance the person's own uploads from the web or the phone
|
||||
// draw on. The cookie / wildcard bucket is unchanged.
|
||||
const rl = scoped
|
||||
? await checkRateLimit(
|
||||
`documents-upload:token:${auth.session.id}`,
|
||||
resolveDocumentUploadLimitPerHour(),
|
||||
UPLOAD_WINDOW_MS,
|
||||
)
|
||||
: await checkRateLimit(
|
||||
`documents-upload:${user.id}`,
|
||||
UPLOAD_LIMIT_PER_HOUR,
|
||||
UPLOAD_WINDOW_MS,
|
||||
);
|
||||
const bucketKey = scoped
|
||||
? `documents-upload:token:${auth.session.id}`
|
||||
: `documents-upload:${user.id}`;
|
||||
const rl = await checkRateLimit(
|
||||
bucketKey,
|
||||
scoped ? resolveDocumentUploadLimitPerHour() : UPLOAD_LIMIT_PER_HOUR,
|
||||
UPLOAD_WINDOW_MS,
|
||||
);
|
||||
// A slot pays for a document stored. An upload that turns out to be one
|
||||
// already held (same key, same bytes) hands its slot back, so re-sends —
|
||||
// a Paperless workflow firing on every edit, a re-run over an archive —
|
||||
// never eat into the allowance.
|
||||
const refundSlot = () => refundRateLimit(bucketKey).catch(() => {});
|
||||
if (!rl.allowed) {
|
||||
const response = apiError("Too many uploads. Try again later.", 429, {
|
||||
errorCode: "documents.inbound.rateLimited",
|
||||
@@ -356,8 +421,23 @@ async function processUpload(
|
||||
);
|
||||
}
|
||||
|
||||
const sourceSystem = parsed.data.sourceSystem ?? null;
|
||||
const sourceId = sourceSystem ? (parsed.data.sourceId ?? null) : null;
|
||||
// The key from the query string, or from the form fields; if a caller sends
|
||||
// both they have to agree.
|
||||
const formSystem = parsed.data.sourceSystem ?? null;
|
||||
const formId = formSystem ? (parsed.data.sourceId ?? null) : null;
|
||||
if (
|
||||
queryKey &&
|
||||
formSystem !== null &&
|
||||
(formSystem !== queryKey.sourceSystem || formId !== queryKey.sourceId)
|
||||
) {
|
||||
return apiError(
|
||||
"The source key in the address and in the form differ.",
|
||||
422,
|
||||
{ errorCode: "documents.inbound.invalidMetadata" },
|
||||
);
|
||||
}
|
||||
const sourceSystem = queryKey?.sourceSystem ?? formSystem;
|
||||
const sourceId = queryKey?.sourceId ?? formId;
|
||||
const ctx: UploadContext = {
|
||||
userId: user.id,
|
||||
scoped,
|
||||
@@ -370,23 +450,12 @@ async function processUpload(
|
||||
// person deleted is refused a second copy — live and tombstoned rows both
|
||||
// count (the unique index has no `deleted_at` predicate), and past the purge
|
||||
// the ledger remembers.
|
||||
if (sourceSystem && sourceId) {
|
||||
const keyed = await findBySourceKey(user.id, sourceSystem, sourceId);
|
||||
if (keyed)
|
||||
return keyed.deletedAt
|
||||
? deletedResponse(ctx, keyed.id)
|
||||
: duplicateResponse(ctx, keyed);
|
||||
const purged = await prisma.documentImportKey.findUnique({
|
||||
where: {
|
||||
userId_sourceSystem_sourceId: {
|
||||
userId: user.id,
|
||||
sourceSystem,
|
||||
sourceId,
|
||||
},
|
||||
},
|
||||
select: { id: true },
|
||||
});
|
||||
if (purged) return deletedResponse(ctx, null);
|
||||
if (sourceSystem && sourceId && !queryKey) {
|
||||
const match = await findSourceKey(user.id, sourceSystem, sourceId);
|
||||
if (match) {
|
||||
await refundSlot();
|
||||
return answerSourceKey(ctx, match);
|
||||
}
|
||||
}
|
||||
|
||||
let buffer: Buffer;
|
||||
@@ -449,6 +518,13 @@ async function processUpload(
|
||||
omit: { contentEncrypted: true },
|
||||
});
|
||||
if (existing) {
|
||||
// An import sending bytes that are already stored under another key (or
|
||||
// none) is answered with that document — and the key is remembered for
|
||||
// it, so once the person deletes the document this key stays deleted too.
|
||||
if (sourceSystem && sourceId) {
|
||||
await rememberSourceAlias(user.id, existing.id, sourceSystem, sourceId);
|
||||
}
|
||||
await refundSlot();
|
||||
return duplicateResponse(ctx, existing);
|
||||
}
|
||||
|
||||
@@ -507,6 +583,7 @@ async function processUpload(
|
||||
: new Date(),
|
||||
sourceSystem,
|
||||
sourceId,
|
||||
aiReadDeferred: ctx.aiDeferred,
|
||||
},
|
||||
omit: { contentEncrypted: true },
|
||||
});
|
||||
@@ -533,11 +610,14 @@ async function processUpload(
|
||||
});
|
||||
} catch (err) {
|
||||
if (err instanceof QuotaExceededError) {
|
||||
// The figures are the owner's; a write-only token learns only that the
|
||||
// vault is full.
|
||||
return apiError("Storage quota exceeded.", 413, {
|
||||
errorCode: "documents.inbound.quotaExceeded",
|
||||
reason: "quotaExceeded",
|
||||
quotaBytes: limits.quotaBytes,
|
||||
usedBytes: err.usedBytes,
|
||||
...(scoped
|
||||
? {}
|
||||
: { quotaBytes: limits.quotaBytes, usedBytes: err.usedBytes }),
|
||||
});
|
||||
}
|
||||
if (isP2002(err)) {
|
||||
@@ -545,18 +625,23 @@ async function processUpload(
|
||||
// source key, or the same bytes. Surface the winner exactly as the fast
|
||||
// paths above would have.
|
||||
if (sourceSystem && sourceId) {
|
||||
const keyed = await findBySourceKey(user.id, sourceSystem, sourceId);
|
||||
if (keyed) {
|
||||
return keyed.deletedAt
|
||||
? deletedResponse(ctx, keyed.id)
|
||||
: duplicateResponse(ctx, keyed);
|
||||
const match = await findSourceKey(user.id, sourceSystem, sourceId);
|
||||
if (match) {
|
||||
await refundSlot();
|
||||
return answerSourceKey(ctx, match);
|
||||
}
|
||||
}
|
||||
const winner = await prisma.inboundDocument.findFirst({
|
||||
where: { userId: user.id, contentSha256, deletedAt: null },
|
||||
omit: { contentEncrypted: true },
|
||||
});
|
||||
if (winner) return duplicateResponse(ctx, winner);
|
||||
if (winner) {
|
||||
if (sourceSystem && sourceId) {
|
||||
await rememberSourceAlias(user.id, winner.id, sourceSystem, sourceId);
|
||||
}
|
||||
await refundSlot();
|
||||
return duplicateResponse(ctx, winner);
|
||||
}
|
||||
}
|
||||
throw err;
|
||||
}
|
||||
@@ -651,24 +736,10 @@ async function processUpload(
|
||||
);
|
||||
}
|
||||
|
||||
/**
|
||||
* Find a document by its import source key, live or tombstoned. The blob
|
||||
* column is never selected.
|
||||
*/
|
||||
function findBySourceKey(
|
||||
userId: string,
|
||||
sourceSystem: DocumentSourceSystemValue,
|
||||
sourceId: string,
|
||||
): Promise<SerialisableDocument | null> {
|
||||
return prisma.inboundDocument.findFirst({
|
||||
where: { userId, sourceSystem, sourceId },
|
||||
omit: { contentEncrypted: true },
|
||||
});
|
||||
}
|
||||
|
||||
/** Authenticate and reserve memory capacity before any request-body read. */
|
||||
async function postUpload(request: Request): Promise<Response> {
|
||||
// The one route that names `documents:write`. Every other vault leg —
|
||||
// One of the two routes that name `documents:write` (the other is the
|
||||
// source-key lookup beside it). Every other vault leg —
|
||||
// list, detail, original, thumbnail, bulk, AI — declares no scope and so
|
||||
// refuses the token (`bearer-scope-enforcement-guard.test.ts`).
|
||||
const auth = await requireAuth(DOCUMENTS_WRITE_SCOPE);
|
||||
|
||||
@@ -0,0 +1,100 @@
|
||||
/**
|
||||
* `GET /api/documents/inbound/source?sourceSystem=…&sourceId=…` — does the
|
||||
* vault already hold this import source key? (#1038)
|
||||
*
|
||||
* Lets an importer skip a document before downloading it from the source
|
||||
* system at all: a nightly re-sync of an archive HealthLog already holds then
|
||||
* costs one small request per document instead of a download and an upload.
|
||||
*
|
||||
* It answers only what the upload itself would answer for the same key —
|
||||
* `known`, the document `id`, and whether the owner `deleted` it — so a
|
||||
* `documents:write` token learns nothing here it could not learn by uploading.
|
||||
* That token is the reason the route exists, and it is the second of the two
|
||||
* routes the scope reaches (`bearer-scope-enforcement-guard.test.ts`).
|
||||
*/
|
||||
import { apiHandler, isScopedCredential, requireAuth } from "@/lib/api-handler";
|
||||
import {
|
||||
apiError,
|
||||
apiSuccess,
|
||||
apiValidationError,
|
||||
sanitiseZodIssues,
|
||||
} from "@/lib/api-response";
|
||||
import { DOCUMENTS_WRITE_SCOPE } from "@/lib/documents/scopes";
|
||||
import { findSourceKey } from "@/lib/documents/source-key";
|
||||
import { annotate } from "@/lib/logging/context";
|
||||
import { requireModuleEnabled } from "@/lib/modules/gate";
|
||||
import { checkRateLimit, rateLimitHeaders } from "@/lib/rate-limit";
|
||||
import { documentSourceKeySchema } from "@/lib/validations/inbound-documents";
|
||||
|
||||
export const dynamic = "force-dynamic";
|
||||
|
||||
/**
|
||||
* Lookups per caller per hour. Generous on purpose: an importer asks once per
|
||||
* document in the source, so a re-sync of a few thousand documents has to fit
|
||||
* in an hour. A lookup reads three indexed rows and stores nothing.
|
||||
*/
|
||||
const LOOKUP_LIMIT_PER_HOUR = 5000;
|
||||
const LOOKUP_WINDOW_MS = 60 * 60 * 1000;
|
||||
|
||||
export const GET = apiHandler(async (request: Request) => {
|
||||
const auth = await requireAuth(DOCUMENTS_WRITE_SCOPE);
|
||||
const { user } = auth;
|
||||
|
||||
const gate = await requireModuleEnabled(user.id, "inboundDocuments");
|
||||
if (!gate.enabled) return gate.response;
|
||||
|
||||
const rl = await checkRateLimit(
|
||||
isScopedCredential(auth)
|
||||
? `documents-source:token:${auth.session.id}`
|
||||
: `documents-source:${user.id}`,
|
||||
LOOKUP_LIMIT_PER_HOUR,
|
||||
LOOKUP_WINDOW_MS,
|
||||
);
|
||||
if (!rl.allowed) {
|
||||
const response = apiError("Too many lookups. Try again later.", 429, {
|
||||
errorCode: "documents.inbound.rateLimited",
|
||||
});
|
||||
for (const [k, v] of Object.entries(rateLimitHeaders(rl))) {
|
||||
response.headers.set(k, v);
|
||||
}
|
||||
return response;
|
||||
}
|
||||
|
||||
const url = new URL(request.url);
|
||||
const parsed = documentSourceKeySchema.safeParse({
|
||||
sourceSystem: url.searchParams.get("sourceSystem") ?? undefined,
|
||||
sourceId: url.searchParams.get("sourceId") ?? undefined,
|
||||
});
|
||||
if (!parsed.success) {
|
||||
return apiValidationError(
|
||||
"Invalid source key",
|
||||
sanitiseZodIssues(parsed.error.issues),
|
||||
422,
|
||||
{ errorCode: "documents.inbound.invalidMetadata" },
|
||||
);
|
||||
}
|
||||
|
||||
const match = await findSourceKey(
|
||||
user.id,
|
||||
parsed.data.sourceSystem,
|
||||
parsed.data.sourceId,
|
||||
);
|
||||
annotate({
|
||||
action: { name: "documents.vault.sourceLookup" },
|
||||
meta: {
|
||||
sourceSystem: parsed.data.sourceSystem,
|
||||
known: match !== null,
|
||||
deleted: match?.state === "deleted",
|
||||
},
|
||||
});
|
||||
return apiSuccess({
|
||||
known: match !== null,
|
||||
id:
|
||||
match === null
|
||||
? null
|
||||
: match.state === "live"
|
||||
? match.document.id
|
||||
: match.id,
|
||||
deleted: match?.state === "deleted",
|
||||
});
|
||||
});
|
||||
@@ -9,7 +9,8 @@
|
||||
* that file for the reasoning behind each guard. In short:
|
||||
*
|
||||
* - `permissions` is a literal, so no request shape reaches it;
|
||||
* - `documents:write` is accepted by exactly one route, the upload, on the
|
||||
* - `documents:write` is accepted by the upload and by the source-key lookup
|
||||
* that answers what the upload would, both on the
|
||||
* holder's own record, and the answer it gets back is a receipt rather than
|
||||
* the stored row — it cannot list, read, download, edit or delete a
|
||||
* document, reach the AI legs, or mint another token;
|
||||
|
||||
@@ -152,6 +152,9 @@ export const WIPE_MODELS = [
|
||||
// import after the wipe skip files for a decision about a record that no
|
||||
// longer exists.
|
||||
"DocumentImportKey",
|
||||
// Further import keys that resolved to a document by content; a child of
|
||||
// the document, removed before it so the counts are truthful.
|
||||
"DocumentSourceAlias",
|
||||
"ExtractedFact",
|
||||
"ImportJob",
|
||||
|
||||
|
||||
@@ -206,6 +206,18 @@ describe("maybeAutoStageLabFacts", () => {
|
||||
expect(res).toEqual({ staged: true, facts: 1 });
|
||||
});
|
||||
|
||||
it("stages nothing for an import held back from AI reading (#1038)", async () => {
|
||||
findFirst.mockResolvedValue({
|
||||
kind: "LAB_RESULT",
|
||||
status: "STORED",
|
||||
aiReadDeferred: true,
|
||||
_count: { facts: 0 },
|
||||
});
|
||||
const res = await maybeAutoStageLabFacts("u1", "d1");
|
||||
expect(res).toEqual({ staged: false, reason: "deferred" });
|
||||
expect(mockExtract).not.toHaveBeenCalled();
|
||||
});
|
||||
|
||||
it("is idempotent — a non-STORED / already-staged document is left alone", async () => {
|
||||
findFirst.mockResolvedValue({
|
||||
kind: "OTHER",
|
||||
|
||||
@@ -189,6 +189,7 @@ export type AutoStageOutcome =
|
||||
| "modules-off"
|
||||
| "not-eligible"
|
||||
| "already-handled"
|
||||
| "deferred"
|
||||
| "no-text"
|
||||
| "not-lab"
|
||||
| "budget"
|
||||
@@ -256,11 +257,19 @@ export async function maybeAutoStageLabFacts(
|
||||
// leave a non-STORED / already-staged document alone.
|
||||
const doc = await prisma.inboundDocument.findFirst({
|
||||
where: { id: documentId, userId, deletedAt: null },
|
||||
select: { kind: true, status: true, _count: { select: { facts: true } } },
|
||||
select: {
|
||||
kind: true,
|
||||
status: true,
|
||||
aiReadDeferred: true,
|
||||
_count: { select: { facts: true } },
|
||||
},
|
||||
});
|
||||
if (!doc || doc.status !== "STORED" || doc._count.facts > 0) {
|
||||
return { staged: false, reason: "already-handled" };
|
||||
}
|
||||
// An import held back from AI reading stages nothing on its own; the
|
||||
// explicit read clears the marker before it calls here.
|
||||
if (doc.aiReadDeferred) return { staged: false, reason: "deferred" };
|
||||
|
||||
// Same provider + consent gate the auto-index external path uses.
|
||||
const provider = await resolveIndexProvider(userId);
|
||||
|
||||
@@ -0,0 +1,73 @@
|
||||
/**
|
||||
* Import source keys (#1038): where a `(sourceSystem, sourceId)` pair sent by
|
||||
* an importer already lands in a person's vault.
|
||||
*
|
||||
* A key can be held in three places, checked in this order:
|
||||
*
|
||||
* 1. the document row itself (`InboundDocument.sourceSystem/sourceId`), live
|
||||
* or tombstoned — the key the document was stored under;
|
||||
* 2. `DocumentSourceAlias` — a further key an import sent for bytes that were
|
||||
* already stored, so the upload answered with the existing document;
|
||||
* 3. `DocumentImportKey` — a key whose document the purge has removed.
|
||||
*
|
||||
* The first two resolve to a document (live → duplicate, tombstoned →
|
||||
* deleted); the third only to "deleted". The upload route and the lookup
|
||||
* route both ask here, so they cannot disagree about what a key means.
|
||||
*/
|
||||
import { prisma } from "@/lib/db";
|
||||
import type { DocumentSourceSystemValue } from "@/lib/validations/inbound-documents";
|
||||
import type { SerialisableDocument } from "@/lib/documents/store";
|
||||
|
||||
export type SourceKeyMatch =
|
||||
| { state: "live"; document: SerialisableDocument }
|
||||
| { state: "deleted"; id: string | null };
|
||||
|
||||
export async function findSourceKey(
|
||||
userId: string,
|
||||
sourceSystem: DocumentSourceSystemValue,
|
||||
sourceId: string,
|
||||
): Promise<SourceKeyMatch | null> {
|
||||
const own = await prisma.inboundDocument.findFirst({
|
||||
where: { userId, sourceSystem, sourceId },
|
||||
omit: { contentEncrypted: true },
|
||||
});
|
||||
const viaAlias = own
|
||||
? null
|
||||
: await prisma.documentSourceAlias.findUnique({
|
||||
where: {
|
||||
userId_sourceSystem_sourceId: { userId, sourceSystem, sourceId },
|
||||
},
|
||||
select: { document: { omit: { contentEncrypted: true } } },
|
||||
});
|
||||
const document = own ?? viaAlias?.document ?? null;
|
||||
if (document) {
|
||||
return document.deletedAt
|
||||
? { state: "deleted", id: document.id }
|
||||
: { state: "live", document };
|
||||
}
|
||||
const purged = await prisma.documentImportKey.findUnique({
|
||||
where: {
|
||||
userId_sourceSystem_sourceId: { userId, sourceSystem, sourceId },
|
||||
},
|
||||
select: { id: true },
|
||||
});
|
||||
return purged ? { state: "deleted", id: null } : null;
|
||||
}
|
||||
|
||||
/**
|
||||
* Remember that `sourceId` in `sourceSystem` was answered with `documentId`
|
||||
* (same bytes, stored earlier). Without this the key would be forgotten, and
|
||||
* once the person deleted that document the next import run would store it
|
||||
* again. Idempotent; a key already held elsewhere is left alone.
|
||||
*/
|
||||
export async function rememberSourceAlias(
|
||||
userId: string,
|
||||
documentId: string,
|
||||
sourceSystem: DocumentSourceSystemValue,
|
||||
sourceId: string,
|
||||
): Promise<void> {
|
||||
await prisma.documentSourceAlias.createMany({
|
||||
data: [{ userId, documentId, sourceSystem, sourceId }],
|
||||
skipDuplicates: true,
|
||||
});
|
||||
}
|
||||
@@ -672,6 +672,8 @@ export const NOT_IN_BACKUP_MODELS: Readonly<Record<string, string>> = {
|
||||
"When this host last put this account's copy in this operator's bucket, and how big it was. It describes one deployment's relationship with one bucket, so restoring it elsewhere would assert an off-host copy that host has never written.",
|
||||
DocumentImportKey:
|
||||
"The source keys of imported documents the person deleted and the purge then removed, kept so a re-run of the importer does not store them again. The backup carries only live documents, so the tombstones that hold the same memory before the purge do not travel either; carrying the ledger alone would restore half of one decision. After a restore, a deliberate re-import can bring such a document back, and the person deletes it again.",
|
||||
DocumentSourceAlias:
|
||||
"Further import source keys that were answered with an already stored document because the bytes matched. The next import run writes the same alias again the moment it meets the same bytes, so a restore loses nothing a re-run does not rebuild; and for a deleted document it restores no less than the tombstones and ledger beside it do, which do not travel either.",
|
||||
ImportJob:
|
||||
"A job record pointing at an uploaded file that the backup does not carry, so restoring it would resurrect a task with nothing to work on.",
|
||||
InviteToken:
|
||||
|
||||
@@ -177,6 +177,8 @@ export interface DocumentBackupEntry {
|
||||
*/
|
||||
sourceSystem?: string | null;
|
||||
sourceId?: string | null;
|
||||
/** Canonical DR only: the import's hold on automatic AI reading. */
|
||||
aiReadDeferred?: boolean;
|
||||
}
|
||||
|
||||
export interface RecordsBackupOptions {
|
||||
@@ -194,6 +196,7 @@ interface DisasterRecoveryDocumentRow {
|
||||
summaryState: string;
|
||||
lastIndexAttemptAt: Date | null;
|
||||
lastIndexOutcome: string | null;
|
||||
aiReadDeferred: boolean;
|
||||
updatedAt: Date;
|
||||
}
|
||||
|
||||
@@ -306,6 +309,7 @@ export async function buildRecordsBackupSection(
|
||||
lastIndexOutcome: true,
|
||||
sourceSystem: true,
|
||||
sourceId: true,
|
||||
aiReadDeferred: true,
|
||||
createdAt: true,
|
||||
updatedAt: true,
|
||||
},
|
||||
@@ -563,6 +567,7 @@ export async function buildRecordsBackupSection(
|
||||
summaryState: dr.summaryState,
|
||||
lastIndexAttemptAt: dr.lastIndexAttemptAt?.toISOString() ?? null,
|
||||
lastIndexOutcome: dr.lastIndexOutcome,
|
||||
aiReadDeferred: dr.aiReadDeferred,
|
||||
};
|
||||
}
|
||||
|
||||
|
||||
@@ -2030,6 +2030,9 @@ export async function restoreBackup(
|
||||
? new Date(document.lastIndexAttemptAt)
|
||||
: null,
|
||||
lastIndexOutcome: document.lastIndexOutcome ?? null,
|
||||
// An older file has no marker: those documents predate the
|
||||
// hold and restore as ordinary ones.
|
||||
aiReadDeferred: document.aiReadDeferred ?? false,
|
||||
sourceSystem: document.sourceSystem ?? null,
|
||||
sourceId: document.sourceSystem
|
||||
? (document.sourceId ?? null)
|
||||
|
||||
@@ -100,6 +100,14 @@ describe("runSummaryCatchUpForUser", () => {
|
||||
});
|
||||
});
|
||||
|
||||
it("leaves imports held back from AI reading alone (#1038)", async () => {
|
||||
serveDocuments(1);
|
||||
await runSummaryCatchUpForUser("user-1");
|
||||
expect(findMany.mock.calls[0]![0]!.where).toMatchObject({
|
||||
aiReadDeferred: false,
|
||||
});
|
||||
});
|
||||
|
||||
it("stops at the documented cap instead of queueing a whole vault", async () => {
|
||||
serveDocuments(MAX_ENQUEUES_PER_RUN + 50);
|
||||
|
||||
|
||||
@@ -140,6 +140,23 @@ beforeEach(() => {
|
||||
});
|
||||
|
||||
describe("runDocumentSummaryJob — gating", () => {
|
||||
it("never summarises an import held back from AI reading (#1038)", async () => {
|
||||
vi.mocked(prisma.inboundDocument.findFirst).mockResolvedValue({
|
||||
id: "doc-1",
|
||||
summaryEncrypted: null,
|
||||
aiReadDeferred: true,
|
||||
} as never);
|
||||
|
||||
await runDocumentSummaryJob({ userId: "user-1", documentId: "doc-1" });
|
||||
|
||||
expect(resolveDocumentVisionProvider).not.toHaveBeenCalled();
|
||||
expect(runDocumentSummary).not.toHaveBeenCalled();
|
||||
for (const [arg] of vi.mocked(prisma.inboundDocument.updateMany).mock
|
||||
.calls) {
|
||||
expect(arg.data).toEqual({ summaryState: "NONE" });
|
||||
}
|
||||
});
|
||||
|
||||
it("does no provider work when the opt-in is OFF (only the PENDING heal may write)", async () => {
|
||||
vi.mocked(documentAutoReadEnabled).mockResolvedValue(false);
|
||||
|
||||
|
||||
@@ -49,18 +49,39 @@ export async function purgeExpiredDocumentTombstones(
|
||||
const keyed = await tx.inboundDocument.findMany({
|
||||
where: {
|
||||
...expired,
|
||||
sourceSystem: { not: null },
|
||||
sourceId: { not: null },
|
||||
OR: [
|
||||
{ sourceSystem: { not: null }, sourceId: { not: null } },
|
||||
{ sourceAliases: { some: {} } },
|
||||
],
|
||||
},
|
||||
select: {
|
||||
userId: true,
|
||||
sourceSystem: true,
|
||||
sourceId: true,
|
||||
// Further keys the import answered with this document (same bytes).
|
||||
// They cascade away with the row, so they are copied here first.
|
||||
sourceAliases: { select: { sourceSystem: true, sourceId: true } },
|
||||
},
|
||||
select: { userId: true, sourceSystem: true, sourceId: true },
|
||||
});
|
||||
if (keyed.length > 0) {
|
||||
const keys = keyed.flatMap((row) => [
|
||||
...(row.sourceSystem && row.sourceId
|
||||
? [
|
||||
{
|
||||
userId: row.userId,
|
||||
sourceSystem: row.sourceSystem,
|
||||
sourceId: row.sourceId,
|
||||
},
|
||||
]
|
||||
: []),
|
||||
...row.sourceAliases.map((alias) => ({
|
||||
userId: row.userId,
|
||||
sourceSystem: alias.sourceSystem,
|
||||
sourceId: alias.sourceId,
|
||||
})),
|
||||
]);
|
||||
if (keys.length > 0) {
|
||||
await tx.documentImportKey.createMany({
|
||||
data: keyed.map((row) => ({
|
||||
userId: row.userId,
|
||||
sourceSystem: row.sourceSystem!,
|
||||
sourceId: row.sourceId!,
|
||||
})),
|
||||
data: keys,
|
||||
skipDuplicates: true,
|
||||
});
|
||||
}
|
||||
|
||||
@@ -120,7 +120,14 @@ export async function runSummaryCatchUpForUser(
|
||||
break;
|
||||
}
|
||||
const rows: { id: string }[] = await prisma.inboundDocument.findMany({
|
||||
where: { userId, deletedAt: null, summaryEncrypted: null },
|
||||
// An import held back from AI reading (`aiRead=defer`) stays held back
|
||||
// when the opt-in is flipped on: the person reads those deliberately.
|
||||
where: {
|
||||
userId,
|
||||
deletedAt: null,
|
||||
summaryEncrypted: null,
|
||||
aiReadDeferred: false,
|
||||
},
|
||||
select: { id: true },
|
||||
orderBy: { id: "asc" },
|
||||
take: PAGE_SIZE,
|
||||
|
||||
@@ -98,7 +98,7 @@ export async function runDocumentSummaryJob(
|
||||
// re-enqueue after a duplicate upload, or a boot backfill overlap).
|
||||
const existing = await prisma.inboundDocument.findFirst({
|
||||
where: { id: documentId, userId, deletedAt: null },
|
||||
select: { id: true, summaryEncrypted: true },
|
||||
select: { id: true, summaryEncrypted: true, aiReadDeferred: true },
|
||||
});
|
||||
if (!existing || existing.summaryEncrypted) {
|
||||
annotate({
|
||||
@@ -107,6 +107,19 @@ export async function runDocumentSummaryJob(
|
||||
});
|
||||
return;
|
||||
}
|
||||
// An import held back from AI reading is never summarised automatically,
|
||||
// whichever path enqueued it; the on-demand route clears the marker first.
|
||||
if (existing.aiReadDeferred) {
|
||||
await prisma.inboundDocument.updateMany({
|
||||
where: { id: documentId, userId, summaryState: "PENDING" },
|
||||
data: { summaryState: "NONE" },
|
||||
});
|
||||
annotate({
|
||||
action: { name: "documents.summary.autoSkipped" },
|
||||
meta: { documentId, reason: "deferred" },
|
||||
});
|
||||
return;
|
||||
}
|
||||
|
||||
// The opt-in IS the trigger: when it is OFF the auto path does nothing (the
|
||||
// on-demand route stays the only way to summarise). Fail-closed on a missing
|
||||
|
||||
@@ -1651,7 +1651,7 @@ export const authPaths: NonNullable<ZodOpenApiObject["paths"]> = {
|
||||
summary: "Mint a document-upload token",
|
||||
description:
|
||||
"Mints a Bearer scoped to exactly `documents:write` and audits the mint. THE RESPONSE CARRIES THE RAW TOKEN, once; it is stored as an HMAC and no path re-reveals it.\n\n" +
|
||||
"**What it can do.** `POST /api/documents/inbound` on its owner's own record, and nothing else. It is the door another document system pushes through: a Paperless-ngx workflow posting a newly tagged document, or an import script copying an archive. Uploads through it draw on a bucket of their own, keyed on the token (default 120 an hour, operator-tunable via `DOCUMENT_UPLOAD_LIMIT_PER_HOUR`), and get back a receipt (`id`, `duplicate`, and `deleted` when the source key belongs to a document the owner deleted) rather than the stored row.\n\n" +
|
||||
"**What it can do.** `POST /api/documents/inbound` on its owner's own record, and `GET /api/documents/inbound/source`, which answers whether a source key is already held so an importer can skip a download. Nothing else. It is the door another document system pushes through: a Paperless-ngx workflow posting a newly tagged document, or an import script copying an archive. Uploads through it draw on a bucket of their own, keyed on the token (default 120 an hour, operator-tunable via `DOCUMENT_UPLOAD_LIMIT_PER_HOUR`), and get back a receipt (`id`, `duplicate`, and `deleted` when the source key belongs to a document the owner deleted) rather than the stored row.\n\n" +
|
||||
"**What it cannot do.** List, read, download, preview, edit, delete or bulk-act on a document, reach the document-AI routes, act on a shared record, or mint another token. Every other route names no scope and refuses it 403.\n\n" +
|
||||
"Minting requires a COOKIE SESSION, for the reason the measurement mint gives: a day-lived native access token must not be able to leave behind a year-lived credential. Gated by the operator's instance-wide API switch. Body capped at 16 KiB; 10 mints per user per minute, and at most 10 live document tokens held at once. Tokens appear in `GET /api/tokens` and are revoked at `DELETE /api/tokens/{id}`.",
|
||||
requestBody: {
|
||||
|
||||
@@ -399,13 +399,31 @@ export const inboundDocumentPaths: NonNullable<ZodOpenApiObject["paths"]> = {
|
||||
},
|
||||
},
|
||||
post: {
|
||||
parameters: [idempotencyKeyParameter],
|
||||
parameters: [
|
||||
idempotencyKeyParameter,
|
||||
{
|
||||
name: "sourceSystem",
|
||||
in: "query",
|
||||
required: false,
|
||||
schema: { type: "string", enum: [...DOCUMENT_SOURCE_SYSTEMS] },
|
||||
description:
|
||||
"Import source key, first half. Sent here instead of (or agreeing with) the form field, it is checked before the body is read or the hourly allowance is charged.",
|
||||
},
|
||||
{
|
||||
name: "sourceId",
|
||||
in: "query",
|
||||
required: false,
|
||||
schema: { type: "string", maxLength: DOCUMENT_SOURCE_ID_MAX },
|
||||
description:
|
||||
"Import source key, second half. Both halves or neither; half a key is 422.",
|
||||
},
|
||||
],
|
||||
tags: ["Documents"],
|
||||
summary: "Store a document (no extraction)",
|
||||
description:
|
||||
'STORE-ONLY upload. Stores the raw document ENCRYPTED at rest with `status: STORED` and runs NO extraction — provider-free, no AI consent / budget / egress. A file is always filable, even with no document-scan provider configured. `multipart/form-data`: a `file` plus optional `title`, `kind`, `documentDate` (YYYY-MM-DD), and repeated `episodeIds` form fields (pre-link to the caller\'s illness/condition episodes). Accepted types (magic-byte sniffed, never the wire Content-Type): PDF/JPEG/PNG/WebP/GIF render inline; Office (docx/xlsx/pptx/doc/xls/ppt), text/CSV/Markdown/RTF, TIFF, HEIC/HEIF, XML/JSON are stored verbatim and served download-only. HEIC is stored as-is but attachment-only — prefer transcoding to JPEG client-side for inline preview parity. Error contract: `413` with `meta.reason = "fileTooLarge"` (+ `maxFileBytes`) or `"quotaExceeded"` (+ `quotaBytes`, `usedBytes`); `415` with `meta.reason = "unsupportedType"`. A same-user duplicate (same bytes) returns 200 + `meta.duplicate: true` with the existing row — not an error. `Idempotency-Key` honoured. Read `GET /api/documents/inbound/usage` for the effective limits before offering an upload. AI extraction is a separate opt-in action — see `POST /api/documents/inbound/{id}/extract`.\n\n' +
|
||||
"**Importing from another system (v1.39.2).** Optional `sourceSystem` (`PAPERLESS`, `PAPRA`, `OTHER`) and `sourceId` (its id there, printable, up to 128 characters; needs `sourceSystem`) key the upload. A re-send with a known key is a duplicate: 200 with the live row (`meta.duplicate: true`), or, when the owner DELETED that document, 200 with `data.deleted: true` and `meta.deleted: true` and nothing stored; this holds after the 30-day purge too. `aiRead=defer` holds back automatic AI reading for this upload: the thumbnail and a local text index still run, the summary and lab staging do not, so the document waits for the person to read it deliberately. Absent, behaviour is unchanged.\n\n" +
|
||||
"**Narrow token.** Also reachable with a `documents:write` Bearer (minted at `POST /api/tokens/documents`), and it is the only route that scope reaches. Such a caller draws on a bucket of its own, keyed on the token (default 120 an hour, `DOCUMENT_UPLOAD_LIMIT_PER_HOUR`, clamped 1-1000; the 429 carries `Retry-After` and `X-RateLimit-*`), and gets a `DocumentUploadReceipt` instead of the stored row. A cookie or wildcard caller keeps the 60-an-hour per-user bucket and the full row. With the vault module off the answer is 403 `module.disabled` for every caller.",
|
||||
"**Importing from another system (v1.39.2).** Optional `sourceSystem` (`PAPERLESS`, `PAPRA`, `OTHER`) and `sourceId` (its id there, printable, up to 128 characters; needs `sourceSystem`) key the upload. A re-send with a known key is a duplicate: 200 with the live row (`meta.duplicate: true`), or, when the owner DELETED that document, 200 with `data.deleted: true` and `meta.deleted: true` and nothing stored; this holds after the 30-day purge too. The key may instead ride the query string (`?sourceSystem=…&sourceId=…`): it is then answered before the body is read and before the hourly allowance is charged, so a re-send of a document already held costs neither; a key in both places must agree (else 422). A key that is answered with an existing document by its bytes is remembered for that document, so deleting it keeps that key deleted too. `OTHER` is one shared namespace per account: two different systems both sent as `OTHER` must not reuse each other\'s ids. An upload answered as a duplicate or as deleted hands its slot of the hourly allowance back. `aiRead=defer` holds back automatic AI reading for this upload: the thumbnail and a local text index still run, the summary and lab staging do not, and the document stays out of the summary catch-up until the person reads it with AI or generates its summary. Absent, behaviour is unchanged.\n\n" +
|
||||
"**Narrow token.** Also reachable with a `documents:write` Bearer (minted at `POST /api/tokens/documents`), which reaches only this route and the source-key lookup `GET /api/documents/inbound/source`. Such a caller draws on a bucket of its own, keyed on the token (default 120 an hour, `DOCUMENT_UPLOAD_LIMIT_PER_HOUR`, clamped 1-1000; the 429 carries `Retry-After` and `X-RateLimit-*`), and gets a `DocumentUploadReceipt` instead of the stored row; its 413 `quotaExceeded` carries no `quotaBytes` / `usedBytes`. The response is an untagged union (row or receipt) on purpose: a discriminator would need a new field on the row every existing client decodes. A cookie or wildcard caller keeps the 60-an-hour per-user bucket and the full row. With the vault module off the answer is 403 `module.disabled` for every caller.",
|
||||
requestBody: {
|
||||
required: true,
|
||||
content: {
|
||||
@@ -485,6 +503,48 @@ export const inboundDocumentPaths: NonNullable<ZodOpenApiObject["paths"]> = {
|
||||
},
|
||||
},
|
||||
},
|
||||
"/api/documents/inbound/source": {
|
||||
get: {
|
||||
tags: ["Documents"],
|
||||
summary: "Look up an import source key",
|
||||
description:
|
||||
"Whether the vault already holds a document under this import source key, so an importer can skip a document before downloading it from the source system. Answers exactly what an upload with the same key would: `known`, the document `id` (null once a deleted document was purged), and whether the owner `deleted` it. Checks the document's own key, keys remembered for it when an import sent the same bytes under another key, and the keys of purged documents. Reachable with a `documents:write` Bearer (the scope's second and last route) or a cookie session; stores nothing. 5000 lookups an hour per token (per person for a session); gated on the `inboundDocuments` module.",
|
||||
parameters: [
|
||||
{
|
||||
name: "sourceSystem",
|
||||
in: "query",
|
||||
required: true,
|
||||
schema: { type: "string", enum: [...DOCUMENT_SOURCE_SYSTEMS] },
|
||||
},
|
||||
{
|
||||
name: "sourceId",
|
||||
in: "query",
|
||||
required: true,
|
||||
schema: { type: "string", maxLength: DOCUMENT_SOURCE_ID_MAX },
|
||||
},
|
||||
],
|
||||
responses: {
|
||||
"200": {
|
||||
description: "The answer for this key.",
|
||||
content: {
|
||||
"application/json": {
|
||||
schema: dataEnvelope(
|
||||
z
|
||||
.object({
|
||||
known: z.boolean(),
|
||||
id: z.string().nullable(),
|
||||
deleted: z.boolean(),
|
||||
})
|
||||
.meta({ id: "DocumentSourceLookup" }),
|
||||
"DocumentSourceLookupEnvelope",
|
||||
),
|
||||
},
|
||||
},
|
||||
},
|
||||
...stdResponses,
|
||||
},
|
||||
},
|
||||
},
|
||||
"/api/documents/inbound/capability": {
|
||||
get: {
|
||||
tags: ["Documents"],
|
||||
|
||||
@@ -1636,6 +1636,7 @@ const documentBackupSchema = z
|
||||
// restore cannot write a system the detail sheet has no name for.
|
||||
sourceSystem: z.enum(DOCUMENT_SOURCE_SYSTEMS).nullable().optional(),
|
||||
sourceId: z.string().max(DOCUMENT_SOURCE_ID_MAX).nullable().optional(),
|
||||
aiReadDeferred: z.boolean().optional(),
|
||||
summary: z.string().nullable().optional(),
|
||||
createdAt: isoDateTime.optional(),
|
||||
updatedAt: isoDateTime.optional(),
|
||||
|
||||
@@ -697,6 +697,26 @@ export function toDocumentSourceSystem(
|
||||
/** Max length of an imported document's id in its source system. */
|
||||
export const DOCUMENT_SOURCE_ID_MAX = 128;
|
||||
|
||||
/**
|
||||
* The document's id in its source system. Printable ASCII only: it is a key,
|
||||
* not prose, and it is echoed on the detail sheet.
|
||||
*/
|
||||
const documentSourceIdSchema = z
|
||||
.string()
|
||||
.trim()
|
||||
.min(1)
|
||||
.max(DOCUMENT_SOURCE_ID_MAX)
|
||||
.regex(/^[\x21-\x7e]+$/u, "Expected printable characters without spaces");
|
||||
|
||||
/**
|
||||
* A source key on its own — the query-string form the upload accepts ahead of
|
||||
* the body, and the lookup route's query. Both halves required.
|
||||
*/
|
||||
export const documentSourceKeySchema = z.object({
|
||||
sourceSystem: z.enum(DOCUMENT_SOURCE_SYSTEMS),
|
||||
sourceId: documentSourceIdSchema,
|
||||
});
|
||||
|
||||
export const documentCreateSchema = z
|
||||
.object({
|
||||
title: z.string().trim().min(1).max(DOCUMENT_TITLE_MAX).optional(),
|
||||
@@ -710,13 +730,7 @@ export const documentCreateSchema = z
|
||||
* The document's id in `sourceSystem`. Printable ASCII only: it is a key,
|
||||
* not prose, and it is echoed on the detail sheet.
|
||||
*/
|
||||
sourceId: z
|
||||
.string()
|
||||
.trim()
|
||||
.min(1)
|
||||
.max(DOCUMENT_SOURCE_ID_MAX)
|
||||
.regex(/^[\x21-\x7e]+$/u, "Expected printable characters without spaces")
|
||||
.optional(),
|
||||
sourceId: documentSourceIdSchema.optional(),
|
||||
/**
|
||||
* `defer` holds back automatic AI reading for this upload: the thumbnail
|
||||
* and a local text index still run, the summary and the lab staging do
|
||||
|
||||
@@ -830,6 +830,9 @@ async function seedEveryTwoEndedModel(prisma: PrismaClient): Promise<void> {
|
||||
// re-run of the importer after a restore stores the page twice.
|
||||
sourceSystem: "PAPERLESS",
|
||||
sourceId: "4711",
|
||||
// Held back from automatic AI reading; a restore that dropped the
|
||||
// marker would hand the page to the summary catch-up.
|
||||
aiReadDeferred: true,
|
||||
},
|
||||
});
|
||||
|
||||
@@ -1515,7 +1518,12 @@ describe("every model the plan claims two-ended survives a real restore", () =>
|
||||
expect({
|
||||
sourceSystem: vaultDocument.sourceSystem,
|
||||
sourceId: vaultDocument.sourceId,
|
||||
}).toEqual({ sourceSystem: "PAPERLESS", sourceId: "4711" });
|
||||
aiReadDeferred: vaultDocument.aiReadDeferred,
|
||||
}).toEqual({
|
||||
sourceSystem: "PAPERLESS",
|
||||
sourceId: "4711",
|
||||
aiReadDeferred: true,
|
||||
});
|
||||
const threads = await prisma.coachConversation.findMany({
|
||||
where: { userId: OWNER_ID },
|
||||
orderBy: { createdAt: "asc" },
|
||||
|
||||
@@ -295,7 +295,7 @@ describe("document import through a documents:write token", () => {
|
||||
expect((await own.json()).data.title).toBeNull();
|
||||
});
|
||||
|
||||
it("413s past the quota with the figures, storing nothing", async () => {
|
||||
it("413s past the quota without the figures, storing nothing", async () => {
|
||||
await seedUser({ inboundDocuments: true });
|
||||
token = await mintDocumentToken();
|
||||
await getPrismaClient().user.update({
|
||||
@@ -306,10 +306,10 @@ describe("document import through a documents:write token", () => {
|
||||
const res = await post(upload(PNG_1X1, {}, "x.png"));
|
||||
expect(res.status).toBe(413);
|
||||
const body = await res.json();
|
||||
expect(body.meta).toMatchObject({
|
||||
reason: "quotaExceeded",
|
||||
quotaBytes: 10,
|
||||
});
|
||||
// The figures are the owner's; the token hears only that it is full.
|
||||
expect(body.meta.reason).toBe("quotaExceeded");
|
||||
expect(body.meta).not.toHaveProperty("quotaBytes");
|
||||
expect(body.meta).not.toHaveProperty("usedBytes");
|
||||
expect(
|
||||
await getPrismaClient().inboundDocument.count({ where: { userId } }),
|
||||
).toBe(0);
|
||||
@@ -325,6 +325,172 @@ describe("document import through a documents:write token", () => {
|
||||
});
|
||||
});
|
||||
|
||||
describe("source keys that reach a document by its bytes (#1038)", () => {
|
||||
it("a key answered with an existing document stays deleted after delete and purge", async () => {
|
||||
await seedUser({ inboundDocuments: true });
|
||||
token = await mintDocumentToken();
|
||||
const prisma = getPrismaClient();
|
||||
|
||||
// Stored by hand first, no key.
|
||||
asCookie();
|
||||
const manual = await post(upload(pdf("same")));
|
||||
expect(manual.status).toBe(201);
|
||||
const id = (await manual.json()).data.id as string;
|
||||
|
||||
// The import sends the same bytes under two keys: both answered with it.
|
||||
asToken();
|
||||
for (const sourceId of ["K1", "K2"]) {
|
||||
const res = await post(
|
||||
upload(pdf("same"), { sourceSystem: "PAPRA", sourceId }),
|
||||
);
|
||||
expect(res.status).toBe(200);
|
||||
expect((await res.json()).data).toEqual({ id, duplicate: true });
|
||||
}
|
||||
expect(
|
||||
await prisma.documentSourceAlias.count({ where: { documentId: id } }),
|
||||
).toBe(2);
|
||||
|
||||
// Deleted: both keys answer "deleted", even with different bytes.
|
||||
asCookie();
|
||||
const byId = await import("@/app/api/documents/inbound/[id]/route");
|
||||
await (byId.DELETE as unknown as (r: Request, c: Ctx) => Promise<Response>)(
|
||||
new Request(`http://localhost/api/documents/inbound/${id}`, {
|
||||
method: "DELETE",
|
||||
}),
|
||||
ctx(id),
|
||||
);
|
||||
asToken();
|
||||
const tomb = await post(
|
||||
upload(pdf("rescan"), { sourceSystem: "PAPRA", sourceId: "K2" }),
|
||||
);
|
||||
expect((await tomb.json()).data).toEqual({
|
||||
id,
|
||||
duplicate: true,
|
||||
deleted: true,
|
||||
});
|
||||
|
||||
// Purged: the aliases move to the ledger with the row.
|
||||
const { purgeExpiredDocumentTombstones } =
|
||||
await import("@/lib/jobs/document-purge");
|
||||
await purgeExpiredDocumentTombstones(
|
||||
prisma as never,
|
||||
new Date(Date.now() + 31 * 86_400_000),
|
||||
);
|
||||
expect(await prisma.documentSourceAlias.count()).toBe(0);
|
||||
expect(
|
||||
(
|
||||
await prisma.documentImportKey.findMany({
|
||||
where: { userId },
|
||||
select: { sourceId: true },
|
||||
orderBy: { sourceId: "asc" },
|
||||
})
|
||||
).map((k) => k.sourceId),
|
||||
).toEqual(["K1", "K2"]);
|
||||
for (const sourceId of ["K1", "K2"]) {
|
||||
const res = await post(
|
||||
upload(pdf("same"), { sourceSystem: "PAPRA", sourceId }),
|
||||
);
|
||||
expect((await res.json()).data).toEqual({
|
||||
id: null,
|
||||
duplicate: true,
|
||||
deleted: true,
|
||||
});
|
||||
}
|
||||
expect(await prisma.inboundDocument.count({ where: { userId } })).toBe(0);
|
||||
});
|
||||
});
|
||||
|
||||
describe("re-sends cost no allowance and no body read (#1038)", () => {
|
||||
it("answers a query-string key before the body and the bucket, and the lookup agrees", async () => {
|
||||
await seedUser({ inboundDocuments: true });
|
||||
token = await mintDocumentToken();
|
||||
process.env.DOCUMENT_UPLOAD_LIMIT_PER_HOUR = "1";
|
||||
asToken();
|
||||
|
||||
const key = "?sourceSystem=PAPERLESS&sourceId=77";
|
||||
const first = await post(
|
||||
new Request(`http://localhost/api/documents/inbound${key}`, {
|
||||
method: "POST",
|
||||
body: (() => {
|
||||
const f = new FormData();
|
||||
f.append("file", new Blob([new Uint8Array(pdf("x"))]), "x.pdf");
|
||||
return f;
|
||||
})(),
|
||||
headers: { authorization: `Bearer ${token}` },
|
||||
}),
|
||||
);
|
||||
expect(first.status).toBe(201);
|
||||
|
||||
// The one slot is spent; re-sends still answer, with no body at all.
|
||||
for (let i = 0; i < 3; i++) {
|
||||
const again = await post(
|
||||
new Request(`http://localhost/api/documents/inbound${key}`, {
|
||||
method: "POST",
|
||||
headers: { authorization: `Bearer ${token}` },
|
||||
}),
|
||||
);
|
||||
expect(again.status).toBe(200);
|
||||
expect((await again.json()).data.duplicate).toBe(true);
|
||||
}
|
||||
|
||||
const lookup = await import("@/app/api/documents/inbound/source/route");
|
||||
const get = (q: string) =>
|
||||
(lookup.GET as unknown as (r: Request) => Promise<Response>)(
|
||||
new NextRequest(`http://localhost/api/documents/inbound/source${q}`, {
|
||||
headers: { authorization: `Bearer ${token}` },
|
||||
} as never),
|
||||
);
|
||||
const known = await get(key);
|
||||
expect(known.status).toBe(200);
|
||||
expect((await known.json()).data).toMatchObject({
|
||||
known: true,
|
||||
deleted: false,
|
||||
});
|
||||
const unknown = await get("?sourceSystem=PAPERLESS&sourceId=78");
|
||||
expect((await unknown.json()).data).toEqual({
|
||||
known: false,
|
||||
id: null,
|
||||
deleted: false,
|
||||
});
|
||||
expect((await get("?sourceSystem=PAPERLESS")).status).toBe(422);
|
||||
|
||||
// A new document still meets the spent bucket.
|
||||
const blocked = await post(upload(pdf("new")));
|
||||
expect(blocked.status).toBe(429);
|
||||
});
|
||||
|
||||
it("a duplicate by bytes hands its slot back", async () => {
|
||||
await seedUser({ inboundDocuments: true });
|
||||
token = await mintDocumentToken();
|
||||
process.env.DOCUMENT_UPLOAD_LIMIT_PER_HOUR = "2";
|
||||
asToken();
|
||||
expect((await post(upload(pdf("one")))).status).toBe(201);
|
||||
// Charged and handed back each time; without the refund the second of
|
||||
// these would already meet a spent bucket.
|
||||
expect((await post(upload(pdf("one")))).status).toBe(200);
|
||||
expect((await post(upload(pdf("one")))).status).toBe(200);
|
||||
expect((await post(upload(pdf("two")))).status).toBe(201);
|
||||
expect((await post(upload(pdf("three")))).status).toBe(429);
|
||||
});
|
||||
});
|
||||
|
||||
describe("aiRead=defer is kept on the row (#1038)", () => {
|
||||
it("marks the upload and leaves others alone", async () => {
|
||||
await seedUser({ inboundDocuments: true });
|
||||
token = await mintDocumentToken();
|
||||
asToken();
|
||||
const deferred = await post(upload(pdf("d"), { aiRead: "defer" }));
|
||||
const plain = await post(upload(pdf("p")));
|
||||
const rows = await getPrismaClient().inboundDocument.findMany({
|
||||
where: { userId },
|
||||
select: { id: true, aiReadDeferred: true },
|
||||
});
|
||||
const byId = new Map(rows.map((r) => [r.id, r.aiReadDeferred]));
|
||||
expect(byId.get((await deferred.json()).data.id)).toBe(true);
|
||||
expect(byId.get((await plain.json()).data.id)).toBe(false);
|
||||
});
|
||||
});
|
||||
|
||||
describe("a documents:write token reaches nothing else", () => {
|
||||
it("is refused by every other vault leg, the mints and the measurement write", async () => {
|
||||
await seedUser({ inboundDocuments: true });
|
||||
|
||||
Reference in new issue
Block a user