fix(scripts): skip known documents before download in the importer

Ask HealthLog whether a document is already held before downloading it
from Paperless-ngx or Papra, and send the source key in the query string.
Pin Paperless API version 9 (Paperless-ngx 2.16 and later), apply --since
locally for Paperless too, fall back to the file name for a blank title,
stop with a clear message when the HealthLog address redirects, and say
only that storage is full when the figures are withheld. The guide names
the minimum versions and the shared OTHER namespace.

Refs #1038

docs/integrations/document-import.md

@@ -30,7 +30,9 @@ API Tokens**. It expires after a year unless you revoke it sooner.
 
 ## 2. New documents from Paperless-ngx (workflow)
 
-In Paperless-ngx, open **Workflows** and add one:
+This needs Paperless-ngx 3.0 or later (the `{{doc_id}}` placeholder and the
+double-brace placeholder syntax below arrived there). In Paperless-ngx, open
+**Workflows** and add one:
 
 - **Triggers:** add _Document Added_ and _Document Updated_. On each, set
   the filter to "has any of these tags" and pick the tag you use for health
@@ -71,7 +73,8 @@ and let the workflow handle what comes after.
 ## 3. The archive: the import script
 
 The script is a single file, `scripts/import-documents.mjs` in the
-HealthLog repository. It needs Node 22 and nothing else, and it runs on any
+HealthLog repository. It works with Paperless-ngx 2.16 or later (API
+version 9) and current Papra releases. It needs Node 22 and nothing else, and it runs on any
 machine that can reach both systems. Download it:
 
 ```sh
@@ -136,14 +139,20 @@ What it does:
 - The document date is the date the source has for the document. When the
   source has none, the script uses the day the document was added there, and
   lists those documents in its summary so you can correct the date.
-- Running it again is safe. HealthLog skips documents it already has, and
-  documents you deleted in HealthLog stay deleted. An interrupted run simply
-  picks up where it stopped.
+- Running it again is safe. Before downloading a document the script asks
+  HealthLog whether it already has it, so documents already there are
+  neither downloaded nor sent again, and a re-run over a large archive is
+  quick. Documents you deleted in HealthLog stay deleted, also after they
+  are removed for good. The same goes for a file you had already uploaded
+  by hand: the import recognises it by its content, and if you delete it
+  later, the import leaves it deleted. An interrupted run simply picks up
+  where it stopped.
 - When HealthLog asks it to slow down, it waits as long as HealthLog says
   and carries on. By default a document token may upload 120 documents an
-  hour, so an archive of a thousand documents takes an evening. Uploads you
-  make yourself in the web app or on your phone are counted separately and
-  are not held up.
+  hour, so an archive of a thousand documents takes an evening. Documents
+  HealthLog already has do not count towards that. Uploads you make yourself
+  in the web app or on your phone are counted separately and are not held
+  up.
 - At the end it prints how many documents were imported, how many were
   already there, how many you had deleted, and every document it skipped
   with the reason. It exits with `0` when everything went through, `3` when
@@ -159,7 +168,9 @@ the script holds that back, so a whole archive does not turn into a
 thousand AI requests at once. Imported documents still get a preview and
 are searchable by their text where the file has a text layer.
 
-To have documents read later, open one and choose **Read with AI** or
+HealthLog remembers that these documents were held back: turning on
+automatic AI reading later does not send them to your AI provider either.
+To have documents read, open one and choose **Read with AI** or
 **Generate summary**, or use **Index all for search** in Documents for the
 ones that had no text to search. Pass `--ai-read` if you do want every
 imported document read as it arrives.
@@ -193,6 +204,15 @@ until they are removed for good 30 days later. Run the script with
 the admin area, for everyone or for one account. If storage runs out during
 an import, the script stops and says so; raise the limit and run it again.
 
+## Source ids
+
+Documents are recognised by the pair of source system and document id:
+`PAPERLESS` plus the Paperless document id, `PAPRA` plus the Papra document
+id. Anything else is sent as `OTHER`, and `OTHER` is one shared set of ids
+per account: if you push documents from two other systems, make sure their
+ids cannot collide, for example by prefixing them (`nextcloud-123`,
+`scanner-123`).
+
 ## For operators
 
 `DOCUMENT_UPLOAD_LIMIT_PER_HOUR` sets how many documents one document token

scripts/__tests__/import-documents.test.ts

@@ -86,12 +86,40 @@ async function healthlog(
     deleted?: string[];
     tooLarge?: string[];
     moduleOff?: boolean;
+    /** An older HealthLog with no lookup route. */
+    noLookup?: boolean;
+    /** The address redirects elsewhere (http → https, a moved host). */
+    redirect?: string;
   } = {},
 ) {
   const stored = new Set<string>();
   const received: Received[] = [];
+  const lookups: string[] = [];
   let throttled = false;
   const url = await serve(async (req, url, body) => {
+    if (opts.redirect) {
+      return {
+        status: 301,
+        headers: { location: `${opts.redirect}${url.pathname}` },
+      };
+    }
+    const queryKey = `${url.searchParams.get("sourceSystem")}:${url.searchParams.get("sourceId")}`;
+    if (
+      url.pathname === "/api/documents/inbound/source" &&
+      req.method === "GET" &&
+      !opts.noLookup
+    ) {
+      lookups.push(queryKey);
+      const deleted = opts.deleted?.includes(queryKey) ?? false;
+      const known = deleted || stored.has(queryKey);
+      return {
+        status: 200,
+        json: {
+          data: { known, id: known ? queryKey : null, deleted },
+          error: null,
+        },
+      };
+    }
     if (url.pathname !== "/api/documents/inbound" || req.method !== "POST") {
       return { status: 404, json: { data: null, error: "Not found" } };
     }
@@ -130,8 +158,11 @@ async function healthlog(
         size = v.size;
       }
     }
+    // The key rides the query string, where HealthLog reads it first.
+    fields.sourceSystem = url.searchParams.get("sourceSystem") ?? "";
+    fields.sourceId = url.searchParams.get("sourceId") ?? "";
     received.push({ auth: req.headers.authorization, fields, filename, size });
-    const key = `${fields.sourceSystem}:${fields.sourceId}`;
+    const key = queryKey;
     if (opts.tooLarge?.includes(key)) {
       return {
         status: 413,
@@ -168,7 +199,7 @@ async function healthlog(
       json: { data: { id: key, duplicate: false }, error: null },
     };
   });
-  return { url, received };
+  return { url, received, lookups };
 }
 
 /** Paperless-ngx v10 stand-in: two pages of documents, one outside the tag. */
@@ -238,7 +269,8 @@ async function paperless() {
             },
             {
               id: 2,
-              title: "Arztbrief",
+              // Blank in the source: the file name stands in.
+              title: "   ",
               created: null,
               added: "2024-04-01T09:30:00Z",
               tags: [7],
@@ -379,6 +411,7 @@ describe("import-documents.mjs — Paperless-ngx", () => {
     });
     // No created date: filed under the day it was added, and said so.
     expect(hl.received[1].fields.documentDate).toBe("2024-04-01");
+    expect(hl.received[1].fields.title).toBe("brief");
     expect(hl.received[1].fields.kind).toBeUndefined();
     expect(first.stdout).toMatch(/Imported: 2/);
     expect(first.stdout).toMatch(/filed under the day it was added/);
@@ -387,7 +420,8 @@ describe("import-documents.mjs — Paperless-ngx", () => {
     // The API contract the script leans on.
     const docs = pl.seen.filter((s) => s.path.startsWith("/api/documents/?"));
     expect(docs[0].path).toContain("tags__id__all=7");
-    expect(docs[0].accept).toBe("application/json; version=10");
+    // API v9: the lowest the current Paperless-ngx releases accept.
+    expect(docs[0].accept).toBe("application/json; version=9");
     expect(docs[0].auth).toBe("Token pl_test");
     // `next` was followed on the source's own origin.
     expect(docs[1].path).toContain("page=2");
@@ -398,11 +432,20 @@ describe("import-documents.mjs — Paperless-ngx", () => {
       "/api/documents/2/download/?original=true",
     ]);
 
-    // A second run changes nothing: HealthLog answers duplicate for both.
+    // A second run changes nothing: HealthLog already holds both, and the
+    // lookup says so before anything is downloaded or uploaded again.
+    const downloadsBefore = pl.seen.filter((s) =>
+      s.path.includes("/download/"),
+    ).length;
     const second = await runScript(args, env);
     expect(second.code).toBe(0);
     expect(second.stdout).toMatch(/Imported: 0/);
     expect(second.stdout).toMatch(/Already in HealthLog: 2/);
+    expect(pl.seen.filter((s) => s.path.includes("/download/")).length).toBe(
+      downloadsBefore,
+    );
+    expect(hl.received).toHaveLength(2);
+    expect(hl.lookups).toContain("PAPERLESS:1");
   });
 
   it("leaves a document deleted in HealthLog alone", async () => {
@@ -481,6 +524,73 @@ describe("import-documents.mjs — Paperless-ngx", () => {
   });
 });
 
+describe("import-documents.mjs — edges", () => {
+  const env = { HEALTHLOG_TOKEN: "hlk_test", PAPERLESS_TOKEN: "pl_test" };
+
+  it("still imports against a HealthLog without the lookup", async () => {
+    const hl = await healthlog({ noLookup: true });
+    const pl = await paperless();
+    const res = await runScript(
+      [
+        "paperless",
+        "--paperless-url",
+        pl.url,
+        "--healthlog-url",
+        hl.url,
+        "--tag",
+        "HealthLog",
+      ],
+      env,
+    );
+    expect(res.code).toBe(0);
+    expect(res.stdout).toMatch(/Imported: 2/);
+  });
+
+  it("stops on a redirecting HealthLog address and names where it points", async () => {
+    const hl = await healthlog({ redirect: "https://health.example" });
+    const pl = await paperless();
+    const res = await runScript(
+      [
+        "paperless",
+        "--paperless-url",
+        pl.url,
+        "--healthlog-url",
+        hl.url,
+        "--tag",
+        "HealthLog",
+      ],
+      env,
+    );
+    expect(res.code).toBe(1);
+    expect(res.stderr).toMatch(/redirects to https:\/\/health\.example/);
+  });
+
+  it("applies --since itself when the server ignores the filter", async () => {
+    const hl = await healthlog();
+    const pl = await paperless();
+    const res = await runScript(
+      [
+        "paperless",
+        "--paperless-url",
+        pl.url,
+        "--healthlog-url",
+        hl.url,
+        "--tag",
+        "HealthLog",
+        "--since",
+        "2024-04-01",
+      ],
+      env,
+    );
+    expect(res.code).toBe(0);
+    // Document 1 (created 2024-03-05) is before the cut and never fetched.
+    expect(hl.received.map((r) => r.fields.sourceId)).toEqual(["2"]);
+    expect(pl.seen.some((s) => s.path.startsWith("/api/documents/1/"))).toBe(
+      false,
+    );
+  });
+});
+
 describe("import-documents.mjs — Papra", () => {
   it("imports by tag since a date, and skips what HealthLog refuses as too large", async () => {
     const hl = await healthlog({ tooLarge: ["PAPRA:doc_b"] });

scripts/import-documents.mjs

@@ -283,7 +283,9 @@ function filenameFrom(response, fallback) {
 function paperless(options) {
   const headers = {
     Authorization: `Token ${options.sourceToken}`,
-    Accept: "application/json; version=10",
+    // API version 9 (Paperless-ngx 2.16 and later): the lowest the current
+    // servers still accept, and the one where `created` became a plain date.
+    Accept: "application/json; version=9",
   };
   const base = options.baseUrl;
   const label = "Paperless-ngx";
@@ -345,10 +347,17 @@ function paperless(options) {
         }
         const created = dayOf(doc.created);
         const added = dayOf(doc.added);
+        const date = created ?? added;
+        // Checked here as well: an older server may ignore the date filter.
+        if (options.since && date && date < options.since) continue;
         yield {
           sourceId: String(doc.id),
-          title: doc.title || doc.original_file_name || `Paperless ${doc.id}`,
-          date: created ?? added,
+          title: titleOf(
+            doc.title,
+            doc.original_file_name,
+            `Paperless ${doc.id}`,
+          ),
+          date,
           dateFallback: created ? null : added ? "added" : "none",
           filename: doc.original_file_name ?? null,
           kindKeys:
@@ -429,7 +438,11 @@ function papra(options) {
           if (options.since && date && date < options.since) continue;
           yield {
             sourceId: String(doc.id),
-            title: stripExtension(doc.name) || `Papra ${doc.id}`,
+            title: titleOf(
+              stripExtension(doc.name),
+              doc.originalName,
+              `Papra ${doc.id}`,
+            ),
             date,
             dateFallback: documentDate ? null : created ? "added" : "none",
             filename: doc.originalName ?? doc.name ?? null,
@@ -480,6 +493,19 @@ function dayOf(value) {
   return /^\d{4}-\d{2}-\d{2}$/.test(day) ? day : null;
 }
 
+/**
+ * A title that is not blank: the source's title, else its file name without
+ * the extension, else a placeholder naming the source id. HealthLog refuses a
+ * blank title.
+ */
+function titleOf(title, filename, fallback) {
+  return (
+    (typeof title === "string" ? title.trim() : "") ||
+    stripExtension(filename).trim() ||
+    fallback
+  );
+}
+
 function stripExtension(name) {
   if (typeof name !== "string") return "";
   return name.replace(/\.[A-Za-z0-9]{1,5}$/, "");
@@ -495,8 +521,68 @@ function kindFor(options, doc) {
   return options.kind;
 }
 
+/** The source key as query parameters, which HealthLog reads before the body. */
+function keyQuery(source, doc) {
+  return new URLSearchParams({
+    sourceSystem: source.system,
+    sourceId: doc.sourceId.slice(0, SOURCE_ID_MAX),
+  });
+}
+
+async function readBody(response) {
+  try {
+    return await response.json();
+  } catch {
+    return null;
+  }
+}
+
+/** Refusals that end the run, whichever HealthLog request met them. */
+function stopOnRefusal(response, body, url) {
+  if (response.status >= 300 && response.status < 400) {
+    const location = response.headers.get("location");
+    throw new StopError(
+      `The HealthLog address redirects${location ? ` to ${new URL(location, url).origin}` : ""}. Use that address with --healthlog-url.`,
+    );
+  }
+  if (response.status === 401) {
+    throw new StopError(
+      "HealthLog refused the token (401). It may be expired or revoked; create a new document token.",
+    );
+  }
+  if (response.status === 403) {
+    if (body?.meta?.errorCode === "module.disabled") {
+      throw new StopError(
+        "The document vault is switched off in HealthLog. Turn on Documents in Settings, then run the script again.",
+      );
+    }
+    throw new StopError(
+      "HealthLog refused the request (403). Use a document token from Settings > API & Tokens; other tokens cannot upload documents.",
+    );
+  }
+}
+
+/**
+ * Ask HealthLog whether it already holds this document, before downloading
+ * it from the source. Null when the answer is "no", or when the HealthLog
+ * version has no lookup yet (the upload then decides, as before).
+ */
+async function lookup(options, source, doc) {
+  const url = `${options.healthlogUrl}/api/documents/inbound/source?${keyQuery(source, doc)}`;
+  const response = await request("HealthLog", url, {
+    headers: {
+      Authorization: `Bearer ${options.healthlogToken}`,
+      Accept: "application/json",
+    },
+  });
+  const body = await readBody(response);
+  stopOnRefusal(response, body, url);
+  if (!response.ok || !body?.data?.known) return null;
+  return body.data.deleted ? { outcome: "deleted" } : { outcome: "duplicate" };
+}
+
 async function upload(options, source, doc, file) {
-  const url = `${options.healthlogUrl}/api/documents/inbound`;
+  const url = `${options.healthlogUrl}/api/documents/inbound?${keyQuery(source, doc)}`;
   const kind = kindFor(options, doc);
   const response = await request(
     "HealthLog",
@@ -513,22 +599,16 @@ async function upload(options, source, doc, file) {
         new Blob([file.bytes], { type: file.type }),
         file.filename,
       );
-      form.append("title", doc.title.trim().slice(0, TITLE_MAX));
+      form.append("title", doc.title.slice(0, TITLE_MAX));
       if (doc.date) form.append("documentDate", doc.date);
       if (kind) form.append("kind", kind);
-      form.append("sourceSystem", source.system);
-      form.append("sourceId", doc.sourceId.slice(0, SOURCE_ID_MAX));
       if (!options.aiRead) form.append("aiRead", "defer");
       return form;
     },
   );
 
-  let body = null;
-  try {
-    body = await response.json();
-  } catch {
-    body = null;
-  }
+  const body = await readBody(response);
+  stopOnRefusal(response, body, url);
   const meta = body?.meta ?? {};
   switch (response.status) {
     case 201:
@@ -537,23 +617,10 @@ async function upload(options, source, doc, file) {
       return body?.data?.deleted
         ? { outcome: "deleted" }
         : { outcome: "duplicate" };
-    case 401:
-      throw new StopError(
-        "HealthLog refused the token (401). It may be expired or revoked; create a new document token.",
-      );
-    case 403:
-      if (meta.errorCode === "module.disabled") {
-        throw new StopError(
-          "The document vault is switched off in HealthLog. Turn on Documents in Settings, then run the script again.",
-        );
-      }
-      throw new StopError(
-        "HealthLog refused the upload (403). Use a document token from Settings > API & Tokens; other tokens cannot upload documents.",
-      );
     case 413:
       if (meta.reason === "quotaExceeded") {
         throw new StopError(
-          `Your HealthLog storage is full (${mib(meta.usedBytes)} of ${mib(meta.quotaBytes)} used). An admin can raise the limit in the admin area, for everyone or just for your account.`,
+          "Your HealthLog storage is full. An admin can raise the limit in the admin area, for everyone or just for your account.",
         );
       }
       return {
@@ -642,6 +709,19 @@ async function run(argv, env = process.env) {
         continue;
       }
 
+      // Asked first so a document HealthLog already holds is not downloaded.
+      const known = await lookup(options, source, doc);
+      if (known) {
+        if (known.outcome === "duplicate") {
+          tally.duplicate++;
+          out(`  present   ${where}`);
+        } else {
+          tally.deleted++;
+          out(`  deleted   ${where} (you deleted it in HealthLog; left alone)`);
+        }
+        continue;
+      }
+
       const file = await source.download(doc);
       if (file.skip) {
         tally.skipped.push({ where, reason: file.skip });