feat(labs): add local-OCR text mode for image-blind AI providers

Add an opt-in in-browser OCR path so a user whose provider cannot read
images (ChatGPT-OAuth/Codex, a text-only model) can still scan a lab
report. The capability probe gains a `mode` field: `vision` when the
provider reads the image directly, `text` when local OCR is enabled and
any provider is configured, or null. The extract route grows a text
branch that accepts the in-browser-OCR'd text as JSON and structures it
through the existing schema + review/commit flow; the raw image never
reaches the server in text mode. New `labsLocalOcrEnabled` user column
(migration 0184, default off) gates it, with a dedicated PATCH endpoint.
This commit is contained in:
Marc-André Bombeck committed 2026-06-20 16:50:06 +02:00
1 parent fa37941add
commit 3d039d5220
13 files changed
+634 -77

No files matched your search

+36 -7
View File
@@ -6285,12 +6285,15 @@ paths:
post:
tags:
- Labs
summary: Extract lab readings from a photo or PDF
description: Read-only (NOT idempotent) vision extraction. Accepts a multipart `file` (JPEG/PNG/WebP, or PDF on an
Anthropic vision provider; ≤ 12 MiB). Gated by AI consent, a 6/hour rate bucket, and the per-day token budget.
The raw upload lives in memory only and is never persisted or logged. Returns proposed rows for the mandatory
human review screen — nothing is written. Extracted text is treated as untrusted (prompt-injection); the review
step is the safety boundary.
summary: Extract lab readings from a photo, PDF, or OCR'd text
description: "Read-only (NOT idempotent) extraction. Two modes by content-type. VISION (`multipart/form-data`): a `file`
(JPEG/PNG/WebP, or PDF on an Anthropic vision provider; ≤ 12 MiB) is run through the user's vision-capable
provider; the upload lives in memory only and is never persisted or logged. TEXT (`application/json`, opt-in
local OCR): the browser OCR's the image (tesseract.js) and POSTs `{ mode: \"text\", text }` — only the extracted
text reaches the server, so a text-only provider (e.g. ChatGPT-OAuth) reaches the same review/commit flow. Both
modes are gated by AI consent, a 6/hour rate bucket, and the per-day token budget, and return proposed rows for
the mandatory human review screen — nothing is written. Extracted content is treated as untrusted
(prompt-injection); the review step is the safety boundary."
requestBody:
required: true
content:
@@ -6304,6 +6307,9 @@ paths:
description: The lab-report image or PDF. Validated by magic-byte MIME sniff, not the wire Content-Type.
required:
- file
application/json:
schema:
$ref: "#/components/schemas/OcrTextExtractRequest"
responses:
"200":
description: Proposed rows for review.
@@ -8429,6 +8435,18 @@ components:
type: string
required:
- ids
OcrTextExtractRequest:
type: object
properties:
mode:
type: string
const: text
text:
type: string
description: The in-browser-OCR'd lab-report text. The raw image never reaches the server in this mode.
required:
- mode
- text
DisclaimerAckRequest:
type: object
properties:
@@ -19234,21 +19252,32 @@ components:
properties:
available:
type: boolean
mode:
anyOf:
- type: string
enum:
- vision
- text
- type: "null"
reason:
anyOf:
- type: string
enum:
- no-provider
- text-only-model
- enable-local-ocr
- type: "null"
pdfSupported:
type: boolean
required:
- available
- mode
- reason
- pdfSupported
additionalProperties: false
description: Whether the caller's configured AI provider can read images (drives the UI's scan affordance). `reason`
description: Whether the caller's configured AI provider can ingest a lab report (drives the UI's scan affordance).
`mode` is `vision` when the provider reads the image directly, `text` when the image is OCR'd in-browser and
only the extracted text is sent (opt-in local OCR for text-only providers), or null when unavailable. `reason`
explains an unavailable state; `pdfSupported` is true only for an Anthropic vision provider (native PDF). No
provider call is made.
OcrExtractEnvelope: