With 16.4.0 a document sheet opened through ?doc= stays open after Back,
and the [ / ] day keys sometimes leave ?day= where it was. Three end-to-end
journeys fail 3 of 40 repeated runs on 16.4.0 and 0 of 80 on 16.3.8 with
every other update in place, and CI hit the same journeys on both
viewports. 16.3.8 already carries every published security fix, so
Dependabot skips 16.4.0 for next and the two packages that move with it.
Next 16.4.0, pg-boss 12.37.1, the AWS SDK 3.1146, @napi-rs/canvas 1.0.10,
@node-rs/argon2 2.2.2, @simplewebauthn/server 14.0.3, TanStack Query and
Virtual, lucide-react, mmdb-lib, nodemailer, otpauth, pg, radix-ui,
react-day-picker, react-hook-form, tailwind-merge and yaml.
pg-boss moves its schema from 42 to 45 (three additive migrations, nullable
columns and one partial index, no table rewrite); the worker applies them on
start as before. The retention test pins the installed version, so it moves
to 12.37.1 after re-checking every source line it asserts.
zod stays on 4.4.3. 4.5 requires seconds in `z.iso.datetime()` and 4.6
rewrites nullable unions in the generated JSON Schema into `type` arrays,
which reshapes the OpenAPI document and the MCP tool schemas. Dependabot
skips zod 4.5 and later until that move is made on purpose.
The live evaluation now captures the tool calls a generation makes, with
their arguments, and hands them to the judge. runScenarioLive puts a
scenario to the model in the chat turn's own context (system prompt,
tool-mode and dialog addenda, DATA INVENTORY, EARLIER TABLES, transcript)
over the real tool catalogue, answers each call with a stand-in result,
and reads the reply through the turn's clarification and chip parsers.
runScenarioJudge grades each scenario a few times with the same
evaluateScenario the deterministic suite uses and reports a pass rate per
scenario; the server-only scenarios and checks are left out.
Like the existing judge it runs only with COACH_EVAL_API_KEY, from the
nightly workflow's non-blocking step, and never throws.
The delegated-writes spec checked for the invitation form's level control
right after page load, before the form had painted, found nothing and
skipped every test on every run, CI included.
- Give the journey its own account pair, seeded and reset in global setup,
so it no longer invites the same pair as the account-sharing journey
(a second live grant between two accounts is refused with 409).
- Wait for the form through its data-slot and fail when the WRITE level
control is missing instead of skipping.
- Revoke the journey's grant in afterAll.
- Keep it out of the mobile project in the config rather than through a
runtime skip.
Add two checks so a skip cannot hide a spec again:
- e2e-conditional-skip-guard.test.ts requires every conditional
test.skip/test.fixme in e2e/ to carry a reason and refuses a bare
test.skip().
- scripts/check-e2e-skipped-specs.mjs reads the Playwright JSON reports of
both shards in the e2e gate job and fails when a spec file skipped all of
its tests across every project, apart from an allowlist with reasons.
A commit that reaches main without a pull request, such as an asset
uploaded through the web UI, was never analysed until the weekly run.
Run on push to main as well. Pull-request runs still cancel their
superseded heads; push runs are grouped per commit and never cancelled,
so back-to-back merges each keep their own analysis.
The Trivy install step piped a script fetched from a tag straight into a
shell. A tag can be moved to other content, so the job ran whatever the
ref pointed at, with the runner's privileges, and the Scorecard check
flagged it as an unpinned download-then-run.
The script now comes from the commit the tag points at, is checked
against its SHA-256 before it runs, and the binary version stays what it
was.
The compose ecosystem added last change did its job on the first run and
immediately proposed 16.15 to 18.6. A PostgreSQL major changes the
on-disk format, so merging that would hand self-hosters a container that
cannot read their data directory without pg_upgrade or a dump and
restore. Minor releases keep arriving as reviewable PRs; a major is a
project decision with a migration path behind it, not a weekly bump.
The guard grows a case for the ignore rule, so dropping it fails rather
than quietly re-opening the door.
The bundled PostgreSQL was pinned to 16.14 while 16.15 had been out for
five weeks. The pin itself is deliberate, a database change should be
reviewable rather than whatever the registry serves on pull day, but
nothing watched it: Dependabot was configured with the docker ecosystem,
which reads the Dockerfile and never opens a compose file, so no update
PR was ever raised. CI and the integration suite meanwhile run against
the floating 16-alpine, so the shipped database was older than the
tested one.
Adds the docker-compose ecosystem so each PostgreSQL minor release
arrives as a reviewable PR, moves the pin to 16.15, and pins both halves
with a test that fails if the exact minor or the watch is dropped.
Refs #987
S1. The journey signs in with a real account and POSTs that account's
password to whatever BASE_URL names, and the workflow took BASE_URL as
free text. Anyone with repo write, or a leaked token, could dispatch the
run at a collector of their own and keep a working credential; secret
masking guards the log, never the egress.
The host now has to be on a short const allowlist in the script, checked
before anything is sent, with exit 2 and a named refusal otherwise. The
workflow input is a choice between the two hosted instances, so the form
cannot offer a third; the script's own check still stands behind it for a
dispatch that goes round the form.
--self-test covers the refusal, the prefix lookalike a startsWith check
would have waved through included.
Post-publish verify proves the published image boots. Nothing proved that
on a running instance a person can sign in, write a reading and read it
back — which is exactly the shape of the defects this line keeps
rediscovering: a value written and never read.
`scripts/synthetic-journey.mjs` walks that journey in plain Node 22 with no
dependencies, so it runs from a workstation, a runner or the deploy host.
It warms `/`, `/api/health`, `/api/version` and the measurements list first
and discards those timings (the first hit on each route after a deploy is a
cold start), then asserts five legs in order: the served version equals the
expected one, the password flow signs in through a cookie jar, a weight
reading is written with an Idempotency-Key, the same row reads back with
its value, unit and source, and the reversible delete leaves the account
clean. One line per leg with the elapsed time; the first failure ends the
run non-zero with the status and a body excerpt that has every token shape
redacted out of it.
The workflow is `workflow_dispatch` only, with `base_url` and
`expected_version` as inputs and the credentials read from repository
secrets. It is deliberately not wired into the deploy path: the target
instance and the secrets are the maintainer's call.
Mutation proof, against a production build served from
`.next/standalone/server.js` on a throwaway Postgres.
Green:
warm 35 ms (discarded)
ok version 2 ms serves 1.38.13
ok sign in 78 ms session cookie held
ok write 64 ms wrote cmtup9lwu0007zhovn0mxajdz
ok read back 8 ms 77.7 kg (MANUAL)
ok delete 27 ms account left clean
synthetic journey green
Read leg broken deliberately — the same server behind a throwaway forwarder
that empties the `GET /api/measurements` list and changes nothing else:
ok version 4 ms serves 1.38.13
ok sign in 38 ms session cookie held
ok write 40 ms wrote cmtupa4ck000gzhovwnij64fz
FAILED read back 8 ms read back: the list came back without the reading
status 200
body {"data":{"measurements":[],"meta":{"total":0}},"error":null}
synthetic journey RED on "read back"
Version leg broken the same way, by asking the instance for the previous
tag:
FAILED version 3 ms version: instance serves "1.38.13", expected "v1.38.12"
synthetic journey RED on "version"
`--self-test` carries that proof in the repository: it runs the journey
against an in-process instance nine times, once with nothing broken and
once per broken leg, and fails if a broken leg still comes back green. The
workflow runs it before it touches the real instance.
Gates: typecheck, `node --check`, lint (the three known warnings, no
errors), prettier, the workflow and CI-determinism guards.
The audit had moved off the pull_request trigger entirely, which left a window in which a tag could be cut from a merge the audit had not seen. It runs on pull requests again as an advisory job (continue-on-error), stays blocking on the push run for main and on the daily schedule, and a tag is only ever cut from a green push run.
The hero-geometry check stopped skipping under CI in v1.38 and started failing
instead, which was the point: it had caught its own launch failure and passed
for months, measuring nothing. The install step went into the quality job of
`security.yml` and nowhere else, and a second job runs the same suite — the
auto-merge job for dependency bumps. Every Dependabot pull request has failed
since on `browserType.launch: Executable doesn't exist`, four of them on
2026-09-06 alone (run 34043840314), on an error that says nothing about the
bump. One workflow was fixed; the class was not.
That job installs the same headless shell now, off the same cache key. The
error the check throws names both jobs rather than sending a reader to the
workflow that was already correct, and a guard sweeps the workflow directory so
a third job cannot run the suite with no browser anywhere in it. Its limit is
written down: the sweep is file-scoped, so a workflow with two jobs where only
one installs still satisfies it.
Gates:
CI=1, browser path hidden exit=1, "This check is the gate in CI and it
has no browser to measure with"
CI=1, browser present exit=0, 4 tests passed (it measures)
install step deleted from the guard fails: ["dependabot-auto-merge.yml"]
auto-merge workflow
`pnpm audit --prod` answers a question about the published advisory database,
not about the diff. The same lockfile passes on Monday and fails on Tuesday
because someone else published, and the branch that turns red changed nothing:
seventeen of the forty-one Security & Quality failures in the last thirty days
were advisory publications on branches that touched no dependency
(GHSA-jqff-g426-hqxp, GHSA-fph4-wmhf-6fwf, GHSA-f65p-4m7j-42xc and
GHSA-5jgf-p345-68v8 in one run alone). A gate that fails for a reason the author
cannot act on is a gate people re-run instead of read.
It keeps running on every push to `main` and once a day on a new schedule
trigger, which is the cadence the question actually has. The other three jobs
sit out the scheduled run. Nothing about what ships is loosened: the container
scan still builds and scans the image on every push to `main`, and the daily
Trivy workflow still scans the lockfile with dev dependencies included and the
published image.
Gate — the trigger set, read back from the parsed workflow:
triggers: push, pull_request, schedule
quality | if: github.event_name != 'schedule'
dependency-audit | if: github.event_name != 'pull_request'
secret-scan | if: github.event_name != 'schedule'
docker-security | if: github.event_name == 'push' && github.ref == 'refs/heads/main'
The scanner job installed Trivy through apt, which runs `apt-get update`
across every repository on the runner image. On the v1.38.12 merge
commit the runner's preinstalled Chrome repository answered with a
corrupt index ("Hash Sum mismatch") twice in a row, and the release gate
went red on a step that has nothing to do with the image under test.
The step now fetches the pinned release binary through Trivy's own
installer, which downloads the tarball for that exact version and checks
it against the published checksums. No apt, so no unrelated repository
can fail the job.
The workflow granted contents: write and pull-requests: write at the top
level, so every step in the run held them, including the checkout and
the dependency install that execute code from the pull request being
merged. Scorecard flags the shape, and the shape is the problem: a
compromised transitive in a Dependabot branch would run with a token
that can write to the repository.
The workflow default is contents: read now, and the two write scopes sit
on the auto-merge job, which is the only job there is and the only place
they are spent. gh pr merge --auto needs contents: write to enable
merge-on-green and pull-requests: write to act on the pull request; the
major-bump comment needs pull-requests: write. The metadata fetch and
the test run need neither. Nothing about which updates merge
automatically changes.
Three of the last four pushes to main failed the integration workflow
without a single failing test. Shard 1 of 3 was still logging green
results when the 25-minute job timeout cancelled it; the same tests in
the same shard had run in 13 minutes the day before. A slow runner day
doubles every test's wall time, and a guard under twice the ordinary
run cannot absorb that.
Four shards keep each slice around fifty files, and the guard moves to
40 minutes, three times an ordinary run. A wedged test still fails
through vitest's own 60-second budget, so the job timeout only has to
catch a container that never boots. The fan-in job is unchanged; it
still refuses a matrix with any cancelled shard.
init, analyze and upload-sarif are the same release pinned in three
workflows. Dependabot opened them as three separate pull requests, so each
run mixed two versions and analyze rejected the config init had written:
"Loaded a configuration file for version 4.37.7, but running version
4.37.9". None of the three could go green alone, and the code-scanning
analysis stayed red while they waited on each other.
Bump all five references together and group the action in dependabot.yml so
the next release arrives as one pull request.
The hero-geometry check has never measured anything in CI. It launches
Chromium in a hook, catches the launch failure, and every case then
returns early — green. The quality job installs no browsers, so every
run of it since the file landed printed
[hero geometry] SKIPPED — no Chromium build available (Error:
browserType.launch: Executable doesn't exist at
/home/runner/.cache/ms-playwright/chromium_headless_shell-1234/ch)
and passed. The file's own header warned about exactly this trap and
then fell into it.
Both ends move. The quality job installs the headless shell before the
unit suite — `--only-shell chromium`, because that is the build
`chromium.launch()` resolves to and the one the failing runs named, with
a cache on the lockfile hash so the download is paid once. And the check
now refuses to be optional: with CI set, a launch failure throws out of
the hook and fails the suite with the step to restore. Without CI it
still skips, through the test context so the skip and its reason land in
the report rather than in a console line nobody reads.
No panel number changed. The measurements the check reads back are the
same ones the reserve was set against: panels of 400 / 428 / 456 / 484
px for three to six rows, 420 px for three rows plus the basis line,
against a 448 px reserve.
Dependabot treats `init`, `analyze` and `upload-sarif` as separate
dependencies, so it raised `init` to 4.37.7 in one PR and left `analyze` on
4.37.6 in the same workflow. The action refuses that pairing outright:
"Loaded a configuration file for version '4.37.7', but running version
'4.37.6'", and the CodeQL job failed on the very PR doing the bump.
All five pinned references move together here: init and analyze in
codeql.yml, and the three upload-sarif steps in scorecard.yml and trivy.yml.
The commit SHA is the one the v4.37.7 tag dereferences to, which is also the
SHA both Dependabot PRs pin.
The determinism guard reads the job that runs the browser, which is now
the shard matrix rather than the gate that only carries the required
check name. The report artifact is named per shard so the two uploads
cannot clobber each other.
The single job passed at about eighteen minutes against a twenty-minute
cap, and the settings geometry measurements pushed it over, so the run
died at the timeout with everything to that point green. Same remedy as
the integration suite, split rather than raise: two Playwright shards,
each with its own database and production build, and a gate job carrying
the exact name branch protection requires that fails when any shard
failed or was cancelled.
Two shards cleared in about fourteen minutes on a quiet day, but with six
pull requests running side by side the larger shard hit the same 25-minute
guard the split was meant to escape. Three shards keep the biggest slice
around sixty files, which clears the guard with real headroom on the
slowest runner observed. The gate job is unchanged.
The suite outgrew its runner. The 25-minute guard was sized for 128 files
taking about 13 minutes; the sharing release took the suite to 180 files and
past the cap on CI runners, so every run died at the timeout with every test
to that point green, reported as cancelled. The guard's own rule applies:
split, don't raise.
The files run as two vitest shards, each with its own Postgres testcontainer.
A gate job carries the exact name branch protection requires and fails when
any shard failed or was cancelled, so a half-green matrix cannot satisfy it.
vitest is invoked directly because pnpm passes the -- separator through to
the script literally: the flag never applied through the script form, and
both "shards" ran the full suite. Proven by paired local runs: the script
form ran 180 files twice, the direct form splits 90/90 with 927 plus 689
tests, all green.
Most of what Scorecard looks for this repository already does: Actions
pinned by commit SHA, a security policy with a private reporting path,
Dependabot with auto-merge, keyless signing on the release image, a
branch-protected trunk. The value is not the score, it is that the checks
keep running after the person who set them up has moved on, and that
anyone evaluating the project can read the result without being given CI
access.
publish_results is on, which is what makes the score citable. It reads
repository metadata and the workflow files; nothing about the code or its
contents goes anywhere. The token is an OIDC exchange, the same mechanism
the release workflow already uses to sign the image.
No badge in the README yet. The public API has no entry for this
repository until the first run publishes one, and the badge URL currently
renders 'invalid repo path'. It goes in once there is a score behind it.
dependabot[bot]