kreeader-ratchet

The detection ratchet: CV does a pass, ML does a pass, LLM does a pass — and we measure which gets closest. Learnings flow strictly downhill: LLM findings train ML; ML findings become CV rules/thresholds; the goal is always MORE done by plain CV, not less.

Hard boundary (RULES-level): LLMs never ship in the kreeader product. They live here, offline, as teachers and graders — labeling, tie-breaking, and correction-triage that would otherwise cost Andrew's time. If an LLM insight matters, it exits this repo as an ML training signal or a CV rule, never as a runtime dependency.

The three passes

pass	what runs	source of truth for scoring
CV	the shipped geometric ladder via cv.DecidePage (kreeader-cv v0.3.2, PaneDetectorVersion 16 — see the pin) — the exact production policy	pane_corrections (independent/drawn rows ONLY — confirms never grade)
ML	the learned strip-classifier rung + trained pane model	same
LLM	local VLMs (ollama + HF) prompted for pane boxes per page	same

Every pass on every page records: boxes, per-box confidence where the pass has one, wall time, model/version identity, and IoU vs ground truth — the strict accounting is the point (RULES.md rules 7/8). Pass results and push records are append-only JSONL under results/; selections, manifests, and derived reports are rebuilt in place.

Feedback into the layout editor (pane workbench)

Every sweep's disagreements are flagged into the pane workbench (review flags + detection-issue samples via the server API) so Andrew can SEE what each pass did on real pages and correct where all three are wrong — corrections which then feed the ordinary training loop.

What a flag actually buys (corrected 2026-08-28)

A review_flags row buys exactly two things:

1. a row in the pane workbench's review queue (GET /api/flags), which is the list Andrew works through. This is the reliable one.
2. eligibility for the active-learning queue's confident-wrong lane — activelearn.queryConfidentWrong will consider a flagged page that the gain lane's doubtedGate excludes for being too confident.

It does not buy a place in the queue, and it does not exempt the page from the cohort term that favours the biggest publisher. flag_publishers.py originally claimed the lane "is scored without the cohort term"; only lane assignment is cohort-free (Candidate.IsConfidentButWrong reads DetectScore and Flagged and nothing else). Ranking inside the lane is SelectMMR over Breakdown.Gain, and Candidate.Score includes w.Cohort*cohort (gain.go:376); the lane's pool is separately truncated to roughly 500 rows by ORDER BY pages.detect_score DESC (queue.go:285, called at queue.go:140). Pages flagged nearest the 0.85 threshold — which is what flag_publishers.py picks — sort toward the back of that cut.

Full correction, with the measured numbers, in results/pubdiversity-20260828/CORRECTIONS.md.

--max is a safety limit; state it when you override it

flag_results.py documents --max at 40, "so a sweep cannot bury the workbench queue". flag_publishers.py defaults it to 400 and the 2026-08-28 push used that — ten times the sibling's documented limit, and the run record did not say so. The review queue went from 68 to 332 entries.

Rule for any future push: if --max exceeds 40, the run record must state the override, why, and the resulting size of the review queue. A flag push is a claim on Andrew's attention and on GPU time (every model adoption re-detects every flagged book — panetrain/auto.go:593), so the size of that claim is part of the result, not a footnote.

One deliberate exception, stated here so it is not mistaken for drift: layout_discovery.py --push-proposals is uncapped by default (--max-proposals 0). What --max protects is a WRITE that claims attention or GPU time — a review flag, a template the detector starts snapping pages to. A layout PROPOSAL does neither: it upserts one pending row, is never re-asked once decided, and is reviewed in batch by design (kreeader-server#184). Capping the nightly push would not make it safer, only partial. What paces it is --rate.

What a flag's life actually looks like (and where it leaks)

A push writes two kinds of row, and they do not die together:

row	written by	deleted by
review_flags	POST /api/comics/{id}/page/{n}/flag, or the mirrored direct INSERT	six shipped paths — a drawn correction (api/debug.go:184), the one-click confirm (api/positivehooks.go:66), DELETE .../flag (api/debug.go:300), a corrections import (api/debug.go:553), a yes question verdict (api/debug.go:1058), comic deletion
page_failure_tags	POST .../tags, or the mirrored direct INSERT	only DELETE .../tags (api/debug.go:848) — which nothing calls automatically

So working a flag orphans its tags. Andrew clears a page, the flag disappears from the review queue, and the two ratchet:* tag rows stay forever. The 2026-08-28 publisher push wrote 528 tag rows across 21 distinct tag strings; once those 264 flags are worked, scorepanes -bytag reports 21 buckets describing pages that are no longer flagged at all. This is a real defect in what the harness leaves behind, not a cosmetic one — the numbers in a -bytag report are about a set that no longer exists.

Fixing it properly (tags dying with their flag, or a flag carrying its own reason so tags are unnecessary — db.ReviewFlag has no note column, which is why tags are used as one) is server-side work and out of this repo's scope. What this repo ships is the broom: harness/revert_flags.py --cleanup-orphans.

Ordering the workbench queue (flag_batch.py)

harness/flag_batch.py builds a human's working queue rather than a corpus lever: books ranked worst-detected first, so the top of the pane workbench is where the detector is least sure.

The whole ordering rests on one line of the server, which is worth re-checking before any future run because nothing else enforces it — api/debug.go handleListFlags is Order("review_flags.created_at DESC") with no limit and no secondary key. So:

• flags are inserted best book first, and the worst book — written last — carries the newest timestamp and lands on top;
• created_at is unix seconds, so a fast loop would drop the whole batch into one or two of them and lose the order to the planner. Each row gets an explicit timestamp instead, one second apart, ending at the push moment (never the future, which would sort above a flag Andrew raises tomorrow).

Per-page uncertainty is activelearn.Candidate.Uncertainty reimplemented term for term, from page_detection_signals deduplicated to the newest row per page (that table carries ~200k rows for ~196k pages; a naive join double-counts every re-detected page). The cohort term is excluded on purpose — it is a diversity device that rewards the biggest publisher, not a measure of doubt.

Two judgement calls it makes, both recorded rather than hidden:

• it does not drop books whose detector found no panelled page. That signature is shared by "a handbook with no panels, where whole is correct" and "a real comic where detection failed on every page", which is the most valuable correction there is. Measured on the 2026-08-29 run, such a gate would have dropped 53 of the top 150 and most were real comics. It reports panes_pages per book instead.
• ranking purely by uncertainty returns a queue that is mostly fallback pages (82% whole on that run) and mostly Marvel (93%). That is what the library is; the run record says so instead of engineering it away.

python -m harness.flag_batch                                  # dry run
python -m harness.flag_batch --run corrections-batch-20260829 --apply

It writes its record as results/<run>/flags.jsonl in flag_results.py's schema, so the revert below works against it with no changes.

Reverting a push

harness/revert_flags.py deletes exactly the rows a push created — and it is dry run unless --apply; nothing here reverts anything by accident.

# what would be deleted, counted, by tag
python -m harness.revert_flags --run pubdiversity-20260828

# actually delete it
python -m harness.revert_flags --run pubdiversity-20260828 --apply

# separately: ratchet tags whose review_flag is gone (the orphans above)
python -m harness.revert_flags --cleanup-orphans          # dry run
python -m harness.revert_flags --cleanup-orphans --apply

The set is identified by the run record, never by a time window. results/<run>/publisher-flags.jsonl recorded every push as it happened, including the exact tag strings and whether each row was inserted or was already there, so:

• a review_flags row is deleted only where the record says this run inserted it — a row the record calls already flagged predates the push and is left alone;
• a page_failure_tags row is deleted by its exact recorded (comic_id, page_num, tag) triple.

Reading the tags back out of the record rather than recomputing them is not belt-and-braces: common.publisher_key changed after that push (it keeps non-Latin letters now), so recomputing today would name rows that are not in the database. A created_at BETWEEN window would have been easier and wrong — it would take any flag a human raised in the same minutes.

pane_corrections are never touched. --skip-corrected (db transport only) additionally leaves alone any page that has been drawn on since the push, so a revert cannot erase the review trail of work that actually happened.

Layout

• harness/ — the sweep runner (book selection, three passes, scoring, flag push). Python; runs on butcher (2070S) and rhaegar (A3000).
• models/ — model registry: every VLM tried, its prompt/parse contract, and its measured record. Negative results stay recorded.
• results/ — append-only pass/push JSONL plus rebuilt selections, manifests, and derived artifacts.
• reports/ — human summaries per sweep; the running leaderboard.
• deploy/ — the systemd user units for the work that recurs on its own (today: nightly layout discovery). Nothing here is installed by checking the repo out; see "Nightly recurring-layout discovery".
• roadmap/ — the ratchet design (double CV↔ML and triple CV↔ML↔LLM loops) and what has been promoted downhill so far.

Running a sweep

# one command, end to end (selection -> CV -> ML -> LLM -> leaderboard -> dry-run flags)
/btrstore/andrew/ml/ratchet-venv/bin/python -m harness.run_sweep \
    --ollama-models ollama:qwen2.5vl:7b,ollama:minicpm-v,ollama:granite3.2-vision,ollama:moondream \
    --hf-models hf:florence2-large-region,hf:florence2-large-od \
    --sample-per-book 4 --build

# add --flag to actually push review flags (the only stage that writes anything)

Stages are independently re-runnable — harness.select_books, harness.cv_ml_pass, harness.llm_pass, harness.leaderboard, harness.flag_results — and --run <id> resumes into an existing sweep directory. Resuming appends pass results and push records; selection and derived artifacts are regenerated in place.

Exit codes: 0 ran, 2 a stage failed, 3 a prerequisite was missing, 4 no model produced a scoreable result, 5 scoring/reporting failed.

Where things run, and why

	butcher (RTX 2070 SUPER, 8 GB)	rhaegar (RTX A3000, 6 GB)
CV + ML passes	yes — the comic archives and OpenCV 5 live there	no OpenCV, no library
ollama VLMs	yes, short keep_alive so the studio engines get their VRAM back	—
transformers VLMs	—	yes, niced

The CV/ML pass is a Go binary (harness/cvpass/main.go, built as ratchetcv on butcher) that calls cv.DetectPage and processing.DecidePage — the exact seams a scan calls. It is driven from here by harness/cv_ml_pass.py; --build re-syncs a pinned kreeader-server worktree and compiles there, --build-local compiles the same worktree on rhaegar against the page cache.

The pin: which policy the CV pass actually is

"The CV pass IS the shipped policy" is a claim about one revision. The binary is compiled from a PINNED kreeader-server worktree, and harness/PIN.json records exactly which:

kreeader-server	6ad15e13f3d0f6df63bc89c73de7a51d22cec2ea
kreeader-cv	v0.3.2
PaneDetectorVersion	16
geometric rungs	6 (the sixth is keyline)
arm label	post-134

harness/selftest.py checks that file against the worktree's git state, against kreeader-cv's own source in the module cache, and against the built binaries themselves — go build stamps vcs.revision and the module graph into the executable, so the binary can be asked what it was compiled from rather than trusted. cv_ml_pass.py refuses to build from a worktree that is not at the pinned commit.

Numbers taken across a pin move are not comparable. The 2026-08-27 sweep ran arm pre-134 (kreeader-server 82e719c, kreeader-cv v0.2.0, PaneDetectorVersion 13, five rungs) — a policy whose low-confidence tail re-derived a layout instead of keeping the voted one, and which had no keyline rung. Its shipped rows and this pin's shipped rows measure two different detectors. Every run record names its arm; a leaderboard that does not is not evidence (kreeader-ratchet#5). The pre-134 binary is kept at /btrstore/andrew/ml/ratchetcv-v0.2.0-82e719c so that arm stays runnable.

Measuring a LADDER change needs revoted, not shipped

ratchetcv's shipped candidate feeds DecidePage the comic's STORED detector_profile. That is right when the ladder is fixed and wrong when the ladder MOVES: a new rung is never the stored vote, so every added rung would measure as a no-op by construction — while production re-detects the book on a PaneDetectorVersion bump and re-runs chooseProfile over its pages before deciding any of them.

harness/rungmeasure/main.go (folded in from the kreeader-server #23/#4 rung work) emits both candidates per page: shipped, and revoted — DecidePage with cv.ChooseProfile re-run over the comic. The re-vote runs over the comic's CORRECTED pages only, so it approximates production's vote rather than reproducing it; both arms of any comparison use the same approximation. Run it with python -m harness.cv_ml_pass --where local --tool rungmeasure.

correction_accuracy is a multiplier, not a sentinel (corrected 2026-08-29)

The job rows the CV/ML pass consumes carry the comic's correction accuracy, and the shipped policy applies it as conf = pageConfidence * (0.5 + 0.5*acc). There is therefore no out-of-band value to reach for: 0.0 is a real measurement ("the voted rung reproduces this book's corrections not at all"), and a marker value does not opt out of the scaling, it scales by something nobody meant.

The 2026-08-27 sweep wrote -1.0 as a "no evidence" sentinel, which scales by exactly zero — three of its fourteen books ran with every page's confidence at 0.0, missing the gate unconditionally and shipping the fallback (kreeader-ratchet#4). A book with no recorded accuracy is a book with no correction evidence, and kreeader-cv's value for that is NoCorrectionEvidence = 1.0, mirrored as common.NO_CORRECTION_EVIDENCE and checked against the library's own source by harness/selftest.py. Measured effect and the re-run that quantifies it: results/20260827T022212-e67b90/CORRECTIONS.md and results/issue4-recheck-20260829/.

Environment

• venvs: /btrstore/andrew/ml/ratchet-venv (rhaegar), /ssd/1TB/work/ratchet-venv (butcher)
• credentials: ~/.config/kreeader-ratchet/db.env (mode 600, outside the repo) — create it with python -m harness.common --bootstrap-creds. Optional api.env holds KREEADER_BEARER= for the API flag transport.
• No secret is ever logged, printed, or committed.

Nightly recurring-layout discovery

Layout discovery is ONGOING, not a one-off study (kreeader-ratchet#9). A systemd user timer runs one pass at 03:30 and pushes every recurring cluster to kreeader-server#184's batch verification queue, where a human confirms or rejects them without touching a single detected layout.

Why every cluster and not just the correction-anchored ones: a proposal adopts nothing. --apply-templates turns a candidate into a template the detector snaps real pages to, so it stays anchored-only and stays manual. --push-proposals only asks, so the unanchored recurring layouts — the ones no human has drawn yet — are exactly what the queue needs to see. The server upserts a pending proposal, remembers a rejection, and never re-asks a decided cluster, which is what makes the same push safe to repeat every night.

mkdir -p ~/.config/systemd/user
ln -sf "$PWD/deploy/kreeader-ratchet-discovery.service" ~/.config/systemd/user/
ln -sf "$PWD/deploy/kreeader-ratchet-discovery.timer"   ~/.config/systemd/user/
systemctl --user daemon-reload
systemctl --user enable --now kreeader-ratchet-discovery.timer

systemctl --user list-timers kreeader-ratchet-discovery.timer   # next elapse
systemctl --user start kreeader-ratchet-discovery.service       # run it now
journalctl --user -u kreeader-ratchet-discovery.service -n 100  # what it did

• Linger. A user timer only runs while the user has a session unless lingering is on: sudo loginctl enable-linger andrew (check with loginctl show-user andrew --property=Linger). Without it the 03:30 run happens only on nights someone happens to be logged in.
• Credentials. The unit is a user unit precisely so it reads ~/.config/kreeader-ratchet/db.env (read-only DB) and api.env (KREEADER_BEARER=). With no bearer the run prints what is missing and exits 3 — a red unit in the journal, never a quiet no-op.
• Interpreter. ExecStart uses /usr/bin/python3. To run it from a venv instead, systemctl --user edit kreeader-ratchet-discovery.service and override ExecStart= (blank line first) rather than editing the committed unit; the same drop-in is where --rate or --max-proposals belong if the nightly push ever needs pacing or a cap.
• Run ids. The timer never passes --run, so every night gets its own layout-discovery-<timestamp> snapshot under results/. Reusing an existing id is refused outright (exit 2) — the candidate set is one clustering pass, not an event log.
• Outcomes. results/<run>/layout-proposal-pushes.jsonl records one row per push: created, updated and already_decided are all success-equivalents, and a refusal keeps the server's whole reply body so a 400 that names its cause is never flattened to "HTTP 400". Each row (ratchet.layout.proposal_push.v3) keeps both sides: at the top level what discovery asked and measured, and under server what the reply said it filed — the proposal id, the scope key the server resolved for itself and the snapshot it took of the rep page. The push sends a REFERENCE, never boxes: scope, rep_comic_id, rep_page, the evidence counts and the run id, because the server snapshots the rep page itself. Exit codes: 0 ran, 2 refused its arguments/run id, 3 no bearer, 4 at least one push failed.
• Members. The push also names WHO recurs (kreeader-server#188): a ranked members array of {comic_id, page_num, similarity, disagreeing}, one entry per member page, because kreeader-ui#30's verification grid is a row of books with each book's matching pages beneath it rather than a count. The order is coverage-first — the representative's page, then the best page of every other book by descending similarity, then the remaining pages — and the server's 200-row cap is applied in that order, so a truncation always costs an extra page of a book already on screen before it costs a book. The full list is on the candidate row in layout-candidates.jsonl; the push record keeps members_sent beside member_pages, which is how a night where the cap bit is visible in the morning. Sending no members key at all means "keep the rows already stored", so an empty list is never sent.

Follow-up: registering the run as an external task

kreeader-server's docs/external-tasks.md seam would make a missed night visible (external_task_missed) instead of merely absent. It is not wired, and cannot be from this repo alone: the report endpoint (POST /api/admin/tasks/external/{name}/report) accepts only names that are declared in the server's own registry — an undeclared id is a 404 by design, so nothing here can register itself. It also needs an admin credential, not the layout-editor KREEADER_BEARER this harness holds.

The exact steps, in kreeader-server:

1. Add an entry to taskRegistry in api/healthtasks.go beside progress-report: id layout-discovery, External: true, CadenceSeconds: 24*60*60, GraceSeconds: 6*60*60 (a nightly job that slips past 03:30 for a reboot is not yet a problem; a morning with no run is), schedule: externalSchedule(24*60*60, "rhaegar systemd user timer"), status: externalStatus("layout-discovery", 24*60*60), run: func(string) error { return ErrTaskExternal }.
2. Document it in that repo's docs/external-tasks.md "Declared today" table.
3. Then, here: give the unit an admin token of its own (a second key in api.env, e.g. KREEADER_ADMIN_TOKEN=) and add an ExecStartPost= that POSTs {started_at, finished_at, result} — ok, or error when the run exited nonzero. Not before step 1: a report against an undeclared name is a 404, which would make the timer look broken while the run itself was fine.

What this repo will not do

• Write pane corrections. Humans draw ground truth. There is no code path here that can write one.
• Grade against confirms. Only independently drawn corrections score anything; detector-copied rows would grade the detector against itself.
• Ship a model. Nothing under models/ is a runtime dependency of kreeader, now or later.
• Mint itself a credential. The workbench endpoints are behind requireLayoutEditor; the harness uses a token if one is provided and a narrow direct write if not. It does not grant itself the scope.

Load discipline

Every page-image miss makes the server open an archive and extract a page. On 2026-08-27 a full-corpus fetch running alongside the detector starved butcher badly enough that it stopped completing SSH banner exchanges and stopped answering :3444, while the detector itself kept going. Misses are now throttled (RATCHET_FETCH_INTERVAL, default 0.25s); cache hits are free. Do not run a page-fetching pass and the CV/ML pass against the same box at the same time — run_sweep.py sequences them for this reason.

The tools

harness/select_books.py	pick the sweep's books, measure the axes, write the manifest, ground truth, job list and the stored baseline
harness/fetch_pages.py	fill the local page cache from the server, slowly and resumably
harness/PIN.json	the kreeader-server commit + kreeader-cv version the CV pass IS; selftest checks it against the worktree, the library and the built binaries
harness/cvpass/main.go	ratchetcv — the CV and ML passes, via the shipped detection seams
harness/rungmeasure/main.go	rungmeasure — ratchetcv plus the revoted candidate a LADDER change has to be measured by
harness/policy_tail_arms.py	build and run the four kreeader-server#134 policy arms from their own commits
harness/policy_tail_table.py	recompute the #134 four-arm table from tracked files alone — no library, no DB, no GPU
harness/cv_ml_pass.py	drive it, --where remote (on the library box) or --where local (against the cache)
harness/llm_pass.py	the VLM passes: ollama and transformers backends, one prompt contract, --skip-done
harness/scoring.py	the ONE metric every pass is graded by
harness/leaderboard.py	recompute every published number from results/. Ingestion is strict: a crash-truncated FINAL row is tolerated but printed in the report, interior corruption refuses to publish (exit 3), and headline eligibility counts pages a model SCORED, not pages it attempted
harness/logo_grading.py	read stored cv/ml logo proposals and their accepted-trace IoU; write per-target reports and machine leaderboard rows. Read-only DB stage; --input runs offline
harness/non_marvel_worklist.py	list the non-Marvel shelf, worst correction coverage first, and emit a flag-ready JSONL worklist plus human report. Read-only; it never raises flags
harness/layout_discovery.py	cluster page layouts within a series, then across a publisher, and propose the RECURRING ones as layout-template candidates — human corrections anchor a cluster and become its representative. Read-only DB stage; --input runs offline. Two transports, two claims: --apply-templates ADOPTS and so sends correction-anchored candidates only, --push-proposals ASKS and so sends every recurring cluster to #184's review queue — that one is what the nightly timer in deploy/ runs
harness/flag_results.py	push review flags for sweep disagreements (--max 40), or consume a generated non-Marvel worklist only with explicit --apply-worklist
harness/flag_publishers.py	push review flags at the NON-Marvel shelf, one publisher quota at a time — the corpus-diversity lever, not a sweep stage (--max 400, see the --max rule above)
harness/flag_batch.py	build a human's correction queue: books ranked worst-detected first, inserted best-first so created_at DESC puts the worst on top. Caps by BOOKS, not by flags — state the count when you run it
harness/revert_flags.py	delete exactly the flags/tags a push wrote, keyed on its run record; also --cleanup-orphans. Dry run unless --apply
harness/run_sweep.py	chain all of it, with explicit exit codes
harness/selftest.py	offline checks for the parts that can lie quietly
harness/verify_placement.py	prove two placements of the CV pass detect identically
harness/reparse.py	re-read stored raw responses under a newer parse contract
harness/manifest.py	record what a sweep did and what went wrong while it did it
harness/sweep_report.py	regenerate a report's numbers from the leaderboard