How this marketplace became what it is, told as the decisions that shaped it:
what forced each one, what was rejected, and a receipt you can check — then where proof
stands today, and what is planned next. 37 decisions shown of 45 recorded; the rest stay in the
data file with their cut reasons.
In progress. A learning journey in progress, and largely machine-written. This repository is eight weeks old and has one maintainer. The research documents, the adversarial critics and skeptics, the graders and readers in the evals, most of the commits, and the draft of this page were produced by language models the maintainer directed; the maintainer chose what to keep, and the page says so wherever a critic, judge or reader is named. Entries are added as decisions land and receipts are re-checked. A model-run fact-check of this page's own draft against its receipts found eleven claims the receipts contradicted; each correction was approved by the maintainer and is listed at the end of the page. Read this as a working record held to its receipts. It is not a finished retrospective, and the planned section near the end is what has not been built.
The through-line. One question, asked at escalating altitude: what does a green check actually show? Every era is one more turn of that question. Some turns were forced by a failure the previous answer recorded, some by a gap it left open. Underneath runs a second question the same discipline keeps answering: who is allowed to do what, and what shows it. That is where the protocol, the compiler and the routing boundary come from. The story runs in three acts: earning green, building the gate, the gate checked against its own work. Eras are threads of work; they are not calendar slots. Several overlap by a day, and the dates say so. It ends, for now, on the layers of checking that are planned but not yet built.
Act I — Earning green
A green check is worth nothing until it has been seen rejecting something: first a broken plugin, then a broken check, then a claim the product made about itself.
Checks before the product
July 9–18, 2026
The eval harness is as old as the plugins it guards; five more shipped under it in July. From birth, a green check has to be earned, and from its first full day, shown able to reject.
2026-07-10
The eval harness is as old as the repo
Forced by
Paid model calls in CI cannot gate every pull request unless cost is controlled by structure. Restraint is not enough, and a required check that gets skipped would deadlock a PR forever.
Decided
A cost-isolated pipeline: detect → run (a path-filtered matrix) → aggregate, where only the static aggregate job is ever a required check. The first plugin was committed on the evening of July 9; the three-tier harness was the third commit, thirty minutes later; the cost-isolated design that shaped CI was written the next day and merged as PR #5 on July 11. "Isolation is scoped to cost, not to tier."
Rejected
Making each paid leg a required check (skip = deadlock), or running paid tiers on every PR regardless of what changed.
The founding invariant: never delete before the backup is confirmed present
Forced by
The first plugin, graveyard, deletes GitHub repositories: the one mistake that cannot be fixed afterward.
Decided
The original is deleted only after its backup bundle is confirmed present. The model never deletes anything itself: it emits a guarded script the user reviews and runs. Every eval tier exists first to defend this clause.
A discipline that costs too much to run on every change will be skipped exactly when it matters.
Decided
cheap (offline, free, sub-second, before every commit) → behavioral (an LLM rubric checking that the prose steers a model, run when prose changes) → deep (a sandboxed cross-harness agent honoring the invariant end-to-end, for safety changes). Subject models are cheap; the grader is Sonnet, "so pass/fail stays trustworthy without paying Opus prices."
Rejected
One expensive test suite for everything, or trusting prose changes because "no script changed."
The counterfeit corpus: a gate must be seen rejecting something
Forced by
The corpus README's own rule: "A gate that never rejected anything would also pass every good plugin — so 'green on the real repo' says nothing about whether the gate actually discriminates."
Decided
Eleven deliberately broken fixtures on the harness's first full day (seventeen today), plus one all-valid baseline. Calibration runs first (the baseline must pass, or "every rejection below is meaningless"); each fixture must fail with the expected message, because "the substring check is what proves the right gate fired, not merely that something failed." One fixture is a structurally perfect plugin whose only defect is a weakened safety invariant.
Rejected
Trusting positive-path green. This is the repo's first shift from "we pass" to "we discriminate, for the right reason", the move every later era repeats at higher altitude.
What it left open The harness could now reject a broken plugin. Then thirty days of silence: no commits between July 18 and August 17. Nobody had yet asked whether it could reject a broken check, and a wave of new skills was about to ship with checks no one had shown could fail.
Fourteen skills, and the fake checks among them
August 17–24, 2026
Shipping fourteen skills took four days, nine of them in sixteen hours. Review then found that many of their checks were fake, and the checks were rebuilt so that could not recur.
2026-08-21
The discipline wave: working habits become checkable skills
Forced by
The maintainer's working discipline (interview before work, classify blast radius, verify before claiming, hypothesize before fixing) lived in habit and not in artifacts.
Decided
Nine skills in sixteen hours: grill-me, semver-gate, tracer-bullets, verify-before-claim, diagnosing-bugs, docs-hygiene, context-handoff, codebase-design, wayfinder. Each encodes one invariant a model can be held to, with a red-by-default eval from birth.
Rejected
One monolithic "best practices" skill: each invariant gets its own gate so each can be measured, and killed, alone.
The fake-check discovery: checks that can never fail
Forced by
Within a day of the wave, review found eval checks across six plugins that grep a load-bearing phrase which also appears in a heading or the check's own prose, so "the operative rule can be deleted or inverted and the check still passes." The repair pass itself introduced five new instances of the same defect, one inside the very group just rewritten to fix it: "this is what the pattern produces by default, not a one-off."
Decided
Checks assert exit codes on fixtures that exercise the behavior, never message strings that merely mention it. Every guard must be shown turning red when its rule is reverted.
Rejected
Treating a grep for the rule's wording as evidence the rule holds.
The behavioral tier: "it has been reporting green without ever running"
Forced by
The tier that checks prose steers a model had a wiring defect: it reported success while never executing. Forty workflow runs of green meant nothing.
Decided
Fix the wiring, and adopt the doctrine that a leg which did not run must say so out loud: skipped tiers now emit an explicit "green because it did not run, NOT because it passed" warning wherever skipping is possible. Its calibration cases were replaced in the same sweep wherever they were shown unable to fail.
Rejected
Quietly fixing the wiring without changing what green is allowed to claim.
With the checks finally biting, two plugins' value claims exceeded what measurement supported, and one skill was found partially redundant.
Decided
Narrow docs-hygiene and fleet-playbook-curator "to what measurement shows earns their slot," and surface verify-before-claim's redundancy finding instead of burying it. The evals are allowed to testify against the product they belong to.
Rejected
Treating a plugin's claimed value as fixed and the eval's job as confirming it.
The failures of the previous week were not local but general: irreversible actions without rehearsed undo, scope creep, rebuilt helpers, silent egress, endless fix loops.
Decided
Five agent-discipline plugins in one commit (prove-the-undo, scope-fence, find-before-build, egress-gate, stop-rule), each a rule the week had just demonstrated the need for, each with behavioral packs following.
Rejected
No alternative was recorded: the five rules were the week's own list, and the PR argues for them without arguing against anything.
What it left open With the checks made to bite, the discipline it enforced still lived in the maintainer's head and in twenty-four separate skills. Nothing composed them, and nothing could show a multi-step piece of work was actually finished.
Act II — Building the gate
The working discipline becomes a protocol that catches its own defects, gets names that carry its rules and research that settles its shape, and finally compiles.
Formalizing the gate
August 26, 2026
The working discipline became a protocol: designed adversarially, then built by running it on itself, which caught defects in its own construction.
2026-08-26
Red Gate is designed adversarially: 27 findings, five fatal
Forced by
The raw idea ("define verifiable criteria up front, then build") was about to become the protocol governing everything else. A flattering review would have been worthless.
Decided
A 12-agent design workflow: parallel surveys, three divergent candidate designs, two judges, and two adversarial critics whose 27 findings (five fatal, two found independently by both critics) are kept as a permanent fatal-flaw ledger, "because a future edit that quietly undoes one of them would reopen a known hole."
Rejected
Designing the protocol in one pass and trusting the designer; a flattering review; and fixing the critics' findings without recording them.
The red gate itself: a verifier must fail before the work begins
Forced by
The refinement table's verdict on the raw idea: "Criteria written but never executed are self-report. A model writes criteria it already believes it meets."
Decided
No build work starts until an executable verifier runs and rejects the current state on every checkable criterion. No criterion ever goes green except by that same pinned verifier, run independently of whoever did the work. Taste criteria that can't be command-checked get a capped, human-countersigned WITNESS verdict declared up front, never a downgrade after the fact.
Rejected
"BEGIN defines verifiable criteria", the raw idea's version, which the design's own refinement table calls too weak in ten recorded ways.
Exit code 127 is a FAIL: the greenfield deadlock, fixed twice
Forced by
"A crash is not red" deadlocked all new work: checking an unbuilt CLI exits 127, so the gate could never be shown failing legitimately. The fix was then reintroduced by review feedback four minutes after the protocol's birth commit, and had to be re-fixed.
Decided
"Not red" means only harness FAULT (exit 99, dirty preflight). A check command exiting non-zero (127 included) is a legitimate FAIL, which is exactly what red-from-birth requires.
Rejected
Requiring the check to run cleanly before its failure counts, which quietly forbids gating anything that doesn't exist yet.
Red shows a check fails now; mutation control shows it is coupled
Forced by
The fatal-flaw ledger's example: "grep -q RETRY src/client.go goes green when a comment containing RETRY is added." A check can be red today and still measure nothing.
Decided
A positive control at arming plus mutation control at judgment: revert the behavior and the check must turn red, restore it and the check must return green. A check still passing after its behavior is reverted "is WITNESS, not green." This is the fake-check lesson from era 2, promoted into the protocol itself.
Rejected
Accepting red-at-birth as sufficient evidence a criterion is real.
Graduated autonomy: round gates classified by blast radius
Forced by
An unconditional human stop at every round gate trains rubber-stamping; approval fatigue is, as the gap analysis put it two days later, "the one signal that decides whether the whole model works or trains rubber-stamping."
Decided
Round gates are classified PATCH / MINOR / MAJOR by semver-gate's blast-radius test (any single MAJOR property makes the whole gate MAJOR). PATCH auto-passes, MINOR auto-passes with a prominent flag and standing veto, MAJOR stops. Derived ratification keeps auto-ratification from becoming self-ratification: "the human approved these exact criteria once, at plan approval, at a higher altitude." An auto-pass that cannot cite its qualifying conditions is a protocol violation.
Rejected
The original unconditional human gate, superseded within hours of the protocol's birth.
The round-zero rule: gate the deciding, not just the building
Forced by
You cannot write behavioral criteria for an idea you have not decided how to build yet, so a naive red gate blocks exactly the work that needs structure most.
Decided
The protocol's round-zero rule: "Start at the first round whose criteria you can write without already knowing the answer. If you cannot write criteria for the work, write criteria for the artifact that will tell you what the work is." The first verifiable output of a vague idea is a decision brief, and the gate arms against that.
Rejected
Treating "I don't know what to do yet" as a blocker to the red gate, or exempting exploratory work from gates entirely.
The protocol, run on itself, catches its own defects
Forced by
Slice 2 (the independent judge) was built under the protocol it implements. Its verifier came back 6/6 green, and would have been accepted.
Decided
Mutation control demoted two of the six green criteria: "both stayed green when the drift gate was reverted — satisfied by the ordinary FAIL path, not by drift detection." The pinned contract was not edited; a fresh corrective round opened instead. There the gate refused to arm ("all 4 criteria green before any work … criteria that are already true are not a build contract"), and a further defect was caught en route: asserting on a drift message passes whether or not the gate acts, so assert the exit code on a passing fixture instead. The corrective round then closed unbuilt, with the finding that regression tests for already-correct behavior belong in the eval tier (green-expected), not in a Red Gate contract (red-first). That is the permanent boundary between the two check systems, and the coupled checks moved into the plugin's cheap tier.
Rejected
Accepting the 6/6 green run, which every process without mutation control would have done.
The corpus verdict: prose invariants should compile
Forced by
An 88-pattern survey of the field, written the same night as the protocol, found the same invariants Red Gate states as prose already moving into code elsewhere: hook handlers, declared tool classes, pinned constraint blocks.
Decided
The corpus's verdict: "Today Red Gate is prose a model is asked to honor; it should become a small set of declarations … that a compiler turns into harness hooks." Six patterns adopted now, each citing outside evidence: the case for pinning rests on a published compaction study in which violations rose to 59% when a constraint dropped out of context and stayed at 0% when pinned; this repo has not measured that itself. Ten patterns were deferred with named blockers, and ten rejected despite industry adoption because "adoption alone is not fit." The verdict became a shipped plugin three days later.
Rejected
Adopting patterns because the field did: mixture-of-agents, durable-execution runtimes, dynamic speaker selection and seven others are rejected by name, with reasons.
What it left open The protocol worked on itself, and the field survey written the same night had already delivered its verdict: prose a model is asked to honor is not enforcement. But before anything could compile, the vocabulary had to stop pointing the wrong way and the stage structure had to rest on more than instinct.
Names, prior art, and the gap analysis
August 28, 2026
Research paid down the design's debts: names that carry the rules, a settled answer on loop structure, and a gap analysis that turned the skepticism inward.
2026-08-28
Lexicon v2: names that make violations sound wrong
Forced by
The glossary's case: "BEGIN/MIDDLE/END say when, not what you must do — MIDDLE is the emptiest word in the protocol, naming the actual work as filler between two other things." Plus negative names (UNVERIFIABLE points away from the required behavior) and one collision (two unrelated "envelopes").
Decided
ARM / TRACE / JUDGE, SCOUT / WIDEN, WITNESS, MANDATE, FAULT. Four rules govern them: verbs over positions; names that make violations audible ("judging your own work," "unarmed dispatch"); say the required behavior, never its absence; one word, one meaning. Migration is staged and gated: prose now, docs sweep cheap-tier-gated, mechanical renames MAJOR and human-gated. Committed run records are never rewritten: "they are provenance."
Rejected
Living with the founding vocabulary, or renaming everything in one drive-by commit.
No Reflect phase: reflection lives between iterations
Forced by
The obvious next feature was a fourth per-round stage (Reflect). The prior-art note checked fourteen sources before adding it.
Decided
The prior-art note's finding: "In mature loop designs, reflection lives between iterations, not inside every one." The three-stage skeleton stays; reflection gets teeth as gate obligations instead: a mandatory one-line lesson in every gates.log entry, the next round's arming reads the prior ledger first, and consolidation is a gated obligation instead of an honor-system phase (Beck marks refactor "optionally"; Fowler calls skipping it TDD's most common failure).
Rejected
A per-round Reflect phase (duplicates stage 3 and the scout/plan round types); Kolb's cycle as precedent ("produces learning rather than verified work … no verification stage at all"); self-refinement configurations later shown unreliable.
The gap analysis's own diagnosis: "The owner builds superb defenses against the agent's dishonesty and almost none against the world's", and "the marketplace preaches measurement and closes zero loops on itself."
Decided
A 14-agent analysis with three skeptics who tried to refute every claimed gap: 39 candidates, 34 survived, ranked by leverage. The verification pass found a live command injection in the flagship irreversible-action safety path, and admitted its own flaw (worktrees branched from a stale base voided its top-level verdicts) instead of hiding it. All ten top-ranked gaps shipped within two days, each new guard shown red under mutation.
Rejected
Three gaps explicitly rejected with reasons and revisit conditions: an MCP distribution surface would be "the repo's first unguarded surface"; OTel telemetry is the wrong weight class; estimation calibration is noise at this sample size.
What it left open With the names settled and the loop shape defended, the corpus verdict from two nights earlier was the only debt left: the invariants were still prose a model was asked to honor. What was missing was enforcement: a protocol that compiles.
The protocol compiles
August 28–29, 2026
Prose invariants became compiled artifacts (deterministic, content-hashed, fail-closed), and the design was corrected three times by actually running it.
2026-08-28
Agents are compiled, not stored
Forced by
The deep-dive page's opening: "Agent personas are normally hand-written prose blobs: unversioned, unattributable, silently over-privileged, non-reproducible, impossible to diff. When one 'works,' nobody can say which sentence did the work, and when it grows a new power, nobody approved it."
Decided
agent-compiler: agents become build outputs of small, versioned registry modules. Same registry, query and compiler version ⇒ byte-identical image and hash; every emitted unit carries provenance; conflicts, missing dependencies and over-ceiling effects are compile errors and never warnings. The boundary rule: "Natural language may select behavior; it may not silently define it." No effect ceiling anywhere is itself a compile error: an unconstrained agent must be asked for explicitly.
Rejected
Hand-authored persona prose; also execution inside the plugin, cut and not deferred, because the harness is already the scheduler: "rebuilding retries/locks/approvals inside a plugin would duplicate the harness badly."
"Three decisions changed between this design and the build, each forced by actually running the thing." The flagship: the first image hash covered the whole-registry revision, and the metamorphic check immediately caught an unrelated module changing an existing image's hash, "a defect no amount of re-reading the design had surfaced."
Decided
The hash covers content-bearing fields only; the kernel moved out of the CI safety glob instead of widening it; one authoring format instead of a YAML dependency. A typo'd frontmatter key that compiled cleanly in the first demonstration became a fail-closed diagnostic (BAD_MODULE_KEY) with a pinned fixture, a demonstration miss converted into a compile error.
Rejected
Whole-image hashing; tolerant parsing ("anything outside the grammar is a compile error, never a guess"); deciding the CI-cost question silently instead of in the implementation PR.
ADR 0001: identities are views, preferences are domains
Forced by
New content (identities, imported taste) tempted new module kinds, and "every language addition is effectively permanent once modules are published against it."
Decided
Populate existing coordinates instead of adding new ones: an identity is a view module (selectors, traits, an effect ceiling, "never a prose persona blob"); a preference is a behavior scoped by domains. Accepted in a grill session, with the reopening condition recorded in the ADR itself: if should-strength rules read forced as the preference corpus grows, revisit a dedicated block kind.
Rejected
A first-class identity kind and a preference block, the handoff's own "premature ontology explosion" warning.
The repo publishes evidence against its own plugin
Forced by
The with/without gallery seed for agent-compiler was judged by an independent reader, who found that "on pure task-completion grounds this is a case where the ungated answer is arguably better, not just different."
Decided
Publish the verdict verbatim on the card, alongside the counterweight (the ungated answer has no capability boundary, "a real gap"). The gallery's rule is absolute: it shows truth, including where a skill's effect is modest or adverse, because "a demonstration with no misses is a sales pitch."
Rejected
Cherry-picking a friendlier scenario; scenario selection is deterministic precisely so it can't be.
What it left open Compiled or not, every check so far was legible only to the maintainer who wrote it. None of it could be followed by a reader who had not been there, the same maintainer a month later included.
Act III — The gate checked against its own work
Checks are turned outward into pages a reader can follow; the protocol overreaches and its scope is narrowed; and the statistical gate that scores every pack is caught excusing real failures, by the tier it was scoring. No outside user appears in this act: the reader it serves is anyone who was not in the room, the maintainer a month later included.
Publishing the checks
August 29–30, 2026
The checks became pages a reader can follow: a gallery that publishes weak results out loud, randomized scale gauntlets, and this timeline.
2026-08-29
The gallery: checks made readable, weak results out loud
Forced by
Green checks convince the maintainer who wrote them; a reader who did not needs to see what a skill actually changes, and a curated gallery is one cherry-pick away from advertising.
Decided
Every published pair is a real, provenanced model run captured from the eval tier, never hand-written; scenario selection is deterministic; refresh lands as a review-gated PR on a schedule, never a direct push; a plugin with no real pair "simply has no card — the gallery never fabricates one." Each card's judged divergence (stark, moderate or subtle) is published verbatim, and the subtle ones are named as the places to ask whether a skill earns its context.
Scale gauntlets: no hiding behind a single lucky fixture
Forced by
The gauntlet's own header: "The cheap tier proves each gate once, on one fixture … a defect that only shows up on a particular criteria count, a partially-green contract, or a particular mutation order cannot hide behind a single lucky fixture."
Decided
Randomized, seeded stress suites for both flagship plugins: 324 kernel invocations across seeded random registries for the compiler, and 25 randomized, isolated round lifecycles for the protocol, each exercising every gate, drift path and fault injection. Behavioral packs with negative controls came with them. The packs promptly earned their keep in reverse: a red leg exposed a defect in the rubric itself (it contradicted the protocol's own round-zero rule), which was fixed without softening the genuine failure the same leg had caught.
Rejected
Calling one green fixture enough; and softening a rubric to make a red leg pass.
The gate-outcome ledger: watching for rubber-stamping
Forced by
Graduated autonomy works only if approvals stay meaningful; approval fatigue is "the one signal that decides whether the whole model works or trains rubber-stamping."
Decided
Every gate entry now records its disposition (approved / flagged / vetoed) and lesson; a generated corpus index makes prior verifiers reusable, with the rule that a shape demoted under mutation control "must NOT be reused as proof."
Rejected
Treating human sign-off as self-evidently meaningful forever.
This timeline: the internal ledgers, turned outward
Forced by
The repo records its decisions in detail (refinement tables, fatal-flaw ledgers, gate lessons, reasoned rejections), but all of it inward, where a reader who was not there can't follow what happened or why.
Decided
Curate the existing ledgers into this page: each entry is a decision with the problem that forced it, the alternative it rejected, and a receipt you can check. It is generated deterministically from a committed data file, guarded by the cheap tier like every other published surface. Per the protocol doc's own words about itself, "this document is itself one turn of the loop."
Rejected
Auto-generating from git history (chronology without why); a narrative with claims a reader must take on faith.
What it left open Turning outward also meant deciding what the protocol was for. The PR that made it the default interactive workflow also made it the router for everything, and the maintainer's own review caught the overreach within the hour.
The routing boundary
August 29 – September 2, 2026
Once it was made the default router, the protocol overreached within a single PR. The correction drew the boundary that now governs routing: a specialist owns the domain procedure; Redgate is the verification envelope around how it is executed.
2026-08-30
Redgate is the envelope, not the router
Forced by
The PR that set out to make Redgate the default interactive workflow had first made it "the default router for planning, research, design, building, debugging, refactoring, review, deployment, multi-agent coordination, security/auth, external writes, and destructive actions": every domain pointed at one skill, with specialist procedures at risk of being displaced by gate ceremony.
Decided
Route to the most-specific applicable specialist skill or recipe for the work itself; compose Redgate around nontrivial work only when execution benefits from explicit falsifiable criteria, iterative verified rounds, or a classified human gate. The root AGENTS.md now reads: "A specialist owns the domain procedure; Redgate reinforces how that procedure is executed". Work fully handled by a specialist "does not acquire Redgate ceremony merely because it is nontrivial." The measurement-narrows rule, applied to the flagship: even the protocol that verifies everything else only keeps the job it earns.
Rejected
The universal-router framing the same PR had shipped an hour earlier (auto-triggering Redgate from task shape across every domain), corrected in flight instead of defended.
Research into a prior-art project (zpratt/lousy-agents) posed the architectural question directly: should automations be governed one semantic layer above harness configuration, with AGENTS.md, hooks, MCP servers and scheduled tasks as projections of a canonical model instead of the model itself? And with Redgate just demoted from universal router, something had to own the coordinating role it gave up.
Decided
Agent OS is, in the curation lens's words, "a design and control plane for agent automations" (taxonomy, recipes, triggers, adapters, portfolio curation) and "not another agent runtime." Redgate "is deliberately a different layer": an optional execution policy an Automation may recommend, never a dependency. The canonical ontology starts at seven concepts (Lane, Workstream, Automation, Trigger, Recipe, Adapter, Evidence); native files are "projections or referenced capabilities, never canonical taxonomy nodes merely because they are files." Cross-harness support is rated per capability from evidence: every cell starts unassessed, and a rating may not be inferred "from generic harness reputation, a prose instruction, or the presence of a file." The first release must be useful when it can only discover, classify, design, diagnose, reconcile and propose; mutation deepens adapter by adapter.
Rejected
Forking the prior art ("as prior art, not as a fork target") or copying its physical construct taxonomy; importing every harness construct into the canonical model; first-class Actor, Policy, Memory or Runtime nodes in v1 ("unless evidence forces promotion"); and one misleading overall support level per harness.
Under the old single-skill ROUTE line, a payment-webhook incident with explicit evidence demands had two defensible answers, diagnosing-bugs or redgate: "a coin-flip no threshold fixes." A negative control run against a validator stub that accepted the legacy form showed it: 17 failures, the stub accepting both.
Decided
A typed routing result (specialist | envelope | guards | interaction_owner) with a fail-closed validator enforcing eight rules, one of which is that redgate is never the specialist. A trajectory pack adds a STEP contract whose cross-field invariants encode the protocol (a MAJOR gate means proceed=no unless approved; silence or an adjacent approval is not consent). Three live-run passes then corrected the graders by data: the grader had been reading the model's reasoning trace, so a correct final line was rejected; legacy scenarios pin only the specialist; discipline skills grade active in either role.
Rejected
Exact-tuple grading on every legacy scenario (relaxed by measurement, owner-approved); role metadata in the roster; treating the terse twelve-repo retirement request's envelope=none as a grading defect; it is logged as a measured capability finding instead.
What it left open Drawing the boundary meant running the routing tier live, again and again. Those runs are where the statistical gate that scores every pack was caught excusing real failures.
The gate that excused failures
September 2–4, 2026
Caught while landing the routing tier: the statistical gate had been quietly excluding real failures as transport faults. Scoring them against the floor turned two packs red across three PRs at once, and the protocol's own pack caught its driver scheduling a landing to happen unattended.
2026-09-02
The statistical gate stops excusing real failures
Forced by
pass-rate.sh treated any row carrying an .error as a transport FAULT and excluded it from the floor, even when the row also carried a real assertion failure. Observed while the routing tier's live runs were being scored: a pack could fail its rubric and still report PASS: "exit 0 — fail-open." Green that was noise, on the tier whose whole job is to say whether prose steers a model.
Decided
FAULT only on genuine transport errors; a real assertion failure scores against the floor. Shown red first against the unfixed script on a fixture ("4/4 valid = 1.00, 1 FAULT excluded, PASS") and green after ("4/5 valid = 0.80, FAIL"); the fixture is now a cheap-tier self-test row so reverting the guard goes red offline. The immediate consequence was the point: two packs went red across three open PRs at once, and no single-plugin hardening could go green alone, so one integration PR carried the gate fix and both plugins' prose together.
Rejected
Calling the newly red legs flakes. Under the honest gate, a red leg is a verdict until shown otherwise, the same rule the repo applied to itself in era 2, now applied to the tier that grades everything else.
Caught by its own pack: landing and destruction are their own MAJOR gates
Forced by
With the gate scoring real failures, redgate's own behavioral pack scored 1/3 on its outage scenario. The grader's reasons, verbatim: the driver "commits to 'execute TRACE→JUDGE→land on main autonomously' and bakes table truncation into the automated criteria" (merging and truncating both scheduled to happen unattended once the user picks an option), and "never states or commits to proving check.sh FAILING on the current broken queue before the fix is built."
Decided
A blanket approval is an adjacent approval and never gate consent: never for landing, never for a destructive step, never something that fires after the human leaves. Landing on main and every irreversible action are their own MAJOR gates, confirmed at the moment they would happen; a single "ratify and I'll do the rest autonomously" option is a protocol violation. Red first is stated as its own line in every plan, with the observed failure quoted. A coded allow is permission to run a tool, never consent at a gate. Prose only; the pack went 3/3 on every scenario with the calibration control still passing.
Rejected
Bundling landing and cleanup into one ratification because the user said "don't ask me anything", the exact rubber-stamp shape the graduated-autonomy design set out to prevent, found in the protocol's own driver by the protocol's own verifier.
The next layers of checking get their rules before they get built
Forced by
Landing the typed routing tier took three live-run passes, and each pass taught a rule the hard way: a grader reading the model's reasoning trace rejected correct answers; a prompt rule stated as an example list ("walked through, interviewed, consulted") pulled rows to the wrong skill; a negative scenario that pinned every slot graded the label instead of the routing. The plan's own line: "They are cheaper to obey than to rediscover."
Decided
A phase-2 testing plan written against where the baseline actually is instead of the original sketch: four layers (decision-point probes, plan audit, trajectory runs with artifact audit, cross-plugin composition), each with what it proves, what it structurally cannot, and PR-sized increments. Seven rules bind every layer, each citing the run that taught it: grade the reply, not the reasoning trace; define by property, never by example list; a negative scenario negates one slot; discipline skills have no stable slot yet; roster prose alone does not carry the envelope; selection is not injection; prove every new gate red first. Two of the rules become cheap-tier standing orders in the PR that first needs them.
Rejected
Stated as prohibitions on every layer: no grading a model's chain of thought; no "flake" category, "FAULT is a transport error; everything else is a verdict"; no must-fire scenario without its must-not-fire twin, because over-activation is the signature failure of an envelope skill and "the only way to see it is to give it room to happen"; and no layer promoted to a required check on the PR that creates it.
What it left open The plan named the objection it could not yet answer: every behavioral verdict in this repository rested on one cheap subject model and one model grader. An outside reader would say the packs measure that pair, not the skill. The next days were spent giving that objection a number.
One subject, one grader
September 5–6, 2026
A subject-model matrix found, on its first run, that the calibration controls are model-specific: a bait tuned so the baseline's bare behaviour fails to gate does not transfer to a model that gates unaided, so a with-skill green on that model measures nothing. A blind labelling sheet was drawn so a human can grade the grader, and its own pull request caught a rubric that graded staffing instead of the skill's rule.
2026-09-06
Calibration controls are model-specific
Forced by
Every pack ran one cheap subject on OpenRouter and one grader, so every behavioral verdict was a statement about that pair. The adversarial read of this page put it first: two of three repeats on one model is a majority, not a measurement, and nothing showed the prose steered any other model. #102 turned the objection into four measurements in leverage order, the first of them a matrix of subjects.
Decided
A manual, advisory subject-model matrix: the same packs, two more subjects, per-provider scoring in which only the baseline decides and every other provider is tagged advisory. The promotion rule was corrected before the first run: a subject is promotable on a pack only if the pack's calibration control still passes under it, because a control that fails means the bare model already does what the skill asks and the with-skill green measures nothing. First run, 34002674057: wayfinder's controls held under both new subjects; semver-gate's two controls each failed 0/3 under a different subject, bare Haiku already stopping to ask for sign-off, bare Llama already refusing to route around a structural block. Neither is promotable on semver-gate as the pack stands.
Rejected
Promoting a subject on its pass rate alone, which would have declared both subjects green on semver-gate on scenarios where the skill was doing no work. Making the matrix a required check: it is manual dispatch, never scheduled, and its cost is stated per run. Reading a 0/3 at n=3 as more than three samples: the comment that reports the numbers says per-cell variance is real and must travel with every figure.
A human grades the grader, blind, against the rubric that graded
Forced by
Measurement 2 of #102: how often does the model grader agree with a human reading the same rubric? The results artifacts live on a host this session cannot reach, and a sheet drawn by hand would carry the drawer's knowledge of the verdicts. The sampling had to happen where the artifact is, and the verdicts had to be present for the agreement script yet impossible to read by accident while labelling.
Decided
A manual calibration-sheet workflow: given a run id, it downloads the results artifact on the runner, draws a seeded blind sheet of twenty rows per pack (scenario, request, output, empty label; no verdict, no provider; each row keyed by a hash of scenario and output), seals the grader's verdicts as base64 beside it, and commits both to a branch based on the commit that produced the run, so the rubric next to the sheet is the one that graded those verdicts while the sampler comes from the dispatch ref. agreement.py joins the two label sets on the hash and reports percent agreement, Cohen's kappa, the confusion matrix, and the disagreements. The first sheets, from run 34002674057, are on main with every label blank.
Rejected
Committing decoded verdicts and relying on discipline not to look, which a review bot proposed as the simpler option. Basing the branch on the dispatch ref, which a second review bot caught as a P1: an older run's rubric could differ from the one beside the sheet and silently corrupt the kappa. Reading anything into the wayfinder sheet's kappa before it exists: that run was all-green, so expected agreement is high whatever the human does, and the semver-gate sheet, with six real failures, is the one that carries information.
A rubric that graded staffing, not the rule, is regraded on the record
Forced by
The sheet pull request's own behavioral tier went red on wayfinder: the "basically merging any minute" scenario passed one of three. Both failing outputs computed the frontier correctly, excluded WAYF-021 because its dependency was still open, then put one engineer on the unstarted ticket and declined to add a second to the one whose pull request was already up. The grader failed them against a PASS clause that said to name both tickets as dispatchable now.
Decided
Classified as a grading defect under the plan's regrade discipline and recorded with the run number in the pack header and the pull request body: the skill's rule is frontier-only dispatch, and whether to double-staff a ticket someone is already working is not something it grades. The clause now grades the frontier set and the WAYF-021 exclusion and says in words that declining to double-staff still passes; the hard-failure clause, any dispatch of WAYF-021, is unchanged. The change is monotone, so verdicts already sealed in the sheet keep their meaning. A reviewer caught the first rewording still saying "eligible for dispatch now" and it was reworded again before merge.
Rejected
Re-running the job as a flake: the outputs were real and the plan says "FAULT is a transport error; everything else is a verdict." Skipping or loosening the scenario without a classification. Leaving the sheet pull request red until the rubric drifted back: the regrade was merged to main and ported into the sheet branch so the same job re-ran against it.
What it left open The sheets are on main with every label blank. The number that answers "AI grades AI" is the one a human has not yet written, and the layers that would check a plan, a trajectory, and a composition are still planned. That is the horizon below.
Where the checks stand today
Where the story has arrived: the ladder of checks as it runs today, cheapest first, each rung catching what the cheaper rungs structurally cannot. "Proves" on this ladder means one thing: a check that has been seen failing on the case it exists to catch. Nothing here is proof in any stronger sense. Every machine rung is documented, drift-guarded, and says out loud when it did not run; the last two are human gates and say so. Each rung names the decision on this page that created it.
cheap — deterministic, offline, before every commit
required
Proves
Scripts parse, manifests are valid, wiring holds both ways, load-bearing sentences survive (a presence grep is a tripwire that the sentence still exists, each one shown red when its sentence is deleted; it is never mistaken for evidence the sentence works, which is the behavioral rung's job), executed script gates still bite, published surfaces are in sync with their data, and the testing doc's own inventory has not drifted.
Cannot prove
Whether any sentence still means anything to a model. It greps and parses; it never runs one.
counterfeit — the cheap gate must be seen rejecting
required
Proves
A corpus of deliberately broken plugins is rejected for the right reason, after the untouched baseline is proven green — including one that is structurally perfect and only weakens the safety invariant.
Cannot prove
Anything about a gate the corpus has no fixture for.
grader model — the judge must exist before a verdict counts
required
Proves
The grader slug in every behavioral pack resolves to a real, reachable model before any paid leg is judged, so a run can never be scored by a nonexistent judge.
Cannot prove
Anything about the subject model or the rubric: only that the judge answers the phone.
One cheap subject model given the skill behaves as the skill demands, judged by a rubric with a stub-skill negative control, under the statistical spine, with a no-tools clause so a harness artifact is never mistaken for a skill failure.
Cannot prove
Multi-round protocol behavior, composition between plugins, or anything a judge can be fooled about. One skill, one turn, one subject model: nothing about the models that will actually run these skills.
routing — the roster picks a composition, not a name
advisory
Proves
With every skill description in context, requests route to a typed specialist | envelope | guards | interaction_owner line validated by eight fail-closed rules; a trajectory leg grades redgate's next move at frozen decision points.
Cannot prove
That the routed-to skill then does anything right in a live run.
Nothing, by design: it warns when a PR touches more than one plugin's paid surface, because a red paid leg on such a PR is ambiguous and a rerun re-bills every touched plugin.
Cannot prove
Block anything. It always exits 0 and is deliberately kept out of the required set, so a legitimate multi-plugin change is never hard-blocked by a nudge.
The deterministic machinery holds across seeded random registries (324 kernel invocations by default) and randomized round lifecycles (25 isolated runs by default), so a defect cannot hide behind one lucky fixture.
deep — a real agent, in a sandbox, across harnesses
required (aggregate)
Proves
The safety invariant survives an end-to-end run by a real coding agent, with an oracle/nop calibration floor so a broken verifier cannot read green. Two plugins carry a pack today: graveyard and fleet-playbook-curator.
Cannot prove
Anything about non-safety skills, and nothing at all when its switch is off, which the required check announces instead of hiding.
published surfaces — the gallery and this timeline
review-gated
Proves
Every card is a real, provenanced model run and every timeline entry carries receipts; both pages are generated from committed data and re-verified at publish time.
Cannot prove
That a captured pair is representative, or that a receipt supports its claim: a human reviews both.
subject-model matrix — the same packs on other subjects, advisory
manual
Proves
Whether a pack's calibration controls still hold under a subject other than the baseline, which is the only condition under which that subject's with-skill green means anything. Seen failing: semver-gate's two controls each went 0/3 under a different subject on the first run.
Cannot prove
Decide a merge, or claim more than n=3 per cell. Only the baseline provider decides; every other provider is scored and tagged advisory.
calibration sheet — a human grades the grader, blind
human
Proves
Nothing until a human fills the labels; then, percent agreement and Cohen's kappa between the human and the model grader on the same rows, with the sample size stated. Draws from a run's own artifact and bases the sheet on the commit that produced it.
Cannot prove
Make a human's labels correct, or narrow a kappa at n=20. Never a required check; never calls a model.
The statistical spine. Every model-driven rung shares one statistical spine: repeated trials, a k-of-N pass-rate floor over valid samples only, transport faults separated from verdicts and never scored, and fail-closed starvation, so a scenario with too few valid samples is "never tested" and never green. The numbers are small and stated here so no rung claims more: three repeats per scenario at a 0.6 floor (five at 0.8 for routing), one subject model (nvidia/nemotron-3-ultra on OpenRouter) and one grader (claude-sonnet-5) across every pack. Two of three is a majority, not a measurement. Widening the subject models exists as a manual, advisory matrix that has run once, and the human half of the grading loop exists as blind sheets whose labels are still blank (#102).
Planned, not built
The next turns of the question, planned and receipted but not yet built. Each is stated as what it would show. Per the plan's own rules, none becomes a required check on the PR that creates it.
not built · in progress · labels pending
#102 — more than one subject model, and a human in the grading loop
How often does the model grader agree with a human reading the same rubric, and does a second grader agree with the first?
The plan
What has landed is in the story above: the matrix ran once and found the controls model-specific, and blind sheets of twenty rows per pack sit on main with every label blank. What remains is the human half of measurement 2, an hour of labelling and one run of agreement.py per pack, then measurement 3, grader self-consistency and a cross-family second grader to bound what two of three can mean, and measurement 4, real-agent runs for the protocol under the deep tier. Until the labels exist, the grader's agreement with a human is unmeasured, and the wayfinder sheet will carry little information even then because its run was all-green.
Depends on
an hour of labelling; a second grader key for measurement 3
At a frozen mid-run state, does the model's next move obey the protocol, and does it stay quiet when it should?
The plan
Six probe families the trajectory pack lacks (gate classification under mandate, coded allow is not consent, T0 pass-through, re-pin refusal, JUDGE independence, surface-specific probes), every must-fire with a must-not-fire twin, because over-activation is the signature failure of an envelope skill. Graveyard becomes the second stateful skill under the same step contract.
Does the model's plan apply its dependencies before any work, and how much of the envelope does the roster prose carry on its own?
The plan
A typed PLAN header validated by a contract in the route-contract mould, then an anchored rubric with a stub-skill control. The corpus is the routing tier's composition scenarios plus their terse twins: the same problem stated without the evidence words that made the composition promptable. The twins' pass rate is the measurement the strategy asked for. The Agent OS row waits, as a marked placeholder, on #85.
Does a real agent leave the run directory in the state the protocol requires, against evidence it cannot forge?
The plan
The reward is a deterministic post-hoc audit, no judge in the loop, cross-checked against provenance the agent cannot write: the gate-responder's own log, a root-owned shim recording every verifier execution, and the sandbox's transcript. The fixture set includes the forgery case, a byte-perfect run directory with no matching shim log, and the audit must go red on it. N ≥ 3 trials per scenario on the same pass-rate arbiter.
Depends on
its audit script and fixtures, landed red-first in the cheap tier
Installed together, as a user would install them, do redgate, a specialist, and Agent OS behave at the boundary each one claims?
The plan
A real installation inside the sandbox (install-smoke shows a plugin is installable, never that it is installed and discoverable), with discoverability as the first assertion, then one task per boundary claim under the L3 audit. Dollars per run; manual dispatch and release tags only.
Depends on
L3's audit, #85's recorded boundary, #84's taxonomy
Which trigger classes are first-class (implied, user, dependency, state gate, always-on, offer-only), and can a skill's description become a compact routing index compiled from one canonical declaration?
The plan
Descriptions currently carry six different control-flow meanings in prose; manual compression does not scale across the fleet. The design targets routing quality per token, generated harness-specific descriptions from one source, and CI checks against description bloat and drift. When it lands, the routing tier's discipline rows return to exact-slot grading in the same PR.
What is the smallest Agent OS context that causes a repeatable improvement in automation design without adding semantic failures or unnecessary context?
The plan
Baseline, taxonomy, recipe-aware and full-Agent-OS contexts compared on the same six scenarios with blind judgments and a per-call cost ledger, under a hard $0.05 cap and deterministic cost preflight. The design PR stays unmerged until the implementation boundary is recorded from evidence.
Can a language world model be calibrated into a test double (never an oracle) that exposes an agent to hundreds of stateful, adversarial environment variants cheaply enough to matter?
The plan
Shadow and advisory only: simulator output can never block or approve a merge. Paired action-tape calibration against a deterministic fixture, six perturbation families each with a clean twin, specialist-only versus specialist-plus-envelope on the same seeds, and a graduation bar stated up front (at least 95% schema-valid observations, 90% outcome-class agreement, zero false-green MAJOR transitions), ending in a measured go, narrow, or stop. "A negative result is valid." A reproducible local endpoint is a separate prerequisite, so hardware setup cannot swallow the first tracer bullet.
Depends on
#100 for the calibration phase; Phase A is buildable offline
not built · planned · enforced in the PR that first needs them
Standing orders the plan adds
Can the rules learned from landing the routing tier be made to bite mechanically, so they are obeyed instead of rediscovered?
The plan
A reasoning-trace guard (every pack grading a typed line declares that it grades the reply, not the thinking), a planned-tier consistency check (a layer is never both planned and live), a regrade discipline (every relabel records whether it fixed a label defect, a grading defect, or measured a capability gap), and findings recorded on the tracking issue instead of in a PR that will be squashed.
A model-run fact-check of this page's draft against its own receipts, approved line by line by the maintainer. Eleven claims the receipts contradicted, with what the page said, what the record says, and the receipt that decided it. Eight smaller wording fixes ride in the same commit. The corrections landed in one commit.