Phase 5: validate the v4 phase-runner template on the live runs #11

New issue

Open

opened 2026-08-11 14:56:19 +00:00 by jbr870 · 1 comment

jbr870 commented

2026-08-11 14:56:19 +00:00

Owner

What this is

Validation of work that has already landed, not a change to it. Produces a run report, not a diff.

The v4 run template (decision 9: the driver dispatches each autonomous step to a phase-runner subagent
and takes back only a fixed return contract) has never been exercised end to end. The verity features
currently being driven on it are this validation — no separate run is scheduled, because a real run
under real conditions is exactly what was asked for and manufacturing a second one would be cheaper
evidence.

Nothing is asked of a run while it happens. Every item below is recoverable from the transcript
afterwards, so the observation costs nothing and cannot bias the run.

The measurements (v3 baselines from six measured runs, 2026-08-05 → 08-07)

Metric	v3 baseline	v4 passes if	v4 has failed if
Peak main-thread context	487–840k (49–84% of window)	stays near the 52–76k session baseline, low six figures at worst	still climbs monotonically with phases
Context at `/integrate` entry	590–730k	≈ the same as at `/technical-plan` entry	it tracks the v3 curve
Main-thread resend	208–620M tok	low tens of millions	over ~100M
Compactions	0–4	0	any

The strongest signal is the shape, not the peak. v3's curve was monotone and phase-shaped in every
one of six runs. Flat driver context across phases ⇒ the model works. Still stepping up at each phase
boundary ⇒ the driver is absorbing what it was told not to.

The six questions the numbers cannot answer

Did the driver leak? Count its Read calls on phase artifacts, reports, source files, procedure
docs during steps 3–7 (discipline says zero; targeted helper one-liners excepted). A leak that was
necessary is a defect in the return contract, not in the driver's obedience.
Was the return contract enough? Look for the driver asking a runner for more, or re-deriving from
the forge what the contract should have carried. ≤30 lines was a guess.
How did runners die, and what did re-dispatch cost? Dead/garbage runners, whether the one
permitted re-dispatch recovered cleanly, whether anything degraded to inline execution.
Did the QA loop survive as a single runner? Longest phase, most likely to need re-dispatch per
round. If one runner cannot hold it, that is a design amendment, not a bug.
Is run-notes.md doing its job? Did later phases rediscover operational facts an earlier phase
learned (ports, credentials, quirks)?
Did wu-status.md hold as a resume checkpoint through a session-limit kill or a dead /develop
runner — the case it was written for.

What a negative result buys

If ordering needs enforcing — steps skipped, gates bypassed, the sweep quietly dropped — that is
Fork B's stated promotion criterion and a /pipeline conductor gets built against something observed
rather than guessed. A v4 that merely underperforms its arithmetic is not that; it is a tuning
question about the contract's width.

Method

Per-request input_tokens + cache_read_input_tokens + cache_creation_input_tokens on main-chain
assistant messages; phase boundaries located by Skill invocations; sidechains excluded. Write the
result up in sdlc-dogfood-run-reports/ in the existing format, with the table above filled in.

Source: sdlc-context-economy-analysis.md §5 + the plan's Phase 5.

## What this is Validation of work that has **already landed**, not a change to it. Produces a run report, not a diff. The v4 run template (decision 9: the driver dispatches each autonomous step to a phase-runner subagent and takes back only a fixed return contract) has never been exercised end to end. The verity features currently being driven on it *are* this validation — no separate run is scheduled, because a real run under real conditions is exactly what was asked for and manufacturing a second one would be cheaper evidence. **Nothing is asked of a run while it happens.** Every item below is recoverable from the transcript afterwards, so the observation costs nothing and cannot bias the run. ## The measurements (v3 baselines from six measured runs, 2026-08-05 → 08-07) | Metric | v3 baseline | v4 passes if | v4 has failed if | |---|---|---|---| | Peak main-thread context | 487–840k (49–84% of window) | stays near the 52–76k session baseline, low six figures at worst | still climbs monotonically with phases | | Context at `/integrate` entry | 590–730k | ≈ the same as at `/technical-plan` entry | it tracks the v3 curve | | Main-thread resend | 208–620M tok | low tens of millions | over ~100M | | Compactions | 0–4 | 0 | any | **The strongest signal is the shape, not the peak.** v3's curve was monotone and phase-shaped in every one of six runs. Flat driver context across phases ⇒ the model works. Still stepping up at each phase boundary ⇒ the driver is absorbing what it was told not to. ## The six questions the numbers cannot answer 1. **Did the driver leak?** Count its `Read` calls on phase artifacts, reports, source files, procedure docs during steps 3–7 (discipline says zero; targeted helper one-liners excepted). A leak that was *necessary* is a defect in the return contract, not in the driver's obedience. 2. **Was the return contract enough?** Look for the driver asking a runner for more, or re-deriving from the forge what the contract should have carried. ≤30 lines was a guess. 3. **How did runners die, and what did re-dispatch cost?** Dead/garbage runners, whether the one permitted re-dispatch recovered cleanly, whether anything degraded to inline execution. 4. **Did the QA loop survive as a single runner?** Longest phase, most likely to need re-dispatch per round. If one runner cannot hold it, that is a design amendment, not a bug. 5. **Is `run-notes.md` doing its job?** Did later phases rediscover operational facts an earlier phase learned (ports, credentials, quirks)? 6. **Did `wu-status.md` hold as a resume checkpoint** through a session-limit kill or a dead `/develop` runner — the case it was written for. ## What a negative result buys If **ordering** needs enforcing — steps skipped, gates bypassed, the sweep quietly dropped — that is Fork B's stated promotion criterion and a `/pipeline` conductor gets built against something observed rather than guessed. A v4 that merely **underperforms its arithmetic** is not that; it is a tuning question about the contract's width. ## Method Per-request `input_tokens + cache_read_input_tokens + cache_creation_input_tokens` on main-chain assistant messages; phase boundaries located by `Skill` invocations; sidechains excluded. Write the result up in `sdlc-dogfood-run-reports/` in the existing format, with the table above filled in. *Source: `sdlc-context-economy-analysis.md` §5 + the plan's Phase 5.*

jbr870 commented

2026-08-13 11:46:38 +00:00

Author

Owner

Two live-validation datapoints from the verity desktop, 2026-08-13, reported first-hand by the verity-release session mid-/develop on verity #230 (i.e. against the CURRENT skill text):

The subagent completion-report failure recurred post-fix. An Implementer went idle without sending its report — commit and journal on disk, work genuinely done, report never arrived; the lead had to detect completion by inspecting the tree. Reporter's words: "a recurring shape here, not a one-off." The develop/SKILL.md report-delivery rule did not prevent it, so the v4 validation this issue tracks should treat report delivery as an open question, not a regression test of a fixed bug.
Agent-name collisions across concurrent sessions. From the verity #206/#207-era learnings (learning:v1 on the forge): spawning a name another session already holds silently renames your agent, and messages addressed to the original name reach the stranger — observed twice in one run, once nearly recording "the expert panel could not run" as fact off a stranger's plausible reply. Suggests the template needs session-unique name salting.

Evidence added by /meta-retrospective 2026-08-13 (second pass — verity drain).

Two live-validation datapoints from the verity desktop, 2026-08-13, reported first-hand by the verity-release session mid-`/develop` on verity #230 (i.e. against the CURRENT skill text): 1. **The subagent completion-report failure recurred post-fix.** An Implementer went idle without sending its report — commit and journal on disk, work genuinely done, report never arrived; the lead had to detect completion by inspecting the tree. Reporter's words: "a recurring shape here, not a one-off." The develop/SKILL.md report-delivery rule did not prevent it, so the v4 validation this issue tracks should treat report delivery as an open question, not a regression test of a fixed bug. 2. **Agent-name collisions across concurrent sessions.** From the verity #206/#207-era learnings (learning:v1 on the forge): spawning a name another session already holds silently renames your agent, and messages addressed to the original name reach the stranger — observed twice in one run, once nearly recording "the expert panel could not run" as fact off a stranger's plausible reply. Suggests the template needs session-unique name salting. *Evidence added by /meta-retrospective 2026-08-13 (second pass — verity drain).*

jbr870 referenced this issue from a commit

2026-08-13 12:06:38 +00:00

meta-retrospective: second pass 2026-08-13 — verity drain (8 learnings watchlisted, gate validated live)

jbr870 referenced this issue

2026-08-14 13:01:43 +00:00

Feature-230 time study: 13 ranked pipeline-efficiency changes #37

jbr870 referenced this issue

2026-08-14 13:01:50 +00:00

wait-discipline: an idle signal is not completion — name the tree-state check #44

jbr870 referenced this issue

2026-08-25 09:33:41 +00:00

wait-discipline: an idle signal is not completion — name the tree-state check #44