wait-discipline: no retry/resume protocol for transient API errors or quota stalls #46
Labels
No labels
component:adapters
component:lifecycle
component:qa
component:setup
component:shared
component:worktrees
phase/accepted
phase/backlog
phase/deployed
phase/developing
phase/integrating
phase/planning
phase/qa
phase/requirements
phase/uat
priority:critical
priority:critical
priority:high
priority:high
priority:low
priority:low
priority:medium
priority:medium
type:bug
type:chore
type:docs
type:feature
type:infra
type:tech-debt
No milestone
No project
No assignees
1 participant
Notifications
Due date
No due date set.
Dependencies
No dependencies set.
Reference
jbr870/devwork-skills#46
Loading…
Add table
Add a link
Reference in a new issue
No description provided.
Delete branch "%!s()"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
PREQ: Run supervisor — out-of-run resume and liveness watch for the unattended driver
Created: 2026-08-25
Problem
An operator running an unattended pipeline run loses the run whenever the driver session's
own turn is ended by a self-clearing condition. Nothing inside the session can retry a call the
session never got to make — by the time the error exists, there is no turn left to run the retry
in. The run then sits dead until a human happens to look.
Two recorded incidents, one shape:
/dev:integratehitAPI Error: 529 Overloadedand stopped; 8m 47sof dead time until the operator typed "Try again" — recovered only because a human was watching.
reset because nothing was scheduled against it (verity slot5-164 / slot2-167).
Users:
survive self-clearing stalls without them.
from the run's durable record rather than indistinguishable from slowness.
Current state: wait-discipline §3a covers every call made inside a live turn, and
phase-runner §5 re-dispatches a runner that dies. A dead driver turn is a documented
limitation: "a stop that costs whatever the operator's absence costs."
Proposed Solution
A supervisor that lives outside the driver session and watches an unattended run. Supervision
is opt-in at run launch — the operator engages the supervisor when starting an unattended
run; it is never inferred onto an attended session.
The resumability rule (definitional, settled by review). A run is resumable when there is
positive evidence that no turn is in flight and the last turn ended on a self-clearing
cause — a transient error or a quota/rate-limit kill. An idle session whose turn died on a 529
is resumable (that is the founding incident); a session that may still be mid-turn is not, and an
operator-initiated stop is never a self-clearing cause and is never auto-resumed. "Dead"
(session gone) and "idle turn-end" both resume; "possibly mid-work" only alerts.
transient/overload errors — classified by the rule below, not an open-ended family): resume
with bounded, backoff-spaced attempts; when the bound is exhausted, escalate naming the
attempt count. The bound is per incident — a later, distinct incident gets a fresh bound.
resume against it — never a guessed interval. A reset time already in the past resumes
immediately. Classification rule: a stated reset/retry-after accompanies the error →
scheduled path; otherwise → bounded-retry path (this settles 429's double role).
alert-only. The supervisor surfaces the stall and never writes into the session.
normally, and at most one supervisor watches a given run (a second attach is refused).
mutations before trusting the tree (wait-discipline rule 5), and when the project declares a
post-resume claims-reconstruction step, the resume path invokes it. The semantics of
reclaiming open claims are #65's, defined once #43's ledger lands — this feature only provides
the hook.
forge record (the project's declared run-traceability surface); when the project declares a
notification channel, escalations are additionally delivered through it.
threshold ship as stated suite defaults which a project may override in a declared
configuration block; the values in force are the ones the acceptance criteria are judged
against.
Scope: Full (supervisor + liveness watch, alert-only on ambiguity) — chosen over
quota-only-minimal, over notify-only (challenged by review; rejected because an unattended
overnight run has nobody reachable to receive the notification — the quota incident lost 3h 17m
for exactly that reason), and over absorb-#65 (reclaim semantics depend on #43's in-flight
ledger).
User Stories
automatically with bounded retries, so that an unattended run survives a 529 without me
watching.
time is lost past a reset the platform already told us about.
nudged, so that wedged runs become visible without risking duplicate work from double-driving.
a stall and its handling are auditable and distinguishable from mere slowness.
Acceptance Criteria
classification rule), when the supervisor detects the run is resumable, then the run is
resumed without operator input, with attempts bounded and backoff-spaced per the declared
(or default) values.
attempt fails, then the supervisor stops retrying and escalates, naming the attempt count
and the error.
detects it, then the reset time is recorded and the resume is scheduled against that time
(immediately, if the reset is already past), and the run resumes without operator input.
it follows the bounded-retry path per the classification rule, escalating on exhaustion
with what it observed.
when the declared (or default) stall threshold passes, then the supervisor escalates and
does not resume, nudge, or otherwise write into the session.
evidence of no turn in flight, a self-clearing cause, and not an operator-initiated stop.
supervisor stands down without intervening, and this is visible in the run's record.
attach, then the attach is refused — at most one supervisor per run.
harness surface is unavailable or refuses), when it detects this, then it escalates with
what it observed rather than acting on ambiguity.
supervisor resumes a run, then that step is invoked before phase work continues (the
step's own semantics are #65, out of scope here).
resume, a stall alert, a stand-down), when it happens, then it is written to the run's
durable forge record — and additionally delivered via the project's declared notification
channel, when one is declared.
Out of Scope
the post-resume hook, not absorbed; depends on #43's ledger).
decision; revisit only with evidence that alert-only strands wedged runs.
across a scheduled quota resume) is not itself supervised. The recursion stops here,
deliberately: the supervisor is the safety net, not a guaranteed one. Its absence is visible
in the run record (interventions stop appearing).
opt-in at unattended launch only.
2ceacfe).Dependencies
2ceacfe): this feature completes theescalation story those sections explicitly defer to "a supervisor outside the run."
session/turn state and triggers resumes through whatever surface the harness actually offers —
its real contract must be verified before design (/dev:technical-plan §2.2b), not assumed. This
is the load-bearing feasibility risk for every resume AC; see Notes.
invoke; itself downstream of #43's claims ledger (in flight).
the bash/jq helper tier; nothing stack- or project-specific baked in. The declared tunables
block must obey the declared-block parseability rules (scalars + flat lists).
Timeline
Notes
rule; stack-agnostic per the suite's portability baseline.
— every resume AC presupposes it. /dev:technical-plan §2.2b must verify the real contract
before design; if the surface cannot support positive turn-state evidence, the feature falls
back to notify-only and this PREQ returns for re-scoping.
must state how each condition is induced or simulated — an AC only checkable by waiting for a
real 529 is not a testable case.
alternative — rejected: unattended means nobody receives the notification for hours), and
don't-build; #65 interfaced (hook), not absorbed; liveness = alert-only; resumable = idle
turn-end on a self-clearing cause (settles the panel's AC1-vs-AC6 contradiction — the
founding incident is resumable, operator stops never are); tunables declared with suite
defaults; alerts = forge record always + declared channel when present.
/meta-amend outcome — protocol written; issue stays OPEN on AC2
Commit
2ceacfeonmain. AC1 and AC3 met; AC2 partly met and honestly not reachable from skill text — see the last section.Verified at HEAD (
3b82a77)The gap held.
wait-discipline.mdrule 3 ("A stall is a decision point, not an exit") enumerated three dispositions — act, escalate, or report-and-stop on a session/usage kill — all of which assume the stall needs a human or ends the run. Nothing covered "the call failed for a reason that will clear on its own." The third bullet came closest and pointed the wrong way: it says say-so-and-stop, where the right answer for a quota reset is wait-and-continue.The amendment
New §3a — "A self-clearing stall is waited out — not escalated, and not stopped on", with the two cases distinguished as AC1 asks:
phase-runner.md§5 now names a transient API error as one of the causes its existing re-dispatch already covers — and makes the consequence explicit: that re-dispatch is the unattended resume for dispatched phase work. Plus two specifics: name the error in the appended line (so three transient failures are distinguishable from slowness), and wait out a stated reset rather than burning the single retry against a wall.Gate
scripts/lint-conventions.sh— clean.Desk-check against the originating scenario
phase-runner.md§5 re-dispatches once; phases resume from the latest Phase Outcome, so the resume is unattended. Covered/dev:integratestopped by a529on 2026-08-14, 8m 47s of dead time until the operator typed "Try again"AC2 — why this issue stays open
"An unattended run that hits a 529 resumes without operator input."
Met for phase work executed in a dispatched runner: the driver survives, sees the runner die, and re-dispatches. Not met for the case actually recorded, which is the driver's own turn being ended by the harness. Nothing inside a session can retry a call that session never got to make — by the time the error exists, there is no turn left to run the retry in. That needs a supervisor outside the run (a re-prompting wrapper, a scheduled resume, a background driver), which is a capability, not a text amendment, and outside this lane.
I have written that limitation into the procedure rather than leaving it implicit, so a reader does not infer coverage that isn't there — and paired it with the one thing text can do in that case: say so in the report, so the stall does not read as slowness.
Remaining work on this issue: decide whether the suite wants an out-of-run supervisor for the driver, and if so route it to
/dev:requirements. That is a scope call, not an amendment. Everything else here is done.What validates it next
For the covered half: the next run that hits a transient error in a helper call or a dispatched runner — the signal is an attempt count appearing in a phase report rather than a stop. For AC3 specifically, the next quota stall; the bar is that the reset time appears in the record and the wait is measured against it rather than guessed.
Linked: this issue is relates_to #65 (recorded by the devwork pipeline).
Test Plan: run-supervisor (issue #46)
Prerequisites
The state the scenarios need (how each condition is induced/simulated is stated per scenario in
plain terms; the concrete mechanism is decided at design time — the PREQ requires only that every
condition be inducible rather than waited for):
supervision engaged at launch (supervision is opt-in at launch)
reset time, one carrying none
threshold), and a second setup with no override (suite defaults in force)
visible trace, and a second setup with no such step declared
Required Test Data
Test Scenarios
Scenario 1: Transient turn-death is resumed automatically
Acceptance criterion: "Given a supervised unattended run whose driver turn ends on a transient error (per the classification rule), when the supervisor detects the run is resumable, then the run is resumed without operator input, with attempts bounded and backoff-spaced per the declared (or default) values."
spacing is consistent with the values in force (defaults, in this setup)
Expected outcome: the run finishes its phase work as if the stall had been mere slowness; the record distinguishes it from slowness.
Scenario 2: Retry bound exhausted → escalation naming the count
Acceptance criterion: "Given the per-incident retry bound is exhausted without a successful resume, when the last attempt fails, then the supervisor stops retrying and escalates, naming the attempt count and the error."
Expected outcome: bounded behavior — neither a single give-up nor an infinite loop; the operator can read exactly what was tried.
Scenario 3: Quota kill with stated reset → scheduled resume
Acceptance criterion: "Given a driver killed by quota exhaustion with a stated reset time, when the supervisor detects it, then the reset time is recorded and the resume is scheduled against that time (immediately, if the reset is already past), and the run resumes without operator input."
Expected outcome: the wait is measured against the platform's own reset, not guessed; zero time lost past the reset beyond ordinary latency.
Scenario 4: Quota kill with a reset already past → immediate resume
Acceptance criterion: same as Scenario 3 (the "immediately, if the reset is already past" clause).
Expected outcome: no wait against a wall that has already fallen.
Scenario 5: Quota kill with no discoverable reset → bounded-retry path
Acceptance criterion: "Given a quota kill with no discoverable reset time, when the supervisor detects it, then it follows the bounded-retry path per the classification rule, escalating on exhaustion with what it observed."
Expected outcome: no blind polling and no silent stop; the classification rule's else-branch is visibly taken.
Scenario 6: Stalled but possibly mid-work → alert only, no writes
Acceptance criterion: "Given a driver that stops making progress without positive evidence that its turn is over, when the declared (or default) stall threshold passes, then the supervisor escalates and does not resume, nudge, or otherwise write into the session."
state is exactly as the stall left it
Expected outcome: the wedged run becomes visible; the supervisor never risks double-driving a session that might still be working.
Scenario 7: Operator-initiated stop is never auto-resumed
Acceptance criterion: "Given any resume, when the supervisor starts it, then the resumability rule held: positive evidence of no turn in flight, a self-clearing cause, and not an operator-initiated stop."
evidence that no turn was in flight and the self-clearing cause it acted on
Expected outcome: a human's decision to stop a run is final for the supervisor; resumes happen only on recorded positive evidence.
Scenario 8: Clean completion → supervisor stands down
Acceptance criterion: "Given a supervised run that completes normally, when the driver finishes, then the supervisor stands down without intervening, and this is visible in the run's record."
Expected outcome: supervision of a healthy run is invisible except for its stand-down entry.
Scenario 9: Second attach refused
Acceptance criterion: "Given a run that already has a supervisor attached, when a second supervisor attempts to attach, then the attach is refused — at most one supervisor per run."
confirm exactly one resume happens)
Expected outcome: no double-resume is possible from doubled supervision.
Scenario 10: Harness surface unavailable → escalate, don't act
Acceptance criterion: "Given the supervisor cannot read the driver's state or cannot trigger a resume (the harness surface is unavailable or refuses), when it detects this, then it escalates with what it observed rather than acting on ambiguity."
observed
Expected outcome: blindness is treated as ambiguity — surfaced loudly, never acted on.
Scenario 11: Declared post-resume reconstruction step is invoked
Acceptance criterion: "Given a project that declares a post-resume claims-reconstruction step, when the supervisor resumes a run, then that step is invoked before phase work continues."
supervised unattended run
invocation and no error about its absence
Expected outcome: the hook fires when declared, is a no-op when not, and always precedes resumed phase work.
Scenario 12: Everything is on the record; channel delivery when declared
Acceptance criterion: "Given any supervisor intervention or escalation …, when it happens, then it is written to the run's durable forge record — and additionally delivered via the project's declared notification channel, when one is declared."
produced appears in the run's durable forge record, legible to a reader who was not present
arrives on that channel as well as in the record
carries it, completely
Expected outcome: a retrospective reader can reconstruct every stall and its handling from the record; the channel is additive, never the only copy.
Scenario 13: Declared tunables are the values in force
Acceptance criterion: the "per the declared (or default) values" clauses of the retry and stall criteria.
suite defaults), run Scenario 2's steps and Scenario 6's steps
Expected outcome: the declaration is honored; the acceptance criteria are judged against whichever values are in force.
Notes
harness surface are induced — the PREQ requires them inducible; the mechanism is a design
decision and will be named when lanes are annotated.
route these to non-browser lanes.
Software Requirements: run-supervisor
Context
An operator who starts an unattended pipeline run (one Claude Code driver session driven by
run-template.md) loses the run whenever the driver's own turn is ended by a self-clearingcondition — a transient API error (the recorded 8m47s dead 529) or a usage-quota kill with a
stated reset (the recorded ~3h17m lost past the reset). Nothing inside the session can retry a
call the session never made. This feature is the supervisor outside the run that wait-discipline
§3a explicitly defers to: opt-in at unattended launch, resuming only under the PREQ's
resumability rule, alert-only on ambiguity, everything on the durable forge record.
Approaches Considered
Approach A: Deterministic shell supervisor daemon
Summary: A helper-tier bash daemon, spawned detached at attach, that polls the driver
session's transcript, classifies turn state mechanically, and resumes via the harness CLI — no
model in the watch loop.
Pros: Classification is fully deterministic given the verified transcript contract
(structured
apiErrorStatus,isApiErrorMessage,stop_reason); zero quota consumed whilewatching (model calls only for actual resumes); fits the portability baseline (bash ≥ 3.2 + jq);
wholly auditable — every branch is a recorded rule, not a judgement.
Cons: A daemon lifecycle to manage (lock, stale-lock recovery, detach); clock-time parsing
for quota resets needs a probed
datecapability.Effort: Medium
Approach B: Supervisor as a second Claude session
Summary: A
/dev:supervisesession (or scheduled agent) watches the driver with modeljudgement, using harness watch tools.
Pros: Judgement available for ambiguous states; no bespoke daemon code.
Cons: Shares the exact failure domain it guards — a quota kill takes out watcher and watched
together (the founding quota incident would have killed both); burns quota continuously;
judgement is unnecessary — the classification inputs are structured fields; harder to make
auditable ("the model decided" vs a recorded rule).
Effort: Medium
Approach C: Scheduler-hosted stateless checks (cron / systemd user timer)
Summary: No long-lived process; a check script re-run every N minutes by the host scheduler,
with all state externalized.
Pros: Partially survives host reboot; no daemon liveness question.
Cons: Platform-specific installation surface (cron vs systemd vs launchd) against a
stack-agnostic suite; attach/stand-down become install/uninstall of scheduler entries (messy,
easy to leak); sub-minute stall responsiveness gone; and the reboot-survival benefit buys little
— supervisor HA is explicitly out of scope in the PREQ.
Effort: Medium-High
Decision
Selected: Approach A — deterministic shell supervisor daemon.
Rationale: The watch decision needs no model: the harness records carry structured
discriminators (verified at plan time), and a watcher that consumes quota or dies of the same
quota kill it guards against fails exactly when needed. A single detached daemon with an atomic
lock and probed capabilities matches the suite's existing helper-tier engineering and keeps the
whole behavior testable with a stub harness.
Architecture
Component Overview
plugin/skills/supervise/SKILL.md— the/dev:superviseskill: engagement at unattendedlaunch, status, detach, the
supervisor:declared-block schema, and the typedsupervisor-event:v1record convention.plugin/skills/supervise/bin/supervisor-attach.sh— preflight (transcript readable;claudeCLI present; jq; probe-by-doing for clock-time parsing), atomic lock acquisition, state
init, daemon spawn (detached,
setsid), synchronous engage verification (waits boundedfor the daemon's first heartbeat, exits non-zero without it), attach record posted.
plugin/skills/supervise/bin/supervisor-watch.sh— the daemon loop (poll default 30 s):read transcript tail + mtime, classify, act, heartbeat.
plugin/skills/supervise/bin/supervisor-status.sh— one-screen answer: engaged? lastheartbeat (stale = loud)? interventions so far? next scheduled action?
plugin/skills/supervise/bin/supervisor-detach.sh— stand-down: kill by recorded PID/pgid(never pattern-kill), release lock, post stand-down record; handles an already-dead daemon
(stale state cleanup, recorded).
$DEVWORK_ROOT/supervisor/{session-id}/(mode 0700) —lock/(atomicmkdir),state.json(issue, cwd, tunables in force, incident/attempt counters, scheduled resume,alerted-condition dedup keys, daemon PID + start time; written atomically via temp +
mv),daemon.log.post-comment.sh(adapter-dispatched forge writes),phase-outcome-read-latest.sh(run-terminal-state check),run-template.md(gains theopt-in engage line),
wait-discipline.md§3a (limitation paragraph now points here).Data Flow
supervisor-attach.sh --session "$CLAUDE_CODE_SESSION_ID" --issue N(the env var isverified present in driver shells). Attach refuses a second supervisor (lock held + live
PID/start-time match), reclaims a stale lock loudly, and only reports "supervision engaged"
after observing the daemon's first heartbeat.
yields: healthy / resumable-transient / resumable-quota / not-self-clearing /
run-complete / ambiguous-stall / operator-stop.
cd {recorded cwd}(validated: exists, matches attach-time record) →claude -p --resume {session-id} --output-format json "<fixed resume prompt>". The promptopens with a visible marker line (
[run-supervisor resume — incident I, attempt A/B]),instructs re-verification of in-flight mutations (wait-discipline rule 5), invokes the
project-declared
post_resume_commandfirst when declared (hook failure ⇒ escalate, do notcontinue silently), then continues driving. While the resume child runs, the daemon is in a
resume-in-flightstate: no classification of its own resume as stall or new incident;outcome judged on child rc + envelope
is_error, then transcript progression.operator-stop acknowledgment, stand-down, notify-channel failure) is posted to the feature
issue via
post-comment.shas a typedsupervisor-event:v1body:{event, cause, evidence: {tail_uuid, apiErrorStatus, error_text}, attempt, bound, next_action, next_eta}— thesynthetic error line and classification metadata only, never other transcript content.
The declared
notify_command, when present, is additionally invoked; its failure is itselfrecorded (the record is always the primary copy).
next_state∈{uat, accepted, deployed} — checked only when the tail shows a clean turn end), on operator
stop (acknowledged visibly: "operator stop observed — standing down"), or on explicit
detach. Lock released, stand-down record posted.
Classification rule (from the verified contract)
The last assistant record decides the base class — never the literal last line, because
harness bookkeeping records (
last-prompt,mode) legitimately follow it and are ignored.Non-bookkeeping records after the last assistant record override it: an operator interrupt
appends a plain user record (
[Request interrupted by user]) after the error record, and thatoverride is a hard operator-stop. Classification lives in one seam — a single
classify-transcripthelper function/file, the only code coupled to the un-versioned harnessrecord schema (schema drift lands in one place, and the seam is unit-testable in the shell
harness; same for the isolated clock-time parser):
isApiErrorMessage:true, text matches quota-reset pattern (resets <clock> (<TZ>))isApiErrorMessage:true,apiErrorStatus∈ {429, 500, 502, 503, 529} or connection-closed textisApiErrorMessage:true, anything else (401, …)stop_reason == "end_turn", run at terminal PO statestop_reason == "end_turn", run not terminalclaudeCLI refusesImmediately before each resume attempt the transcript is re-classified in full — not
merely diffed — because detection and action can be hours apart (a quota sleep): if the tail
changed or the class is no longer resumable (operator returned, another writer, new records),
the resume is aborted and the observation recorded — closing the check-to-act race at the
moment that matters. Alerts are deduplicated per condition
(keyed on tail uuid / stall-onset): one alert per distinct condition, re-alert only on state
change.
External Data Contracts
~/.claude/projects/{cwd-slug}/{session-id}.jsonltail + mtimestop_reason:"end_turn"; turn-death =isApiErrorMessage:true+apiErrorStatus(int) +error+ human text; quota text carries clock-time reset + TZ; interrupt = plain user record; seeexternal-contracts/harness-session-transcript.sample.jsonrecorded→external-contracts/harness-session-transcript.provenance.jsonclaude -p --resume {sid} --output-format json{session_id, result, is_error, subtype}; interactive-created sessions also resume headlessrecorded→external-contracts/harness-resume-cli.provenance.jsonBoth boundaries verified by real reads/probes at plan time — no preconditions. The transcript
record schema is un-versioned harness internals; drift is a Technical Risk below, mitigated by
classifying on the minimal stable core and the surface-unavailable escape (AC-9).
Key Decisions
claude -p --resumeheadless, driver's own cwd-pgives one full agentic turn; no TTY needednotify_command/post_resume_commandexec'd with event data as argv/env only — never string-interpolated into shellmkdir+ PID/start-time liveness; stale lock reclaimed loudlyprobe_pin); fail closed to bounded-retrydate -dis not in the baseline; capability is behavioral, never provenancesupervisor-event:v1comment; metadata + synthetic error line onlyTechnical Risks
isApiErrorMessage,stop_reason,apiErrorStatus); attach preflight re-validates readability; unparseable ⇒ surface-unavailable escalation (AC-9), never a guess-presume gives one agentic turn; a driver ending that turn awaiting a cross-turn harness notification exits and the run parkssupervisor-status.shreports staleness loudly; absence of interventions visible in the run recordExpert Review
Reviewers
date -dnot in baseline. All absorbed (classification rule + Key Decisions).pre-resume re-classification after long quota sleeps; last-assistant-record (not
literal-last-line) classification, since bookkeeping records follow it. All absorbed.
Changes Made
hard operator-stop state (Backend), with visible acknowledgment + stand-down (UX); and
anchored on the last assistant record with bookkeeping records ignored (Architect).
to un-versioned harness internals (Architect).
hours-long quota-sleep gap (Architect); daemon death detectable at next attach is recorded
on the forge issue during stale-lock reclaim (Architect).
supervisor-status.sh; alert dedup once-per-condition (UX).already-past rule (Backend, Security).
resume-in-flightdaemon state so the loop never classifies its own resume (Backend).partial-JSONL-line tolerance; detach by recorded PID; cwd validated before
cd;transcript checked to be a regular file owned by the invoking user before automated action;
notify_commandfailure itself recorded; resume prompt carries a visible marker line(Security, Backend, UX).
post_resume_commandfailure semantics fixed: escalate, never proceed silently (Backend).Noted (not actioned)
ships; a declared re-alert interval is deliberately not added until evidence shows
single-shot alerts get missed. Rejected for this slice on simplicity grounds.
ship; active watching-the-watcher is the PREQ's explicit HA out-of-scope. Already covered by
out-of-scope finding and the heartbeat visibility that did ship.
Acceptance Criteria
Verification suite:
plugin/skills/supervise/test/supervisor-flow.sh(+ fixtures) — a shellintegration suite authored by this feature (TDD, red-first per
red-markers.mdresolutionorder), running the daemon against synthetic transcripts built from the recorded real shapes
(provenance-tagged, never hand-guessed), a stub
claudeon PATH, thelocal-fsadapter as theforge target, and a stub
notify_command. Executor confirmed runnable here: bash + jq + theexisting
_shared/procedures/test/ adapter-test/harness conventions.supervisor-event:v1carries tail-uuid evidence + cause; interrupt fixture never resumessupervise/bin/*— noeval, nosh -cwith substituted event fields; suite fixture with hostile error text executes nothingbash scripts/lint-conventions.sh(existing gate, extended to the new bin/)No temporary scaffolding is introduced (no flags, aliases, or dual paths). The project's
observability mode is
none; the supervisor's forge records are the feature's own function,already first-class as AC-11 — no separate Observability & Audit subsection applies.
Implementation Scope
Areas
plugin/skills/supervise/SKILL.mdsupervisor:block schema (flat scalars: retry_bound, backoff_base_seconds, backoff_multiplier, stall_threshold_minutes, poll_interval_seconds, notify_command, post_resume_command),supervisor-event:v1record conventionplugin/skills/supervise/bin/{supervisor-attach,supervisor-watch,supervisor-status,supervisor-detach}.shplugin/skills/supervise/test/(suite + provenance-tagged fixtures + stubclaude+ stub notify/hook)plugin/skills/_shared/run-template.md$CLAUDE_CODE_SESSION_IDplugin/skills/_shared/procedures/wait-discipline.md§3a/dev:supervisescripts/lint-conventions.sh.claude-plugin/marketplace/plugin listing if skills are enumeratedFile Boundaries
The supervise skill (SKILL.md + bin/ + test/) is one coherent unit — single writer. The
run-template and wait-discipline edits are one-paragraph touches, independent of the helpers and
of each other. Lint-gate extension is independent. No shared-file contention with in-flight
siblings expected (#43 owns claims-ledger surfaces; #61 owns devcontainer) — but
_shared/edits are the known collision class: coordinate merge timing at a phase boundary.
Dependencies & Sequencing
Helpers and their test suite co-evolve (TDD). SKILL.md documents what the helpers do — draft
alongside, finalize after helper behavior settles. Template/procedure pointer edits can land
any time after the skill name is fixed. Everything else is parallel-safe.
Constraints & Non-Goals
Constraints:
mapfile/declare -A; POSIX-only glue inanything the skill emits into an operator shell; no python/node.
_shared/procedures/bin/*+ active adapter — never inline forge CLI.supervisor:declared block obeys the declared-block parseability rules: flat scalarsonly, schema stated in the skill.
notify_command,post_resume_command) are the project's stack — readfrom CLAUDE.md, never defaulted to any concrete tool.
Non-goals (do NOT build):
post_resume_commandhook here).Test Plan: run-supervisor (issue #46)
Prerequisites
The state the scenarios need (how each condition is induced/simulated is stated per scenario in
plain terms; the concrete mechanism is decided at design time — the PREQ requires only that every
condition be inducible rather than waited for):
supervision engaged at launch (supervision is opt-in at launch)
reset time, one carrying none
threshold), and a second setup with no override (suite defaults in force)
visible trace, and a second setup with no such step declared
Required Test Data
Test Scenarios
Scenario 1: Transient turn-death is resumed automatically
Acceptance criterion: "Given a supervised unattended run whose driver turn ends on a transient error (per the classification rule), when the supervisor detects the run is resumable, then the run is resumed without operator input, with attempts bounded and backoff-spaced per the declared (or default) values."
Lane: integration-covered —
plugin/skills/supervise/test/supervisor-flow.shspacing is consistent with the values in force (defaults, in this setup)
Expected outcome: the run finishes its phase work as if the stall had been mere slowness; the record distinguishes it from slowness.
Scenario 2: Retry bound exhausted → escalation naming the count
Acceptance criterion: "Given the per-incident retry bound is exhausted without a successful resume, when the last attempt fails, then the supervisor stops retrying and escalates, naming the attempt count and the error."
Lane: integration-covered —
plugin/skills/supervise/test/supervisor-flow.shExpected outcome: bounded behavior — neither a single give-up nor an infinite loop; the operator can read exactly what was tried.
Scenario 3: Quota kill with stated reset → scheduled resume
Acceptance criterion: "Given a driver killed by quota exhaustion with a stated reset time, when the supervisor detects it, then the reset time is recorded and the resume is scheduled against that time (immediately, if the reset is already past), and the run resumes without operator input."
Lane: integration-covered —
plugin/skills/supervise/test/supervisor-flow.shExpected outcome: the wait is measured against the platform's own reset, not guessed; zero time lost past the reset beyond ordinary latency.
Scenario 4: Quota kill with a reset already past → immediate resume
Acceptance criterion: same as Scenario 3 (the "immediately, if the reset is already past" clause).
Lane: integration-covered —
plugin/skills/supervise/test/supervisor-flow.shExpected outcome: no wait against a wall that has already fallen.
Scenario 5: Quota kill with no discoverable reset → bounded-retry path
Acceptance criterion: "Given a quota kill with no discoverable reset time, when the supervisor detects it, then it follows the bounded-retry path per the classification rule, escalating on exhaustion with what it observed."
Lane: integration-covered —
plugin/skills/supervise/test/supervisor-flow.shExpected outcome: no blind polling and no silent stop; the classification rule's else-branch is visibly taken.
Scenario 6: Stalled but possibly mid-work → alert only, no writes
Acceptance criterion: "Given a driver that stops making progress without positive evidence that its turn is over, when the declared (or default) stall threshold passes, then the supervisor escalates and does not resume, nudge, or otherwise write into the session."
Lane: integration-covered —
plugin/skills/supervise/test/supervisor-flow.shstate is exactly as the stall left it
Expected outcome: the wedged run becomes visible; the supervisor never risks double-driving a session that might still be working.
Scenario 7: Operator-initiated stop is never auto-resumed
Acceptance criterion: "Given any resume, when the supervisor starts it, then the resumability rule held: positive evidence of no turn in flight, a self-clearing cause, and not an operator-initiated stop."
Lane: integration-covered —
plugin/skills/supervise/test/supervisor-flow.shevidence that no turn was in flight and the self-clearing cause it acted on
Expected outcome: a human's decision to stop a run is final for the supervisor; resumes happen only on recorded positive evidence.
Scenario 8: Clean completion → supervisor stands down
Acceptance criterion: "Given a supervised run that completes normally, when the driver finishes, then the supervisor stands down without intervening, and this is visible in the run's record."
Lane: integration-covered —
plugin/skills/supervise/test/supervisor-flow.shExpected outcome: supervision of a healthy run is invisible except for its stand-down entry.
Scenario 9: Second attach refused
Acceptance criterion: "Given a run that already has a supervisor attached, when a second supervisor attempts to attach, then the attach is refused — at most one supervisor per run."
Lane: integration-covered —
plugin/skills/supervise/test/supervisor-flow.shconfirm exactly one resume happens)
Expected outcome: no double-resume is possible from doubled supervision.
Scenario 10: Harness surface unavailable → escalate, don't act
Acceptance criterion: "Given the supervisor cannot read the driver's state or cannot trigger a resume (the harness surface is unavailable or refuses), when it detects this, then it escalates with what it observed rather than acting on ambiguity."
Lane: integration-covered —
plugin/skills/supervise/test/supervisor-flow.shobserved
Expected outcome: blindness is treated as ambiguity — surfaced loudly, never acted on.
Scenario 11: Declared post-resume reconstruction step is invoked
Acceptance criterion: "Given a project that declares a post-resume claims-reconstruction step, when the supervisor resumes a run, then that step is invoked before phase work continues."
Lane: integration-covered —
plugin/skills/supervise/test/supervisor-flow.shsupervised unattended run
invocation and no error about its absence
Expected outcome: the hook fires when declared, is a no-op when not, and always precedes resumed phase work.
Scenario 12: Everything is on the record; channel delivery when declared
Acceptance criterion: "Given any supervisor intervention or escalation …, when it happens, then it is written to the run's durable forge record — and additionally delivered via the project's declared notification channel, when one is declared."
Lane: integration-covered —
plugin/skills/supervise/test/supervisor-flow.shproduced appears in the run's durable forge record, legible to a reader who was not present
arrives on that channel as well as in the record
carries it, completely
Expected outcome: a retrospective reader can reconstruct every stall and its handling from the record; the channel is additive, never the only copy.
Scenario 13: Declared tunables are the values in force
Acceptance criterion: the "per the declared (or default) values" clauses of the retry and stall criteria.
Lane: integration-covered —
plugin/skills/supervise/test/supervisor-flow.shsuite defaults), run Scenario 2's steps and Scenario 6's steps
Expected outcome: the declaration is honored; the acceptance criteria are judged against whichever values are in force.
Notes
harness surface are induced — the PREQ requires them inducible; the mechanism is a design
decision and will be named when lanes are annotated.
route these to non-browser lanes.
Lane annotation (added by /dev:technical-plan, 2026-08-25)
integration-covered—plugin/skills/supervise/test/supervisor-flow.sh,the shell integration suite this feature itself authors (TDD red-first in /dev:develop; each
scenario is bound to it by the SREQ Acceptance Criteria table). The executor is confirmed
runnable in this environment: bash + jq (suite preflight green), same harness conventions as
the existing
_shared/procedures/test/and adaptertest/suites. The suite drives thesupervisor daemon against synthetic transcripts built from the plan-time recorded harness
record shapes (
external-contracts/provenance), a stubclaudeon PATH, thelocal-fsadapter as forge target, and stub notify/post-resume commands — so every "induce/simulate"
prerequisite in this plan is satisfied without waiting for a real 529.
e2e-browser,config-variant, or standalonehuman-uat: thisproject declares no browser surface (
qa_domains.not_applicable: e2e/a11y/security-browser),tunable variants run as per-invocation suite inputs (no server or restart exists), and no
scenario requires live-external writes. Human judgement enters at the UAT gate as declared
(
uat.url_source: dogfood a supervised run against a real project), which is the run-leveloverlay on top of these lanes, not a replacement for any one scenario.
Decomposition notes: WU-1 is the SREQ's single-writer coherent unit (skill doc, helper tier,
TDD suite with provenance-tagged fixtures from the feature folder's external-contracts/). WU-2
is artifact-shape (two one-paragraph pointer edits in _shared/) — Test Writer skipped per the
artifact-shape exception; it runs in wave 2 so its engage line documents the flags WU-1's
attach helper actually ships. Two SREQ scope areas need no change at all, verified against the
tree: scripts/lint-conventions.sh (its find glob already scans any plugin/skills//bin/.sh)
and the plugin manifests (skills are not enumerated). Single shared worktree: execution is
sequential (Test Writer → Implementer per WU, WU-1 before WU-2).
Development Journal — WU-46-3-1 (supervise skill unit)
2026-08-25T17:45 DISCOVERY: the lint gate refuses
/dev:superviseuntil a sibling WU widens itContext: planning
plugin/skills/supervise/SKILL.md, whose natural spelling of the skill's owninvocation is
/dev:supervise.Pattern (PRODUCT FINDING — for the lead's finding pipeline):
scripts/lint-conventions.shhardcodes the 14 shipped skills in
SHIPPED_SKILLSand reports every/dev:<name>outside thatlist as a violation ("unknown /dev: skill reference").
superviseis the 15th, and the lint file isowned by a sibling WU, which I am told not to touch. My own AC-14 requires
bash scripts/lint-conventions.shto exit 0, so this WU cannot write the token/dev:superviseinto any shipped markdown until the sibling adds
supervisetoSHIPPED_SKILLS.Why it matters: the constraint is invisible from the SREQ — it only shows up as a red AC-14 the
first time a new skill names itself. Two consequences the lead must route: (a) the sibling WU that
extends the lint gate must add
supervisetoSHIPPED_SKILLS(otherwiserun-template.mdandwait-discipline.md, which are supposed to point at/dev:supervise, cannot land either); and(b) more generally, adding a skill to the suite has an undocumented second step in the meta lane.
Until (a) lands I refer to the skill by its bare name in SKILL.md prose.
2026-08-25T18:05 DISCOVERY: the pre-resume abort must test "still resumable", not "same cause"
Context: implementing the SREQ's "full re-classification immediately before every resume
attempt — abort on change". The obvious reading is "abort if the class differs from the class
that opened the incident".
Pattern: that reading is wrong, and AC-4 is the case that proves it. A quota kill whose
resume fails comes back as a 529, so the cause legitimately shifts
resumable-quota -> resumable-transientbetween attempts of the same incident. Aborting onthat shift caps the incident at one attempt and the declared bound never applies. The correct
abort test is two separate questions: is the class still in the resumable family (if not,
the run moved on — close the incident), and has the tail record moved out from under the
schedule (if so, somebody else wrote — re-aim, but keep the attempt count).
Why it matters: keeping the attempt count on a moved tail is the load-bearing half. Resetting
it there turns "another writer is appending" into an unbounded resume loop, which is exactly the
failure the bound exists to prevent. The incident — and therefore the bound — is identified by
the incident being open, never by the cause staying constant.
2026-08-25T18:10 DISCOVERY: attach must heartbeat before it classifies, and the pre-resume re-check is what makes AC-9a deterministic
Context: AC-9a chmods the transcript unreadable immediately after attach returns and asserts
no resume ever fires. Whether the daemon has already classified and scheduled by then is a race
the test cannot control.
Pattern: two things make it deterministic rather than lucky. (1) The daemon writes its first
heartbeat before its first classification, so attach's synchronous engage check means "this
process is alive and owns the session", not "it has already decided something". (2) The SREQ's
pre-resume full re-classification catches the surface going away between scheduling and
firing — so even when the daemon did schedule a resume before the chmod, the resume aborts and
escalates instead of firing. The security requirement and the test's determinism are the same
mechanism; neither was added for the other.
Dev journal — WU-46-3-2 (wire the run supervisor into the suite surfaces)
2026-08-25T18:05 DISCOVERY:
test/supervisor-flow.shAC-9a is timing-flakyContext: Running the WU-1 suite as this WU's third verification gate. It failed on run 1
(
FAIL - AC-9a no resume attempted once the transcript is unreadable: expected [0] got [1]),passed on run 2, failed again on run 3 — same assertion each time, no other failure.
Pattern: Not caused by this WU. The suite reads no markdown at all (
grep -n 'SKILL.md' plugin/skills/supervise/test/supervisor-flow.shfinds nothing), and the only non-markdown filethis WU touches is
scripts/lint-conventions.sh, which the suite does exercise — AC-14 runs itand reports
okon every run, including the failing ones.The race is in the scenario itself:
scenario_ac9_unreadableseeds a transient529on thetranscript, attaches with
poll_interval_seconds=1/backoff_base_seconds=1, and only thenruns
chmod 000on the transcript. Attach returns after the daemon's first heartbeat, so thedaemon is already polling a readable transcript carrying a resumable cause. Whether it fires a
resume before the
chmodlands is a sub-second coin flip. The assertion wants "no resume onceunreadable", but it counts all resumes, including a legitimate one issued while the transcript
was still readable.
Why it matters: It is a false red on a gate other WUs must judge themselves by, and it is in
WU-46-3-1's territory (
test/*is off-limits to this WU), so it is reported up rather than fixedhere.
2026-08-25T18:20 COMPLETE
Hardest part: Deciding how far the "(v4)" → "(v5)" rename should reach in
run-template.md—the execution-model paragraph names the current version twice while also narrating v3 history, so
only the two current-version mentions moved and every historical mention (and every version-log
row) stayed put.
If I did this again: Run the WU-1 suite once before making any edit, to have a pre-edit
baseline for a flake like AC-9a instead of having to reason backwards from what the suite reads.
Linked: this issue is sibling #255 (recorded by the devwork pipeline).
Linked: this issue is sibling #255 (recorded by the devwork pipeline).
Linked: this issue is sibling #255 (recorded by the devwork pipeline).
UAT prerequisites — issue #46, run supervisor (
/dev:supervise)Where the reviewable build is
Per the project's
uat.url_source, verbatim: "no deploy — review the merged skill text on main and dogfood a run against a real project". Modelocal, no PR (uat.open_pr: false).Until
/dev:integratemerges, the reviewable tree is the worktree.claude/worktrees/devwork-46at commit8de068e(branchfeature/46-run-supervisor). The skill text to read:plugin/skills/supervise/SKILL.md. The helpers:plugin/skills/supervise/bin/.Authentication
None. Everything is local: the supervisor reads
~/.claude/projects/<cwd-slug>/<session-id>.jsonlas your own user and posts records via the project's forge adapter (this repo:tea-cliagainst git.wihslon.com, your existingtealogin).Scenario-specific test data
claude,local-fsforge sandboxes) — nothing persists for you to inspect except the suite itself:plugin/skills/supervise/test/supervisor-flow.sh(181 checks green at8de068e).supervisor-attach.sh --session "$CLAUDE_CODE_SESSION_ID" --issue <N>). The engage refuses to report success until the daemon's first heartbeat.supervisor-status.sh --session <id>gives the one-screen answer (engaged, heartbeat age, interventions, next action).supervisor-detach.shstands it down.Setup beyond normal startup
None. Preflight is self-contained (transcript readable,
claudeCLI on PATH, jq, clock-parse probe); a failing preflight names what is missing and refuses loudly.Scenario list (from
test-plan:v1— the same activity as this walk)All 13 scenarios ran green in the integration lane; the dogfood walk is the run-level overlay, not a re-execution of each:
Suggested dogfood focus (what the suite could not observe): a real long quota sleep (the suite accepts V-2's composition argument — the live fire at a future reset is the one path only a real run exercises), and the
-presume giving one agentic turn (SREQ Technical Risk: a driver parking after a cross-turn wait lands in the alert-only ambiguous class — confirm the alert is legible when it happens).Known accepted residuals (so the walk doesn't re-derive them)
Thirteen accepted findings across the five QA rounds, each with recorded reasoning on the issue (highlights: crash-window record gap under the HA non-goal; nohup-fallback orphan; pid-only identity on exotic boxes; weekly-limit futile-but-bounded retries; V-2 future-reset live fire covered by composition — the dogfood walk is its observer; the marker paste-quote tradeoff; the absent-bound status default). Four robustness edges deferred to issue #255 (retry bound across re-attach, lock-reclaim TOCTOU, stall blind spots, abort-re-aim churn bound).