jbr870/devwork-skills

Fork 0

/develop: validate against the real committed content, not only fixtures #48

New issue

Closed

opened 2026-08-14 13:01:55 +00:00 by jbr870 · 1 comment

jbr870 commented

2026-08-14 13:01:55 +00:00

Owner

Rank 12 of 13 in #37 · caught a blocking defect that every gate had passed

Evidence

/develop on #230 finished with all gates green: typecheck 0, lint 0, build 0, 4,187 tests passing,
469 browser tests passing, plus explicit probe runs. Every test used fixtures.

Forty minutes into QA, the orchestrator parsed the real shipping content —
release-notes/v0.1.0.md and v0.2.0.md — and found the feature's first blocking defect: the
markdown parser broke wrapped bullets into stray paragraphs. Its note: "this is exactly the failure
mode the playbook's §6 fixture rule exists to catch."

That defect went on to cost 2h 21m and four fix rounds (#40), and would have shipped a
release-notes page that rendered the project's own release notes wrong.

The same lesson recurred at /promote: writing v0.2.1.md the model parser-verified the new file
before committing it, precisely because of this history — and separately discovered that a QA-added
test asserted "exactly the two committed entries", which the new file broke.

What happens now

test-workflow.md and the domain briefs are fixture-based by design, which is right for
determinism. Nothing requires a single pass over the real artifact the feature will actually serve.

Proposed change

Where a feature reads or renders content committed in the repository, require one validation
case that exercises the real content set, not a fixture — and require it in /develop's
cross-unit verification rather than only in QA. Fixtures stay the unit-level default; the real
content is a smoke check on the artifact that ships.

Related: watchlist RC-W9 ("baseline the suite before speccing") is the adjacent idea at the other
end of the phase.

Acceptance

The rule names the trigger condition (feature consumes committed content) and the check.
/develop cross-unit verification runs it before posting a terminal outcome.
A test over real content states that it is intentionally coupled to committed data.

**Rank 12 of 13** in #37 · **caught a blocking defect that every gate had passed** ## Evidence `/develop` on #230 finished with all gates green: typecheck 0, lint 0, build 0, 4,187 tests passing, 469 browser tests passing, plus explicit probe runs. Every test used **fixtures**. Forty minutes into QA, the orchestrator parsed the **real shipping content** — `release-notes/v0.1.0.md` and `v0.2.0.md` — and found the feature's first blocking defect: the markdown parser broke wrapped bullets into stray paragraphs. Its note: *"this is exactly the failure mode the playbook's §6 fixture rule exists to catch."* That defect went on to cost 2h 21m and four fix rounds (#40), and would have shipped a release-notes page that rendered the project's own release notes wrong. The same lesson recurred at `/promote`: writing `v0.2.1.md` the model parser-verified the new file before committing it, precisely because of this history — and separately discovered that a QA-added test asserted "exactly the two committed entries", which the new file broke. ## What happens now `test-workflow.md` and the domain briefs are fixture-based by design, which is right for determinism. Nothing requires a single pass over the real artifact the feature will actually serve. ## Proposed change Where a feature reads or renders **content committed in the repository**, require one validation case that exercises the real content set, not a fixture — and require it in `/develop`'s cross-unit verification rather than only in QA. Fixtures stay the unit-level default; the real content is a smoke check on the artifact that ships. Related: watchlist RC-W9 ("baseline the suite before speccing") is the adjacent idea at the other end of the phase. ## Acceptance - The rule names the trigger condition (feature consumes committed content) and the check. - `/develop` cross-unit verification runs it before posting a terminal outcome. - A test over real content states that it is intentionally coupled to committed data.

jbr870 added the

labels

2026-08-14 13:01:55 +00:00

jbr870 referenced this issue

2026-08-14 13:02:37 +00:00

Feature-230 time study: 13 ranked pipeline-efficiency changes #37

jbr870 referenced this issue

2026-08-14 13:02:39 +00:00

Feature-230 time study: 13 ranked pipeline-efficiency changes #37

jbr870 referenced this issue from a commit

2026-08-25 09:36:07 +00:00

develop: validate the real committed content before handing off, not only fixtures

jbr870 closed this issue

2026-08-25 09:36:07 +00:00

jbr870 commented

2026-08-25 09:36:37 +00:00

Author

Owner

/meta-amend outcome — amended

Commit 2e0e43e on main.

Verified at HEAD (`8440c90`)

The gap held. Nothing in /dev:develop required a pass over real content, and the nearest existing rule did not cover this case: qa-playbook.md §6's "Synthetic fixtures at an external boundary" reasons from "the repo holds the schema, not the rows" — the seeded-datastore case, where the fixture is a guess because the real data lives elsewhere. Content committed in the tree is the inverse: the real set is right there, which makes a fixture a worse excuse, not the same one. And §6 is a QA-time audit, one phase after the point where the defect was cheap.

The amendment

develop/SKILL.md 3.7 step 1b — Real-content smoke check, placed after the wide-tier suite run and before the terminal Phase Outcome (AC2). Lettered 1b deliberately: the file already uses 2b/2c, and renumbering would have broken the 3.7 step 2 reference at SKILL.md:49.

Trigger (AC1): the feature reads, parses, renders, or transforms content committed in this repository — docs, release notes, seed/config/schema files, fixtures-as-product, content collections, migrations; anything the shipped artifact serves from the tree rather than from a request. If it doesn't apply, say so in one line and move on.
The check (AC1): one pass of the feature's real code path over the whole committed set — not a sample, not a fixture — with the output inspected. A crash, a swallowed item, or visibly wrong structure is a failure of the step, handled exactly like a failing test in step 1 (spawn the fix teammate). Fixtures stay the unit-level default; this is explicitly framed as a smoke check on the artifact that ships.
AC3 — the test declares its coupling: a comment at the top, in the project's own comment syntax, carrying both halves — it reads the real set rather than a fixture, and content edits may legitimately change it. The rationale is written down: without the declaration the next reader sees brittle data-coupling and swaps in a fixture, which is the defect coming back; and the brittleness is a real cost the person who trips over it needs to recognise as deliberate.
One clause the issue's second incident earned: assert properties over the set, not a snapshot of it. "Every committed item parses", "nothing is dropped" survive the next content addition; "exactly the two entries currently committed" does not — and that exact test was written on this same feature and broke at /promote when v0.2.1.md landed. Both halves of the issue's evidence are now covered by the rule, not just the first.
Carry-forward: a new Finding trigger in step 3 for the check applied but could not be run (content not committed yet, code path undrivable outside a deployment). Rationale in the text: otherwise QA inherits a green suite that has only ever seen fixtures and has no way to know. A check that ran needs no Finding, and a defect it found was fixed in 1b like any other failing test.
qa-playbook.md §6 gains the inverse-fixture bullet, pointing at 1b rather than restating it, and telling the audit two things it would otherwise get wrong: pick up the pass when 1b is absent or a Finding says it couldn't run, and do not flag a declared real-content test as brittle coupling.

Gate

scripts/lint-conventions.sh — clean.

Desk-check against the originating scenario (verity #230)

The feature rendered a release-notes page from release-notes/v0.1.0.md and v0.2.0.md — content committed in the repo, so the trigger fires unambiguously. Step 1b runs the parser over both files and inspects the output; wrapped bullets breaking into stray paragraphs is "visibly wrong structure", so the step fails and the fix teammate is spawned inside /dev:develop, before the terminal Phase Outcome — roughly forty minutes and one phase earlier than where QA actually caught it, and in the phase that already owns a fix-teammate path.

The second incident checks out too: the QA-added test asserting "exactly the two committed entries" is precisely what the properties-over-snapshot clause forbids, and it is the one that broke at /promote on the next release file.

Worth stating plainly: this does not claim the four fix rounds (#40) disappear. The parser bug still has to be fixed and is still a hard one. What moves is when it is found — inside the phase that can cheaply fix it, rather than forty minutes into QA against a suite that had already declared itself green.

What validates it next

The next feature that consumes committed content. Two things to watch: whether the trigger is recognised at all (it is a self-assessment, and "we have fixtures for that" is the tempting wrong answer), and whether the declared-coupling comment actually gets written — AC3's value is entirely in surviving the next reader, which no run in this repo will demonstrate for months.

## /meta-amend outcome — amended Commit `2e0e43e` on `main`. ### Verified at HEAD (`8440c90`) The gap held. Nothing in `/dev:develop` required a pass over real content, and the nearest existing rule did not cover this case: `qa-playbook.md` §6's *"Synthetic fixtures at an external boundary"* reasons from **"the repo holds the schema, not the rows"** — the seeded-datastore case, where the fixture is a guess because the real data lives elsewhere. Content committed in the tree is the **inverse**: the real set is right there, which makes a fixture a worse excuse, not the same one. And §6 is a QA-time audit, one phase after the point where the defect was cheap. ### The amendment **`develop/SKILL.md` 3.7 step 1b — Real-content smoke check**, placed after the wide-tier suite run and before the terminal Phase Outcome (AC2). Lettered `1b` deliberately: the file already uses `2b`/`2c`, and renumbering would have broken the `3.7 step 2` reference at SKILL.md:49. - **Trigger (AC1):** the feature *reads, parses, renders, or transforms content committed in this repository* — docs, release notes, seed/config/schema files, fixtures-as-product, content collections, migrations; anything the shipped artifact serves from the tree rather than from a request. If it doesn't apply, say so in one line and move on. - **The check (AC1):** one pass of the feature's real code path over the **whole committed set** — not a sample, not a fixture — with the output inspected. A crash, a swallowed item, or visibly wrong structure is a **failure of the step**, handled exactly like a failing test in step 1 (spawn the fix teammate). Fixtures stay the unit-level default; this is explicitly framed as a smoke check on the artifact that ships. - **AC3 — the test declares its coupling:** a comment at the top, in the project's own comment syntax, carrying *both* halves — it reads the real set rather than a fixture, **and** content edits may legitimately change it. The rationale is written down: without the declaration the next reader sees brittle data-coupling and swaps in a fixture, which is the defect coming back; and the brittleness is a real cost the person who trips over it needs to recognise as deliberate. - **One clause the issue's second incident earned:** *assert properties over the set, not a snapshot of it.* "Every committed item parses", "nothing is dropped" survive the next content addition; "exactly the two entries currently committed" does not — and that exact test was written on this same feature and broke at `/promote` when v0.2.1.md landed. Both halves of the issue's evidence are now covered by the rule, not just the first. - **Carry-forward:** a new Finding trigger in step 3 for *the check applied but could not be run* (content not committed yet, code path undrivable outside a deployment). Rationale in the text: otherwise QA inherits a green suite that has only ever seen fixtures and has no way to know. A check that ran needs no Finding, and a defect it found was fixed in 1b like any other failing test. - `qa-playbook.md` §6 gains the inverse-fixture bullet, pointing at 1b rather than restating it, and telling the audit two things it would otherwise get wrong: pick up the pass when 1b is absent or a Finding says it couldn't run, and **do not flag a declared real-content test as brittle coupling**. ### Gate `scripts/lint-conventions.sh` — clean. ### Desk-check against the originating scenario (verity #230) The feature rendered a release-notes page from `release-notes/v0.1.0.md` and `v0.2.0.md` — content committed in the repo, so the trigger fires unambiguously. Step 1b runs the parser over both files and inspects the output; wrapped bullets breaking into stray paragraphs is "visibly wrong structure", so the step fails and the fix teammate is spawned **inside `/dev:develop`**, before the terminal Phase Outcome — roughly forty minutes and one phase earlier than where QA actually caught it, and in the phase that already owns a fix-teammate path. The second incident checks out too: the QA-added test asserting *"exactly the two committed entries"* is precisely what the properties-over-snapshot clause forbids, and it is the one that broke at `/promote` on the next release file. Worth stating plainly: this does **not** claim the four fix rounds (#40) disappear. The parser bug still has to be fixed and is still a hard one. What moves is *when* it is found — inside the phase that can cheaply fix it, rather than forty minutes into QA against a suite that had already declared itself green. ### What validates it next The next feature that consumes committed content. Two things to watch: whether the trigger is recognised at all (it is a self-assessment, and "we have fixtures for that" is the tempting wrong answer), and whether the declared-coupling comment actually gets written — AC3's value is entirely in surviving the *next* reader, which no run in this repo will demonstrate for months.

No milestone

No project

No assignees

1 participant

Notifications

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference

jbr870/devwork-skills#48

No description provided.

Rows
Columns