Add audit subcommand for rule review-and-promote workflow #9

New issue

Open

opened 2026-04-20 19:37:39 +00:00 by jbr870 · 0 comments

jbr870 commented

2026-04-20 19:37:39 +00:00

Owner

Summary

Add a claude-permit audit subcommand that turns audit-log observations into actionable rule candidates. Complements #1 (stats subcommand): #1 answers "what happened?", this one answers "what rule should I add next?".

The system already auto-promotes GREEN LLM decisions to auto-rules.toml. The review-and-promote workflow is about the cases auto-promote deliberately skips — patterns that keep hitting the LLM tier (YELLOW / latency / token cost) or pass through to Claude Code's built-in rules without a deterministic match.

Relationship to #1

Overlap: both parse audit.jsonl, both do frequency aggregation and time windowing.
Distinct value: this issue produces rule suggestions (concrete TOML snippets) and backtests (how many historical entries a proposed rule would have matched). #1 produces numeric summaries only.
Implementation hint: if #1 ships first, this can build on its audit-parsing code — share, don't duplicate.

Also related: the existing /audit skill already provides an interactive review workflow. This subcommand is the non-interactive, CLI-reportable counterpart.

What's needed

1. Analysis

claude-permit audit parses audit.jsonl and surfaces actionable insights:

Most frequent passthrough commands (candidates for new allow rules)
Most frequent LLM evaluations with their scores (GREEN = should be a rule, YELLOW / RED = worth reviewing)
Commands that repeatedly hit the LLM tier (wasting tokens / latency)
Grouping by command pattern (e.g., all git diff variants, all npm run variants)

2. Rule suggestion

Given a passthrough / LLM-tier pattern, emit a concrete config.toml rule (tool regex + field regex) that would catch it. Include a dry-run / backtest:

"This rule would have matched N entries from the last 7 days."

3. Documented workflow

Step-by-step guide for the review-and-promote cycle: when to do it, what to look for, how to validate new rules before committing them.

Design considerations

Keep it simple — a CLI report that outputs to stdout is sufficient to start. No TUI, no database.
The audit log can grow large. Filtering by date range and event type is essential.
Rule suggestions should be conservative: prefer specific patterns over broad ones to avoid over-permitting.

Context

Severity: Feature gap
Affected: User workflow, config.toml
Discovered: 2026-02-26
Migrated from devwork/inbox.md.

## Summary Add a `claude-permit audit` subcommand that turns audit-log observations into actionable rule candidates. Complements #1 (stats subcommand): #1 answers *"what happened?"*, this one answers *"what rule should I add next?"*. The system already auto-promotes GREEN LLM decisions to `auto-rules.toml`. The review-and-promote workflow is about the cases auto-promote deliberately skips — patterns that keep hitting the LLM tier (YELLOW / latency / token cost) or pass through to Claude Code's built-in rules without a deterministic match. ## Relationship to #1 - **Overlap:** both parse `audit.jsonl`, both do frequency aggregation and time windowing. - **Distinct value:** this issue produces *rule suggestions* (concrete TOML snippets) and *backtests* (how many historical entries a proposed rule would have matched). #1 produces numeric summaries only. - **Implementation hint:** if #1 ships first, this can build on its audit-parsing code — share, don't duplicate. Also related: the existing `/audit` skill already provides an interactive review workflow. This subcommand is the non-interactive, CLI-reportable counterpart. ## What's needed ### 1. Analysis `claude-permit audit` parses `audit.jsonl` and surfaces actionable insights: - Most frequent passthrough commands (candidates for new allow rules) - Most frequent LLM evaluations with their scores (GREEN = should be a rule, YELLOW / RED = worth reviewing) - Commands that repeatedly hit the LLM tier (wasting tokens / latency) - Grouping by command pattern (e.g., all `git diff` variants, all `npm run` variants) ### 2. Rule suggestion Given a passthrough / LLM-tier pattern, emit a concrete `config.toml` rule (tool regex + field regex) that would catch it. Include a **dry-run / backtest**: > "This rule would have matched N entries from the last 7 days." ### 3. Documented workflow Step-by-step guide for the review-and-promote cycle: when to do it, what to look for, how to validate new rules before committing them. ## Design considerations - Keep it simple — a CLI report that outputs to stdout is sufficient to start. No TUI, no database. - The audit log can grow large. Filtering by date range and event type is essential. - Rule suggestions should be conservative: prefer specific patterns over broad ones to avoid over-permitting. ## Context Severity: Feature gap Affected: User workflow, `config.toml` Discovered: 2026-02-26 Migrated from `devwork/inbox.md`.

jbr870 added the

observability

enhancement

labels

2026-04-20 19:37:39 +00:00

jbr870 referenced this issue

2026-04-20 19:42:10 +00:00

Add stats subcommand #1

jbr870 referenced this issue from a commit

2026-04-20 20:05:18 +00:00

docs(inbox): migrate inbox items to forge issues

jbr870 referenced this issue