Turning LLM-Assisted Code Review into a Bounded, Measured System
Personal project: making review behavior explicit, reproducible, measured, and portable
This is a personal, self-initiated systems project, not professional production experience. It does not replace human review or claim production adoption; it explores the engineering boundaries required for dependable LLM-assisted review.
Problem
LLMs can produce useful observations about code, but useful observations alone do not make a dependable reviewer. Once a reviewer participates in an engineering workflow, the difficult questions are about the system around the model: what was reviewed, which revision the result applies to, who owns the review, what counts as blocking, whether an old finding still holds, and which actions the reviewer is allowed to take.
I built this project to make those behaviors explicit rather than leaving them to an ad-hoc conversation or a model's interpretation on each run.
Context
The repository provides two portable Code Review Agent Skills. One reviews a local implementation delta before a pull request exists; the other reviews an existing GitHub pull request and can deliver either a passive report or, when independently authorized, a GitHub review. Both consume one shared review standard while keeping their target-specific inspection and delivery mechanics separate.
Constraints
- Local and GitHub review needed to apply the same evidence, severity, and blocking standards without pretending their targets or delivery mechanisms were identical.
- Review authority had to stay bounded: implementation code remains read-only, self-review analysis cannot become self-approval, and a reasoned verdict does not itself grant permission to mutate GitHub.
- Every result needed to refer to a defined target and revision. A changed pull-request HEAD cannot inherit a previous green result without review.
- Prior comments and resolved findings could inform a review, but could not be treated as automatically true against changed code.
- The core behavior needed to remain portable across Agent Skills-compatible runtimes rather than depend on one provider's orchestration features.
My Role
Sole designer and implementer: defining the review semantics, policy boundaries, skill-specific runbooks, validation and test strategy, packaging model, capability architecture, evaluation infrastructure, and documentation architecture for both review surfaces.
Architecture / Approach
The project reads as three layers built in sequence: bound the reviewer, measure the reviewer, then scale the reviewer without loosening those bounds.
Shared Review Standard
├── Local Code Review
└── GitHub PR Review
Capability Architecture (declarative ownership, progressive loading)
└── capability.yaml per capability -> generated package manifest
Benchmark / Evaluation Subsystem
├── deterministic structural corpus tests
└── behavioral proof against the live reviewer
Capability architecture and declarative ownership. Individual policy areas, such as parallel execution, GitHub publication, scale-driven review depth, or specialist-depth analysis, are each declared by a capability.yaml manifest: what it loads and when, the activation predicate that gates it (fail-closed by default), the policy files it owns, the other capabilities it requires, explicit negative constraints, and a pointer to the benchmark corpus that proves it. A generator derives the packaging manifest directly from those declarations, and a check fails the build if the committed package manifest and a freshly generated one diverge. That replaced an earlier state where packaging and scope lived in hand-maintained files that had to be kept in sync by convention rather than by construction.
Progressive loading. Everything a Skill's policy files describe is instruction text loaded into the model's context, so loading every capability unconditionally is expensive even when most of it is irrelevant to a given review. Progressive loading makes each capability's instructions conditional on its own activation predicate, so a review only pays for the policy surface it actually needs.
Evaluation subsystem. A benchmark corpus of realistic review scenarios, spanning correctness, security, quality, no-op, and several capability-specific structural cases, backs two distinct kinds of validation that answer different questions. Deterministic tests assert corpus and fixture shape, never an LLM judgment. Behavioral proof runs the unmodified Skills through the real reviewer path against real fixtures and scores the output against the corpus using a match/near-miss/no-match rubric on location and defect axes, with a stored baseline for regression comparison and drift detection that opens a tracking issue when live behavior diverges from that baseline. Execution is scheduled rather than ad hoc: a lighter sentinel pass runs every few days and a fuller pass runs weekly, replacing an earlier per-pull-request CI check that was retired once scheduled execution covered the same ground more cheaply.
Key Decisions
- Make governance a first-class part of review. Reviewer ownership, read-only boundaries, target scope, action authority, and re-review behavior are explicit contracts rather than assumptions surrounding the model invocation.
- Derive the decision mechanically from severity, and make that mechanism unavoidable. P0 and P1 findings block; P2 findings do not, and the final decision is derived once from the reconciled finding set. A real regression, described below, showed that stating this rule in prose was not enough on its own.
- Bind review state to an exact revision. The GitHub review records and revalidates the reviewed HEAD before delivery. If the HEAD changes, the earlier result remains evidence about an older state, not approval of the new one.
- Treat review history as evidence, not truth. Prior findings, resolved conversations, and settled decisions are classified against the current target. They can be reused, marked resolved, or re-evaluated, but are never copied forward mechanically.
- Make severity-adjacent signals deterministic, not advisory. Review depth (standard/elevated/deep) is derived from a fixed catalog of risk signals and thresholds, and every finding carries a closed-set confidence value. Both are inspectable metadata: neither can move a finding's P0/P1/P2 severity or the mechanically derived decision, so richer signal never becomes a backdoor for judgment to drift.
A Real Regression: When a Clean P2 Finding Blocked Anyway
Dogfooding the reviewer against its own repository surfaced a case where a review containing only a single P2 finding rendered a blocking verdict instead of the mechanically required clean result, contradicting the project's own decision rule.
The policy prose describing "P0/P1 block, P2 does not" was correct, but nothing in the review procedure forced the model to state an explicit P0/P1 tally before writing the decision. Without that forcing step, the decision could drift toward the apparent severity of the wording rather than the actual counted severity, even with a worked P2-only-clean example already present in the reference material. The fix moved the rule from prose into procedure: both skills' decision steps now require stating the concrete P0/P1 tally as the explicit basis for the decision, and the P2 count is excluded as an input by construction, not by convention.
That fix was then encoded as regression evidence rather than left as a one-off patch: a small benchmark corpus adds a P2-only case with mild wording, a P2-only case with deliberately urgent wording (to catch wording-driven drift specifically), and a zero-finding clean case, all exercised through the real reviewer path rather than a unit test that only checks fixture shape. The failure mode is now something the evaluation subsystem watches for on every scheduled run, not something that depends on remembering to check it by hand.
Evidence-Driven Progressive Loading
Rather than rolling progressive loading out across every capability at once, the project treated it as a hypothesis to test on one capability first. The sequence was: record a baseline of what a review always loads regardless of relevance, extract one capability's instructions behind an activation predicate, measure the resulting reduction in loaded instruction surface, run a live behavioral proof to confirm the unmodified Skills still reviewed correctly with the capability loading conditionally, and only then decide, at an explicit checkpoint, whether to generalize.
The measured baseline for the GitHub PR review Skill is roughly 81,000 words of instruction loaded unconditionally before any conditional loading existed. Extracting the first capability behind an activation predicate reduced the instruction surface a review must apply, when that capability is not relevant, by roughly 8 to 11 percent depending on which Skill invoked it. The checkpoint after that first extraction deliberately did not greenlight a batch rollout: it authorized extending the same pattern to exactly one more capability, not the remaining set, so that each generalization step is backed by its own measurement rather than inherited confidence from the last one. That second capability's extraction, measured the same way, produced a smaller reduction, roughly 3 to 5 percent, consistent with a capability whose instructions are already a smaller share of the total. The broader migration remains partial by design: only a couple of the project's declared capabilities have been carried through this full measure-then-generalize cycle so far, with the rest still loading unconditionally until each earns the same evidence.
Trade-offs
Explicit policies, capability manifests, and behavioral evaluation make the repository considerably larger than a compact skill definition, but that extra structure is the point of the project: review behavior, packaging boundaries, and instruction-loading behavior can all be inspected, tested, measured, and changed without silently altering unrelated semantics. Runtime-neutral packaging also avoids provider lock-in, at the cost of keeping optional runtime capabilities outside the portable core and always preserving a sequential fallback.
The system deliberately does not execute code from a reviewed repository. That limits dynamic verification, but preserves the read-only boundary and avoids treating access to untrusted changes as permission to run them.
Behavioral benchmarks are bounded empirical evidence about the fixtures they cover, not proof of correctness for every possible review. The evaluation subsystem surfaces this honestly on its own: its structural defect-matching axis is, per the project's own drift analysis, not populated by real reviewer output in practice, so exact-match scoring falls back to a coarser text-similarity heuristic more often than the design intended. That is treated as a known limitation of the benchmark's precision, not a settled result.
Impact / Outcome
The result is no longer just two packaged review skills. It is a bounded review system: one shared evidence, severity, and decision contract behind two review surfaces; declarative capability ownership that generates its own packaging manifest instead of relying on hand-kept files staying in sync; progressive loading that pays only for the policy surface a review actually needs, with measured evidence behind each extraction rather than assumption; and a benchmark and evaluation subsystem that distinguishes deterministic structural correctness from behavioral correctness, tracks drift against a stored baseline, and runs on a recurring schedule rather than only when someone remembers to run it. The project demonstrates how a natural-language runtime artifact can still be developed with conventional engineering discipline: canonical ownership, explicit contracts, tests, compatibility boundaries, reproducible packaging, empirical evaluation, and a versioned release process with deterministic SemVer classification from the changelog.
The project has also picked up early external distribution: the recipe was accepted and shipped on FlowStacks as code-review-skill-independent-verdict-contract, where its keyless verification gate passes 4/4 fixtures, and it is listed in Awesome AI Workflows. It was previously accepted into two external Agent Skill directories, Kodus's Awesome Agent Skills and open-agent-craft/awesome-agent-skills. This is distribution and listing evidence, not a production-adoption or usage-scale claim.
Lessons
- The hard part is governance, not bug finding. Model capability does not answer who owns a review, which actions are authorized, or what state a decision refers to.
- A review decision needs identity. A result is meaningful only when it is tied to a particular target and revision; a moving branch or pull-request HEAD cannot inherit trust by implication.
- A correct rule in prose is not a correct rule in practice. The P2-only regression happened with the right policy already written down; fixing it needed a forcing procedural step, not better wording, and a regression case that would catch it again.
- Deterministic tests and behavioral benchmarks answer different questions. Structural tests confirm a fixture or policy file has the shape it claims to have. Behavioral evaluation is the only way to find out whether the actual reviewer, run against real input, behaves the way the policy says it should.
- Probabilistic reviewer behavior needs bounded empirical evaluation, not just contracts. Explicit rules constrain what the system is allowed to do; measuring it against a corpus is what shows whether it is actually doing that, and by how much a change moved the needle.
- Architecture should earn generalization through measurement, not momentum. One successful extraction is a data point, not a mandate to roll the same pattern out everywhere. The project's own checkpoint discipline, generalize by exactly one step and re-measure, is what keeps that distinction real instead of aspirational.
Current Direction
Capability ownership today is mostly logical: each capability's capability.yaml declares what it owns, but the files it points to and the benchmark corpus that proves it still live in shared, general-purpose directories rather than directories scoped to that capability. The current direction is turning that logical ownership into physical repository structure incrementally, one deliberate step at a time rather than as a single restructuring pass. The benchmark and evaluation subsystem is being separated out from general documentation first, as its own owned area, and capability-scoped benchmark corpora are expected to move alongside their owning capabilities as that separation lands, rather than all at once.
The incremental pace is deliberate, not just cautious by default. During design work for the target directory layout, a proposed top-level platform/ directory was checked against the language runtime it would ship in and found to collide with Python's own standard-library platform module: a top-level package on the import path with that name would have shadowed the stdlib module for every later import platform in the same process, including a sandbox-capability-detection call site that depends on it for a security-relevant check. That was caught and corrected to runtime_platform/ at the design stage, before any files were moved, and documented as the reasoning behind the rename rather than left as a silent decision. It is a small example of a habit this project tries to hold itself to: treat a structural migration as something to verify against the concrete runtime it targets before it touches a single file, not just as something to be careful about in the abstract.