Turning LLM-Assisted Code Review into a Bounded System

This is a personal, self-initiated systems project, not professional production experience. It does not replace human review or claim production adoption; it explores the engineering boundaries required for dependable LLM-assisted review.

Problem

LLMs can produce useful observations about code, but useful observations alone do not make a dependable reviewer. Once a reviewer participates in an engineering workflow, the difficult questions are about the system around the model: what was reviewed, which revision the result applies to, who owns the review, what counts as blocking, whether an old finding still holds, and which actions the reviewer is allowed to take.

I built this project to make those behaviors explicit rather than leaving them to an ad-hoc conversation or a model's interpretation on each run.

Context

The repository provides two portable Code Review Agent Skills. One reviews a local implementation delta before a pull request exists; the other reviews an existing GitHub pull request and can deliver either a passive report or, when independently authorized, a GitHub review. Both consume one shared review standard while keeping their target-specific inspection and delivery mechanics separate.

Constraints

• Local and GitHub review needed to apply the same evidence, severity, and blocking standards without pretending their targets or delivery mechanisms were identical.
• Review authority had to stay bounded: implementation code remains read-only, self-review analysis cannot become self-approval, and a reasoned verdict does not itself grant permission to mutate GitHub.
• Every result needed to refer to a defined target and revision. A changed pull-request HEAD cannot inherit a previous green result without review.
• Prior comments and resolved findings could inform a review, but could not be treated as automatically true against changed code.
• The core behavior needed to remain portable across Agent Skills-compatible runtimes rather than depend on one provider's orchestration features.

My Role

Sole designer and implementer: defining the review semantics, policy boundaries, skill-specific runbooks, validation and test strategy, packaging model, and documentation architecture for both review surfaces.

Architecture / Approach

The architecture separates shared review semantics from surface-specific execution:

Shared Review Standard
    ├── Local Code Review
    └── GitHub PR Review

The shared layer owns scope, evidence requirements, P0/P1/P2 severity, the mechanical blocking rule, review context, prior-review evidence, ownership, and the canonical finding shape. The local skill applies those contracts to committed, staged, unstaged, and untracked Git changes and returns one structured report. The GitHub skill applies the same contracts to a pull-request delta, then handles GitHub-specific concerns such as exact-HEAD revalidation, inline placement, authorization, and batched delivery.

Repository context, optional requirement context, and existing review evidence remain distinct from the review target. They can focus reasoning and explain intent, but they never widen the change whose defects determine the result. Candidate findings from sequential or parallel analysis also pass through one reconciliation stage, so deduplication, final severity, and the decision have a single owner.

Key Decisions

• Make governance a first-class part of review. Reviewer ownership, read-only boundaries, target scope, action authority, and re-review behavior are explicit contracts rather than assumptions surrounding the model invocation.
• Derive the decision mechanically from severity. P0 and P1 findings block; P2 findings do not. The final clean or blocking result is derived once from the reconciled finding set, so approval semantics cannot drift into a separate free-form judgment.
• Bind review state to an exact revision. The GitHub review records and revalidates the reviewed HEAD before delivery. If the HEAD changes, the earlier result remains evidence about an older state, not approval of the new one.
• Treat review history as evidence, not truth. Prior findings, resolved conversations, and settled decisions are classified against the current target. They can be reused, marked resolved, or re-evaluated, but are never copied forward mechanically.
• Keep shared policy central and delivery layers thin. One canonical policy set prevents local and GitHub review from developing different quality standards, while each skill retains only the mechanics specific to its review surface.

Trade-offs

Explicit policies and validation make the repository larger than a compact skill definition, but that extra structure is the point of the project: review behavior can be inspected, tested, packaged, and changed without silently altering unrelated semantics. Runtime-neutral packaging also avoids provider lock-in, at the cost of keeping optional runtime capabilities outside the portable core and always preserving a sequential fallback.

The system deliberately does not execute code from a reviewed repository. That limits dynamic verification, but preserves the read-only boundary and avoids treating access to untrusted changes as permission to run them.

Impact / Outcome

The result is a pair of independently packaged review skills with one review standard, explicit local and GitHub targets, evidence-backed finding contracts, shared severity and decision semantics, review/re-review workflows, and validation covering policy structure, decision behavior, packaging boundaries, and GitHub-specific state handling. The project demonstrates how natural-language runtime artifacts can still be developed with conventional engineering discipline: canonical ownership, explicit contracts, tests, compatibility boundaries, and reproducible packaging.

Lessons

• The hard part is governance, not bug finding. Model capability does not answer who owns a review, which actions are authorized, or what state a decision refers to.
• A review decision needs identity. A result is meaningful only when it is tied to a particular target and revision; a moving branch or pull-request HEAD cannot inherit trust by implication.
• Prior review history is evidence, not truth. Old conclusions become useful inputs to reconcile against current code, not a substitute for inspecting it.
• Shared semantics prevent surface drift. Local and GitHub review can use different delivery mechanics while retaining one standard for evidence, severity, scope, and blocking behavior.
• Natural-language systems still need software-engineering discipline. Policy ownership, validation, tests, packaging, and compatibility matter even when the runtime artifact is an Agent Skill rather than an application binary.

What I Would Continue Exploring

The project is intentionally evolving as new workflow failure modes become visible. The repository documents two areas that are not yet implemented: loading prior reviewed state to support fully stateful re-review, and movement-tolerant finding identity across revisions. Both require careful failure behavior so ambiguous history falls back to a fresh review instead of manufacturing confidence.