Decision in brief
Require a human owner to explain the behavior, affected dependencies, failure modes, and recovery path for every material change. Use deterministic tests for known invariants and keep automated review advisory. If nobody can explain a consequential code path or prove it behaves as intended, split, rewrite, or hold the patch.

AI-generated code review checklist by change risk
Begin with the issue and expected behavior. Ask the author to show what changed, where execution enters the code, what state can change, and how the old behavior can be restored. Compare this description with the actual diff. Review added dependencies, configuration, data handling, interfaces, and tests that changed alongside implementation.
Use one independently specified regression test. Where practical, run it on the baseline and the candidate patch so you can see that it detects the original failure. Re-read the assertions when the agent authored both code and tests. Give higher-risk changes an owner who knows the subsystem and a tested recovery path before rollout.
| Change or review need | Review workflow | Evidence before proceeding |
|---|---|---|
| Repeatable rule or known invariant | Deterministic CI such as a focused test, type check, or linter | An independent check demonstrates the expected behavior and guards against regression. |
| Author needs private feedback before pushing | Local review of the staged diff where repository data policy permits | A human verifies each consequential finding and checks which context the tool received. |
| Team needs shared pull request feedback | Trial an already approved Copilot or Codex integration, or a dedicated bot such as CodeRabbit | Reviewers can reproduce useful findings, and the pilot bounds permissions, noise, and billing. |
| The patch changes permissions, persistence, or an unfamiliar critical path | Subsystem-owner review with staging and rollback | The owner explains failure cases, verifies required checks, and rehearses recovery. |
What two HN discussions reveal about review effort
In the Ask HN discussion about shipping code people do not understand, cadamsdotcom argued for tests observed failing before implementation. Use this as a review check. Ask for the smallest independent failing case that explains why the change is needed and confirm the reviewer can replay it.
In a separate review thread, ocharles reported finding private pre-review worthwhile. ohans reported finding automatic PR comments noisy in the same discussion. Evaluate author-side and team-visible feedback separately. A suggestion that helps an author may still interrupt every reviewer when posted to a shared queue.
Pilot one review workflow before changing merge policy
- Choose eight closed, sanitized, or synthetic changes. Include clean diffs and benign known regressions in formatting, state transitions, or expected test behavior. Have a maintainer write the expected findings and mark which cases are merge-blocking.
- Run the existing human and CI process, then run one advisory tool on the same commits with approved repository scope. For local tools, verify what context is sent to the provider. For PR integrations, record trigger, feedback visibility, repository permissions, and charges.
- Count reproducible findings, missed blocking cases, false or duplicate comments, and total human triage minutes. On a sample patch, ask an independent reviewer to explain the changed path and replay an acceptance test and the rollback.
- Keep the tool advisory and widen its use only if the pilot retains required human approval and all designated blocking regressions are caught by the combined process. Continue only if review effort or useful coverage improves. Stop on a missed blocker, unapproved access, or unclear ownership of rollback.