How to review every pull request as commits land

Human review arrives in a batch, late, after the author has moved on. An agent can read each push as it lands instead. The risk is noise: a naive bot re-reviews the entire diff on every commit, and the team mutes it within a week.

Code review is built for this. It reviews only the commits it has not seen yet, it never re-comments a line it already reviewed, and it files a finding only where it can name the input that breaks the line.

Turn it on

Code review is one organization-wide setting, off by default. Enable it on the Settings tab of /reviews in the dashboard. No YAML required: every pull request in the organization is then reviewed by the built-in pipeline.

That pipeline is two agents. A cheap pr-description agent keeps the pull request description's summary current. The bugs reviewer reads the range on the frontier model, briefed to find defects: logic errors, unhandled edge cases, error handling, concurrency, security holes, regressions in behavior callers depend on, and migration hazards. It reads the surrounding code, not the diff alone, because most real findings depend on a caller or a guard the diff does not show.

What keeps that from becoming noise is the reviewer's own bar, not a stage after it. It may file a finding only when it can point at the line that is wrong and name the input or state that breaks it, so a hedge is a finding it deletes rather than softens, and its comments run a few sentences with the fix in a GitHub suggestion where one fits. If you want an adversarial second read of its findings anyway, add a gatekeeper.

When priya-shah opens "Expire pending trades on account deactivation" and pushes three more commits over the afternoon, the pipeline runs four times, each time reading only what that push added. The third push drops a rest-period check and the comment lands on that line, with no repetition of the first review.

Scope it with a pipeline file

A file is only needed to change something, and where you commit it decides what it governs. Commit code_review.yaml to the root of your .ellipsis repository to scope review across the whole organization:

ellipsis:
  version: v1
  kind: code_review
  name: Backend review

pull_requests:
  repositories: [splitshift-api, splitshift-web]
  base: [main, release/*]
  draft: false
  for:
    users: true
    bots: false

kind: code_review is what makes this a pipeline rather than an agent config, and it is required. Everything else is optional: this file changes the watch set and inherits the built-in reviewer, the description agent, and the budget from the platform.

A committed file makes its filters authoritative for the repositories it governs. Scoping this file to splitshift-api and splitshift-web means "and nothing else", so a pull request in splitshift-mobile gets no review at all rather than falling back to the unscoped default. That is the point of scoping: Ellipsis never reviews, or bills you for, something your file excluded.

for.bots: false is what turns bot pull requests off. The built-in pipeline reviews them, and declaring a pull_requests: block narrows the audience to humans unless the block writes for back out, so this file has to say what it means either way.

A single repository that needs something different commits its own root code_review.yaml, which replaces this file for that repository, filters included. So splitshift-mobile can review itself even though the organization-wide file left it out. pull_requests.repositories belongs only here, in the .ellipsis repository's copy: a repository's own file already reviews only its own pull requests, and declaring the key there is a configuration error.

Add a specialist reviewer

Declaring review: replaces the built-in reviewer, so a pipeline that wants a specialist alongside a general review declares both. Here the specialist only starts when a migration changes:

ellipsis:
  version: v1
  kind: code_review
  name: Backend review with a migration specialist

pull_requests:
  repositories: [splitshift-api]

review:
  # The general pass, on every pull request the pipeline matches.
  - name: correctness
    claude:
      system: |
        Review this diff for defects a reviewer would ask to have fixed
        before merge: logic that does not do what the change intends,
        broken error handling, and regressions in behavior.
  - name: migration-safety
    claude:
      model: claude-opus-4-8
      system: |
        You review database migrations for production safety.

        Check for locking behavior that blocks writes on a large table,
        missing backfills for new non-null columns, missing indexes on
        new foreign keys, and rollout ordering that breaks if the
        migration and the code deploy in the wrong order.

        Comment on the specific line with the specific change you want.
        A migration that is safe as written gets a one-line
        confirmation, not invented concerns.
    pull_requests:
      paths: ["**/migrations/**"]

budget:
  run: 12.00

Two reviewers now run on a migration pull request, and only correctness on the rest. The built-in bugs reviewer does not run: declaring review: replaces it, and there is no way to append to it, so anything you want beside a specialist is a reviewer you write.

Per-reviewer filters

migration-safety carries its own pull_requests: block, so it only runs when a migration is actually in the diff. A reviewer that filters out costs nothing: no sandbox starts and no model call is made, and its share of the run budget goes to the reviewers that did run.

A reviewer's block takes paths, base, and head only. repositories belongs to the pipeline, which owns the watch set. When dani-okoro adds a non-null expired_at column, migration-safety runs alongside correctness and the review lands before anyone else looks:

migrations/0042_expired_at.py:14 — adding a NOT NULL column with a
default rewrites the whole table under an ACCESS EXCLUSIVE lock.
On trade_requests (~50M rows) that blocks writes for minutes.

Split it: add the column nullable, backfill in batches, then set
NOT NULL in a follow-up migration once the backfill completes.

On a pull request that touches no migrations, that reviewer never starts.

Run only your own reviewer

Declaring a single reviewer is all it takes, since review: replaces the built-in one. That is the right shape when your own prompt is the review:

ellipsis:
  version: v1
  kind: code_review
  name: House review

pull_requests:
  repositories: [splitshift-api]

review:
  - name: house-rules
    claude:
      model: claude-haiku-4-5-20251001
      system: |
        Review this diff against the conventions in CONTRIBUTING.md
        and the service patterns in docs/architecture.md. Comment only
        where the diff departs from a documented convention, and cite
        the document. Skip anything a linter catches.

budget:
  run: 4.00

A pipeline may declare at most 8 reviewers. Asking for more is a validation error rather than a silent trim. Reviewers run in parallel, so several cost latency only once.

Add a gatekeeper

Findings post as written: the built-in pipeline runs no stage between a reviewer and your pull request. That is the right default for one careful reviewer, and the wrong one once you run several eager ones of your own, because two reviewers word the same defect two ways and mechanical deduplication catches only identical findings. Declare a filter: agent and nothing posts until it approves:

filter:
  name: gatekeeper
  claude:
    model: claude-opus-5
    system: |
      Approve only findings a staff engineer would raise in review.
      Reject style opinions, anything a linter catches, and any claim
      you have not verified against the surrounding code.

It is a second frontier-model read of the same diff, so it roughly doubles a review's cost and its latency, and it spends from the same budget.run. Rejected findings are not lost: they stay on the reviews dashboard with the reason, and the posted review states how many of the drafted comments made it through. Gatekeeper covers tuning it.

Budgets

budget.run caps one pass over one pull request, across every stage agent of that pass together. Reviewers that declare no budget of their own split what is left equally:

budget:
  run: 10.00
  day: 60.00
  week: 200.00

day and week are this pipeline's trailing spend, checked before a review starts. A pipeline over its daily cap reviews nothing until the window rolls, which is the guard against a push storm turning into a bill.

Reviewers fan out, so one push is several sessions. Keep the frontier model on the pass that reads every diff, since that is where a missed defect costs you, and use a cheap one for a narrow specialist whose prompt does most of the work.

Cross-repository context

pull_requests.repositories is the watch set. environment.repositories is what gets cloned. They are independent, so a reviewer can read a pull request in one repository with another checked out for context:

ellipsis:
  version: v1
  kind: code_review
  name: API reviewer with web context

pull_requests:
  repositories: [splitshift-api]

environment:
  repositories:
    - name: splitshift-web
    - name: splitshift-shared

Reviews trigger only on splitshift-api pull requests, and splitshift-web plus splitshift-shared are on disk while reviewing. The triggering repository is always cloned, so it does not need listing.

This is how a reviewer judges a change against callers it does not contain: an API pull request that changes a response shape can be read against the web client that consumes it. It needs no second pipeline file.

One pipeline per pull request

Exactly one pipeline reviews a given pull request, and which repository holds the file decides which: the repository's own code_review.yaml when it has one, otherwise the one in your .ellipsis repository. Two files can never contend for one repository, so nothing has to be tie-broken and no scope has to be kept from overlapping. The .ellipsis repository is never itself reviewed, which is what keeps one filename unambiguous.

A repository file replaces the organization-wide one rather than layering onto it, so anything above that you still want, copy across. And a specialist reviewer is a reviewer inside one pipeline, not a second file: that is what per-reviewer pull_requests: filters are for.

Next

Keep descriptions current on the same pushes with How to keep pull request descriptions up to date, review one narrow class of change with a dedicated agent in How to review database migrations before they merge, or see Budgets for what a cap does when it is reached.