An honest look at CodeRabbit, from someone who reviews real MRs
I review Drupal merge requests for a living — the kind where a well-meaning hook_form_alter quietly breaks three unrelated forms — so when a robot offered to take the first pass for me, I was equal parts curious and defensive. CodeRabbit has been on my radar for a while, so I finally sat down to work out what it actually is, what it is genuinely good at, and where it gets on your nerves.
What CodeRabbit actually is
CodeRabbit is AI-powered code review that plugs straight into your pull and merge requests. It installs in a couple of clicks on GitHub, GitLab, Azure DevOps and Bitbucket, with editor extensions for VS Code, Cursor and Windsurf, plus a CLI for people who live in the terminal.
The company throws big numbers around — it claims 6M repositories analysed, 75M defects found, and 15,000+ customers including NVIDIA. Importantly, it is not trying to replace your linters. It appears to sit on top of them, wiring in tools like ESLint, Ruff and Semgrep, then adding an LLM layer that reasons about the diff in the context of your codebase.
How it behaves on a PR
Open a pull request and CodeRabbit does a handful of things:
- Writes a TL;DR summary and a walkthrough of the change, sometimes with a small sequence diagram.
- Leaves line-by-line inline comments with severity levels, and offers a Fix with AI button.
- Lets you argue back — you chat with the bot, and it records "learnings" so it stops nagging about things your team has already settled.
- Supports path and AST-based instructions, custom checks, and will generate unit tests and docstrings on request.
An AI reviewer that never gets tired of leaving comments is a blessing and a threat in exactly equal measure.
The noise question
Here is the honest part: CodeRabbit has a reputation for being chatty. An independent audit by the Lychee project looked at 28 reviewed PRs and classed the comments as roughly 35% genuine improvements, 21% nitpicks and 15% useless, with the rest somewhere in between. A Martian benchmark reportedly ranked it last on precision for offline PRs. The blunt Reddit verdicts run along the lines of "very verbose," "half of it useless," and "ended up wasting more time than it saved."
CodeRabbit is not deaf to this. It appears to run a separate judge model that scores each finding and drops ones it cannot ground, and it ships three review profiles — quiet, chill (the default) and assertive — plus a YAML config so you can point it at what you actually care about:
reviews:
profile: chill
high_level_summary: true
auto_review:
enabled: true
drafts: false
path_filters:
- "web/modules/custom/**"
- "!web/**/*.generated.php"
path_instructions:
- path: "web/modules/custom/**/*.php"
instructions: |
Enforce Drupal coding standards and dependency injection.
Flag direct \Drupal:: calls in classes that could inject services.
tools:
phpstan:
enabled: trueWould I put it on my Drupal MRs?
Probably — with the dial turned down and expectations set. On pricing, as stated on their site: public open-source repos are free forever, the Free tier covers public and private repos with a 14-day Pro Plus trial, Pro is $24 per user per month billed annually, Pro Plus is $48, and Enterprise is custom with a self-hosting option. It appears to be SOC 2 Type II certified with zero data retention after review, which matters if your day job has an infosec team.
My take: treat it as a tireless first-pass filter that catches the boring stuff before a human looks — not as a substitute for someone who understands why the change exists. The real risk isn't false positives; it's over-reliance, the quiet temptation to rubber-stamp whatever the bot already approved.
If you want to see it move rather than read about it, the official channel has a short VS Code walkthrough: