Tricking GitHub Copilot into reviewing PRs in Gitea

Xe Iaso wrote about building a review bot that uses a self-hosted LLM to review GitHub pull requests. When I learned that some "AI" CLIs have SDKs, I wondered whether I could build something similar while leaving the loop and execution environment to a billion-dollar corporation.

Disclaimer: I chose Copilot because it was free, and I liked the novelty of using a GitHub thing somewhere other than GitHub. These companies also don't need more money for a silly experiment.

Contrary to the title, I didn't actually "trick" Copilot. These AI coding CLIs have SDKs that let you interact with them programmatically, kinda like a RESTful API. The API is as non-deterministic as the CLI, and the agent can still run its normal loop and call tools. Copilot was convenient here, but this pattern is common across these kinds of tools.

I should also note that I have reservations about AI tooling in general (a topic for another post that's been sitting in my drafts). This post is about a pattern I found interesting, not an endorsement of any particular product, and especially not an endorsement/condoning of the practices of the companies behind them.

The approach#

Xe's setup triggers via a GitHub Actions workflow when someone comments /reviewbot on a PR. It uses an OpenAI-compatible API backed by a self-hosted model running on a DGX Spark. Its agentic loop can execute Python to analyze the codebase and submit the review. The bot also clones the repo so the model has filesystem access through Python execution.

Instead of an Actions workflow, mine starts from a webhook listener that watches for a specific pattern in PR comments and fires off a review when it matches. A Go process fetches the PR's diff and existing comments, then spins up the Copilot CLI with a prompt built from that context and a set of custom tools. The agent explores the codebase and calls the tools as needed, and when it's done, the review summary gets posted back to the PR.

The tools#

The tools translate the agent's feedback into the specific API calls Gitea expects. The agent has no idea it's talking to Gitea.

post_inline_comment posts a comment on a specific file and line. Its severity is either blocker (must-fix before merge) or suggestion (recommended improvement). The PR author can triage quickly: address the blockers, consider the suggestions.

note_low_confidence records an observation the agent isn't confident about.

submit_review posts the final summary as a top-level PR comment. The agent calls it once at the end of the review.

The note_low_confidence tool came from Angie Jones' post on teaching Copilot to think like a maintainer. She recommends a confidence threshold above 80%, so the reviewer only comments when it's fairly sure something is wrong. Without that, AI reviewers dump every observation as an equally-weighted comment, and people learn to ignore the bot fast.

Note: While we may describe these machines as "thinking", it is important to recognize our anthropomorphization of what is really statistical analysis being done by the model.

I agree with the threshold idea, but my confidence in these models isn't high enough to trust their weighting. If I were building a product or running this somewhere that noisy bot comments would overload developers, I'd probably have the bot stay silent below the threshold. For an experiment, though, I wanted to see everything the model had to say and judge it myself.

Low-confidence observations go into a collapsible section at the bottom of the main review. They stay out of the way while remaining available to anyone curious. This way, I get to be my true Hannah Montana self and have the best of both worlds.

The system prompt#

Angie's post was full of great ideas, like telling the reviewer to skip linting, since CI already catches those errors and does a better job of it.

The prompt also explains the confidence model (high confidence gets an inline comment, low confidence gets a note_low_confidence call) and includes the expected output format for the final summary.

PR context goes in as a structured template: title, author, branches, description, the diff, existing comments (filtered to exclude bot noise). If the diff is too large, it's truncated with a note telling the agent to use git diff on specific files. The agent has the full cloned repo, so it can always look at more than what's in the prompt.

The feedback loop#

Once the agent is done, the review summary comment gets updated. Low-confidence notes are appended in a collapsible <details> section, and usage metadata (token counts, model used, API calls, cost if available) goes in another collapsible section at the bottom.

I include the usage metadata so that anyone reading a bot-generated review can see what produced it and roughly what it cost. A line like "claude-sonnet-4.5, 12 API calls, 45k input tokens" makes the resource use concrete, and hopefully nudges future contributions to stay focused. If a contribution is too large, it can overflow the context window and the review gets less useful.

Beyond Copilot#

Very little of this is Copilot-specific because the SDK, model, and hosting can all change. The custom tools adapt the review loop to Gitea.

Code is left as an exercise to the reader. Although, if you are a VC, I am happy to send you my routing number in exchange for $10 Billion for this unicorn.