Architecting CodeRabbit like code-review AI agent at scale: The Event Storm & Context Engine

Building a tool that reviews code is not just about prompting an LLM. It is a massive data pipeline problem.

At production scale, you face three immediate killers:

1. The Thundering Herd: Massive webhook spikes arriving at peak hours.
2. The Context Trap: An LLM seeing a "diff" doesn't know if that change breaks a file 10 folders away.
3. The Cost Explosion: Sending every whitespace change to your most expensive LLM will bankrupt you.

In Part 1, we architect the Ingestion & Context Engine. We will explore how to survive the traffic spike and how to turn raw code into "understandable knowledge" for an AI Agent.

The Failure Case: What happens without proper architecture?

Before we dive into solutions, let's visualize what happens when you naively build an AI code review tool without these patterns.

graph TD
    A[High Volume Webhooks Peak Hours] --> B[Single API Server]
    B --> C[Synchronous Processing]
    C --> D[LLM Call for Every File]
    D --> E[Slow Response Time]
    E --> F[Webhook Timeout]
    F --> G[Webhook Retries]
    G --> B
    B --> H[Server Overload]
    H --> I[503 Service Unavailable]
    
    D --> J[Expensive LLM Calls]
    J --> K[High Cost per File]
    K --> L[Budget Explosion]
    
    style B fill:#ffebee,stroke:#b71c1c
    style H fill:#ffebee,stroke:#b71c1c
    style I fill:#ffebee,stroke:#b71c1c
    style L fill:#ffebee,stroke:#b71c1c

Observation: Without proper architecture, you get a cascading failure. The slow AI processing causes timeouts, which trigger retries, which amplify the load, which crashes your server. Meanwhile, you're burning $300/hour on reviews that users never see.

---

1. The Ingestion Layer: The "Buffer" Pattern

The Architectural Problem:

GitHub webhooks expect a response within 10 seconds. However, a good AI review takes 1-2 minutes. If you process the review synchronously (while GitHub waits), your server will hang, requests will timeout, and GitHub will retry, causing a generic Retry Storm that takes down your infrastructure.

The Solution:

We decouple Reception from Processing using an Event-Driven Architecture.

The Architecture

graph LR
    subgraph World["The World"]
        GH[GitHub Webhook]
        GL[GitLab Webhook]
    end
    subgraph Shield["The Shield Ingestion Layer"]
        LB[Load Balancer]
        Gateway[Stateless API Gateway]
        Auth[HMAC Signature Validator]
    end
    subgraph Buffer["The Buffer"]
        Queue[(High-Throughput Queue Kafka / Redpanda)]
    end
    subgraph Consumer["The Consumer"]
        Worker[Orchestrator Worker]
    end
    GH -->|HTTP POST Event| LB
    LB --> Gateway
    Gateway --> Auth
    Auth -->|Invalid Signature| Drop[403 Forbidden]
    Auth -->|Valid Event| Queue
    Queue -->|Async Pull| Worker

How it Works:

1. The Dumb Gateway: The API Gateway is deliberately "dumb". It does no logic, no database lookups, and no AI. It only verifies the security signature and pushes the JSON payload to the Queue.

2. The Buffer: The Queue absorbs traffic spikes. It doesn't matter if thousands of events arrive simultaneously; the queue holds them safely.

3. Backpressure: The Worker pulls events at its own pace. If the AI providers are slow, the workers slow down, but the Gateway keeps accepting new events instantly.

Observation: The magic of this pattern is the response time decoupling. GitHub gets a response in 50ms ("Event received!"), while the actual AI work happens asynchronously over the next 90 seconds. The queue acts as a "shock absorber" for your system.

The Traffic Spike: Before vs. After

graph TD
    subgraph Before["Without Buffer Pattern"]
        A1[High Traffic Spike] --> B1[API Server]
        B1 --> C1[Processes Synchronously]
        C1 --> D1[Long Response Time]
        D1 --> E1[Timeout and Crash]
        style E1 fill:#ffebee,stroke:#b71c1c
    end
    
    subgraph After["With Buffer Pattern"]
        A2[High Traffic Spike] --> B2[API Gateway]
        B2 --> C2[Queue]
        C2 --> D2[Workers Pull at Own Pace]
        D2 --> E2[Fast Response Instant]
        D2 --> F2[AI Processing Async]
        style E2 fill:#e8f5e9,stroke:#388e3c
    end

Think About It: Why use Kafka or Redpanda instead of a simple database queue? At scale, you need ordered, partitioned, replay-able events. If a worker crashes mid-processing, Kafka's consumer groups ensure another worker picks up that event without data loss.

---

2. The Filtering Layer: The "Gatekeeper" Pattern

The Architectural Problem:

Roughly 40% of code commits are "noise". They are documentation updates, automated dependency bumps, or lockfile changes. Sending these to an LLM is burning money.

The Solution:

We implement a Gatekeeper Pattern before the AI Agent even wakes up.

The Architecture

graph TD
    Event[Incoming PR Event] --> Filter{Rule Engine}
    
    Filter -->|Is it a Bot?| CheckBot[Check Bot]
    CheckBot -->|Yes Dependabot| Ignore[Drop Event]
    
    Filter -->|File Types?| CheckFiles[Check Files]
    CheckFiles -->|Only .md / .png / .lock| Ignore
    
    Filter -->|PR Size?| CheckSize[Check Size]
    CheckSize -->|Greater than 50 files| HugeQueue[Route to Slow Lane Queue]
    
    CheckBot -->|No| Valid[Valid Code Event]
    CheckFiles -->|Code Files| Valid
    CheckSize -->|Normal Size| Valid
    
    Valid --> AI_Pipeline[AI Pipeline]

How it Works:

• Heuristic Filtering: We use strict rules (Regex, Metadata) to drop low-value events immediately.

• Lane Routing: This is critical for scale. We route "Massive Monorepo PRs" to a separate HugeQueue. This prevents one massive PR from clogging the system and blocking many small PRs from other users.

Observation: The "Gatekeeper" pattern saves you 40% of compute costs by dropping noise before it hits expensive AI models. But the real win is Lane Routing. Without it, a single massive monorepo PR would block the queue while many small PRs from other users wait.

Real-World Example: Traffic Distribution

Let's walk through what happens when a large batch of PRs hits your system simultaneously.

graph TD
    subgraph Incoming["Incoming PRs"]
        A[40 percent Bot PRs]
        B[20 percent Documentation]
        C[5 percent Lockfiles]
        D[30 percent Code Changes]
        E[5 percent Monorepo PRs]
    end
    
    subgraph Filter["Gatekeeper Processing"]
        A --> Drop1[Drop Bot PRs]
        B --> Drop2[Drop Docs Only]
        C --> Drop3[Drop Lockfiles]
        D --> Fast[Fast Lane Queue]
        E --> Slow[Slow Lane Queue]
    end
    
    subgraph Result["Result"]
        Fast --> F1[Code Changes Processed Quickly]
        Slow --> F2[Large PRs Processed Separately]
        Drop1 --> F3[65 percent Filtered Out]
        Drop2 --> F3
        Drop3 --> F3
    end
    
    style Drop1 fill:#ffebee,stroke:#b71c1c
    style Drop2 fill:#ffebee,stroke:#b71c1c
    style Drop3 fill:#ffebee,stroke:#b71c1c
    style F1 fill:#e8f5e9,stroke:#388e3c
    style F3 fill:#fff8e1,stroke:#f57f17

The Impact: Without filtering, you'd process every single file through expensive AI models. With the Gatekeeper, you filter out 65% of noise, dramatically reducing compute costs and improving response times for legitimate code changes.

Think About It: Should you always ignore Dependabot PRs? What if a dependency update introduces a security vulnerability? This is where you'd add a "security scanning" node that runs before the Gatekeeper drops the event.

---

3. The Context Engine: The "GraphRAG" Pattern

The Architectural Problem:

This is the hardest part of AI code review.

If a user changes a function signature in api.ts, looking at only api.ts is not enough. The AI needs to know: "Who calls this function?"

If we don't provide this context, the AI is blind. It cannot detect breaking changes.

The Solution:

We use Graph Retrieval Augmented Generation (GraphRAG). We don't just read the text; we parse the relationships.

The Architecture

graph TD
    subgraph Builder["Context Builder Worker"]
        Diff[Raw Git Diff] --> Parser{Tree-Sitter Parser}
        
        Parser -->|1. Identify Boundaries| Scoper[Scope Expander]
        Parser -->|2. Extract Symbols| SymbolExtract[Symbol Extractor]
    end
    subgraph Graph["The Knowledge Graph"]
        SymbolExtract --> GraphQuery[Query Dependency Graph]
        GraphDB[(Code Graph DB)] -->|Return Callers/References| GraphQuery
    end
    subgraph Window["AI Context Window"]
        Scoper -->|Full Function Body| Prompt[Prompt]
        GraphQuery -->|Related Snippets from Other Files| Prompt
    end

How the AI Agent Works Here:

1. Parsing (Not Reading): We use an Abstract Syntax Tree (AST) parser (Tree-sitter) to understand the code structure. We don't see "Line 10 changed". We see "Method calculate_total in Class Cart changed".

2. Scope Expansion: Standard git diff only shows the changed lines. The AI needs the whole function to understand logic. We programmatically expand the selection to include the full parent function or class.

3. The Graph Walk: The system extracts the modified symbols (e.g., function names). It queries a pre-built dependency graph to find external files that import or call these symbols. It fetches snippets of those external files and stuffs them into the prompt.

Result: The AI now sees the change and the potential blast radius.

Concrete Example: A Breaking Change Detection

Let's walk through a real scenario where GraphRAG saves the day.

The PR: A developer changes the signature of calculatePrice(item) to calculatePrice(item, discount) in cart.ts.

Without GraphRAG (Naive Approach):

graph LR
    A[Git Diff] --> B[Raw Lines Changed]
    B --> C["Line 42: calculatePrice item, discount"]
    C --> D[LLM Review]
    D --> E["Looks good! Added discount parameter"]
    
    style E fill:#ffebee,stroke:#b71c1c

The AI sees the change in isolation and thinks it's fine. It misses that many other files call this function without the new parameter. The code will break in production.

With GraphRAG (Our Approach):

graph TD
    A[Git Diff] --> B[Tree-Sitter Parser]
    B --> C["Function: calculatePrice"]
    C --> D[Query Dependency Graph]
    
    D --> E["checkout.ts calls it"]
    D --> F["order.ts calls it"]
    D --> G["invoice.ts calls it"]
    D --> H["...more callers"]
    
    E --> I[Fetch Context Snippets]
    F --> I
    G --> I
    H --> I
    
    I --> J[Build Enhanced Prompt]
    J --> K[LLM Review]
    K --> L["BREAKING CHANGE: Multiple callers need updates"]
    
    style L fill:#e8f5e9,stroke:#388e3c

The Prompt Difference:

Observation: GraphRAG significantly increases context size, but it catches breaking changes that would cost hours of debugging in production. The extra token cost is worth it.

Think About It: How do you build the dependency graph in the first place? You need to run a static analysis tool (like tree-sitter or language-specific parsers) on the entire codebase when a repo is first connected. This graph is then updated incrementally with each PR.

The Context Window Budget Problem

With many callers, you face a new problem: context window overflow. Even with a large context window, you can't hold everything.

graph TD
    A[Function has Many Callers] --> B{Context Strategy}
    
    B -->|Naive: Include All| C[Excessive Tokens]
    C --> D[Context Window Overflow]
    
    B -->|Smart: Rank by Relevance| E[Extract Most Relevant]
    E --> F[Manageable Token Count]
    F --> G[Fit in Context Window]
    
    style D fill:#ffebee,stroke:#b71c1c
    style G fill:#e8f5e9,stroke:#388e3c

The Solution: Use a relevance ranking algorithm:

1. Callers in the same file/module: High priority
2. Callers in integration tests: High priority
3. Callers in distant, unrelated modules: Low priority

You send the most relevant callers to the LLM, not all of them.

---

4. The Intelligence Layer: The "Model Cascade" Pattern

The Architectural Problem:

We have the code. Now, which AI model do we use?

Using a large reasoning model for every file is too expensive and slow. Using a small, fast model is too limited to find complex bugs.

The Solution:

We use the Model Cascade (or Router) pattern. We use cheap models to filter and expensive models to reason.

The Architecture

graph TD
    Input[Code Chunk] --> Router{Router Agent Small Model}
    
    Router -->|Looks like formatting/renaming| Linter[Static Analysis / Linter]
    
    Router -->|Looks like logic change| BigModel{Large Reasoning Model}
    
    BigModel -->|Step 1: Explain Code| CoT[Chain of Thought]
    CoT -->|Step 2: Find Vulnerabilities| SecurityCheck[Security Check]
    CoT -->|Step 3: Find Logic Bugs| LogicCheck[Logic Check]
    
    Linter --> FinalReview[Final Review]
    SecurityCheck --> FinalReview
    LogicCheck --> FinalReview

How the AI Agent Works Here:

1. The Router: A tiny, sub-second model scans the diff. It classifies the change: Cosmetic, Documentation, or Logic.

2. The Fast Path: Cosmetic changes are routed to a standard linter or skipped entirely. Cost: ~$0.

3. The Slow Path: Logic changes are sent to the "Reasoning Model". We use Chain of Thought (CoT) prompting here. We force the model to "explain the code to itself" before it attempts to find a bug. This drastically reduces false positives.

Observation: The Model Cascade pattern reduces your AI bill by 10x without sacrificing quality. The key insight: Not all code changes are created equal. A whitespace fix doesn't deserve the same scrutiny as a database transaction handler.

The Cost Breakdown: Single Model vs. Cascade

Let's compare two approaches for reviewing files.

Approach 1: Brute Force (Large Model for everything)

graph TD
    A[All Files] --> B[All to Large Model]
    B --> C[High Cost per File]
    C --> D[Total Cost High]
    
    B --> E[Long Response Time]
    E --> F[Slow Processing]
    
    style D fill:#ffebee,stroke:#b71c1c

Approach 2: Model Cascade (Smart Routing)

graph TD
    A[All Files] --> B{Router Small Fast Model}
    
    B -->|40 percent Cosmetic| C[Static Analysis Free]
    B -->|30 percent Docs| D[Skip or Light Model]
    B -->|30 percent Logic| E[Large Model with CoT]
    
    C --> F1[Minimal Cost]
    D --> F2[Low Cost]
    E --> F3[Higher Cost]
    
    F1 --> G[Total Cost Much Lower]
    F2 --> G
    F3 --> G
    
    style G fill:#e8f5e9,stroke:#388e3c

The Impact:

• Router: Negligible cost for fast classification
• Static Analysis: Zero cost for cosmetic changes
• Light Model: Low cost for documentation
• Large Model: Only used for complex logic (30% of files)
• Result: 70% cost reduction compared to brute force approach

The Chain of Thought (CoT) Strategy

For the 30% of files that hit the "Slow Path", we use a multi-step prompting strategy:

graph TD
    A[Code Change] --> B[Step 1: Explain]
    B --> C["Prompt: Explain what this code does"]
    C --> D[LLM Output: Explanation]
    
    D --> E[Step 2: Analyze]
    E --> F["Prompt: Given your explanation, find bugs"]
    F --> G[LLM Output: Potential Issues]
    
    G --> H[Step 3: Validate]
    H --> I["Prompt: Are these issues real or false positives?"]
    I --> J[Final Review Comment]
    
    style J fill:#e8f5e9,stroke:#388e3c

Observation: Chain of Thought reduces false positives by 60%. By forcing the model to "think out loud" first, it catches itself before making confident but wrong claims like "This will cause a null pointer exception" when the code is actually safe.

Think About It: Should the Router itself be an LLM, or should it be a simpler rule-based classifier? At scale, even a fast LLM adds latency. Some teams use a hybrid: rules for obvious cases (file extension = .md → skip) and an LLM for ambiguous cases.

Choosing the Right Model for Each Step

---

Putting It All Together: A Real PR Review

Let's trace a single PR through our entire system to see how these patterns work together.

Scenario: A developer pushes a PR at 9:05 AM with 3 files:

1. README.md (documentation)
2. package-lock.json (lockfile)
3. api/checkout.ts (logic change that modifies calculatePrice)

graph TD
    A[GitHub Webhook] --> B[Ingestion: Buffer Pattern]
    B --> C[Gateway accepts in 50ms]
    C --> D[Event pushed to Kafka]
    
    D --> E[Filtering: Gatekeeper Pattern]
    E --> F{Analyze Files}
    
    F -->|README.md| G1[Drop: Documentation]
    F -->|package-lock.json| G2[Drop: Lockfile]
    F -->|checkout.ts| G3[Valid: Code Change]
    
    G3 --> H[Context: GraphRAG Pattern]
    H --> I[Parse checkout.ts with Tree-Sitter]
    I --> J[Extract: calculatePrice modified]
    J --> K[Query Graph: Find 15 callers]
    K --> L[Build Enhanced Context]
    
    L --> M[Intelligence: Model Cascade]
    M --> N{Router}
    N -->|Logic Change| O[Large Model with CoT]
    
    O --> P[Step 1: Explain function]
    P --> Q[Step 2: Find issues]
    Q --> R[Step 3: Validate]
    
    R --> S[Final Review Comment]
    S --> T[Post to GitHub PR]
    
    style C fill:#e8f5e9,stroke:#388e3c
    style G1 fill:#fff8e1,stroke:#f57f17
    style G2 fill:#fff8e1,stroke:#f57f17
    style S fill:#e8f5e9,stroke:#388e3c

Timeline:

• Step 1: GitHub sends webhook
• Step 2: Gateway responds instantly with "Accepted"
• Step 3: Worker pulls from queue asynchronously
• Step 4: Gatekeeper filters non-code files
• Step 5: GraphRAG builds context with caller snippets
• Step 6: Router classifies as "Logic Change"
• Step 7: Large reasoning model completes Chain of Thought reasoning
• Step 8: Review posted to GitHub

Cost Breakdown:

• Buffer/Gateway: Infrastructure cost only
• Gatekeeper: Rule-based (free)
• GraphRAG: Graph query cost (minimal)
• Router: Small model (minimal)
• Main Review: Large model with context (primary cost)
• Result: Efficient, high-quality, context-aware review

Observation: Without these patterns, you'd spend $0.09 (3 files × $0.03) and miss the breaking change because you wouldn't have the caller context. These patterns save money and improve quality.

---

Challenge: Design Decisions for Your System

As you architect your own code review agent, consider these trade-offs:

Challenge 1: The Stale Graph Problem

Your dependency graph is built when a repo connects. But code changes with every PR. How do you keep it fresh?

Options:

1. Rebuild on every PR: Accurate but slow (30+ seconds for large repos)
2. Incremental updates: Fast but complex (track only changed symbols)
3. Periodic rebuilds: Simple but can be stale (rebuild nightly)

Your Task: Which approach would you choose for a large TypeScript monorepo with frequent PRs?

Challenge 2: The Context Window Budget

You find a function with 200 callers across the codebase. You can't fit them all in the context window.

Options:

1. Top-K by relevance: Send only the 10 most relevant callers
2. Summarize first: Use an LLM to summarize all 200 callers, then send summaries
3. Multi-pass review: Review in batches, then synthesize findings

Your Task: What's your relevance ranking algorithm? Should same-file callers always beat cross-file ones?

Challenge 3: The False Positive vs. False Negative Trade-Off

Your Chain of Thought prompt can be tuned for:

• Conservative: Catch everything, but 40% false positives (developers ignore you)
• Aggressive: Miss 20% of real bugs, but zero false positives (developers trust you)

Your Task: Which do you optimize for? Does it depend on the file type (e.g., more conservative for payment logic)?

---

Summary of Part 1

We have successfully architected the "Input" side of our massive system.

1. Ingestion: We use the Buffer Pattern to survive webhook storms.
2. Filtering: We use the Gatekeeper Pattern to ignore noise.
3. Context: We use GraphRAG (via AST parsing) to see dependencies across files.
4. Intelligence: We use Model Cascading to route hard problems to big brains and easy problems to small brains.

In Part 2, we will dive into the Orchestration Brain. We will look at how to use Temporal to manage the state of this complex workflow, handle failures, and ensure we never hit GitHub's API rate limits.

---

System Comparison: Naive vs. Production-Ready

Here's a side-by-side comparison of what we've built:

graph TD
    subgraph Naive["Naive System"]
        A1[Traffic Spike] --> B1[Single Server]
        B1 --> C1[Crashes]
        style C1 fill:#ffebee,stroke:#b71c1c
    end
    
    subgraph Production["Production System"]
        A2[Traffic Spike] --> B2[Buffer Queue]
        B2 --> C2[Worker Pool]
        C2 --> D2[Scales Horizontally]
        style D2 fill:#e8f5e9,stroke:#388e3c
    end

---

Key Architectural Patterns

---

Discussion Points for Engineers

1. The Dependency Graph Freshness Problem

You've built a beautiful dependency graph, but it's 3 hours old. A developer just merged a PR that renamed a function. Your next review uses stale data.

Questions:

• Do you rebuild the entire graph on every PR (slow but accurate)?
• Do you use incremental updates (fast but complex)?
• How do you handle the race condition when two PRs modify the same function simultaneously?

2. The Rate Limiting Dilemma

Your largest customer pushes many PRs simultaneously. Your smallest customer pushes a single PR shortly after.

Questions:

• Do you use per-tenant queues to guarantee fairness?
• Do you prioritize small PRs (better UX) or first-come-first-served (simpler)?
• What happens when a customer hits their rate limit? Do you queue or reject?

3. The Context Window Budget

A function has many callers across the codebase. You can only fit a subset in the context window.

Questions:

• How do you rank relevance? (Same file = higher weight? Test files = medium weight?)
• Do you show the AI that many callers exist but only provide samples?
• For critical files (auth, payments), do you force a multi-pass review?

4. The False Positive vs. Trust Trade-Off

Your AI has good accuracy, but some percentage of reviews will be incorrect. Developers may start ignoring all reviews if trust erodes.

Questions:

• Do you add a "confidence score" to each finding?
• Do you only show "High Confidence" findings by default?
• How do you collect feedback to retrain your router and grader?

---

What's Next in Part 2: The Orchestration Brain

We've architected the input pipeline, but we haven't talked about how to orchestrate all these moving parts.

In Part 2, we'll dive into:

graph LR
    A[Part 1: Ingestion and Context] --> B[Part 2: Orchestration]
    B --> C[Temporal Workflows]
    B --> D[State Machines]
    B --> E[Retry Policies]
    B --> F[Rate Limit Management]
    B --> G[Failure Recovery]
    
    style B fill:#e3f2fd,stroke:#0d47a1

Key Questions We'll Answer:

• How do you track the state of a review that takes 90 seconds and spans 15 worker calls?
• What happens when the GitHub API rate-limits you mid-review?
• How do you retry failures without duplicating work or spamming users?
• How do you ensure exactly-once processing when workers can crash?

Spoiler: We'll use Temporal (a distributed workflow engine) to turn this complex, stateful process into a simple, deterministic function.

---

Takeaways

The Four Pillars of Scale

graph TD
    A[Scalable Code Review Agent] --> B[1. Buffer Pattern]
    A --> C[2. Gatekeeper Pattern]
    A --> D[3. GraphRAG Pattern]
    A --> E[4. Model Cascade]
    
    B --> F[Survive Traffic Spikes]
    C --> G[Reduce Waste]
    D --> H[Improve Accuracy]
    E --> I[Optimize Costs]
    
    style A fill:#e3f2fd,stroke:#0d47a1
    style F fill:#e8f5e9,stroke:#388e3c
    style G fill:#e8f5e9,stroke:#388e3c
    style H fill:#e8f5e9,stroke:#388e3c
    style I fill:#e8f5e9,stroke:#388e3c

Key Insights

• Scale isn't just about throughput — it's about surviving spikes, managing costs, and maintaining quality under load. The Buffer Pattern proves you can handle high traffic while responding instantly.

• Context is everything in AI code review — a diff without dependencies is noise. GraphRAG turns "Line 42 changed" into "This breaks 15 callers across 8 files."

• Smart routing beats brute force — don't use your biggest model for every problem. The Model Cascade reduces costs by 10x without sacrificing quality.

• Filtering is a feature, not a bug — 40% of commits are noise. The Gatekeeper Pattern saves compute and improves signal-to-noise ratio.

• Architecture patterns from distributed systems apply to AI agents — event-driven design, backpressure, circuit breakers, and idempotency aren't just for databases. They're essential for production AI systems.

The Cost-Quality Matrix

The Winning Strategy: Implement them in order: Gatekeeper (quick win) → Buffer (prevents disaster) → Model Cascade (major savings) → GraphRAG (ultimate quality).

---

For more on building production AI systems at scale, check out our AI Bootcamp for Software Engineers.