Set Up a Multi-Agent AI Code Review Pipeline for Pull Requests

What You'll Build

A GitHub Actions workflow that, when a pull request is opened, runs three AI agents in parallel:

• Security Agent – checks for common vulnerabilities (e.g., SQL injection, insecure deserialization).
• Style Agent – verifies code style consistency and naming conventions.
• Logic Agent – looks for potential bugs or logical errors.

Each agent returns structured feedback (pass/fail + comments), and the pipeline posts a summary as a PR comment.

Prerequisites

• GitHub account and a repository where you can test pull requests.
• Python 3.9+ installed locally for testing.
• An API key from OpenAI (or any LLM supported by LangChain).
• Basic familiarity with GitHub Actions and YAML.

Step 1: Create the LangChain Agent Script

First, set up a Python script that defines the agents and runs the review. Create a file code_review_agents.py in your repository's .github/scripts/ folder.

from langchain.agents import AgentExecutor, create_openai_functions_agent
from langchain.tools import Tool
from langchain.prompts import ChatPromptTemplate
from langchain.schema import SystemMessage, HumanMessage
from langchain.chat_models import ChatOpenAI
import json

# Initialize LLM
llm = ChatOpenAI(model="gpt-4", temperature=0)

# Define agents
def create_agent(name, instructions):
    system_message = SystemMessage(content=instructions)
    prompt = ChatPromptTemplate.from_messages([system_message, HumanMessage(content="{diff}")])
    agent = create_openai_functions_agent(llm, prompt, tools=[])
    return AgentExecutor(agent=agent, tools=[], verbose=True)

security_agent = create_agent(
    "Security",
    "You are a security reviewer. Analyze the provided diff for vulnerabilities. "
    "Return JSON with keys: 'passed' (bool) and 'comments' (list of strings)."
)

style_agent = create_agent(
    "Style",
    "You are a style reviewer. Check for naming conventions, indentation, etc. "
    "Return JSON with keys: 'passed' (bool) and 'comments' (list of strings)."
)

logic_agent = create_agent(
    "Logic",
    "You are a logic reviewer. Find potential bugs or incorrect logic. "
    "Return JSON with keys: 'passed' (bool) and 'comments' (list of strings)."
)

def review_diff(diff_text):
    results = {}
    for agent in [security_agent, style_agent, logic_agent]:
        response = agent.run(diff_text)
        # Assume agent returns valid JSON
        results[agent.agent.name] = json.loads(response)
    return results

if __name__ == "__main__":
    import sys
    diff = sys.argv[1]
    results = review_diff(diff)
    print(json.dumps(results))

Step 2: Create the GitHub Actions Workflow

Create .github/workflows/code-review.yml in your repository:

name: Multi-Agent Code Review
on:
  pull_request:
    types: [opened, synchronize]

jobs:
  review:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v3
        with:
          fetch-depth: 0
      - name: Get diff
        id: diff
        run: |
          git diff origin/${{ github.base_ref }}...HEAD > diff.txt
          echo "DIFF<> $GITHUB_OUTPUT
          cat diff.txt >> $GITHUB_OUTPUT
          echo "EOF" >> $GITHUB_OUTPUT
      - name: Run agents
        id: agents
        run: |
          python .github/scripts/code_review_agents.py "${{ steps.diff.outputs.DIFF }}" > results.json
      - name: Post comment
        uses: actions/github-script@v6
        with:
          script: |
            const fs = require('fs');
            const results = JSON.parse(fs.readFileSync('results.json','utf8'));
            const {data: comments} = await github.rest.issues.listComments({
              owner: context.repo.owner,
              repo: context.repo.repo,
              issue_number: context.issue.number
            });
            // Delete previous comments from this bot (optional)
            for (const c of comments) {
              if (c.user.type === 'Bot') {
                await github.rest.issues.deleteComment({
                  owner: context.repo.owner,
                  repo: context.repo.repo,
                  comment_id: c.id
                });
              }
            }
            let body = '## AI Code Review Results\n\n';
            for (const [agent, res] of Object.entries(results)) {
              const emoji = res.passed ? '✅' : '❌';
              body += `### ${agent} ${emoji}\n`;
              if (res.comments.length > 0) {
                body += res.comments.map(c => `- ${c}`).join('\n') + '\n';
              } else {
                body += 'No issues found.\n';
              }
            }
            await github.rest.issues.createComment({
              owner: context.repo.owner,
              repo: context.repo.repo,
              issue_number: context.issue.number,
              body: body
            });

Step 3: Configure Secrets and Run

Add your OPENAI_API_KEY to the repository secrets in GitHub Settings -> Secrets and variables -> Actions. Then create a pull request to trigger the workflow.

Next Steps

• Customize agent instructions for your specific codebase or language.
• Add more agents for performance, documentation, or testing.
• Use structured outputs with Pydantic to ensure JSON format consistency.
• Experiment with different models like Claude or Gemini via LangChain.