Setting Up a Multi-Agent Code Review Pipeline with GitHub Actions

Modern AI coding tools can do more than just generate code. In this tutorial, you'll set up a multi-agent code review pipeline that runs on every pull request. Each agent focuses on a different area: security, style, and logic. The agents give structured feedback that your team can act on immediately.

Prerequisites

• GitHub repository with write access
• OpenAI API key (or equivalent)
• Basic knowledge of YAML and GitHub Actions

Step 1: Define the Review Agent Prompts

Create a .github/agents directory in your repo. Each agent gets a system prompt that defines its role and output format. Use structured outputs (JSON) to make parsing easy.

Example for a security agent (.github/agents/security.yml):

name: Security Agent
system_prompt: |
  You are a security code reviewer. Review the provided diff for:
  - SQL injection, XSS, command injection
  - Hardcoded secrets
  - Unsafe deserialization
  
  Return JSON array of objects with fields:
  - severity ("critical","high","medium","low")
  - file (string)
  - line_number (integer)
  - description (string, max 200 chars)
  - suggestion (string, max 300 chars)
  
  If no issues, return empty array.

Similarly create style and logic agent YAML files.

Step 2: Build the Orchestrator Script

Write a Python script (.github/scripts/review.py) that:

• Reads the git diff from the PR
• Loads each agent's system prompt
• Calls the LLM with function calling to get structured output
• Aggregates results and posts them as a PR comment

import os, json, yaml
from openai import OpenAI

client = OpenAI(api_key=os.environ['OPENAI_API_KEY'])

def get_diff():
    # Get diff from environment (set by GitHub Action)
    with open(os.environ['GITHUB_EVENT_PATH']) as f:
        event = json.load(f)
    # Simplified: assume diff is passed via env var or file
    return os.environ.get('DIFF', '')

def load_agent(name):
    with open(f'.github/agents/{name}.yml') as f:
        return yaml.safe_load(f)

def review_with_agent(agent, diff):
    response = client.chat.completions.create(
        model='gpt-4',
        messages=[
            {'role': 'system', 'content': agent['system_prompt']},
            {'role': 'user', 'content': f"Diff:\n{diff}"}
        ],
        response_format={'type': 'json_object'}
    )
    return json.loads(response.choices[0].message.content)

if __name__ == '__main__':
    diff = get_diff()
    agents = ['security', 'style', 'logic']
    all_issues = []
    for agent_name in agents:
        agent = load_agent(agent_name)
        issues = review_with_agent(agent, diff)
        for issue in issues:
            issue['agent'] = agent_name
        all_issues.extend(issues)
    
    # Post comment to PR (simplified: print to stdout)
    print(json.dumps(all_issues, indent=2))

Step 3: Create the GitHub Actions Workflow

Create .github/workflows/code-review.yml:

name: Multi-Agent Code Review
on:
  pull_request:
    types: [opened, synchronize]

jobs:
  review:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v3
      - uses: actions/setup-python@v4
        with:
          python-version: '3.11'
      - run: pip install openai pyyaml
      - name: Get diff
        id: diff
        run: |
          git fetch origin ${{ github.base_ref }}
          diff=$(git diff origin/${{ github.base_ref }}..HEAD)
          echo "DIFF<> $GITHUB_ENV
          echo "$diff" >> $GITHUB_ENV
          echo "EOF" >> $GITHUB_ENV
      - name: Run agents
        env:
          OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}
        run: python .github/scripts/review.py > review_output.json
      - name: Post comment
        uses: actions/github-script@v6
        with:
          script: |
            const fs = require('fs');
            const issues = JSON.parse(fs.readFileSync('review_output.json','utf8'));
            let comment = '### Multi-Agent Review Results\n\n';
            if (issues.length === 0) {
              comment += 'No issues found.';
            } else {
              issues.forEach(i => {
                comment += `**${i.agent}** - [${i.severity}] ${i.file}:${i.line_number}\n`;
                comment += `${i.description}\n`;
                comment += `> ${i.suggestion}\n\n`;
              });
            }
            github.rest.issues.createComment({
              issue_number: context.issue.number,
              owner: context.repo.owner,
              repo: context.repo.repo,
              body: comment
            });

Step 4: Test and Iterate

Create a test branch with a few deliberate issues. Open a PR and watch the pipeline run. Fine-tune your agent prompts to reduce false positives. Add more agents for performance, documentation, or accessibility.

Next Steps

• Add agent-specific models (e.g., use GPT-4 for security, GPT-3.5 for style)
• Cache agent responses to avoid repeated reviews on unchanged files
• Integrate with code quality tools (linters, SAST) as additional agents