Parse AI Code Reviews in CI with Structured Outputs

When integrating AI into CI/CD, one pain point is parsing free-text responses. Structured outputs solve this by forcing the AI to return valid JSON matching a schema. This tutorial shows you how to use OpenAI's structured outputs feature to get predictable code review comments that your CI pipeline can parse automatically.

Step 1: Define Your Schema

First, decide what data you need from the AI. For a code review, we want a list of comments with file path, line number, severity, and description. Use Pydantic to define the schema:

from pydantic import BaseModel
from typing import List, Literal

class CodeReviewComment(BaseModel):
    file_path: str
    line_number: int
    severity: Literal["info", "warning", "critical"]
    description: str

class CodeReview(BaseModel):
    comments: List[CodeReviewComment]

Step 2: Call OpenAI with Structured Outputs

Now create a function that sends the diff (or code snippet) to OpenAI and expects a structured response. Use the response_format parameter with your schema:

import openai
from openai import OpenAI

client = OpenAI(api_key="YOUR_API_KEY")

def review_code(diff_text: str) -> CodeReview:
    response = client.beta.chat.completions.parse(
        model="gpt-4o-2024-08-06",
        messages=[
            {
                "role": "system",
                "content": "You are a senior code reviewer. Review the following diff and provide structured comments."
            },
            {
                "role": "user",
                "content": f"Review this diff:\n\n{diff_text}"
            }
        ],
        response_format=CodeReview
    )
    return response.choices[0].message.parsed

Step 3: Parse and Act on the Output

Now that you have a structured CodeReview object, you can iterate over comments and post them as PR annotations. Here's an example that outputs in GitHub Actions format:

if __name__ == "__main__":
    import sys
    diff_path = sys.argv[1]
    with open(diff_path, "r") as f:
        diff = f.read()
    review = review_code(diff)
    for comment in review.comments:
        print(f"::warning file={comment.file_path},line={comment.line_number}::{comment.description}")

Step 4: Integrate into CI Pipeline (GitHub Actions Example)

Create a workflow that runs on pull requests:

name: AI Code Review
on:
  pull_request:
    types: [opened, synchronize]
jobs:
  review:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
        with:
          fetch-depth: 0
      - name: Get diff
        run: git diff origin/main -- . > diff.txt
      - name: Run AI review
        env:
          OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}
        run: |
          pip install openai pydantic
          python review.py diff.txt

Why This Works

By using structured outputs, you get a guaranteed JSON structure – no regex parsing, no markdown extraction. The CI pipeline can directly read the fields and create precise annotations. This pattern works for any AI task where you need machine-readable responses.