When you ask an LLM to review a pull request, you get prose. Prose is hard to parse in a pipeline. You need to extract verdicts, comments, and line numbers reliably. The fix: request structured JSON output with a defined schema, then validate and consume it.

1. Define Your Schema

Start by modeling the result with Pydantic. This gives you type safety and a JSON Schema for free.

from pydantic import BaseModel, Field
from typing import List, Literal

class ReviewComment(BaseModel):
    file: str
    line: int = Field(description="Line number in the diff")
    severity: Literal["error", "warning", "nit"]
    message: str

class PRReview(BaseModel):
    summary: str
    comments: List[ReviewComment]
    verdict: Literal["approve", "request_changes"]

2. Enforce the Schema with the API

Now ask the model to produce JSON matching this schema. In OpenAI, use the response_format parameter with json_schema:

from openai import OpenAI
from pydantic import ValidationError

client = OpenAI()

response = client.chat.completions.create(
    model="gpt-4o",
    messages=[
        {"role": "system", "content": "You are a strict code reviewer. Output only JSON."},
        {"role": "user", "content": f"Review this diff:\n{diff_text}"}
    ],
    response_format={
        "type": "json_schema",
        "json_schema": {
            "name": "PRReview",
            "schema": PRReview.model_json_schema(),
            "strict": True
        }
    }
)

review = PRReview.model_validate_json(response.choices[0].message.content)
Both OpenAI and Anthropic support schema-forced output. For Anthropic, use tool-use with a JSON schema; the principle is identical.

3. Validate and Retry

Even with strict mode, you should still handle malformed responses. Wrap the call in a retry decorator:

import json
from tenacity import retry, stop_after_attempt, retry_if_exception_type

@retry(stop=stop_after_attempt(2), retry=retry_if_exception_type((json.JSONDecodeError, ValidationError)))
def get_review(diff_text: str) -> PRReview:
    # ... call the API and parse the response
    return PRReview.model_validate_json(response.choices[0].message.content)
Strict schema mode will fail if the model omits required fields. Always include a fallback for malformed output, even in a demo.

4. Wire It into CI/CD

Now plug it into a GitHub Action that runs on every pull request:

name: AI PR Review
on:
  pull_request:
    types: [opened, synchronize]

jobs:
  review:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
        with:
          fetch-depth: 0
      - uses: actions/setup-python@v5
        with:
          python-version: '3.12'
      - run: pip install openai pydantic tenacity
      - run: python review.py --diff "${{ github.event.pull_request.diff_url }}"
        env:
          OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}
          GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}

In review.py, fetch the diff, pass it to get_review(), and then post the formatted comments to the pull request.

Now every PR gets a consistent, structured review that you can post or query programmatically. No more regex on prose.

This pattern scales to any AI coding task: commit message generation, test classification, or security triage. Define the schema first, force the model to match it, and your pipeline stays deterministic.