Using Structured Outputs to Parse AI Responses in CI/CD

AI models are powerful, but when you plug them into a CI/CD pipeline, their freeform text responses can break automation. The fix: structured outputs. By forcing the model to return a JSON object matching a predefined schema, you can reliably parse the response and trigger actions. This tutorial walks through a real-world example using GitHub Actions, Python, and OpenAI's JSON mode to generate automated PR descriptions and review summaries.

Step 1: Define Your Schema

The first step is deciding what fields you want the AI to return. For a PR review agent, a typical schema might include: summary (string), issues (array of objects with file, line, severity, comment), and approve (boolean). We'll use Pydantic to define this in Python, but any JSON schema works.

from pydantic import BaseModel
from typing import List

class Issue(BaseModel):
    file: str
    line: int
    severity: str  # 'critical' | 'warning' | 'info'
    comment: str

class Review(BaseModel):
    summary: str
    issues: List[Issue]
    approve: bool

Step 2: Configure the AI Call with JSON Mode

With the schema in hand, we call the model using OpenAI's JSON mode (or any provider's equivalent). Pass the schema via the response_format parameter. The model will return a JSON string that fits the schema.

import openai
from pydantic import ValidationError

client = openai.Client()

def call_structured_review(diff_text: str) -> Review:
    system = "You are a code reviewer. Analyze the diff and return JSON matching the schema."
    user = f"Review this diff:\n{diff_text}"

    response = client.chat.completions.create(
        model="gpt-4o-mini",
        messages=[
            {"role": "system", "content": system},
            {"role": "user", "content": user},
        ],
        response_format={
            "type": "json_schema",
            "json_schema": {"name": "review", "schema": Review.model_json_schema()}
        }
    )

    raw = response.choices[0].message.content
    try:
        return Review.model_validate_json(raw)
    except ValidationError as e:
        raise RuntimeError(f"Failed to parse AI response: {e}")

Step 3: Integrate into a GitHub Action

Now we package the script into a GitHub Action that triggers on pull requests. The action will:

1. Checkout the PR and get the diff between the base and head branches.
2. Call our Python script with the diff.
3. Parse the structured output and post a review on the PR using the GitHub API.

# .github/workflows/ai-review.yml
name: AI Code Review
on:
  pull_request:
    types: [opened, synchronize]

jobs:
  review:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
        with:
          fetch-depth: 0
      - name: Get diff
        id: diff
        run: |
          git diff origin/${{ github.base_ref }}... > diff.txt
          echo "diff=$(cat diff.txt)" >> $GITHUB_OUTPUT
      - name: Run AI review
        id: ai
        env:
          OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}
        run: |
          python -m pip install openai pydantic
          python review.py "${{ steps.diff.outputs.diff }}" > result.json
      - name: Post review
        env:
          GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
        run: |
          python post_review.py result.json

Step 4: Test and Iterate

Open a test PR and watch the action run. If the schema is too rigid, the model might fail to return valid JSON. Adjust the schema or prompt to improve reliability. A common trick: add "strict": true in the json_schema definition to enforce exact types.

Next Steps

• Add a risk_score (integer 1-10) to the schema.
• Use required fields in the schema to ensure critical info is always present.
• Combine multiple AI calls: one for review, one for summary, one for tests.