Using Structured Outputs to Parse AI Responses in CI/CD

When you integrate AI into CI/CD, you quickly hit a wall: AI models return natural language, not structured data. A code review agent might say "The function looks fine, but consider adding error handling" – good luck parsing that. But with structured outputs (supported by OpenAI, Anthropic, and others via function calling or JSON mode), you can force the AI to return a predefined JSON schema. This tutorial shows you how to build a PR review agent that returns a parseable JSON object, then use it in a GitHub Actions workflow.

What You'll Build

• A Python script that calls an LLM with a structured output schema for code review.
• A GitHub Actions workflow that runs the script on every PR, parses the JSON, and posts comments based on severity.

Prerequisites

Basic Python, a GitHub repo, and an API key from OpenAI or Anthropic. We'll use OpenAI for this example.

Step 1: Define the Output Schema

We want the AI to return something like:

{
  "summary": "string",
  "issues": [
    {
      "severity": "critical" | "warning" | "info",
      "file": "string",
      "line": number,
      "description": "string"
    }
  ]
}

Using OpenAI's function calling, we define this schema as a JSON object. Create schema.py:

from pydantic import BaseModel
from typing import List, Literal

class Issue(BaseModel):
    severity: Literal["critical", "warning", "info"]
    file: str
    line: int
    description: str

class ReviewOutput(BaseModel):
    summary: str
    issues: List[Issue]

Step 2: Build the Review Agent

Create review_agent.py:

import os
import json
from openai import OpenAI
from schema import ReviewOutput

client = OpenAI(api_key=os.environ["OPENAI_API_KEY"])

def fetch_diff():
    # In CI, diff comes from git. For demo, read from env.
    return os.environ.get("PR_DIFF", "")

def run_review(diff: str):
    schema = ReviewOutput.schema()
    function_def = {
        "name": "review_code",
        "description": "Review a code diff and return structured feedback.",
        "parameters": schema
    }
    response = client.chat.completions.create(
        model="gpt-4o-mini",
        messages=[
            {"role": "system", "content": "You are a senior code reviewer. Analyze the diff and output structured feedback."},
            {"role": "user", "content": f"Review this diff:\n{diff}"}
        ],
        functions=[function_def],
        function_call={"name": "review_code"}
    )
    args = response.choices[0].message.function_call.arguments
    return json.loads(args)

if __name__ == "__main__":
    diff = fetch_diff()
    result = run_review(diff)
    print(json.dumps(result, indent=2))

Step 3: Add to GitHub Actions

Create .github/workflows/ai-review.yml:

name: AI Code Review
on: [pull_request]

jobs:
  review:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
        with:
          fetch-depth: 0
      - name: Get diff
        run: |
          git fetch origin ${{ github.base_ref }}
          git diff origin/${{ github.base_ref }}...HEAD > pr_diff.txt
      - name: Run review agent
        env:
          OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}
          PR_DIFF: $(cat pr_diff.txt)
        run: python review_agent.py > review_result.json
      - name: Parse and comment
        uses: actions/github-script@v7
        with:
          script: |
            const fs = require('fs');
            const result = JSON.parse(fs.readFileSync('review_result.json', 'utf8'));
            let comment = `## AI Review\n\n**Summary:** ${result.summary}\n\n`;
            result.issues.forEach(issue => {
              comment += `- [${issue.severity}] ${issue.file}:${issue.line} — ${issue.description}\n`;
            });
            github.rest.issues.createComment({
              owner: context.repo.owner,
              repo: context.repo.repo,
              issue_number: context.issue.number,
              body: comment
            });

Step 4: Test It

Open a pull request with a simple change. The workflow will run, and you'll see a comment like:

Summary: Good overall, but one critical issue found.
- [critical] main.py:12 — Missing input validation on user-supplied data.
- [warning] utils.py:5 — Consider using f-strings instead of concat.
- [info] config.py:1 — Unused import `os`.

Going Further

• Use Anthropic's tool use for similar results.
• Add severity thresholds: auto-request changes if any critical issue.
• Store review results for analytics.

Structured outputs turn AI from a chatbot into a pipeline component. Try it on your next project.