Tame AI Chaos in CI/CD with Structured Outputs

AI is great at writing, but terrible at following formatting rules—unless you force it. In CI/CD, a single malformed JSON response can break your entire pipeline. The fix? Use structured outputs. This tutorial shows you how to make your AI model return predictable, schema-valid data every time, and wire it into a GitHub Action.

Why Structured Outputs?

Normal text or json_object responses often include stray markdown or missing keys. Structured outputs use a JSON schema to guarantee the response matches exactly. This is a game-changer for automation.

Step 1: Define Your Schema

First, we define what the AI must return. Here's a schema for a PR title, summary, and risk level:

{
  "name": "pr_analysis",
  "strict": true,
  "schema": {
    "type": "object",
    "properties": {
      "title": {"type": "string"},
      "summary": {"type": "string"},
      "risk_level": {"enum": ["low", "medium", "high"]},
      "suggested_reviewers": {"type": "array", "items": {"type": "string"}}
    },
    "required": ["title", "summary", "risk_level", "suggested_reviewers"],
    "additionalProperties": false
  }
}

Step 2: Python Script with Structured Output

Now create analyze_pr.py. We'll use the OpenAI SDK in Python with the new strict: true parameter.

import os
import json
from openai import OpenAI

client = OpenAI()

schema = {
  "name": "pr_analysis",
  "strict": true,
  "schema": {
    "type": "object",
    "properties": {
      "title": {"type": "string"},
      "summary": {"type": "string"},
      "risk_level": {"enum": ["low", "medium", "high"]},
      "suggested_reviewers": {"type": "array", "items": {"type": "string"}}
    },
    "required": ["title", "summary", "risk_level", "suggested_reviewers"],
    "additionalProperties": false
  }
}

response = client.chat.completions.create(
    model="gpt-4o-2024-08-06",
    messages=[
        {"role": "system", "content": "Analyze the diff and extract PR metadata."},
        {"role": "user", "content": "Here is the diff:\n" + os.environ["PR_DIFF"]}
    ],
    response_format={
        "type": "json_schema",
        "json_schema": schema
    }
)

# Parse the structured output
result = json.loads(response.choices[0].message.content)

# Now we can use result['title'], result['summary'], etc.
print(json.dumps(result, indent=2))

Step 3: GitHub Action Workflow

Now wire this into a workflow. We'll trigger it on every PR and post a comment with the AI-generated suggestion.

# .github/workflows/pr-analysis.yml
name: PR Analysis
on:
  pull_request:
    types: [opened, synchronize]

jobs:
  analyze:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with:
          python-version: '3.12'
      - run: pip install openai
      - name: Get PR diff
        run: |
          curl -s -H "Authorization: token ${{ secrets.GITHUB_TOKEN }}" \
            "${{ github.event.pull_request.diff_url }}" > diff.txt
      - name: Run AI analysis
        env:
          OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}
          PR_DIFF: $(cat diff.txt)
        run: |
          python analyze_pr.py > result.json
      - name: Post comment
        uses: actions/github-script@v7
        with:
          script: |
            const fs = require('fs');
            const result = JSON.parse(fs.readFileSync('result.json'));
            const body = `## 🤖 AI PR Suggestion\n\n**Title:** ${result.title}\n\n**Summary:** ${result.summary}\n\n**Risk Level:** ${result.risk_level}\n\n**Suggested Reviewers:** ${result.suggested_reviewers.join(', ')}`;
            github.rest.issues.createComment({
              owner: context.repo.owner,
              repo: context.repo.repo,
              issue_number: context.issue.number,
              body: body
            });

Tips & Next Steps

• Use pydantic to define schemas in Python instead of raw JSON.
• Add a fallback if the API call fails—make the job not fail the whole build.
• Extend this pattern to automated code review with multiple agents.