Using Structured Outputs to Parse AI Responses in CI/CD

This tutorial assumes you have a basic understanding of CI/CD pipelines and API calls to large language models.

Why Structured Outputs?

When you ask an AI to review code, generate a PR description, or suggest changes, the response is often free-form text. Parsing that in a script is fragile. By using structured outputs (JSON mode or function calling), you get a guaranteed schema that your CI/CD pipeline can consume directly.

Step 1: Define Your Schema

Decide what fields you need. For a code review, you might want:

{
  "summary": "string",
  "issues": [{"severity": "high|medium|low", "file": "", "line": 0, "description": ""}],
  "approved": boolean
}

Step 2: Call the AI with Structured Outputs

Using OpenAI’s API with JSON mode (or Anthropic / other providers that support it):

import openai

client = openai.OpenAI()

response = client.chat.completions.create(
    model="gpt-4o-mini",
    messages=[
        {"role": "system", "content": "You are a code reviewer. Output valid JSON following the provided schema."},
        {"role": "user", "content": f"Review this diff:\n{diff}"}
    ],
    response_format={"type": "json_object"},
    temperature=0.2
)

import json
result = json.loads(response.choices[0].message.content)

Step 3: Validate the Output

Use JSON schema validation (e.g., jsonschema library) to double‑check the response before proceeding. This prevents malformed responses from breaking your pipeline.

from jsonschema import validate, ValidationError

schema = {
    "type": "object",
    "properties": {
        "summary": {"type": "string"},
        "issues": {
            "type": "array",
            "items": {
                "type": "object",
                "properties": {
                    "severity": {"type": "string", "enum": ["high", "medium", "low"]},
                    "file": {"type": "string"},
                    "line": {"type": "integer"},
                    "description": {"type": "string"}
                },
                "required": ["severity", "file", "line", "description"]
            }
        },
        "approved": {"type": "boolean"}
    },
    "required": ["summary", "issues", "approved"]
}

try:
    validate(instance=result, schema=schema)
except ValidationError as e:
    print("Invalid response from AI:", e)
    exit(1)

Step 4: Integrate into CI/CD

Create a script (e.g., ai_review.py) that accepts a diff or PR and outputs structured data. Then in your CI file (GitHub Actions example):

name: AI Code Review
on:
  pull_request:
    types: [opened, synchronize]

jobs:
  review:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v3
        with:
          fetch-depth: 0
      - name: Get diff
        run: git diff origin/${{ github.base_ref }}...origin/${{ github.head_ref }} > diff.txt
      - name: AI Review
        run: python ai_review.py diff.txt > review.json
      - name: Comment on PR
        uses: actions/github-script@v6
        with:
          script: |
            const fs = require('fs');
            const review = JSON.parse(fs.readFileSync('review.json', 'utf8'));
            let comment = `## AI Review Summary\n${review.summary}\n\n### Issues\n`;
            review.issues.forEach(issue => {
                comment += `- [${issue.severity}] ${issue.file}:${issue.line} - ${issue.description}\n`;
            });
            github.rest.issues.createComment({
                issue_number: context.issue.number,
                owner: context.repo.owner,
                repo: context.repo.repo,
                body: comment
            });

Best Practices

• Use a low temperature (0.0–0.2) to get consistent JSON output.
• Include a retry mechanism in case the AI returns an invalid JSON or schema mismatch.
• Store API keys as secrets in your CI provider.
• Cache the AI response for the same diff to avoid duplicate costs.