Using AI to automate pull request reviews, generate release notes, or triage issues is powerful — but plain text output is fragile in CI. One malformed response and your pipeline breaks. The fix? Structured outputs. By forcing the AI to return JSON that matches a schema, you can parse, validate, and act on it with confidence.

This tutorial shows you a practical pattern: send a well-crafted prompt, request a strict JSON response, validate it against a schema, and use the data in your CI workflow. You'll build this with Python and GitHub Actions, but the same approach applies to any language or CI platform.

Step 1: Define your output schema

First, create a schema that describes exactly what you want from the AI. Python's Pydantic makes this easy and gives you built-in validation.

from pydantic import BaseModel, ValidationError

class PRSummary(BaseModel):
    title: str
    description: str
    changed_files: list[str]
    risk_level: Literal["low", "medium", "high"]
    review_comments: list[dict]  # each dict has 'file' and 'comment'
If you're using JavaScript, pick Zod. The key is having a schema, not the library.

Step 2: Prompt for strict JSON output

Now tell the AI to answer in JSON only. Be explicit about the fields and format. Include the schema in the prompt.

prompt = f"""You are a code reviewer. Analyze the following diff.

Return ONLY valid JSON matching this exact structure:
{{
  "title": "short summary",
  "description": "2-3 sentences",
  "changed_files": ["file1.py", ...],
  "risk_level": "low|medium|high",
  "review_comments": [{{"file": "path", "comment": "note"}}]
}}

Diff:
{diff}
"""
Even with strict prompting, models may occasionally add markdown fences or extra text. Always validate and sanitize.

Step 3: Extract and parse the JSON

In your CI script, strip any false formatting and parse the response. Use the schema to validate immediately.

import json, re

def parse_ai_response(raw: str) -> PRSummary:
    # Remove code fences if present
    raw = re.sub(r"^```(?:json)?|```$", "", raw.strip())
    try:
        data = json.loads(raw)
        return PRSummary(**data)
    except (json.JSONDecodeError, ValidationError) as e:
        raise SystemExit(f"AI output failed validation: {e}")
Make extraction and validation a reusable script in your repo (e.g., scripts/parse_ai_output.py). That way every CI job uses the same logic.

Step 4: Wire it into CI

Create a GitHub Actions job that runs the AI call, parses the result, and posts a PR comment.

name: ai-review
on: pull_request
jobs:
  review:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with: python-version: '3.12'
      - run: pip install pydantic requests
      - name: Run AI script
        env:
          OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}
        run: |
          python scripts/generate_pr_summary.py \
            --diff-file changed_files.txt \
            --output summary.json
      - name: Parse and validate output
        run: python scripts/parse_ai_output.py summary.json

If the AI fails to produce valid JSON, the script exits with a non-zero code and the job fails. That's a feature — you want to catch it early rather than ship a bad automation step.

You now have a deterministic checkpoint. All downstream steps (labels, comments, tests) can safely consume the structured data.

Why this works

Structured outputs move the risk from 'garbage text' to 'valid JSON.' You can always fall back, retry, or log a clean error. Combined with a schema, the AI becomes a reliable API — perfect for autonomous workflows.

Start with one simple use case today. Define a schema, prompt for JSON, validate, and only then take action. Your CI will thank you.