AI agents are great at generating PR descriptions, code review feedback, or release notes. But dumping free-form text into your CI/CD pipeline is a recipe for broken automation. The fix? Structured outputs. By constraining the AI to return valid JSON matching a schema, you can parse its response and use it across your pipeline without fragile string matching.

In this tutorial, you'll build a GitHub Actions job that uses OpenAI's structured outputs (JSON schema mode) to auto-generate a pull request summary, then validates and uses the result in subsequent steps. Everything runs in isolated Python code, so you can adapt it to any CI system.

What you'll build: A pipeline step that calls an LLM with a predefined schema for PR metadata, validates the response with Pydantic, and writes formatted output to a file your other steps can consume.

Prerequisites

  • Basic knowledge of Python and JSON
  • A GitHub repository with GitHub Actions enabled
  • An OpenAI API key (or compatible endpoint)

Step 1: Define Your Output Schema

Start by defining exactly what you want the AI to return. For a PR summary, you might want:

  • title – short string
  • summary – one paragraph
  • breaking_changes – boolean
  • files_changed – array of file paths

In Python, use pydantic to define the schema. This same schema will be passed to the LLM for structured output and used for validation.

from pydantic import BaseModel, Field

class PRSummary(BaseModel):
    title: str = Field(description="Short, descriptive PR title")
    summary: str = Field(description="One-paragraph overview")
    breaking_changes: bool = Field(description="Whether any breaking changes are introduced")
    files_changed: list[str] = Field(description="List of file paths changed")

Step 2: Call the LLM with Structured Outputs

Use the OpenAI SDK and pass your Pydantic model as the response_format. The model must support structured outputs (gpt-4o and later).

from openai import OpenAI
import json

client = OpenAI()

# You'll fetch the diff from the CI environment
diff = open("diff.txt").read()

completion = client.beta.chat.completions.parse(
    model="gpt-4o",
    messages=[
        {"role": "system", "content": "You are a helpful code reviewer. Generate a PR summary from the diff."},
        {"role": "user", "content": diff},
    ],
    response_format=PRSummary,
)

parsed_summary = completion.choices[0].message.parsed

Why this works: The parse method tells the API to produce a JSON object that satisfies the schema. The SDK automatically validates the response and raises an error if it doesn't conform. No more regex on free-text.

Step 3: Validate and Serialize

Even with structured outputs, LLMs can occasionally return a list instead of a string or miss a field. Run the parsed result through Pydantic again to be extra sure:

# parses are already validated, but this catches schema drift
from pydantic import ValidationError

try:
    validated = PRSummary.model_validate(parsed_summary.model_dump())
except ValidationError as e:
    print(f"AI response failed validation: {e}")
    exit(1)

# Write to a file so later pipeline steps can consume it
with open("pr_summary.json", "w") as f:
    f.write(validated.model_dump_json(indent=2))

Step 4: Integrate into GitHub Actions

Now put it all together in a workflow file. You'll need steps to fetch the diff, run your Python script, and then use the generated JSON (e.g., to update a PR comment).

# .github/workflows/pr-summary.yml
name: Generate PR Summary

on:
  pull_request:
    types: [opened, synchronize]

jobs:
  generate:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
        with:
          fetch-depth: 0

      - name: Get diff
        run: git diff origin/main...HEAD > diff.txt

      - name: Set up Python
        uses: actions/setup-python@v5
        with:
          python-version: '3.12'

      - name: Install dependencies
        run: pip install openai pydantic

      - name: Run AI summary generator
        env:
          OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}
        run: python generate_pr_summary.py

      - name: Post summary to PR
        run: |
          TITLE=$(python -c "import json; print(json.load(open('pr_summary.json'))['title'])")
          BODY=$(python -c "import json; print(json.load(open('pr_summary.json'))['summary'])")
          gh pr comment "${{ github.event.pull_request.number }}" --body "**$TITLE** - $BODY"

Watch out: Never hardcode your API key. Use GitHub secrets and pass it via env. Also, the diff can be huge; consider truncating to the first few hundred lines to keep token usage low.

Step 5: Handle Failure Gracefully

If the model is down or returns invalid output, you don't want the whole CI to fail. Add a timeout and a fallback so your pipeline continues:

try:
    completion = client.beta.chat.completions.parse(
        model="gpt-4o",
        messages=...,
        response_format=PRSummary,
        timeout=30,
    )
except Exception as e:
    print(f"AI generation failed: {e}")
    # Fallback: write a generic summary
    with open("pr_summary.json", "w") as f:
        json.dump({"title": "PR Summary", "summary": "No summary generated.", "breaking_changes": False, "files_changed": []}, f)

Now you've got it. Every new PR gets a structured, validated AI summary automatically. The same pattern works for code review comments, issue triage, or even automated tests. You can extend the schema to include risk scores, test suggestions, or deployment notes.

The key takeaway: always define a contract for your AI output. Structured outputs + a validation library turn an unpredictable model into a reliable pipeline component. Start with a single workflow, measure the quality, and iterate.