Structured AI Outputs in CI/CD: Automate PR Summaries with JSON

Structured outputs from AI models let you parse responses predictably, making them ideal for automation in CI/CD pipelines. Instead of free-form text, you get JSON that you can consume directly in scripts. In this tutorial, you'll build a GitHub Action that analyzes code changes, asks GPT-4o to summarize them in a strict JSON format, and posts the result as a PR comment.

1. Define the JSON Schema

First, decide what you want the AI to return. For a PR summary, we need a title, a short description, a list of key changes, and a risk assessment.

{
  "title": "string",
  "description": "string",
  "key_changes": ["string"],
  "risk_level": "low" | "medium" | "high"
}

2. Write the Python Script

Create a file generate_pr_summary.py that reads the diff (from environment variable), calls the OpenAI API with structured outputs, and outputs JSON.

import os, json, openai
from pydantic import BaseModel

class PRSummary(BaseModel):
    title: str
    description: str
    key_changes: list[str]
    risk_level: str

client = openai.OpenAI(api_key=os.environ["OPENAI_API_KEY"])

diff = os.environ.get("PR_DIFF", "")

completion = client.beta.chat.completions.parse(
    model="gpt-4o-2024-08-06",
    messages=[
        {"role": "system", "content": "You summarize code changes in JSON."},
        {"role": "user", "content": f"Analyze this diff and provide a structured summary:\n\n{diff}"}
    ],
    response_format=PRSummary,
)

summary = completion.choices[0].message.parsed
print(summary.model_dump_json(indent=2))

3. Create the GitHub Action Workflow

In your repository, create .github/workflows/pr-summary.yml.

name: PR Summary
on:
  pull_request:
    types: [opened, synchronize]

jobs:
  summarize:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
        with:
          fetch-depth: 0
      - name: Get PR diff
        run: |
          git fetch origin ${{ github.event.pull_request.base.ref }}
          git diff origin/${{ github.event.pull_request.base.ref }}...HEAD > pr_diff.txt
      - name: Generate PR summary
        id: generate
        env:
          OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}
          PR_DIFF: "${{ github.event.pull_request.body }}"  # fallback, we'll use file
        run: |
          export PR_DIFF="$(cat pr_diff.txt)"
          python generate_pr_summary.py > summary.json
      - name: Post comment
        uses: actions/github-script@v7
        with:
          script: |
            const fs = require('fs');
            const summary = JSON.parse(fs.readFileSync('summary.json', 'utf8'));
            const comment = `## PR Summary\n\n**${summary.title}**\n\n${summary.description}\n\n### Key Changes\n${summary.key_changes.map(c => `- ${c}`).join('\n')}\n\n**Risk Level**: ${summary.risk_level}`;
            github.rest.issues.createComment({
              owner: context.repo.owner,
              repo: context.repo.repo,
              issue_number: context.issue.number,
              body: comment
            });

4. Test and Iterate

Open a PR and check the comment. If the schema constraints are too strict, adjust the JSON Schema or prompt. You can extend this to generate test suggestions or code review checklists.

This workflow gives you reliable, parseable AI outputs that fit perfectly into automated CI/CD pipelines. No more guessing what the AI meant — just clean JSON.