Parsing AI Responses in CI/CD with Structured Outputs

AI models are great at generating text, but unstructured responses are hard to parse programmatically. In CI/CD, you need predictable outputs to trigger actions, generate reports, or block merges. Structured outputs (JSON mode) solve this by forcing the model to return a valid JSON object matching your schema.

Prerequisites: Python 3.8+, an OpenAI API key, a GitHub repository with Actions enabled.

Step 1: Define Your Schema

First, decide what information you want from the AI. For a code review, we want: a summary, a list of issues (each with severity, line number, and suggestion), and a final verdict (approve/request changes). Use Pydantic to define the schema.

import openai
from pydantic import BaseModel
from typing import List, Optional

class Issue(BaseModel):
    severity: str  # "critical", "warning", "info"
    line_number: int
    description: str
    suggestion: str

class CodeReview(BaseModel):
    summary: str
    issues: List[Issue]
    verdict: str  # "approve" or "request_changes"

Step 2: Create the Review Function

Write a function that takes a diff (the code changes) and returns the structured review.

def review_code(diff: str) -> CodeReview:
    client = openai.OpenAI()
    response = client.beta.chat.completions.parse(
        model="gpt-4o-mini",
        messages=[
            {"role": "system", "content": "You are a senior code reviewer. Analyze the diff and return a structured review."},
            {"role": "user", "content": f"Review this diff:\n\n{diff}"}
        ],
        response_format=CodeReview
    )
    return response.choices[0].message.parsed

Step 3: Build the CI/CD Script

Create a Python script that fetches the PR diff, calls the review function, and outputs a JSON report. Then use exit code to signal failure.

import sys
import json
import os
from pathlib import Path
from review import review_code  # assume above code is in review.py

def main():
    diff_file = sys.argv[1]
    diff = Path(diff_file).read_text()
    if not diff.strip():
        print("No changes to review.")
        sys.exit(0)
    
    review = review_code(diff)
    report = review.model_dump()
    
    # Write structured output
    with open("review-report.json", "w") as f:
        json.dump(report, f, indent=2)
    
    # Check verdict
    if review.verdict == "request_changes":
        print("❌ Review failed. Issues found:")
        for issue in review.issues:
            print(f"  [{issue.severity}] Line {issue.line_number}: {issue.description}")
        sys.exit(1)
    else:
        print("✅ Review passed.")
        sys.exit(0)

if __name__ == "__main__":
    main()

Step 4: GitHub Action Workflow

Create .github/workflows/ai-review.yml that triggers on pull request events, checks out the repo, installs dependencies, and runs the script.

name: AI Code Review
on:
  pull_request:
    types: [opened, synchronize]

jobs:
  review:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - name: Set up Python
        uses: actions/setup-python@v5
        with:
          python-version: '3.11'
      - name: Install dependencies
        run: |
          pip install openai pydantic
      - name: Get diff
        id: diff
        run: |
          git fetch origin ${{ github.base_ref }} 
          git diff origin/${{ github.base_ref }}...HEAD > diff.txt
      - name: Run AI Review
        env:
          OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}
        run: python ci_review.py diff.txt
      - name: Upload Report
        if: always()
        uses: actions/upload-artifact@v4
        with:
          name: review-report
          path: review-report.json

Step 5: Use the Report in Downstream Jobs

Add a step to post the review result as a PR comment using GitHub's API. Download the artifact and parse the JSON to create a markdown comment.

import json
import os
from github import Github

# Inside a new action step
with open("review-report.json") as f:
    report = json.load(f)

g = Github(os.environ["GITHUB_TOKEN"])
repo = g.get_repo(os.environ["GITHUB_REPOSITORY"])
pr = repo.get_pull(int(os.environ["PR_NUMBER"]))

comment_body = f"## AI Review\n\n**Summary:** {report['summary']}\n\n**Verdict:** {report['verdict']}\n"
for issue in report['issues']:
    comment_body += f"- [{issue['severity']}] Line {issue['line_number']}: {issue['description']}\n"

pr.create_issue_comment(comment_body)

Don't forget to install PyGithub and pass the GITHUB_TOKEN secret.

Conclusion

By using structured outputs, you eliminate guesswork from AI responses. Your CI/CD pipeline can now deterministically parse AI feedback, enforce rules, and produce machine-readable artifacts. This pattern works for PR summaries, vulnerability scanning, documentation generation, and more.