Validate AI Responses in CI/CD with Structured Outputs

When you integrate AI into CI/CD pipelines, one of the biggest pain points is unreliable output. An LLM might return valid JSON with a missing field, an incorrect type, or worse – a plausible-looking but wrong structure. Without validation, these errors propagate silently, causing downstream failures. The solution: use structured outputs with runtime validation. This tutorial shows you how to parse and validate LLM responses in your CI pipeline using Pydantic and catch issues early.

Step 1: Define a Pydantic model for the expected output

Create a Python file (models.py) that describes the exact shape of the data you want from the LLM. For example, if you're generating code review comments:

from pydantic import BaseModel, Field
from typing import List

class ReviewComment(BaseModel):
    file_path: str = Field(description="Path to the file under review")
    line_number: int = Field(ge=1, description="Line number where comment applies")
    severity: str = Field(pattern="^(critical|major|minor|info)$")
    message: str = Field(description="The review comment text")

class ReviewResponse(BaseModel):
    summary: str
    comments: List[ReviewComment]

Step 2: Call the LLM with a system prompt that requests JSON

In your pipeline script, make an API call to the LLM with a system message instructing it to output JSON matching your model's schema. Use the response_format parameter if available (OpenAI supports json_object).

import openai

client = openai.Client()

response = client.chat.completions.create(
    model="gpt-4o",
    response_format={"type": "json_object"},
    messages=[
        {"role": "system", "content": "You are a code reviewer. Output JSON conforming to this schema: " + ReviewResponse.model_json_schema()},
        {"role": "user", "content": f"Review this diff:\n{diff_content}"}
    ],
    temperature=0.1
)

Step 3: Parse and validate the response

After receiving the response, attempt to parse the JSON string into your Pydantic model. If validation fails, log the error and exit with a non-zero code to fail the pipeline step.

import json
from pydantic import ValidationError

raw_content = response.choices[0].message.content
try:
    data = json.loads(raw_content)
    review = ReviewResponse.model_validate(data)
    print("Validation passed!")
except (json.JSONDecodeError, ValidationError) as e:
    print(f"Invalid response: {e}")
    exit(1)

Step 4: Integrate into GitHub Actions

Create a workflow file (.github/workflows/ai-review.yml) that runs the script on pull requests. Use environment variables for your API key.

name: AI Code Review
on: pull_request
jobs:
  review:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with:
          python-version: '3.11'
      - run: pip install openai pydantic
      - env:
          OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}
        run: python review.py

Step 5: Handle retries and edge cases

LLMs occasionally return malformed JSON even with structured prompting. Add a retry loop (up to 3 attempts) with exponential backoff to increase robustness. Also consider logging the raw response for debugging.

import time
for attempt in range(3):
    response = client.chat.completions.create(...)
    try:
        review = ReviewResponse.model_validate(json.loads(response.choices[0].message.content))
        break
    except (json.JSONDecodeError, ValidationError) as e:
        if attempt == 2:
            print("All retries exhausted. Failing.")
            exit(1)
        time.sleep(2 ** attempt)

By following this pattern, you can extend the same concept to any task: generating commit messages, summarizing PR descriptions, or even transforming code. The key is to define your data contracts upfront and let Pydantic enforce them. Start small, test often, and iterate from there.