LLMs are powerful but unpredictable. When you integrate them into CI/CD, you need reliable, parseable outputs. This tutorial shows you how to use OpenAI's Structured Outputs feature to enforce a JSON schema on AI responses, then process them in a GitHub Actions workflow.

Prerequisites: A GitHub repository, an OpenAI API key (with models that support structured outputs, like gpt-4o-mini), and basic familiarity with GitHub Actions.

Step 1: Define Your Structured Output Schema

Instead of asking the LLM to return a free-form description, define a JSON schema. For example, to generate a changelog from a git diff:

{
  "type": "object",
  "properties": {
    "summary": { "type": "string", "description": "Short PR summary" },
    "breaking_changes": { "type": "boolean" },
    "commit_messages": {
      "type": "array",
      "items": { "type": "string" }
    }
  },
  "required": ["summary", "breaking_changes", "commit_messages"]
}

Step 2: Use the OpenAI API with Response Format

Pass the schema as response_format parameter. In Python:

from openai import OpenAI
import json

client = OpenAI(api_key="YOUR_API_KEY")

schema = {
  "type": "object",
  "properties": {
    "summary": {"type": "string"},
    "breaking_changes": {"type": "boolean"},
    "commit_messages": {"type": "array", "items": {"type": "string"}}
  },
  "required": ["summary", "breaking_changes", "commit_messages"]
}

diff = """commit a1b2c3
fix: handle null pointer in login"""

response = client.chat.completions.create(
    model="gpt-4o-mini",
    messages=[
        {"role": "system", "content": "You are a helpful assistant that summarizes git diffs."},
        {"role": "user", "content": f"Summarize the following diff:\n{diff}"}
    ],
    response_format={
        "type": "json_schema",
        "json_schema": {
            "name": "changelog",
            "strict": True,
            "schema": schema
        }
    }
)

# Parse the response - guaranteed valid JSON
output = json.loads(response.choices[0].message.content)
print(output["summary"])
Why this works: The response_format enforces the schema. The API will not return invalid JSON, and it will always contain all required fields. No more regex parsing!.

Step 3: Integrate into a CI/CD Pipeline (GitHub Actions)

Create a workflow that runs on pull requests. It will analyze the diff, call OpenAI, and post a comment.

name: AI Changelog
on:
  pull_request:
    types: [opened, synchronize]

jobs:
  generate-changelog:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
        with:
          fetch-depth: 0

      - name: Get git diff
        id: diff
        run: |
          diff=$(git diff origin/${{ github.base_ref }}...${{ github.head_ref }})
          # Escape for JSON
          diff="${diff//'%'/'%25'}"
          diff="${diff//$'\n'/'%0A'}"
          diff="${diff//$'\r'/'%0D'}"
          echo "diff=$diff" >> $GITHUB_OUTPUT

      - name: Call OpenAI
        id: ai
        uses: fjogeleit/http-request-action@v1
        with:
          url: https://api.openai.com/v1/chat/completions
          method: 'POST'
          customHeaders: '{"Authorization":"Bearer ${{ secrets.OPENAI_API_KEY }}","Content-Type":"application/json"}'
          data: >
            {
              "model": "gpt-4o-mini",
              "messages": [
                {"role": "system", "content": "You return valid JSON matching the schema."},
                {"role": "user", "content": "Summarize this diff: ${{ steps.diff.outputs.diff }}"}
              ],
              "response_format": {
                "type": "json_schema",
                "json_schema": {
                  "name": "changelog",
                  "strict": true,
                  "schema": {
                    "type": "object",
                    "properties": {
                      "summary": {"type": "string"},
                      "breaking_changes": {"type": "boolean"},
                      "commit_messages": {"type": "array", "items": {"type": "string"}}
                    },
                    "required": ["summary", "breaking_changes", "commit_messages"]
                  }
                }
              }
            }

      - name: Parse and comment
        uses: actions/github-script@v7
        with:
          script: |
            const body = JSON.parse(${{ steps.ai.outputs.response }});
            const content = JSON.parse(body.choices[0].message.content);
            const comment = `## AI Changelog\n\n**Summary:** ${content.summary}\n\n**Breaking Changes:** ${content.breaking_changes}\n\n**Commit Messages:**\n${content.commit_messages.map(c => `- ${c}`).join('\n')}`;
            github.rest.issues.createComment({
              issue_number: context.issue.number,
              owner: context.repo.owner,
              repo: context.repo.repo,
              body: comment
            });
Note: The above uses a third-party HTTP action. In production, consider writing a custom action using the OpenAI SDK for better error handling.

Step 4: Handle Errors Gracefully

Even with structured outputs, network errors or API limits can occur. Wrap your calls with retry logic and fallback messages. Example Python snippet:

import time
import openai

for attempt in range(3):
    try:
        response = openai.chat.completions.create(...)
        break
    except openai.APIError as e:
        if attempt == 2:
            print("Fallback: Using default summary.")
            # Use a pre-defined fallback JSON
            break
        time.sleep(2 ** attempt)

Conclusion

Structured outputs eliminate the worst part of LLM integration: parsing unreliable text. By enforcing a schema at the API level, you can safely chain AI calls in CI/CD without validation headaches. Start with a simple changelog generator and extend to code review suggestions, test generation, or automated documentation.

Try it now: Copy the GitHub Actions workflow above, replace the schema with your own, and see the magic on your next PR.