Parsing AI Responses with Structured Outputs in CI/CD

AI is making its way into CI/CD pipelines—generating release notes, reviewing code, and triaging issues. But one common pain point: LLMs return text, not structured data. A freeform response can break your automation if you rely on regex or lucky parsing. The fix is to use structured outputs: enforce a JSON schema on the model response so your pipeline can consume it directly.

Step 1: Define your JSON schema

Start by defining the exact shape you want from the AI. Use JSON Schema (or the API's equivalent) to describe required fields, types, and enums. Here's an example for a PR review agent:

{
  "type": "object",
  "properties": {
    "summary": { "type": "string" },
    "severity": { "enum": ["low", "medium", "high"] },
    "files_changed": { "type": "array", "items": { "type": "string" } }
  },
  "required": ["summary", "severity", "files_changed"]
}

Step 2: Call the model with structured output

Most major providers now support structured output natively. In OpenAI, use the response_format parameter with json_schema and pass your schema. In Anthropic, use tool calling with a dummy tool. Here's a Node.js example using OpenAI:

const response = await openai.chat.completions.create({
  model: "gpt-4o-mini",
  messages: [{ role: "user", content: "Review this diff: ..." }],
  response_format: {
    type: "json_schema",
    json_schema: {
      name: "pr_review",
      schema: prReviewSchema
    }
  }
});

const review = JSON.parse(response.choices[0].message.content);
console.log(review.severity); // "high"

If you're using an API that doesn't support schemas, add a strict system prompt: "Respond only with valid JSON matching this schema: ..." and parse with a fallback.

Step 3: Validate the response in CI

Parsing JSON isn't enough—you need to validate. Use a lightweight library like ajv (Node.js) or jsonschema (Python). This catches missing fields or wrong types before you use them.

import Ajv from "ajv";
const ajv = new Ajv();
const validate = ajv.compile(prReviewSchema);

if (!validate(review)) {
  throw new Error("AI response invalid: " + JSON.stringify(validate.errors));
}

Step 4: Handle errors gracefully

When the model returns invalid JSON—or the schema validator fails—you need a plan. Common strategies:

• Retry with a corrected prompt
• Fall back to a deterministic model or cache
• Fail the pipeline step with a clear message

try {
  const review = parseAndValidate(response);
} catch (err) {
  console.warn("AI review failed, skipping action");
  // or exit(1) if blocking
}

Step 5: Put it in a CI/CD step

Here's a minimal GitHub Actions workflow that runs a script and uses the AI response to comment on the PR:

name: ai-review
on: pull_request
jobs:
  review:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-node@v4
        with: { node-version: 20 }
      - run: npm install
      - run: node review.mjs
        env:
          OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}
        shell: bash

Your review.mjs handles the API call, validation, and posting the comment (via the GitHub API). No fragile text parsing.