Automate PR Descriptions with AI Agents: A Practical CI Workflow

Writing pull request descriptions is tedious. You know it, your team knows it. But AI can handle it for you. In this tutorial, you'll build a practical CI workflow that reads your diff, asks an AI model to summarize it, and posts the result as a PR description — using structured outputs to ensure the AI responds in a format your script can parse.

Prerequisites

• A GitHub repository with at least one pull request
• An OpenAI API key (or any LLM API with structured output support)
• Basic familiarity with GitHub Actions and Node.js

Step 1: Create the Workflow File

Add a new file at .github/workflows/pr-description.yml. This workflow triggers when a PR is opened or synchronized.

name: AI PR Description

on:
  pull_request:
    types: [opened, synchronize]

permissions:
  contents: read
  pull-requests: write

jobs:
  generate:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
        with:
          fetch-depth: 0
      - uses: actions/setup-node@v4
        with:
          node-version: '20'
      - run: npm install openai
      - name: Generate PR Description
        env:
          OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}
          GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
          PR_NUMBER: ${{ github.event.pull_request.number }}
          REPO: ${{ github.repository }}
        run: node .github/scripts/generate-pr-description.mjs

Step 2: Write the AI Script

Create a Node.js script at .github/scripts/generate-pr-description.mjs. This script:

1. Fetches the diff via GitHub's API.
2. Sends the diff to the AI model with a structured output schema.
3. Updates the PR description using the parsed response.

import { Octokit } from "@octokit/rest";
import OpenAI from "openai";

const octokit = new Octokit({ auth: process.env.GITHUB_TOKEN });
const openai = new OpenAI({ apiKey: process.env.OPENAI_API_KEY });

const [owner, repo] = process.env.REPO.split("/");
const prNumber = Number(process.env.PR_NUMBER);

// 1. Fetch the diff
const { data: pr } = await octokit.rest.pulls.get({ owner, repo, pull_number: prNumber });
const { data: diff } = await octokit.rest.pulls.get({
  owner, repo, pull_number: prNumber, mediaType: { format: "diff" }
});

// 2. Ask the AI for a structured PR description
const completion = await openai.chat.completions.create({
  model: "gpt-4o-mini",
  response_format: { type: "json_object" },
  messages: [
    { role: "system", content: "You are a senior dev reviewer. Analyze the diff and return JSON." },
    { role: "user", content: `Generate PR description for ${pr.title}\n\nDiff:\n${diff.slice(0, 8000)}` }
  ],
  tools: [{
    type: "function",
    function: {
      name: "set_pr_description",
      description: "Set the PR description fields",
      parameters: {
        type: "object",
        properties: {
          summary: { type: "string" },
          test_plan: { type: "string" },
          risks: { type: "array", items: { type: "string" } }
        },
        required: ["summary", "test_plan", "risks"]
      }
    }
  }]
});

// 3. Parse the structured output
const toolCall = completion.choices[0].message.tool_calls[0];
const parsed = JSON.parse(toolCall.function.arguments);

const description = `## Summary\n${parsed.summary}\n\n## Test Plan\n${parsed.test_plan}\n\n## Risks\n${parsed.risks.map(r => "- " + r).join("\n")}`;

// 4. Update the PR description
await octokit.rest.issues.update({ owner, repo, issue_number: prNumber, body: description });

console.log("PR description updated successfully!");

Step 3: Add Dependencies

You need @octokit/rest and openai as dependencies. Add a minimal package.json in your .github/scripts folder:

{
  "name": "pr-description-generator",
  "private": true,
  "type": "module",
  "dependencies": {
    "@octokit/rest": "^20.0.2",
    "openai": "^4.52.0"
  }
}

Step 4: Test It

Commit the workflow and script to a branch, then open a pull request. You should see two things:

1. The workflow running in the PR's checks.
2. The PR description being replaced within seconds.

Next Steps

• Add a fallback if the AI call fails (use the original PR body).
• Truncate the diff to avoid token limits — we already do this with slice(0, 8000), but consider using diff --stat for large PRs.
• Use a model that supports response_format: { type: "json_object" } to guarantee JSON even without tools.

This workflow is a template. Swap the model for Claude, Gemini, or a local LLM, and adjust the schema to match your team's PR template. The key insight: structured outputs turn AI from a toy into a reliable automation tool.