feat(model-profiles): plain-English summary for profile refresh PRs (#38218)

Automated model-profile refresh PRs (e.g. #38210) ship a static template
body, so a reviewer has to open *Files changed* and read large blocks of
generated data to learn what actually moved. Because the underlying
profile data is fully structured, we can describe the changes
deterministically — no LLM, no hallucination risk.

This adds a `langchain-profiles summarize` subcommand that compares the
working-tree `_profiles.py` files against a git ref and renders a
skimmable Markdown summary: models added (with a short capability
descriptor), models removed, and per-field capability changes
(context/output tokens, modalities, tool calling, reasoning, etc.),
grouped by provider and capped so huge refreshes stay readable. Profiles
are read with `ast.literal_eval` rather than imported, so the generated
data file is never executed.

Example output for a refresh that adds a model and bumps an output
limit:

```
## Summary of changes

**1 added · 0 removed · 1 changed** across 1 provider(s).

### openai

** 1 added**
- `gpt-6-preview` — 1,000,000 ctx, 128,000 out, text+image+audio in, reasoning, tools

**✏️ 1 changed**
- `gpt-3.5-turbo`: max output tokens 4,096 → 16,384
```

Made by [Open
SWE](https://openswe.vercel.app/agents/9bcbf182-effc-ba9b-0df3-afac620ad152)

---------

Co-authored-by: open-swe[bot] <open-swe@users.noreply.github.com>
This commit is contained in:
Mason Daugherty
2026-06-22 22:15:29 -04:00
committed by GitHub
parent c669da7662
commit f7e87f7ab8
5 changed files with 1017 additions and 3 deletions

View File

@@ -167,6 +167,32 @@ jobs:
exit 1
fi
- name: "📝 Build PR body with change summary"
id: pr-body
env:
PROVIDERS_JSON: ${{ inputs.providers }}
PR_BODY: ${{ inputs.pr-body }}
run: |
# The refresh step modified the working tree without committing, so
# comparing against HEAD yields exactly the refresh's changes.
cli_dir="${GITHUB_WORKSPACE}/${{ steps.cli.outputs.dir }}"
body_file="${RUNNER_TEMP}/pr_body.md"
printf '%s\n\n' "${PR_BODY}" > "${body_file}"
# `summarize` builds the whole summary in memory and prints it once,
# so a failure exits non-zero before any stdout reaches the append —
# the body keeps only the static note, never a half-written summary.
if ! uv run --frozen --project "${cli_dir}" \
langchain-profiles summarize \
--providers "${PROVIDERS_JSON}" \
--base-ref HEAD \
--repo-root "${GITHUB_WORKSPACE}" >> "${body_file}"; then
echo "::warning::Could not generate change summary; see job log."
# Surface the degradation in the PR body too: the warning above only
# lands in the Actions log, which a PR reviewer won't see.
printf '\n> [!NOTE]\n> Automated change summary unavailable — see the workflow run log.\n' >> "${body_file}"
fi
echo "path=${body_file}" >> "$GITHUB_OUTPUT"
- name: "🔑 Generate GitHub App token"
id: app-token
uses: actions/create-github-app-token@bcd2ba49218906704ab6c1aa796996da409d3eb1 # v3
@@ -182,7 +208,7 @@ jobs:
branch: ${{ inputs.pr-branch }}
commit-message: ${{ inputs.pr-title }}
title: ${{ inputs.pr-title }}
body: ${{ inputs.pr-body }}
body-path: ${{ steps.pr-body.outputs.path }}
labels: ${{ inputs.pr-labels }}
add-paths: ${{ inputs.add-paths }}