`build_summary` in `langchain_model_profiles._summary` now wraps each
provider section in a `<details>` toggle when more than one provider has
changes, making multi-provider refresh PR summaries skimmable.
Single-provider summaries are unchanged.
---
When the `refresh_model_profiles` workflow refreshes data for multiple
partner integrations at once, the resulting PR summary can be long and
hard to skim — every provider's added/removed/changed rows are rendered
flat, one after another. This wraps each provider section in a
`<details>`/`<summary>` toggle when more than one provider has changes,
so reviewers can expand only the providers they care about.
Single-provider summaries stay flat since there's nothing to collapse.
The per-provider `### {provider}` heading is stripped inside toggles so
the `<summary>` tag is the sole label — no duplicated provider name.
Also includes two smaller changes that were staged alongside:
- The `refresh_model_profiles` and `_refresh_model_profiles` workflow
PR-body templates now link to the workflow file instead of bare-text
referencing it.
- `AGENTS.md` and `CLAUDE.md` document the LangSmith integration test
tracing setup: the env vars CI sets, the pytest plugin that bridges
`LANGSMITH_TAGS`/`LANGSMITH_METADATA` into `tracing_context`, and the
unit-test isolation approach.
---------
Signed-off-by: Mason Daugherty <github@mdrxy.com>
Co-authored-by: open-swe[bot] <215916821+open-swe[bot]@users.noreply.github.com>
Automated model-profile refresh PRs (e.g. #38210) ship a static template
body, so a reviewer has to open *Files changed* and read large blocks of
generated data to learn what actually moved. Because the underlying
profile data is fully structured, we can describe the changes
deterministically — no LLM, no hallucination risk.
This adds a `langchain-profiles summarize` subcommand that compares the
working-tree `_profiles.py` files against a git ref and renders a
skimmable Markdown summary: models added (with a short capability
descriptor), models removed, and per-field capability changes
(context/output tokens, modalities, tool calling, reasoning, etc.),
grouped by provider and capped so huge refreshes stay readable. Profiles
are read with `ast.literal_eval` rather than imported, so the generated
data file is never executed.
Example output for a refresh that adds a model and bumps an output
limit:
```
## Summary of changes
**1 added · 0 removed · 1 changed** across 1 provider(s).
### openai
**➕ 1 added**
- `gpt-6-preview` — 1,000,000 ctx, 128,000 out, text+image+audio in, reasoning, tools
**✏️ 1 changed**
- `gpt-3.5-turbo`: max output tokens 4,096 → 16,384
```
Made by [Open
SWE](https://openswe.vercel.app/agents/9bcbf182-effc-ba9b-0df3-afac620ad152)
---------
Co-authored-by: open-swe[bot] <open-swe@users.noreply.github.com>
Originally a narrow bump of mypy to `1.20` in four packages. Expanded to
get the whole monorepo onto a single, current mypy and a consistent
type-check configuration, so contributors no longer hit different mypy
versions and divergent behavior depending on which package they touch.
### What changed
- **Unified the mypy pin to `>=2.1.0,<2.2.0`** in every mypy-using
package (6 libs + 14 partners), replacing the previously scattered pins
(`1.10`/`1.17`/`1.18`/`1.19`/`1.20`, with assorted upper bounds).
- **Unified the `[tool.mypy]` base per tier:**
- libs: `plugins = ["pydantic.mypy"]`, `strict = true`,
`enable_error_code = "deprecated"`, `warn_unreachable = true`
- partners: `disallow_untyped_defs = true`
- Normalized style (`disallow_untyped_defs = "True"` string → bool,
quote/key consistency).
- **Fixed the 20 real errors** mypy 2.1 surfaces: `redundant-cast` from
improved narrowing (`core`, `langchain-classic`), a `var-annotated` for
`_LOGGED`, a return-type widening in `langchain-groq`'s
`_convert_from_v1_to_groq` (it can legitimately return a bare `str`),
and stale `type-arg`/`unused-ignore` in `langchain-model-profiles`
tests.
### Deliberate non-uniformity (documented inline in the relevant
`pyproject.toml`s)
Going fully byte-identical would surface ~196 additional errors that are
*not* real bugs, so two settings are kept package-appropriate:
- **`warn_unreachable`** is enabled on every strict lib **except
`core`**, where it false-flags intentional defensive code — including
the SSRF / IP-policy guards in `_security/` — as unreachable.
- **`pydantic.mypy` plugin** is used only on `anthropic` and
`perplexity` (their code is authored against it and reports ~99/~132
errors without it). It is *not* added to the other partners, where it
only flags the public alias constructor API (e.g. `ChatGroq(model=...)`)
in tests rather than finding bugs.
- **`ollama`** is left on its `ty` type checker; it does not use mypy.
---------
Co-authored-by: Mason Daugherty <github@mdrxy.com>
PR #35788 added 7 new fields to the `langchain-profiles` CLI output
(`name`, `status`, `release_date`, `last_updated`, `open_weights`,
`attachment`, `temperature`) but didn't update `ModelProfile` in
`langchain-core`. Partner packages like `langchain-aws` that set
`extra="forbid"` on their Pydantic models hit `extra_forbidden`
validation errors when Pydantic encountered undeclared TypedDict keys at
construction time. This adds the missing fields, makes `ModelProfile`
forward-compatible, provides a base-class hook so partners can stop
duplicating model-profile validator boilerplate, migrates all in-repo
partners to the new hook, and adds runtime + CI-time warnings for schema
drift.
## Changes
### `langchain-core`
- Add `__pydantic_config__ = ConfigDict(extra="allow")` to
`ModelProfile` so unknown profile keys pass Pydantic validation even on
models with `extra="forbid"` — forward-compatibility for when the CLI
schema evolves ahead of core
- Declare the 7 missing fields on `ModelProfile`: `name`, `status`,
`release_date`, `last_updated`, `open_weights` (metadata) and
`attachment`, `temperature` (capabilities)
- Add `_warn_unknown_profile_keys()` in `model_profile.py` — emits a
`UserWarning` when a profile dict contains keys not in `ModelProfile`,
suggesting a core upgrade. Wrapped in a bare `except` so introspection
failures never crash model construction
- Add `BaseChatModel._resolve_model_profile()` hook that returns `None`
by default. Partners can override this single method instead of
redefining the full `_set_model_profile` validator — the base validator
calls it automatically
- Add `BaseChatModel._check_profile_keys` as a separate
`model_validator` that calls `_warn_unknown_profile_keys`. Uses a
distinct method name so partner overrides of `_set_model_profile` don't
inadvertently suppress the check
### `langchain-profiles` CLI
- Add `_warn_undeclared_profile_keys()` to the CLI (`cli.py`), called
after merging augmentations in `refresh()` — warns at profile-generation
time (not just runtime) when emitted keys aren't declared in
`ModelProfile`. Gracefully skips if `langchain-core` isn't installed
- Add guard test
`test_model_data_to_profile_keys_subset_of_model_profile` in
model-profiles — feeds a fully-populated model dict to
`_model_data_to_profile()` and asserts every emitted key exists in
`ModelProfile.__annotations__`. CI fails before any release if someone
adds a CLI field without updating the TypedDict
### Partner packages
- Migrate all 10 in-repo partners to the `_resolve_model_profile()`
hook, replacing duplicated `@model_validator` / `_set_model_profile`
overrides: anthropic, deepseek, fireworks, groq, huggingface, mistralai,
openai (base + azure), openrouter, perplexity, xai
- Anthropic retains custom logic (context-1m beta → `max_input_tokens`
override); all others reduce to a one-liner
- Add `pr_lint.yml` scope for the new `model-profiles` package
Extract additional fields from models.dev into `_model_data_to_profile`:
`name`, `status`, `release_date`, `last_updated`, `open_weights`,
`attachment`, `temperature`
Move the model profile refresh logic from an inline bash script in the
GitHub Actions workflow into a `make refresh-profiles` target in
`libs/model-profiles/Makefile`. This makes it runnable locally with a
single command and keeps the provider map in one place instead of
duplicated between CI and developer docs.
- Sort model profiles alphabetically by model ID (the top-level
`_PROFILES` dictionary keys, e.g. `claude-3-5-haiku-20241022`,
`gpt-4o-mini`) before writing `_profiles.py`, so that regenerating
profiles only shows actual data changes in diffs — not random reordering
from the models.dev API response order
- Regenerate all 10 partner profile files with the new sorted ordering
- Add `text_inputs` and `text_outputs` fields to `ModelProfile`
- Regenerate `_profiles.py` for all providers
## Why
models.dev data includes `'text'` as both an input and output modality,
but we didn't capture it.
models.dev broadly contains models without text input (Whisper/ASR) and
without text output (image generators, TTS).
Without this, downstream consumers can't filter on model text support
(e.g. preventing users from passing text input to an audio-only model).
---
We'd need to also run for Google, AWS and cut releases for all to
propagate
