Fixes missing `run.metadata.usage_metadata` population in
`LangChainTracer` for real LLM/chat traces following #34414
- Fix extraction to read usage from serialized tracer message shape:
`outputs.generations[*][*].message.kwargs.usage_metadata`
- Remove non-serialized direct message shape handling
(`message.usage_metadata`) from extractor to match real tracer output
path
- Clarify tracer docstrings around chat callback naming
(`on_chat_model_start` + shared `on_llm_end`) to reduce ambiguity
## Why
#34414 introduced usage duplication into `run.metadata.usage_metadata`,
but the extractor read `message.usage_metadata`.
In real tracer flow, messages are serialized with `dumpd(...)` during
run completion, so usage metadata lives under
`message.kwargs.usage_metadata`. Because of this mismatch, duplication
did not trigger in real traces.
Adds automatic tool call counting to tracing by means of a new
`store_tool_call_count_in_run()`, which calls on newly added
`count_tool_calls_in_run()`.
Runs on successful LLM completion. Does not run on errored runs.
Adds `usage_metadata` (token counts, etc.) to the run metadata in
`LangChainTracer`.
When an LLM run ends, usage metadata is extracted from all generations
and aggregated using the existing `add_usage` helper, then stored in
`run.extra["metadata"]["usage_metadata"]`.
The original data in outputs remains unchanged.
Also, see #34415
---------
Co-authored-by: Mason Daugherty <github@mdrxy.com>
Co-authored-by: Mason Daugherty <mason@langchain.dev>
ref https://github.com/langchain-ai/langchainjs/pull/9665
Fixes trace persistence for iterator/generator inputs (like
`RunnableGenerator`) where the full input isn't available at chain
start. Instead of POSTing a run with incomplete inputs on start and
PATCHing later, this defers the POST until chain end when inputs are
fully realized.
---------
Co-authored-by: Mason Daugherty <github@mdrxy.com>
* Fix detection of support of context in `asyncio.create_task`
* Fix: in Python 3.14 `asyncio.get_event_loop()` raises an exception if
there's no running loop
* Bump pydantic to version 2.12
* Skips tests with pydantic v1 models as they are not supported with
Python 3.14
* Run core tests with Python 3.14 in CI.
---------
Co-authored-by: Mason Daugherty <mason@langchain.dev>
Co-authored-by: Sydney Runkle <54324534+sydney-runkle@users.noreply.github.com>
This PR uses the [blockbuster](https://github.com/cbornet/blockbuster)
library in langchain-core to detect blocking calls made in the asyncio
event loop during unit tests.
Avoiding blocking calls is hard as these can be deeply buried in the
code or made in 3rd party libraries.
Blockbuster makes it easier to detect them by raising an exception when
a call is made to a known blocking function (eg: `time.sleep`).
Adding blockbuster allowed to find a blocking call in
`aconfig_with_context` (it ends up calling `get_function_nonlocals`
which loads function code).
**Dependencies:**
- blockbuster (test)
**Twitter handle:** cbornet_
Ruff doesn't know about the python version in
`[tool.poetry.dependencies]`. It can get it from
`project.requires-python`.
Notes:
* poetry seems to have issues getting the python constraints from
`requires-python` and using `python` in per dependency constraints. So I
had to duplicate the info. I will open an issue on poetry.
* `inspect.isclass()` doesn't work correctly with `GenericAlias`
(`list[...]`, `dict[..., ...]`) on Python <3.11 so I added some `not
isinstance(type, GenericAlias)` checks:
Python 3.11
```pycon
>>> import inspect
>>> inspect.isclass(list)
True
>>> inspect.isclass(list[str])
False
```
Python 3.9
```pycon
>>> import inspect
>>> inspect.isclass(list)
True
>>> inspect.isclass(list[str])
True
```
Co-authored-by: Eugene Yurtsev <eyurtsev@gmail.com>
- [x] **Adding AsyncRootListener**: "langchain_core: Adding
AsyncRootListener"
- **Description:** Adding an AsyncBaseTracer, AsyncRootListener and
`with_alistener` function. This is to enable binding async root listener
to runnables. This currently only supported for sync listeners.
- **Issue:** None
- **Dependencies:** None
- [x] **Add tests and docs**: Added units tests and example snippet code
within the function description of `with_alistener`
- [x] **Lint and test**: Run make format_diff, make lint_diff and make
test
Check if event stream is closed in memory loop.
Using try/except here to avoid race condition, but this may incur a
small overhead in versions prios to 3.11
Removes required usage of `requests` from `langchain-core`, all of which
has been deprecated.
- removes Tracer V1 implementations
- removes old `try_load_from_hub` github-based hub implementations
Removal done in a way where imports will still succeed, and usage will
fail with a `RuntimeError`.
This PR replaces the memory stream implementation used by the
LogStreamCallbackHandler.
This implementation resolves an issue in which streamed logs and
streamed events originating from sync code would arrive only after the
entire sync code would finish execution (rather than arriving in real
time as they're generated).
One example is if trying to stream tokens from an llm within a tool. If
the tool was an async tool, but the llm was invoked via stream (sync
variant) rather than astream (async variant), then the tokens would fail
to stream in real time and would all arrived bunched up after the tool
invocation completed.
Pydantic's `dict()` function raises an error here if you pass in a
generator. We have a more robust serialization function in lagnsmith
that we will use instead.
Fix some circular deps:
- move PromptValue into top level module bc both PromptTemplates and
OutputParsers import
- move tracer context vars to `tracers.context` and import them in
functions in `callbacks.manager`
- add core import tests
