To make migration easier, things are more backwards compat
Very minimal footprint here
Will need to upgrade migration guide and other docs w/ this change
Adds `ToolRetryMiddleware` to automatically retry failed tool calls with
configurable exponential backoff, exception filtering, and error
handling.
## Example
```python
from langchain.agents import create_agent
from langchain.agents.middleware import ToolRetryMiddleware
from langchain_openai import ChatOpenAI
# Retry up to 3 times with exponential backoff
retry = ToolRetryMiddleware(
max_retries=3,
initial_delay=1.0,
backoff_factor=2.0,
)
agent = create_agent(
model=ChatOpenAI(model="gpt-4"),
tools=[search_tool, database_tool],
middleware=[retry],
)
# Tool failures are automatically retried
result = agent.invoke({"messages": [{"role": "user", "content": "Search for AI news"}]})
```
For advanced usage with specific exception handling:
```python
from requests.exceptions import Timeout, HTTPError
def should_retry(exc: Exception) -> bool:
# Only retry on 5xx errors or timeouts
if isinstance(exc, HTTPError):
return 500 <= exc.response.status_code < 600
return isinstance(exc, Timeout)
retry = ToolRetryMiddleware(
max_retries=4,
retry_on=should_retry,
tools=["search_database"], # Only apply to specific tools
)
```
Largely:
- Remove explicit `"Default is x"` since new refs show default inferred
from sig
- Inline code (useful for eventual parsing)
- Fix code block rendering (indentations)
The one risk point that I can see here is that model + tool call
counting now occurs in the `after_model` hook which introduces order
dependency (what if you have HITL execute before this hook and we jump
early to `model`, for example).
This is something users can work around at the moment and we can
document. We could also introduce a priority concept to middleware.
This is tool emulation middleware. The idea is to help test out an agent
that may have some tools that either take a long time to run or are
expensive to set up. This could allow simulating the behavior a bit.
Easiest to review side by side (not inline)
* Adding `dict` type requests + responses so that we can ship config w/
interrupts. Also more extensible.
* Keeping things generic in terms of `interrupt_on` rather than
`tool_config`
* Renaming allowed decisions -- approve, edit, reject
* Draws differentiation between actions (requested + performed by the
agent), in this case tool calls, though we generalize beyond that and
decisions - human feedback for said actions
New request structure
```py
class Action(TypedDict):
"""Represents an action with a name and arguments."""
name: str
"""The type or name of action being requested (e.g., "add_numbers")."""
arguments: dict[str, Any]
"""Key-value pairs of arguments needed for the action (e.g., {"a": 1, "b": 2})."""
DecisionType = Literal["approve", "edit", "reject"]
class ReviewConfig(TypedDict):
"""Policy for reviewing a HITL request."""
action_name: str
"""Name of the action associated with this review configuration."""
allowed_decisions: list[DecisionType]
"""The decisions that are allowed for this request."""
description: NotRequired[str]
"""The description of the action to be reviewed."""
arguments_schema: NotRequired[dict[str, Any]]
"""JSON schema for the arguments associated with the action, if edits are allowed."""
class HITLRequest(TypedDict):
"""Request for human feedback on a sequence of actions requested by a model."""
action_requests: list[Action]
"""A list of agent actions for human review."""
review_configs: list[ReviewConfig]
"""Review configuration for all possible actions."""
```
New response structure
```py
class ApproveDecision(TypedDict):
"""Response when a human approves the action."""
type: Literal["approve"]
"""The type of response when a human approves the action."""
class EditDecision(TypedDict):
"""Response when a human edits the action."""
type: Literal["edit"]
"""The type of response when a human edits the action."""
edited_action: Action
"""Edited action for the agent to perform.
Ex: for a tool call, a human reviewer can edit the tool name and args.
"""
class RejectDecision(TypedDict):
"""Response when a human rejects the action."""
type: Literal["reject"]
"""The type of response when a human rejects the action."""
message: NotRequired[str]
"""The message sent to the model explaining why the action was rejected."""
Decision = ApproveDecision | EditDecision | RejectDecision
class HITLResponse(TypedDict):
"""Response payload for a HITLRequest."""
decisions: list[Decision]
"""The decisions made by the human."""
```
User facing API:
NEW
```py
HumanInTheLoopMiddleware(interrupt_on={
'send_email': True,
# can also use a callable for description that takes tool call, state, and runtime
'execute_sql': {
'allowed_decisions': ['approve', 'edit', 'reject'],
'description': 'please review sensitive tool execution'},
}
})
Command(resume={"decisions": [{"type": "approve"}, {"type": "reject": "message": "db down"}]})
```
OLD
```py
HumanInTheLoopMiddleware(interrupt_on={
'send_email': True,
'execute_sql': {
'allow_accept': True,
'allow_edit': True,
'allow_respond': True,
description='please review sensitive tool execution'
},
})
Command(resume=[{"type": "approve"}, {"type": "reject": "message": "db down"}])
```
Refactor tool call middleware from generator-based to handler-based
pattern
Simplifies on_tool_call middleware by replacing the complex generator
protocol with a straightforward handler pattern. Instead of yielding
requests and receiving results via .send(),
handlers now receive an execute callable that can be invoked multiple
times for retry logic.
Before vs. After
Before (Generator):
```python
class RetryMiddleware(AgentMiddleware):
def on_tool_call(self, request, state, runtime):
for attempt in range(3):
response = yield request # Yield request, receive result via .send()
if is_valid(response) or attempt == 2:
return # Final result is last value sent to generator
```
After (Handler):
```python
class RetryMiddleware(AgentMiddleware):
def on_tool_call(self, request, handler):
for attempt in range(3):
result = handler(request) # Direct function call
if is_valid(result):
return result
return result
```
Follow up after this PR:
* Rename the interceptor to wrap_tool_call
* Fix the async path for the ToolNode
1. Main fix: when we don't have a response format or middleware, don't
draw a conditional edge back to the loop entrypoint (self loop on model)
2. Supplementary fix: when we jump to `end` and there is an
`after_agent` hook, jump there instead of `__end__`
Other improvements -- I can remove these if they're more harmful than
helpful
1. Use keyword only arguments for edge generator functions for clarity
2. Rename args to `model_destination` and `end_destination` for clarity
This reduces confusion w/ types like `AgentState`, different arg names,
etc.
Second attempt, following
https://github.com/langchain-ai/langchain/pull/33249
* Ability to pass through `cache` and name in `create_agent` as
compilation args for the agent
* Right now, removing `test_react_agent.py` but we should add these
tests back as implemented w/ the new agent
* Add conditional edge when structured output tools are present to allow
for retries
* Rename `tracking` to `model_call_limit` to be consistent w/ tool call
limits
We need in the future (I'm happy to own):
* Significant test refactor
* Significant test overhaul where we emphasize and enforce coverage
