mirror of
https://github.com/hwchase17/langchain.git
synced 2025-08-17 16:39:52 +00:00
feat(docs): add truefoundry ai gateway (#32362)
This PR adds documentation for integrating [TrueFoundry’s AI Gateway](https://www.truefoundry.com/ai-gateway) with Langfuse using the Langraph OpenAI SDK. The integration sends requests through TrueFoundry’s AI Gateway for unified governance, observability, and routing, while Langraph runs on the client side to capture execution traces and telemetry. - Issue: N/A - Dependencies: None - Twitter - https://x.com/truefoundry tests - Not applicable --------- Co-authored-by: Mason Daugherty <mason@langchain.dev> Co-authored-by: Mason Daugherty <github@mdrxy.com> Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
This commit is contained in:
parent
c8df6c7ec9
commit
7f259863e1
101
docs/docs/integrations/providers/truefoundry.mdx
Normal file
101
docs/docs/integrations/providers/truefoundry.mdx
Normal file
@ -0,0 +1,101 @@
|
||||
# TrueFoundry
|
||||
|
||||
TrueFoundry provides an enterprise-ready [AI Gateway](https://www.truefoundry.com/ai-gateway) to provide governance and observability to agentic frameworks like LangChain. TrueFoundry AI Gateway serves as a unified interface for LLM access, providing:
|
||||
|
||||
- **Unified API Access**: Connect to 250+ LLMs (OpenAI, Claude, Gemini, Groq, Mistral) through one API
|
||||
- **Low Latency**: Sub-3ms internal latency with intelligent routing and load balancing
|
||||
- **Enterprise Security**: SOC 2, HIPAA, GDPR compliance with RBAC and audit logging
|
||||
- **Quota and cost management**: Token-based quotas, rate limiting, and comprehensive usage tracking
|
||||
- **Observability**: Full request/response logging, metrics, and traces with customizable retention
|
||||
|
||||
|
||||
## Prerequisites
|
||||
|
||||
Before integrating LangChain with TrueFoundry, ensure you have:
|
||||
|
||||
1. **TrueFoundry Account**: A [TrueFoundry account](https://www.truefoundry.com/register) with at least one model provider configured. Follow quick start guide [here](https://docs.truefoundry.com/gateway/quick-start)
|
||||
2. **Personal Access Token**: Generate a token by following the [TrueFoundry token generation guide](https://docs.truefoundry.com/gateway/authentication)
|
||||
|
||||
## Quickstart
|
||||
|
||||
You can connect to TrueFoundry's unified LLM gateway through the `ChatOpenAI` interface.
|
||||
|
||||
- Set the `base_url` to your TrueFoundry endpoint (explained below)
|
||||
- Set the `api_key` to your TrueFoundry [PAT (Personal Access Token)](https://docs.truefoundry.com/gateway/authentication#personal-access-token-pat)
|
||||
- Use the same `model-name` as shown in the unified code snippet
|
||||
|
||||

|
||||
|
||||
### Installation
|
||||
|
||||
```bash
|
||||
pip install langchain-openai
|
||||
```
|
||||
|
||||
### Basic Setup
|
||||
|
||||
Connect to TrueFoundry by updating the `ChatOpenAI` model in LangChain:
|
||||
|
||||
```python
|
||||
from langchain_openai import ChatOpenAI
|
||||
|
||||
llm = ChatOpenAI(
|
||||
api_key=TRUEFOUNDRY_API_KEY,
|
||||
base_url=TRUEFOUNDRY_GATEWAY_BASE_URL,
|
||||
model="openai-main/gpt-4o" # Similarly you can call any model from any model provider
|
||||
)
|
||||
|
||||
llm.invoke("What is the meaning of life, universe and everything?")
|
||||
```
|
||||
|
||||
The request is routed through your TrueFoundry gateway to the specified model provider. TrueFoundry automatically handles rate limiting, load balancing, and observability.
|
||||
|
||||
### LangGraph Integration
|
||||
|
||||
|
||||
```python
|
||||
from langchain_openai import ChatOpenAI
|
||||
from langgraph.graph import StateGraph, MessagesState
|
||||
from langchain_core.messages import HumanMessage
|
||||
|
||||
# Define your LangGraph workflow
|
||||
def call_model(state: MessagesState):
|
||||
model = ChatOpenAI(
|
||||
api_key=TRUEFOUNDRY_API_KEY,
|
||||
base_url=TRUEFOUNDRY_GATEWAY_BASE_URL,
|
||||
# Copy the exact model name from gateway
|
||||
model="openai-main/gpt-4o"
|
||||
)
|
||||
response = model.invoke(state["messages"])
|
||||
return {"messages": [response]}
|
||||
|
||||
# Build workflow
|
||||
workflow = StateGraph(MessagesState)
|
||||
workflow.add_node("agent", call_model)
|
||||
workflow.set_entry_point("agent")
|
||||
workflow.set_finish_point("agent")
|
||||
|
||||
app = workflow.compile()
|
||||
|
||||
# Run agent through TrueFoundry
|
||||
result = app.invoke({"messages": [HumanMessage(content="Hello!")]})
|
||||
```
|
||||
|
||||
|
||||
## Observability and Governance
|
||||
|
||||

|
||||
|
||||
With the Metrics Dashboard, you can monitor and analyze:
|
||||
|
||||
- **Performance Metrics**: Track key latency metrics like Request Latency, Time to First Token (TTFS), and Inter-Token Latency (ITL) with P99, P90, and P50 percentiles
|
||||
- **Cost and Token Usage**: Gain visibility into your application's costs with detailed breakdowns of input/output tokens and the associated expenses for each model
|
||||
- **Usage Patterns**: Understand how your application is being used with detailed analytics on user activity, model distribution, and team-based usage
|
||||
- **Rate Limiting & Load Balancing**: Configure limits, distribute traffic across models, and set up fallbacks
|
||||
|
||||
## Support
|
||||
|
||||
For questions, issues, or support:
|
||||
|
||||
- **Email**: [support@truefoundry.com](mailto:support@truefoundry.com)
|
||||
- **Documentation**: [https://docs.truefoundry.com/](https://docs.truefoundry.com/)
|
BIN
docs/static/img/gateway-metrics.png
vendored
Normal file
BIN
docs/static/img/gateway-metrics.png
vendored
Normal file
Binary file not shown.
After Width: | Height: | Size: 530 KiB |
BIN
docs/static/img/unified-code-tfy.png
vendored
Normal file
BIN
docs/static/img/unified-code-tfy.png
vendored
Normal file
Binary file not shown.
After Width: | Height: | Size: 408 KiB |
Loading…
Reference in New Issue
Block a user