diff --git a/docs/docs/integrations/providers/truefoundry.mdx b/docs/docs/integrations/providers/truefoundry.mdx new file mode 100644 index 00000000000..a629c41c365 --- /dev/null +++ b/docs/docs/integrations/providers/truefoundry.mdx @@ -0,0 +1,101 @@ +# TrueFoundry + +TrueFoundry provides an enterprise-ready [AI Gateway](https://www.truefoundry.com/ai-gateway) to provide governance and observability to agentic frameworks like LangChain. TrueFoundry AI Gateway serves as a unified interface for LLM access, providing: + +- **Unified API Access**: Connect to 250+ LLMs (OpenAI, Claude, Gemini, Groq, Mistral) through one API +- **Low Latency**: Sub-3ms internal latency with intelligent routing and load balancing +- **Enterprise Security**: SOC 2, HIPAA, GDPR compliance with RBAC and audit logging +- **Quota and cost management**: Token-based quotas, rate limiting, and comprehensive usage tracking +- **Observability**: Full request/response logging, metrics, and traces with customizable retention + + +## Prerequisites + +Before integrating LangChain with TrueFoundry, ensure you have: + +1. **TrueFoundry Account**: A [TrueFoundry account](https://www.truefoundry.com/register) with at least one model provider configured. Follow quick start guide [here](https://docs.truefoundry.com/gateway/quick-start) +2. **Personal Access Token**: Generate a token by following the [TrueFoundry token generation guide](https://docs.truefoundry.com/gateway/authentication) + +## Quickstart + +You can connect to TrueFoundry's unified LLM gateway through the `ChatOpenAI` interface. + +- Set the `base_url` to your TrueFoundry endpoint (explained below) +- Set the `api_key` to your TrueFoundry [PAT (Personal Access Token)](https://docs.truefoundry.com/gateway/authentication#personal-access-token-pat) +- Use the same `model-name` as shown in the unified code snippet + +![TrueFoundry metrics](/img/unified-code-tfy.png) + +### Installation + +```bash +pip install langchain-openai +``` + +### Basic Setup + +Connect to TrueFoundry by updating the `ChatOpenAI` model in LangChain: + +```python +from langchain_openai import ChatOpenAI + +llm = ChatOpenAI( + api_key=TRUEFOUNDRY_API_KEY, + base_url=TRUEFOUNDRY_GATEWAY_BASE_URL, + model="openai-main/gpt-4o" # Similarly you can call any model from any model provider +) + +llm.invoke("What is the meaning of life, universe and everything?") +``` + +The request is routed through your TrueFoundry gateway to the specified model provider. TrueFoundry automatically handles rate limiting, load balancing, and observability. + +### LangGraph Integration + + +```python +from langchain_openai import ChatOpenAI +from langgraph.graph import StateGraph, MessagesState +from langchain_core.messages import HumanMessage + +# Define your LangGraph workflow +def call_model(state: MessagesState): + model = ChatOpenAI( + api_key=TRUEFOUNDRY_API_KEY, + base_url=TRUEFOUNDRY_GATEWAY_BASE_URL, + # Copy the exact model name from gateway + model="openai-main/gpt-4o" + ) + response = model.invoke(state["messages"]) + return {"messages": [response]} + +# Build workflow +workflow = StateGraph(MessagesState) +workflow.add_node("agent", call_model) +workflow.set_entry_point("agent") +workflow.set_finish_point("agent") + +app = workflow.compile() + +# Run agent through TrueFoundry +result = app.invoke({"messages": [HumanMessage(content="Hello!")]}) +``` + + +## Observability and Governance + +![TrueFoundry metrics](/img/gateway-metrics.png) + +With the Metrics Dashboard, you can monitor and analyze: + +- **Performance Metrics**: Track key latency metrics like Request Latency, Time to First Token (TTFS), and Inter-Token Latency (ITL) with P99, P90, and P50 percentiles +- **Cost and Token Usage**: Gain visibility into your application's costs with detailed breakdowns of input/output tokens and the associated expenses for each model +- **Usage Patterns**: Understand how your application is being used with detailed analytics on user activity, model distribution, and team-based usage +- **Rate Limiting & Load Balancing**: Configure limits, distribute traffic across models, and set up fallbacks + +## Support + +For questions, issues, or support: + +- **Email**: [support@truefoundry.com](mailto:support@truefoundry.com) +- **Documentation**: [https://docs.truefoundry.com/](https://docs.truefoundry.com/) diff --git a/docs/sidebars.js b/docs/sidebars.js index 26d7102f2d9..3d175abe971 100644 --- a/docs/sidebars.js +++ b/docs/sidebars.js @@ -27,7 +27,7 @@ module.exports = { }, { type: "category", - link: {type: 'doc', id: 'tutorials/index'}, + link: { type: 'doc', id: 'tutorials/index' }, label: "Tutorials", collapsible: false, items: [{ @@ -38,7 +38,7 @@ module.exports = { }, { type: "category", - link: {type: 'doc', id: 'how_to/index'}, + link: { type: 'doc', id: 'how_to/index' }, label: "How-to guides", collapsible: false, items: [{ @@ -49,7 +49,7 @@ module.exports = { }, { type: "category", - link: {type: 'doc', id: 'concepts/index'}, + link: { type: 'doc', id: 'concepts/index' }, label: "Conceptual guide", collapsible: false, items: [{ @@ -103,7 +103,7 @@ module.exports = { { type: "category", label: "Migrating from v0.0 chains", - link: {type: 'doc', id: 'versions/migrating_chains/index'}, + link: { type: 'doc', id: 'versions/migrating_chains/index' }, collapsible: false, collapsed: false, items: [{ @@ -115,7 +115,7 @@ module.exports = { { type: "category", label: "Upgrading to LangGraph memory", - link: {type: 'doc', id: 'versions/migrating_memory/index'}, + link: { type: 'doc', id: 'versions/migrating_memory/index' }, collapsible: false, collapsed: false, items: [{ @@ -418,7 +418,7 @@ module.exports = { }, ], }, - + ], link: { type: "generated-index", @@ -434,7 +434,7 @@ module.exports = { }, { type: "category", - link: {type: 'doc', id: 'contributing/tutorials/index'}, + link: { type: 'doc', id: 'contributing/tutorials/index' }, label: "Tutorials", collapsible: false, items: [{ @@ -445,7 +445,7 @@ module.exports = { }, { type: "category", - link: {type: 'doc', id: 'contributing/how_to/index'}, + link: { type: 'doc', id: 'contributing/how_to/index' }, label: "How-to guides", collapsible: false, items: [{ @@ -456,7 +456,7 @@ module.exports = { }, { type: "category", - link: {type: 'doc', id: 'contributing/reference/index'}, + link: { type: 'doc', id: 'contributing/reference/index' }, label: "Reference & FAQ", collapsible: false, items: [{ diff --git a/docs/static/img/gateway-metrics.png b/docs/static/img/gateway-metrics.png new file mode 100644 index 00000000000..5c521c892ae Binary files /dev/null and b/docs/static/img/gateway-metrics.png differ diff --git a/docs/static/img/unified-code-tfy.png b/docs/static/img/unified-code-tfy.png new file mode 100644 index 00000000000..3f54355ec06 Binary files /dev/null and b/docs/static/img/unified-code-tfy.png differ