mirror of
https://github.com/hwchase17/langchain.git
synced 2025-04-30 04:45:23 +00:00
0b8e9868e6
# What's Changed? - [x] 1. docs: **docs/docs/integrations/chat/litellm.ipynb** : Updated with docs for litellm_router since it has been moved into the [langchain-litellm](https://github.com/Akshay-Dongare/langchain-litellm) package along with ChatLiteLLM - [x] 2. docs: **docs/docs/integrations/chat/litellm_router.ipynb** : Deleted to avoid redundancy - [x] 3. docs: **docs/docs/integrations/providers/litellm.mdx** : Updated to reflect inclusion of ChatLiteLLMRouter class - [x] Lint and test: Done # Issue: - [x] Related to the issue https://github.com/langchain-ai/langchain/issues/30368 # About me - [x] 🔗 LinkedIn: [akshay-dongare](https://www.linkedin.com/in/akshay-dongare/)
329 lines
12 KiB
Plaintext
329 lines
12 KiB
Plaintext
{
|
|
"cells": [
|
|
{
|
|
"cell_type": "raw",
|
|
"id": "59148044",
|
|
"metadata": {
|
|
"id": "59148044"
|
|
},
|
|
"source": [
|
|
"---\n",
|
|
"sidebar_label: LiteLLM\n",
|
|
"---"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "5bcea387",
|
|
"metadata": {
|
|
"id": "5bcea387"
|
|
},
|
|
"source": [
|
|
"# ChatLiteLLM and ChatLiteLLMRouter\n",
|
|
"\n",
|
|
"[LiteLLM](https://github.com/BerriAI/litellm) is a library that simplifies calling Anthropic, Azure, Huggingface, Replicate, etc.\n",
|
|
"\n",
|
|
"This notebook covers how to get started with using Langchain + the LiteLLM I/O library.\n",
|
|
"\n",
|
|
"This integration contains two main classes:\n",
|
|
"\n",
|
|
"- ```ChatLiteLLM```: The main Langchain wrapper for basic usage of LiteLLM ([docs](https://docs.litellm.ai/docs/)).\n",
|
|
"- ```ChatLiteLLMRouter```: A ```ChatLiteLLM``` wrapper that leverages LiteLLM's Router ([docs](https://docs.litellm.ai/docs/routing))."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "2ddb7fd3",
|
|
"metadata": {},
|
|
"source": [
|
|
"## Table of Contents\n",
|
|
"1. [Overview](#overview)\n",
|
|
" - [Integration Details](#integration-details)\n",
|
|
" - [Model Features](#model-features)\n",
|
|
"2. [Setup](#setup)\n",
|
|
"3. [Credentials](#credentials)\n",
|
|
"4. [Installation](#installation)\n",
|
|
"5. [Instantiation](#instantiation)\n",
|
|
" - [ChatLiteLLM](#chatlitellm)\n",
|
|
" - [ChatLiteLLMRouter](#chatlitellmrouter)\n",
|
|
"6. [Invocation](#invocation)\n",
|
|
"7. [Async and Streaming Functionality](#async-and-streaming-functionality)\n",
|
|
"8. [API Reference](#api-reference)"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "37be6ef8",
|
|
"metadata": {},
|
|
"source": [
|
|
"## Overview\n",
|
|
"### Integration details\n",
|
|
"\n",
|
|
"| Class | Package | Local | Serializable | JS support| Package downloads | Package latest |\n",
|
|
"| :--- | :--- | :---: | :---: | :---: | :---: | :---: |\n",
|
|
"| [ChatLiteLLM](https://python.langchain.com/docs/integrations/chat/litellm/#chatlitellm) | [langchain-litellm](https://pypi.org/project/langchain-litellm/)| ❌ | ❌ | ❌ |  |  |\n",
|
|
"| [ChatLiteLLMRouter](https://python.langchain.com/docs/integrations/chat/litellm/#chatlitellmrouter) | [langchain-litellm](https://pypi.org/project/langchain-litellm/)| ❌ | ❌ | ❌ |  |  |\n",
|
|
"\n",
|
|
"### Model features\n",
|
|
"| [Tool calling](https://python.langchain.com/docs/how_to/tool_calling/) | [Structured output](https://python.langchain.com/docs/how_to/structured_output/) | JSON mode | Image input | Audio input | Video input | [Token-level streaming](https://python.langchain.com/docs/integrations/chat/litellm/#chatlitellm-also-supports-async-and-streaming-functionality) | [Native async](https://python.langchain.com/docs/integrations/chat/litellm/#chatlitellm-also-supports-async-and-streaming-functionality) | [Token usage](https://python.langchain.com/docs/how_to/chat_token_usage_tracking/) | [Logprobs](https://python.langchain.com/docs/how_to/logprobs/) |\n",
|
|
"| :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: |\n",
|
|
"| ✅ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ | ✅ | ✅ | ❌ |\n",
|
|
"\n",
|
|
"### Setup\n",
|
|
"To access ```ChatLiteLLM``` and ```ChatLiteLLMRouter``` models, you'll need to install the `langchain-litellm` package and create an OpenAI, Anthropic, Azure, Replicate, OpenRouter, Hugging Face, Together AI, or Cohere account. Then, you have to get an API key and export it as an environment variable."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "0a2f8164",
|
|
"metadata": {
|
|
"id": "0a2f8164"
|
|
},
|
|
"source": [
|
|
"## Credentials\n",
|
|
"\n",
|
|
"You have to choose the LLM provider you want and sign up with them to get their API key.\n",
|
|
"\n",
|
|
"### Example - Anthropic\n",
|
|
"Head to https://console.anthropic.com/ to sign up for Anthropic and generate an API key. Once you've done this, set the ANTHROPIC_API_KEY environment variable.\n",
|
|
"\n",
|
|
"\n",
|
|
"### Example - OpenAI\n",
|
|
"Head to https://platform.openai.com/api-keys to sign up for OpenAI and generate an API key. Once you've done this, set the OPENAI_API_KEY environment variable."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"id": "7595eddf",
|
|
"metadata": {
|
|
"id": "7595eddf"
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"## Set ENV variables\n",
|
|
"import os\n",
|
|
"\n",
|
|
"os.environ[\"OPENAI_API_KEY\"] = \"your-openai-key\"\n",
|
|
"os.environ[\"ANTHROPIC_API_KEY\"] = \"your-anthropic-key\""
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "74c3ad30",
|
|
"metadata": {
|
|
"id": "74c3ad30"
|
|
},
|
|
"source": [
|
|
"### Installation\n",
|
|
"\n",
|
|
"The LangChain LiteLLM integration is available in the `langchain-litellm` package:"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 2,
|
|
"id": "ca3f8a25",
|
|
"metadata": {
|
|
"id": "ca3f8a25"
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"%pip install -qU langchain-litellm"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "bc1182b4",
|
|
"metadata": {
|
|
"id": "bc1182b4"
|
|
},
|
|
"source": [
|
|
"## Instantiation"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "d439241a",
|
|
"metadata": {},
|
|
"source": [
|
|
"### ChatLiteLLM\n",
|
|
"You can instantiate a ```ChatLiteLLM``` model by providing a ```model``` name [supported by LiteLLM](https://docs.litellm.ai/docs/providers)."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"id": "d4a7c55d-b235-4ca4-a579-c90cc9570da9",
|
|
"metadata": {
|
|
"id": "d4a7c55d-b235-4ca4-a579-c90cc9570da9",
|
|
"tags": []
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"from langchain_litellm import ChatLiteLLM\n",
|
|
"\n",
|
|
"llm = ChatLiteLLM(model=\"gpt-4.1-nano\", temperature=0.1)"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "3d0ed306",
|
|
"metadata": {},
|
|
"source": [
|
|
"### ChatLiteLLMRouter\n",
|
|
"You can also leverage LiteLLM's routing capabilities by defining your model list as specified [here](https://docs.litellm.ai/docs/routing)."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"id": "8d26393a",
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"from langchain_litellm import ChatLiteLLMRouter\n",
|
|
"from litellm import Router\n",
|
|
"\n",
|
|
"model_list = [\n",
|
|
" {\n",
|
|
" \"model_name\": \"gpt-4.1\",\n",
|
|
" \"litellm_params\": {\n",
|
|
" \"model\": \"azure/gpt-4.1\",\n",
|
|
" \"api_key\": \"<your-api-key>\",\n",
|
|
" \"api_version\": \"2024-10-21\",\n",
|
|
" \"api_base\": \"https://<your-endpoint>.openai.azure.com/\",\n",
|
|
" },\n",
|
|
" },\n",
|
|
" {\n",
|
|
" \"model_name\": \"gpt-4o\",\n",
|
|
" \"litellm_params\": {\n",
|
|
" \"model\": \"azure/gpt-4o\",\n",
|
|
" \"api_key\": \"<your-api-key>\",\n",
|
|
" \"api_version\": \"2024-10-21\",\n",
|
|
" \"api_base\": \"https://<your-endpoint>.openai.azure.com/\",\n",
|
|
" },\n",
|
|
" },\n",
|
|
"]\n",
|
|
"litellm_router = Router(model_list=model_list)\n",
|
|
"llm = ChatLiteLLMRouter(router=litellm_router, model_name=\"gpt-4.1\", temperature=0.1)"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "63d98454",
|
|
"metadata": {
|
|
"id": "63d98454"
|
|
},
|
|
"source": [
|
|
"## Invocation\n",
|
|
"Whether you've instantiated a `ChatLiteLLM` or a `ChatLiteLLMRouter`, you can now use the ChatModel through Langchain's API."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 4,
|
|
"id": "8199ef8f-eb8b-4253-9ea0-6c24a013ca4c",
|
|
"metadata": {
|
|
"colab": {
|
|
"base_uri": "https://localhost:8080/"
|
|
},
|
|
"id": "8199ef8f-eb8b-4253-9ea0-6c24a013ca4c",
|
|
"outputId": "a4c0e5f5-a859-43fa-dd78-74fc0922ecb2",
|
|
"tags": []
|
|
},
|
|
"outputs": [
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"content='Neutral' additional_kwargs={} response_metadata={'token_usage': Usage(completion_tokens=2, prompt_tokens=30, total_tokens=32, completion_tokens_details=CompletionTokensDetailsWrapper(accepted_prediction_tokens=0, audio_tokens=0, reasoning_tokens=0, rejected_prediction_tokens=0, text_tokens=None), prompt_tokens_details=PromptTokensDetailsWrapper(audio_tokens=0, cached_tokens=0, text_tokens=None, image_tokens=None)), 'model': 'gpt-3.5-turbo', 'finish_reason': 'stop', 'model_name': 'gpt-3.5-turbo'} id='run-ab6a3b21-eae8-4c27-acb2-add65a38221a-0' usage_metadata={'input_tokens': 30, 'output_tokens': 2, 'total_tokens': 32}\n"
|
|
]
|
|
}
|
|
],
|
|
"source": [
|
|
"response = await llm.ainvoke(\n",
|
|
" \"Classify the text into neutral, negative or positive. Text: I think the food was okay. Sentiment:\"\n",
|
|
")\n",
|
|
"print(response)"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "c361ab1e-8c0c-4206-9e3c-9d1424a12b9c",
|
|
"metadata": {
|
|
"id": "c361ab1e-8c0c-4206-9e3c-9d1424a12b9c"
|
|
},
|
|
"source": [
|
|
"## Async and Streaming Functionality\n",
|
|
"`ChatLiteLLM` and `ChatLiteLLMRouter` also support async and streaming functionality:"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 5,
|
|
"id": "c5fac0e9-05a4-4fc1-a3b3-e5bbb24b971b",
|
|
"metadata": {
|
|
"colab": {
|
|
"base_uri": "https://localhost:8080/"
|
|
},
|
|
"id": "c5fac0e9-05a4-4fc1-a3b3-e5bbb24b971b",
|
|
"outputId": "ee8cdda1-d992-4696-9ad0-aa146360a3ee",
|
|
"tags": []
|
|
},
|
|
"outputs": [
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"Antibiotics are medications that fight bacterial infections in the body. They work by targeting specific bacteria and either killing them or preventing their growth and reproduction.\n",
|
|
"\n",
|
|
"There are several different mechanisms by which antibiotics work. Some antibiotics work by disrupting the cell walls of bacteria, causing them to burst and die. Others interfere with the protein synthesis of bacteria, preventing them from growing and reproducing. Some antibiotics target the DNA or RNA of bacteria, disrupting their ability to replicate.\n",
|
|
"\n",
|
|
"It is important to note that antibiotics only work against bacterial infections and not viral infections. It is also crucial to take antibiotics as prescribed by a healthcare professional and to complete the full course of treatment, even if symptoms improve before the medication is finished. This helps to prevent antibiotic resistance, where bacteria become resistant to the effects of antibiotics."
|
|
]
|
|
}
|
|
],
|
|
"source": [
|
|
"async for token in llm.astream(\"Hello, please explain how antibiotics work\"):\n",
|
|
" print(token.text(), end=\"\")"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "88af2a9b",
|
|
"metadata": {
|
|
"id": "88af2a9b"
|
|
},
|
|
"source": [
|
|
"## API reference\n",
|
|
"For detailed documentation of all `ChatLiteLLM` and `ChatLiteLLMRouter` features and configurations, head to the API reference: https://github.com/Akshay-Dongare/langchain-litellm"
|
|
]
|
|
}
|
|
],
|
|
"metadata": {
|
|
"colab": {
|
|
"provenance": []
|
|
},
|
|
"kernelspec": {
|
|
"display_name": "g6_alda",
|
|
"language": "python",
|
|
"name": "python3"
|
|
},
|
|
"language_info": {
|
|
"codemirror_mode": {
|
|
"name": "ipython",
|
|
"version": 3
|
|
},
|
|
"file_extension": ".py",
|
|
"mimetype": "text/x-python",
|
|
"name": "python",
|
|
"nbconvert_exporter": "python",
|
|
"pygments_lexer": "ipython3",
|
|
"version": "3.12.4"
|
|
}
|
|
},
|
|
"nbformat": 4,
|
|
"nbformat_minor": 5
|
|
}
|