github/langchain
1
0
Fork 0
mirror of https://github.com/hwchase17/langchain.git synced 2025-06-25 08:03:39 +00:00
Code Issues Packages Projects Releases Wiki Activity

Solved issue Implement langchain-litellm #30368 (#30637)

Browse Source 
**PR title**: 
- [x] 1. docs: docs/docs/integrations/providers/LiteLLM.md
- [x] 2. docs: docs/docs/integrations/chat/litellm.ipynb
- [x] 3. libs: libs/packages.yml

- [x] **PR message**:
    - **Description:** Implement langchain-litellm
    - **Issue:** the issue #30368 
    - **Twitter handle:** akshay_d02
    - **LinkedIn Handle** https://linkedin.com/in/akshay-dongare 

- [x] **Add tests and docs**: Done

- [x] **Lint and test**: Done

---------

Co-authored-by: Chester Curme <chester.curme@gmail.com>
This commit is contained in:
Akshay Dongare 2025-04-04 12:12:10 -04:00 committed by GitHub
parent 87e82fe1e8
commit f79473b752
No known key found for this signature in database
GPG Key ID: B5690EEEBB952194
4 changed files with 171 additions and 132 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

241
docs/docs/integrations/chat/litellm.ipynb
View File

@ -3,7 +3,9 @@
{
"cell_type": "raw",
"id": "59148044",
"metadata": {},
"metadata": {
"id": "59148044"
},
"source": [
"---\n",
"sidebar_label: LiteLLM\n",
@ -11,120 +13,139 @@
]
},
{
"attachments": {},
"cell_type": "markdown",
"id": "bf733a38-db84-4363-89e2-de6735c37230",
"metadata": {},
"id": "5bcea387",
"metadata": {
"id": "5bcea387"
},
"source": [
"# ChatLiteLLM\n",
"\n",
"[LiteLLM](https://github.com/BerriAI/litellm) is a library that simplifies calling Anthropic, Azure, Huggingface, Replicate, etc.\n",
"\n",
"This notebook covers how to get started with using Langchain + the LiteLLM I/O library. "
"This notebook covers how to get started with using Langchain + the LiteLLM I/O library.\n",
"\n",
"## Overview\n",
"### Integration details\n",
"\n",
"| Class | Package | Local | Serializable | JS support| Package downloads | Package latest |\n",
"| :--- | :--- | :---: | :---: | :---: | :---: | :---: |\n",
"| [ChatLiteLLM](https://python.langchain.com/docs/integrations/chat/litellm/) | [langchain-litellm](https://pypi.org/project/langchain-litellm/)| ❌ | ❌ | ❌ | ![PyPI - Downloads](https://img.shields.io/pypi/dm/langchain-litellm?style=flat-square&label=%20) | ![PyPI - Version](https://img.shields.io/pypi/v/langchain-litellm?style=flat-square&label=%20) |\n",
"\n",
"### Model features\n",
"| [Tool calling](https://python.langchain.com/docs/how_to/tool_calling/) | [Structured output](https://python.langchain.com/docs/how_to/structured_output/) | JSON mode | Image input | Audio input | Video input | [Token-level streaming](https://python.langchain.com/docs/integrations/chat/litellm/#chatlitellm-also-supports-async-and-streaming-functionality) | [Native async](https://python.langchain.com/docs/integrations/chat/litellm/#chatlitellm-also-supports-async-and-streaming-functionality) | [Token usage](https://python.langchain.com/docs/how_to/chat_token_usage_tracking/) | [Logprobs](https://python.langchain.com/docs/how_to/logprobs/) |\n",
"| :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: |\n",
"| ✅ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ | ✅ | ✅ | ❌ |\n",
"\n",
"### Setup\n",
"To access ChatLiteLLM models you'll need to install the `langchain-litellm` package and create an OpenAI, Anthropic, Azure, Replicate, OpenRouter, Hugging Face, Together AI or Cohere account. Then you have to get an API key, and export it as an environment variable."
]
},
{
"cell_type": "markdown",
"id": "0a2f8164",
"metadata": {
"id": "0a2f8164"
},
"source": [
"## Credentials\n",
"\n",
"You have to choose the LLM provider you want and sign up with them to get their API key.\n",
"\n",
"### Example - Anthropic\n",
"Head to https://console.anthropic.com/ to sign up for Anthropic and generate an API key. Once you've done this set the ANTHROPIC_API_KEY environment variable.\n",
"\n",
"\n",
"### Example - OpenAI\n",
"Head to https://platform.openai.com/api-keys to sign up for OpenAI and generate an API key. Once you've done this set the OPENAI_API_KEY environment variable."
]
},
{
"cell_type": "code",
"execution_count": 1,
"id": "d4a7c55d-b235-4ca4-a579-c90cc9570da9",
"id": "7595eddf",
"metadata": {
"tags": []
"id": "7595eddf"
},
"outputs": [],
"source": [
"from langchain_community.chat_models import ChatLiteLLM\n",
"from langchain_core.messages import HumanMessage"
"## set ENV variables\n",
"import os\n",
"\n",
"os.environ[\"OPENAI_API_KEY\"] = \"your-openai-key\"\n",
"os.environ[\"ANTHROPIC_API_KEY\"] = \"your-anthropic-key\""
]
},
{
"cell_type": "markdown",
"id": "74c3ad30",
"metadata": {
"id": "74c3ad30"
},
"source": [
"### Installation\n",
"\n",
"The LangChain LiteLLM integration lives in the `langchain-litellm` package:"
]
},
{
"cell_type": "code",
"execution_count": 2,
"id": "70cf04e8-423a-4ff6-8b09-f11fb711c817",
"id": "ca3f8a25",
"metadata": {
"tags": []
"id": "ca3f8a25"
},
"outputs": [],
"source": [
"chat = ChatLiteLLM(model=\"gpt-3.5-turbo\")"
"%pip install -qU langchain-litellm"
]
},
{
"cell_type": "markdown",
"id": "bc1182b4",
"metadata": {
"id": "bc1182b4"
},
"source": [
"## Instantiation\n",
"Now we can instantiate our model object and generate chat completions:"
]
},
{
"cell_type": "code",
"execution_count": 3,
"id": "8199ef8f-eb8b-4253-9ea0-6c24a013ca4c",
"id": "d4a7c55d-b235-4ca4-a579-c90cc9570da9",
"metadata": {
"id": "d4a7c55d-b235-4ca4-a579-c90cc9570da9",
"tags": []
},
"outputs": [
{
"data": {
"text/plain": [
"AIMessage(content=\" J'aime la programmation.\", additional_kwargs={}, example=False)"
]
},
"execution_count": 3,
"metadata": {},
"output_type": "execute_result"
}
],
"outputs": [],
"source": [
"messages = [\n",
" HumanMessage(\n",
" content=\"Translate this sentence from English to French. I love programming.\"\n",
" )\n",
"]\n",
"chat(messages)"
"from langchain_litellm.chat_models import ChatLiteLLM\n",
"\n",
"llm = ChatLiteLLM(model=\"gpt-3.5-turbo\")"
]
},
{
"attachments": {},
"cell_type": "markdown",
"id": "c361ab1e-8c0c-4206-9e3c-9d1424a12b9c",
"metadata": {},
"id": "63d98454",
"metadata": {
"id": "63d98454"
},
"source": [
"## `ChatLiteLLM` also supports async and streaming functionality:"
"## Invocation"
]
},
{
"cell_type": "code",
"execution_count": 4,
"id": "93a21c5c-6ef9-4688-be60-b2e1f94842fb",
"id": "8199ef8f-eb8b-4253-9ea0-6c24a013ca4c",
"metadata": {
"tags": []
"colab": {
"base_uri": "https://localhost:8080/"
},
"outputs": [],
"source": [
"from langchain_core.callbacks import CallbackManager, StreamingStdOutCallbackHandler"
]
},
{
"cell_type": "code",
"execution_count": 5,
"id": "c5fac0e9-05a4-4fc1-a3b3-e5bbb24b971b",
"metadata": {
"tags": []
},
"outputs": [
{
"data": {
"text/plain": [
"LLMResult(generations=[[ChatGeneration(text=\" J'aime programmer.\", generation_info=None, message=AIMessage(content=\" J'aime programmer.\", additional_kwargs={}, example=False))]], llm_output={}, run=[RunInfo(run_id=UUID('8cc8fb68-1c35-439c-96a0-695036a93652'))])"
]
},
"execution_count": 5,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"await chat.agenerate([messages])"
]
},
{
"cell_type": "code",
"execution_count": 6,
"id": "025be980-e50d-4a68-93dc-c9c7b500ce34",
"metadata": {
"id": "8199ef8f-eb8b-4253-9ea0-6c24a013ca4c",
"outputId": "a4c0e5f5-a859-43fa-dd78-74fc0922ecb2",
"tags": []
},
"outputs": [
@ -132,41 +153,75 @@
"name": "stdout",
"output_type": "stream",
"text": [
" J'aime la programmation."
"content='Neutral' additional_kwargs={} response_metadata={'token_usage': Usage(completion_tokens=2, prompt_tokens=30, total_tokens=32, completion_tokens_details=CompletionTokensDetailsWrapper(accepted_prediction_tokens=0, audio_tokens=0, reasoning_tokens=0, rejected_prediction_tokens=0, text_tokens=None), prompt_tokens_details=PromptTokensDetailsWrapper(audio_tokens=0, cached_tokens=0, text_tokens=None, image_tokens=None)), 'model': 'gpt-3.5-turbo', 'finish_reason': 'stop', 'model_name': 'gpt-3.5-turbo'} id='run-ab6a3b21-eae8-4c27-acb2-add65a38221a-0' usage_metadata={'input_tokens': 30, 'output_tokens': 2, 'total_tokens': 32}\n"
]
},
{
"data": {
"text/plain": [
"AIMessage(content=\" J'aime la programmation.\", additional_kwargs={}, example=False)"
]
},
"execution_count": 6,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"chat = ChatLiteLLM(\n",
" streaming=True,\n",
" verbose=True,\n",
" callback_manager=CallbackManager([StreamingStdOutCallbackHandler()]),\n",
"response = await llm.ainvoke(\n",
" \"Classify the text into neutral, negative or positive. Text: I think the food was okay. Sentiment:\"\n",
")\n",
"chat(messages)"
"print(response)"
]
},
{
"cell_type": "markdown",
"id": "c361ab1e-8c0c-4206-9e3c-9d1424a12b9c",
"metadata": {
"id": "c361ab1e-8c0c-4206-9e3c-9d1424a12b9c"
},
"source": [
"## `ChatLiteLLM` also supports async and streaming functionality:"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "c253883f",
"metadata": {},
"outputs": [],
"source": []
"execution_count": 5,
"id": "c5fac0e9-05a4-4fc1-a3b3-e5bbb24b971b",
"metadata": {
"colab": {
"base_uri": "https://localhost:8080/"
},
"id": "c5fac0e9-05a4-4fc1-a3b3-e5bbb24b971b",
"outputId": "ee8cdda1-d992-4696-9ad0-aa146360a3ee",
"tags": []
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Antibiotics are medications that fight bacterial infections in the body. They work by targeting specific bacteria and either killing them or preventing their growth and reproduction.\n",
"\n",
"There are several different mechanisms by which antibiotics work. Some antibiotics work by disrupting the cell walls of bacteria, causing them to burst and die. Others interfere with the protein synthesis of bacteria, preventing them from growing and reproducing. Some antibiotics target the DNA or RNA of bacteria, disrupting their ability to replicate.\n",
"\n",
"It is important to note that antibiotics only work against bacterial infections and not viral infections. It is also crucial to take antibiotics as prescribed by a healthcare professional and to complete the full course of treatment, even if symptoms improve before the medication is finished. This helps to prevent antibiotic resistance, where bacteria become resistant to the effects of antibiotics."
]
}
],
"source": [
"async for token in llm.astream(\"Hello, please explain how antibiotics work\"):\n",
" print(token.text(), end=\"\")"
]
},
{
"cell_type": "markdown",
"id": "88af2a9b",
"metadata": {
"id": "88af2a9b"
},
"source": [
"## API reference\n",
"For detailed documentation of all `ChatLiteLLM` features and configurations head to the API reference: https://github.com/Akshay-Dongare/langchain-litellm"
]
}
],
"metadata": {
"colab": {
"provenance": []
},
"kernelspec": {
"display_name": "Python 3 (ipykernel)",
"display_name": "g6_alda",
"language": "python",
"name": "python3"
},
@ -180,7 +235,7 @@
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.9.1"
"version": "3.12.4"
}
},
"nbformat": 4,

18
docs/docs/integrations/providers/litellm.mdx Normal file
View File

@ -0,0 +1,18 @@
# LiteLLM
>[LiteLLM](https://github.com/BerriAI/litellm) is a library that simplifies calling Anthropic, Azure, Huggingface, Replicate, etc.
## Installation and setup
```bash
pip install langchain-litellm
```
## Chat Models
```python
from langchain_litellm.chat_models import ChatLiteLLM
```
See more detail in the guide [here](/docs/integrations/chat/litellm).
## API reference
For detailed documentation of all `ChatLiteLLM` features and configurations head to the API reference: https://github.com/Akshay-Dongare/langchain-litellm

37
docs/docs/integrations/providers/littlellm.md
View File

@ -1,37 +0,0 @@
# LiteLLM
>[LiteLLM](https://docs.litellm.ai/docs/) is a library that simplifies calling Anthropic,
> Azure, Huggingface, Replicate, etc. LLMs in a unified way.
>
>You can use `LiteLLM` through either:
>
>* [LiteLLM Proxy Server](https://docs.litellm.ai/docs/#openai-proxy) - Server to call 100+ LLMs, load balance, cost tracking across projects
>* [LiteLLM python SDK](https://docs.litellm.ai/docs/#basic-usage) - Python Client to call 100+ LLMs, load balance, cost tracking
## Installation and setup
Install the `litellm` python package.
```bash
pip install litellm
```
## Chat models
### ChatLiteLLM
See a [usage example](/docs/integrations/chat/litellm).
```python
from langchain_community.chat_models import ChatLiteLLM
```
### ChatLiteLLMRouter
You also can use the `ChatLiteLLMRouter` to route requests to different LLMs or LLM providers.
See a [usage example](/docs/integrations/chat/litellm_router).
```python
from langchain_community.chat_models import ChatLiteLLMRouter
```

3
libs/packages.yml
View File

@ -583,3 +583,6 @@ packages:
path: .
repo: yigit353/langchain-qwq
provider_page: alibaba_cloud
- name: langchain-litellm
path: .
repo: akshay-dongare/langchain-litellm