community: Memcached LLM Cache Integration (#27323)

## Description
This PR adds support for Memcached as a usable LLM model cache by adding
the ```MemcachedCache``` implementation relying on the
[pymemcache](https://github.com/pinterest/pymemcache) client.

Unit test-wise, the new integration is generally covered under existing
import testing. All new functionality depends on pymemcache if
instantiated and used, so to comply with the other cache implementations
the PR also adds optional integration tests for ```MemcachedCache```.

Since this is a new integration, documentation is added for Memcached as
an integration and as an LLM Cache.

## Issue
This PR closes #27275 which was originally raised as a discussion in
#27035

## Dependencies
There are no new required dependencies for langchain, but
[pymemcache](https://github.com/pinterest/pymemcache) is required to
instantiate the new ```MemcachedCache```.

## Example Usage
```python3
from langchain.globals import set_llm_cache
from langchain_openai import OpenAI

from langchain_community.cache import MemcachedCache
from pymemcache.client.base import Client

llm = OpenAI(model="gpt-3.5-turbo-instruct", n=2, best_of=2)
set_llm_cache(MemcachedCache(Client('localhost')))

# The first time, it is not yet in cache, so it should take longer
llm.invoke("Which city is the most crowded city in the USA?")

# The second time it is, so it goes faster
llm.invoke("Which city is the most crowded city in the USA?")
```

---------

Co-authored-by: Erick Friis <erick@langchain.dev>
This commit is contained in:
Dmitriy Prokopchuk
2024-11-06 22:07:59 -05:00
committed by GitHub
parent cfff2a057e
commit 53b0a99f37
4 changed files with 285 additions and 0 deletions

View File

@@ -2368,6 +2368,102 @@
")"
]
},
{
"cell_type": "markdown",
"id": "7e6b9b1a",
"metadata": {},
"source": [
"## `Memcached` Cache\n",
"You can use [Memcached](https://www.memcached.org/) as a cache to cache prompts and responses through [pymemcache](https://github.com/pinterest/pymemcache).\n",
"\n",
"This cache requires the pymemcache dependency to be installed:"
]
},
{
"cell_type": "code",
"execution_count": 1,
"id": "b2e5e0b1",
"metadata": {},
"outputs": [],
"source": [
"%pip install -qU pymemcache"
]
},
{
"cell_type": "code",
"execution_count": 1,
"id": "4c7ffe37",
"metadata": {},
"outputs": [],
"source": [
"from langchain_community.cache import MemcachedCache\n",
"from pymemcache.client.base import Client\n",
"\n",
"set_llm_cache(MemcachedCache(Client(\"localhost\")))"
]
},
{
"cell_type": "code",
"execution_count": 5,
"id": "a4cfc48a",
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"CPU times: user 32.8 ms, sys: 21 ms, total: 53.8 ms\n",
"Wall time: 343 ms\n"
]
},
{
"data": {
"text/plain": [
"'\\n\\nWhy did the chicken cross the road?\\n\\nTo get to the other side!'"
]
},
"execution_count": 5,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"%%time\n",
"# The first time, it is not yet in cache, so it should take longer\n",
"llm.invoke(\"Tell me a joke\")"
]
},
{
"cell_type": "code",
"execution_count": 7,
"id": "cb3b2bf5",
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"CPU times: user 2.31 ms, sys: 850 µs, total: 3.16 ms\n",
"Wall time: 6.43 ms\n"
]
},
{
"data": {
"text/plain": [
"'\\n\\nWhy did the chicken cross the road?\\n\\nTo get to the other side!'"
]
},
"execution_count": 7,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"%%time\n",
"# The second time it is, so it goes faster\n",
"llm.invoke(\"Tell me a joke\")"
]
},
{
"cell_type": "markdown",
"id": "7019c991-0101-4f9c-b212-5729a5471293",

View File

@@ -0,0 +1,34 @@
# Memcached
> [Memcached](https://www.memcached.org/) is a free & open source, high-performance, distributed memory object caching system,
> generic in nature, but intended for use in speeding up dynamic web applications by alleviating database load.
This page covers how to use Memcached with langchain, using [pymemcache](https://github.com/pinterest/pymemcache) as
a client to connect to an already running Memcached instance.
## Installation and Setup
```bash
pip install pymemcache
```
## LLM Cache
To integrate a Memcached Cache into your application:
```python3
from langchain.globals import set_llm_cache
from langchain_openai import OpenAI
from langchain_community.cache import MemcachedCache
from pymemcache.client.base import Client
llm = OpenAI(model="gpt-3.5-turbo-instruct", n=2, best_of=2)
set_llm_cache(MemcachedCache(Client('localhost')))
# The first time, it is not yet in cache, so it should take longer
llm.invoke("Which city is the most crowded city in the USA?")
# The second time it is, so it goes faster
llm.invoke("Which city is the most crowded city in the USA?")
```
Learn more in the [example notebook](/docs/integrations/llm_caching#memcached-cache)