diff --git a/docs/extras/integrations/vectorstores/redis.ipynb b/docs/extras/integrations/vectorstores/redis.ipynb index 972af86347c..ae17b0e4e68 100644 --- a/docs/extras/integrations/vectorstores/redis.ipynb +++ b/docs/extras/integrations/vectorstores/redis.ipynb @@ -6,20 +6,124 @@ "source": [ "# Redis\n", "\n", - ">[Redis (Remote Dictionary Server)](https://en.wikipedia.org/wiki/Redis) is an in-memory data structure store, used as a distributed, in-memory key–value database, cache and message broker, with optional durability.\n", + "Redis vector database introduction and langchain integration guide.\n", "\n", - "This notebook shows how to use functionality related to the [Redis vector database](https://redis.com/solutions/use-cases/vector-database/).\n", + "## What is Redis?\n", "\n", - "As database either Redis standalone server or Redis Sentinel HA setups are supported for connections with the \"redis_url\"\n", - "parameter. More information about the different formats of the redis connection url can be found in the LangChain\n", - "[Redis Readme](/docs/integrations/vectorstores/redis) file" + "Most developers from a web services background are probably familiar with Redis. At it's core, Redis is an open-source key-value store that can be used as a cache, message broker, and database. Developers choice Redis because it is fast, has a large ecosystem of client libraries, and has been deployed by major enterprises for years.\n", + "\n", + "In addition to the traditional uses of Redis. Redis also provides capabilities built directly into Redis. These capabilities include the Search and Query capability that allows users to create secondary index structures within Redis. This allows Redis to be a Vector Database, at the speed of a cache. \n", + "\n", + "\n", + "## Redis as a Vector Database\n", + "\n", + "Redis uses compressed, inverted indexes for fast indexing with a low memory footprint. It also supports a number of advanced features such as:\n", + "\n", + "* Indexing of multiple fields in Redis hashes and JSON\n", + "* Vector similarity search (with HNSW (ANN) or FLAT (KNN))\n", + "* Vector Range Search (e.g. find all vectors within a radius of a query vector)\n", + "* Incremental indexing without performance loss\n", + "* Document ranking (using [tf-idf](https://en.wikipedia.org/wiki/Tf%E2%80%93idf), with optional user-provided weights)\n", + "* Field weighting\n", + "* Complex boolean queries with AND, OR, and NOT operators\n", + "* Prefix matching, fuzzy matching, and exact-phrase queries\n", + "* Support for [double-metaphone phonetic matching](https://redis.io/docs/stack/search/reference/phonetic_matching/)\n", + "* Auto-complete suggestions (with fuzzy prefix suggestions)\n", + "* Stemming-based query expansion in [many languages](https://redis.io/docs/stack/search/reference/stemming/) (using [Snowball](http://snowballstem.org/))\n", + "* Support for Chinese-language tokenization and querying (using [Friso](https://github.com/lionsoul2014/friso))\n", + "* Numeric filters and ranges\n", + "* Geospatial searches using [Redis geospatial indexing](/commands/georadius)\n", + "* A powerful aggregations engine\n", + "* Supports for all utf-8 encoded text\n", + "* Retrieve full documents, selected fields, or only the document IDs\n", + "* Sorting results (for example, by creation date)\n", + "\n", + "\n", + "\n", + "## Clients\n", + "\n", + "Since redis is much more than just a vector database, there are often use cases that demand usage of a Redis client besides just the langchain integration. You can use any standard Redis client library to run Search and Query commands, but it's easiest to use a library that wraps the Search and Query API. Below are a few examples, but you can find more client libraries [here](https://redis.io/resources/clients/).\n", + "\n", + "| Project | Language | License | Author | Stars |\n", + "|----------|---------|--------|---------|-------|\n", + "| [jedis][jedis-url] | Java | MIT | [Redis][redis-url] | ![Stars][jedis-stars] |\n", + "| [redisvl][redisvl-url] | Python | MIT | [Redis][redis-url] | ![Stars][redisvl-stars] |\n", + "| [redis-py][redis-py-url] | Python | MIT | [Redis][redis-url] | ![Stars][redis-py-stars] |\n", + "| [node-redis][node-redis-url] | Node.js | MIT | [Redis][redis-url] | ![Stars][node-redis-stars] |\n", + "| [nredisstack][nredisstack-url] | .NET | MIT | [Redis][redis-url] | ![Stars][nredisstack-stars] |\n", + "\n", + "[redis-url]: https://redis.com\n", + "\n", + "[redisvl-url]: https://github.com/RedisVentures/redisvl\n", + "[redisvl-stars]: https://img.shields.io/github/stars/RedisVentures/redisvl.svg?style=social&label=Star&maxAge=2592000\n", + "[redisvl-package]: https://pypi.python.org/pypi/redisvl\n", + "\n", + "[redis-py-url]: https://github.com/redis/redis-py\n", + "[redis-py-stars]: https://img.shields.io/github/stars/redis/redis-py.svg?style=social&label=Star&maxAge=2592000\n", + "[redis-py-package]: https://pypi.python.org/pypi/redis\n", + "\n", + "[jedis-url]: https://github.com/redis/jedis\n", + "[jedis-stars]: https://img.shields.io/github/stars/redis/jedis.svg?style=social&label=Star&maxAge=2592000\n", + "[Jedis-package]: https://search.maven.org/artifact/redis.clients/jedis\n", + "\n", + "[nredisstack-url]: https://github.com/redis/nredisstack\n", + "[nredisstack-stars]: https://img.shields.io/github/stars/redis/nredisstack.svg?style=social&label=Star&maxAge=2592000\n", + "[nredisstack-package]: https://www.nuget.org/packages/nredisstack/\n", + "\n", + "[node-redis-url]: https://github.com/redis/node-redis\n", + "[node-redis-stars]: https://img.shields.io/github/stars/redis/node-redis.svg?style=social&label=Star&maxAge=2592000\n", + "[node-redis-package]: https://www.npmjs.com/package/redis\n", + "\n", + "[redis-om-python-url]: https://github.com/redis/redis-om-python\n", + "[redis-om-python-author]: https://redis.com\n", + "[redis-om-python-stars]: https://img.shields.io/github/stars/redis/redis-om-python.svg?style=social&label=Star&maxAge=2592000\n", + "\n", + "[redisearch-go-url]: https://github.com/RediSearch/redisearch-go\n", + "[redisearch-go-author]: https://redis.com\n", + "[redisearch-go-stars]: https://img.shields.io/github/stars/RediSearch/redisearch-go.svg?style=social&label=Star&maxAge=2592000\n", + "\n", + "[redisearch-api-rs-url]: https://github.com/RediSearch/redisearch-api-rs\n", + "[redisearch-api-rs-author]: https://redis.com\n", + "[redisearch-api-rs-stars]: https://img.shields.io/github/stars/RediSearch/redisearch-api-rs.svg?style=social&label=Star&maxAge=2592000\n", + "\n", + "\n", + "## Deployment Options\n", + "\n", + "There are many ways to deploy Redis with RediSearch. The easiest way to get started is to use Docker, but there are are many potential options for deployment such as\n", + "\n", + "- [Redis Cloud](https://redis.com/redis-enterprise-cloud/overview/)\n", + "- [Docker (Redis Stack)](https://hub.docker.com/r/redis/redis-stack)\n", + "- Cloud marketplaces: [AWS Marketplace](https://aws.amazon.com/marketplace/pp/prodview-e6y7ork67pjwg?sr=0-2&ref_=beagle&applicationId=AWSMPContessa), [Google Marketplace](https://console.cloud.google.com/marketplace/details/redislabs-public/redis-enterprise?pli=1), or [Azure Marketplace](https://azuremarketplace.microsoft.com/en-us/marketplace/apps/garantiadata.redis_enterprise_1sp_public_preview?tab=Overview)\n", + "- On-premise: [Redis Enterprise Software](https://redis.com/redis-enterprise-software/overview/)\n", + "- Kubernetes: [Redis Enterprise Software on Kubernetes](https://docs.redis.com/latest/kubernetes/)\n", + "\n", + "\n", + "## Examples\n", + "\n", + "Many examples can be found in the [Redis AI team's GitHub](https://github.com/RedisVentures/)\n", + "\n", + "- [Awesome Redis AI Resources](https://github.com/RedisVentures/redis-ai-resources) - List of examples of using Redis in AI workloads\n", + "- [Azure OpenAI Embeddings Q&A](https://github.com/ruoccofabrizio/azure-open-ai-embeddings-qna) - OpenAI and Redis as a Q&A service on Azure.\n", + "- [ArXiv Paper Search](https://github.com/RedisVentures/redis-arXiv-search) - Semantic search over arXiv scholarly papers\n", + "\n", + "\n", + "## More Resources\n", + "\n", + "For more information on how to use Redis as a vector database, check out the following resources:\n", + "\n", + "- [RedisVL Documentation](https://redisvl.com) - Documentation for the Redis Vector Library Client\n", + "- [Redis Vector Similarity Docs](https://redis.io/docs/stack/search/reference/vectors/) - Redis official docs for Vector Search.\n", + "- [Redis-py Search Docs](https://redis.readthedocs.io/en/latest/redismodules.html#redisearch-commands) - Documentation for redis-py client library\n", + "- [Vector Similarity Search: From Basics to Production](https://mlops.community/vector-similarity-search-from-basics-to-production/) - Introductory blog post to VSS and Redis as a VectorDB." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ - "## Installing" + "## Install Redis Python Client\n", + "\n", + "Redis-py is the officially supported client by Redis. Recently released is the RedisVL client which is purpose built for the Vector Database use cases. Both can be installed with pip." ] }, { @@ -30,7 +134,7 @@ }, "outputs": [], "source": [ - "!pip install redis" + "!pip install redis redisvl openai tiktoken" ] }, { @@ -42,52 +146,97 @@ }, { "cell_type": "code", - "execution_count": null, + "execution_count": 2, "metadata": {}, "outputs": [], "source": [ "import os\n", "import getpass\n", "\n", - "os.environ[\"OPENAI_API_KEY\"] = getpass.getpass(\"OpenAI API Key:\")" + "os.environ[\"OPENAI_API_KEY\"] = getpass.getpass(\"OpenAI API Key:\")\n", + "\n", + "from langchain.embeddings import OpenAIEmbeddings\n", + "embeddings = OpenAIEmbeddings()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ - "## Example" + "## Sample Data\n", + "\n", + "First we will describe some sample data so that the various attributes of the Redis vector store can be demonstrated." ] }, { "cell_type": "code", - "execution_count": 1, + "execution_count": 3, + "metadata": {}, + "outputs": [], + "source": [ + "metadata = [\n", + " {\n", + " \"user\": \"john\",\n", + " \"age\": 18,\n", + " \"job\": \"engineer\",\n", + " \"credit_score\": \"high\",\n", + " },\n", + " {\n", + " \"user\": \"derrick\",\n", + " \"age\": 45,\n", + " \"job\": \"doctor\",\n", + " \"credit_score\": \"low\",\n", + " },\n", + " {\n", + " \"user\": \"nancy\",\n", + " \"age\": 94,\n", + " \"job\": \"doctor\",\n", + " \"credit_score\": \"high\",\n", + " },\n", + " {\n", + " \"user\": \"tyler\",\n", + " \"age\": 100,\n", + " \"job\": \"engineer\",\n", + " \"credit_score\": \"high\",\n", + " },\n", + " {\n", + " \"user\": \"joe\",\n", + " \"age\": 35,\n", + " \"job\": \"dentist\",\n", + " \"credit_score\": \"medium\",\n", + " },\n", + "]\n", + "texts = [\"foo\", \"foo\", \"foo\", \"bar\", \"bar\"]" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## Initializing Redis\n", + "\n", + "The Redis VectorStore instance can be initialized in a number of ways. There are multiple class methods that can be used to initialize a Redis VectorStore instance.\n", + "\n", + "- ``Redis.__init__`` - Initialize directly\n", + "- ``Redis.from_documents`` - Initialize from a list of ``Langchain.docstore.Document`` objects\n", + "- ``Redis.from_texts`` - Initialize from a list of texts (optionally with metadata)\n", + "- ``Redis.from_texts_return_keys`` - Initialize from a list of texts (optionally with metadata) and return the keys\n", + "- ``Redis.from_existing_index`` - Initialize from an existing Redis index\n", + "\n", + "Below we will use the ``Redis.from_documents`` method." + ] + }, + { + "cell_type": "code", + "execution_count": 4, "metadata": { "tags": [] }, "outputs": [], "source": [ - "from langchain.embeddings import OpenAIEmbeddings\n", - "from langchain.text_splitter import CharacterTextSplitter\n", "from langchain.vectorstores.redis import Redis" ] }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "from langchain.document_loaders import TextLoader\n", - "\n", - "loader = TextLoader(\"../../../state_of_the_union.txt\")\n", - "documents = loader.load()\n", - "text_splitter = CharacterTextSplitter(chunk_size=1000, chunk_overlap=0)\n", - "docs = text_splitter.split_documents(documents)\n", - "\n", - "embeddings = OpenAIEmbeddings()" - ] - }, { "cell_type": "markdown", "metadata": {}, @@ -97,90 +246,705 @@ }, { "cell_type": "code", - "execution_count": null, + "execution_count": 5, "metadata": {}, "outputs": [], "source": [ + "from langchain.docstore.document import Document\n", + "\n", + "documents = [Document(page_content=t, metadata=m) for t, m in zip(texts, metadata)]\n", "rds = Redis.from_documents(\n", - " docs, embeddings, redis_url=\"redis://localhost:6379\", index_name=\"link\"\n", + " documents,\n", + " embeddings,\n", + " redis_url=\"redis://localhost:6379\",\n", + " index_name=\"users\"\n", ")" ] }, + { + "cell_type": "code", + "execution_count": 6, + "metadata": {}, + "outputs": [ + { + "data": { + "text/plain": [ + "'users'" + ] + }, + "execution_count": 6, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "rds.index_name" + ] + }, { "cell_type": "markdown", "metadata": {}, "source": [ - "If you're interested in the keys of your entries you have to split your docs in texts and metadatas" + "## Inspecting the Created Index\n", + "\n", + "Once the ``Redis`` VectorStore object has been constructed, an index will have been created in Redis if it did not already exist. The index can be inspected with both the ``rvl``and the ``redis-cli`` command line tool. If you installed ``redisvl`` above, you can use the ``rvl`` command line tool to inspect the index." ] }, { "cell_type": "code", - "execution_count": null, + "execution_count": 7, + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "\u001b[32m16:58:26\u001b[0m \u001b[34m[RedisVL]\u001b[0m \u001b[1;30mINFO\u001b[0m Indices:\n", + "\u001b[32m16:58:26\u001b[0m \u001b[34m[RedisVL]\u001b[0m \u001b[1;30mINFO\u001b[0m 1. users\n" + ] + } + ], + "source": [ + "# assumes you're running Redis locally (use --host, --port, --password, --username, to change this)\n", + "!rvl index listall" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "The ``Redis`` VectorStore implementation will attempt to generate index schema (fields for filtering) for any metadata passed through the ``from_texts``, ``from_texts_return_keys``, and ``from_documents`` methods. This way, whatever metadata is passed will be indexed into the Redis search index allowing\n", + "for filtering on those fields.\n", + "\n", + "Below we show what fields were created from the metadata we defined above" + ] + }, + { + "cell_type": "code", + "execution_count": 8, + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "\n", + "\n", + "Index Information:\n", + "╭──────────────┬────────────────┬───────────────┬─────────────────┬────────────╮\n", + "│ Index Name │ Storage Type │ Prefixes │ Index Options │ Indexing │\n", + "├──────────────┼────────────────┼───────────────┼─────────────────┼────────────┤\n", + "│ users │ HASH │ ['doc:users'] │ [] │ 0 │\n", + "╰──────────────┴────────────────┴───────────────┴─────────────────┴────────────╯\n", + "Index Fields:\n", + "╭────────────────┬────────────────┬─────────┬────────────────┬────────────────╮\n", + "│ Name │ Attribute │ Type │ Field Option │ Option Value │\n", + "├────────────────┼────────────────┼─────────┼────────────────┼────────────────┤\n", + "│ user │ user │ TEXT │ WEIGHT │ 1 │\n", + "│ job │ job │ TEXT │ WEIGHT │ 1 │\n", + "│ credit_score │ credit_score │ TEXT │ WEIGHT │ 1 │\n", + "│ content │ content │ TEXT │ WEIGHT │ 1 │\n", + "│ age │ age │ NUMERIC │ │ │\n", + "│ content_vector │ content_vector │ VECTOR │ │ │\n", + "╰────────────────┴────────────────┴─────────┴────────────────┴────────────────╯\n" + ] + } + ], + "source": [ + "!rvl index info -i users" + ] + }, + { + "cell_type": "code", + "execution_count": 9, + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "\n", + "Statistics:\n", + "╭─────────────────────────────┬─────────────╮\n", + "│ Stat Key │ Value │\n", + "├─────────────────────────────┼─────────────┤\n", + "│ num_docs │ 5 │\n", + "│ num_terms │ 15 │\n", + "│ max_doc_id │ 5 │\n", + "│ num_records │ 33 │\n", + "│ percent_indexed │ 1 │\n", + "│ hash_indexing_failures │ 0 │\n", + "│ number_of_uses │ 4 │\n", + "│ bytes_per_record_avg │ 4.60606 │\n", + "│ doc_table_size_mb │ 0.000524521 │\n", + "│ inverted_sz_mb │ 0.000144958 │\n", + "│ key_table_size_mb │ 0.000193596 │\n", + "│ offset_bits_per_record_avg │ 8 │\n", + "│ offset_vectors_sz_mb │ 2.19345e-05 │\n", + "│ offsets_per_term_avg │ 0.69697 │\n", + "│ records_per_doc_avg │ 6.6 │\n", + "│ sortable_values_size_mb │ 0 │\n", + "│ total_indexing_time │ 0.32 │\n", + "│ total_inverted_index_blocks │ 16 │\n", + "│ vector_index_sz_mb │ 6.0126 │\n", + "╰─────────────────────────────┴─────────────╯\n" + ] + } + ], + "source": [ + "!rvl stats -i users" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "It's important to note that we have not specified that the ``user``, ``job``, ``credit_score`` and ``age`` in the metadata should be fields within the index, this is because the ``Redis`` VectorStore object automatically generate the index schema from the passed metadata. For more information on the generation of index fields, see the API documentation." + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## Querying\n", + "\n", + "There are multiple ways to query the ``Redis`` VectorStore implementation based on what use case you have:\n", + "\n", + "- ``similarity_search``: Find the most similar vectors to a given vector.\n", + "- ``similarity_search_with_score``: Find the most similar vectors to a given vector and return the vector distance\n", + "- ``similarity_search_limit_score``: Find the most similar vectors to a given vector and limit the number of results to the ``score_threshold``\n", + "- ``similarity_search_with_relevance_scores``: Find the most similar vectors to a given vector and return the vector similarities" + ] + }, + { + "cell_type": "code", + "execution_count": 10, + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "foo\n" + ] + } + ], + "source": [ + "results = rds.similarity_search(\"foo\")\n", + "print(results[0].page_content)" + ] + }, + { + "cell_type": "code", + "execution_count": 11, + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Key of the document in Redis: doc:users:a70ca43b3a4e4168bae57c78753a200f\n", + "Metadata of the document: {'user': 'derrick', 'job': 'doctor', 'credit_score': 'low', 'age': '45'}\n" + ] + } + ], + "source": [ + "# return metadata\n", + "results = rds.similarity_search(\"foo\", k=3)\n", + "meta = results[1].metadata\n", + "print(\"Key of the document in Redis: \", meta.pop(\"id\"))\n", + "print(\"Metadata of the document: \", meta)\n" + ] + }, + { + "cell_type": "code", + "execution_count": 12, + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Content: foo --- Score: 0.0\n", + "Content: foo --- Score: 0.0\n", + "Content: foo --- Score: 0.0\n", + "Content: bar --- Score: 0.1566\n", + "Content: bar --- Score: 0.1566\n" + ] + } + ], + "source": [ + "# with scores (distances)\n", + "results = rds.similarity_search_with_score(\"foo\", k=5)\n", + "for result in results:\n", + " print(f\"Content: {result[0].page_content} --- Score: {result[1]}\")" + ] + }, + { + "cell_type": "code", + "execution_count": 13, + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Content: foo --- Score: 0.0\n", + "Content: foo --- Score: 0.0\n", + "Content: foo --- Score: 0.0\n" + ] + } + ], + "source": [ + "# limit the vector distance that can be returned\n", + "results = rds.similarity_search_with_score(\"foo\", k=5, distance_threshold=0.1)\n", + "for result in results:\n", + " print(f\"Content: {result[0].page_content} --- Score: {result[1]}\")" + ] + }, + { + "cell_type": "code", + "execution_count": 14, + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Content: foo --- Similiarity: 1.0\n", + "Content: foo --- Similiarity: 1.0\n", + "Content: foo --- Similiarity: 1.0\n", + "Content: bar --- Similiarity: 0.8434\n", + "Content: bar --- Similiarity: 0.8434\n" + ] + } + ], + "source": [ + "# with scores\n", + "results = rds.similarity_search_with_relevance_scores(\"foo\", k=5)\n", + "for result in results:\n", + " print(f\"Content: {result[0].page_content} --- Similiarity: {result[1]}\")" + ] + }, + { + "cell_type": "code", + "execution_count": 15, + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Content: foo --- Similarity: 1.0\n", + "Content: foo --- Similarity: 1.0\n", + "Content: foo --- Similarity: 1.0\n" + ] + } + ], + "source": [ + "# limit scores (similarities have to be over .9)\n", + "results = rds.similarity_search_with_relevance_scores(\"foo\", k=5, score_threshold=0.9)\n", + "for result in results:\n", + " print(f\"Content: {result[0].page_content} --- Similarity: {result[1]}\")" + ] + }, + { + "cell_type": "code", + "execution_count": 16, + "metadata": {}, + "outputs": [ + { + "data": { + "text/plain": [ + "['doc:users:b9c71d62a0a34241a37950b448dafd38']" + ] + }, + "execution_count": 16, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "# you can also add new documents as follows\n", + "new_document = [\"baz\"]\n", + "new_metadata = [{\n", + " \"user\": \"sam\",\n", + " \"age\": 50,\n", + " \"job\": \"janitor\",\n", + " \"credit_score\": \"high\"\n", + "}]\n", + "# both the document and metadata must be lists\n", + "rds.add_texts(new_document, new_metadata)" + ] + }, + { + "cell_type": "code", + "execution_count": 17, + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "{'id': 'doc:users:b9c71d62a0a34241a37950b448dafd38', 'user': 'sam', 'job': 'janitor', 'credit_score': 'high', 'age': '50'}\n" + ] + } + ], + "source": [ + "# now query the new document\n", + "results = rds.similarity_search(\"baz\", k=3)\n", + "print(results[0].metadata)" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## Connect to an Existing Index\n", + "\n", + "In order to have the same metadata indexed when using the ``Redis`` VectorStore. You will need to have the same ``index_schema`` passed in either as a path to a yaml file or as a dictionary. The following shows how to obtain the schema from an index and connect to an existing index." + ] + }, + { + "cell_type": "code", + "execution_count": 18, "metadata": {}, "outputs": [], "source": [ - "texts = [d.page_content for d in docs]\n", - "metadatas = [d.metadata for d in docs]\n", + "# write the schema to a yaml file\n", + "rds.write_schema(\"redis_schema.yaml\")" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "The schema file for this example should look something like:\n", + "\n", + "```yaml\n", + "numeric:\n", + "- name: age\n", + " no_index: false\n", + " sortable: false\n", + "text:\n", + "- name: user\n", + " no_index: false\n", + " no_stem: false\n", + " sortable: false\n", + " weight: 1\n", + " withsuffixtrie: false\n", + "- name: job\n", + " no_index: false\n", + " no_stem: false\n", + " sortable: false\n", + " weight: 1\n", + " withsuffixtrie: false\n", + "- name: credit_score\n", + " no_index: false\n", + " no_stem: false\n", + " sortable: false\n", + " weight: 1\n", + " withsuffixtrie: false\n", + "- name: content\n", + " no_index: false\n", + " no_stem: false\n", + " sortable: false\n", + " weight: 1\n", + " withsuffixtrie: false\n", + "vector:\n", + "- algorithm: FLAT\n", + " block_size: 1000\n", + " datatype: FLOAT32\n", + " dims: 1536\n", + " distance_metric: COSINE\n", + " initial_cap: 20000\n", + " name: content_vector\n", + "```\n", + "\n", + "**Notice**, this include **all** possible fields for the schema. You can remove any fields that you don't need." + ] + }, + { + "cell_type": "code", + "execution_count": 19, + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "{'id': 'doc:users:8484c48a032d4c4cbe3cc2ed6845fabb', 'user': 'john', 'job': 'engineer', 'credit_score': 'high', 'age': '18'}\n" + ] + } + ], + "source": [ + "# now we can connect to our existing index as follows\n", + "\n", + "new_rds = Redis.from_existing_index(\n", + " embeddings,\n", + " index_name=\"users\",\n", + " redis_url=\"redis://localhost:6379\",\n", + " schema=\"redis_schema.yaml\"\n", + ")\n", + "results = new_rds.similarity_search(\"foo\", k=3)\n", + "print(results[0].metadata)" + ] + }, + { + "cell_type": "code", + "execution_count": 20, + "metadata": {}, + "outputs": [ + { + "data": { + "text/plain": [ + "True" + ] + }, + "execution_count": 20, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "# see the schemas are the same\n", + "new_rds.schema == rds.schema" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## Custom Metadata Indexing\n", + "\n", + "In some cases, you may want to control what fields the metadata maps to. For example, you may want the ``credit_score`` field to be a categorical field instead of a text field (which is the default behavior for all string fields). In this case, you can use the ``index_schema`` parameter in each of the initialization methods above to specify the schema for the index. Custom index schema can either be passed as a dictionary or as a path to a yaml file.\n", + "\n", + "All arguments in the schema have defaults besides the name, so you can specify only the fields you want to change. All the names correspond to the snake/lowercase versions of the arguments you would use on the command line with ``redis-cli`` or in ``redis-py``. For more on the arguments for each field, see the [documentation](https://redis.io/docs/interact/search-and-query/basic-constructs/field-and-type-options/)\n", + "\n", + "The below example shows how to specify the schema for the ``credit_score`` field as a Tag (categorical) field instead of a text field. \n", + "\n", + "```yaml\n", + "# index_schema.yml\n", + "tag:\n", + " - name: credit_score\n", + "text:\n", + " - name: user\n", + " - name: job\n", + "numeric:\n", + " - name: age\n", + "```\n", + "\n", + "In Python this would look like:\n", + "\n", + "```python\n", + "\n", + "index_schema = {\n", + " \"tag\": [{\"name\": \"credit_score\"}],\n", + " \"text\": [{\"name\": \"user\"}, {\"name\": \"job\"}],\n", + " \"numeric\": [{\"name\": \"age\"}],\n", + "}\n", + "\n", + "```\n", + "\n", + "Notice that only the ``name`` field needs to be specified. All other fields have defaults." + ] + }, + { + "cell_type": "code", + "execution_count": 21, + "metadata": {}, + "outputs": [ + { + "name": "stderr", + "output_type": "stream", + "text": [ + "`index_schema` does not match generated metadata schema.\n", + "If you meant to manually override the schema, please ignore this message.\n", + "index_schema: {'tag': [{'name': 'credit_score'}], 'text': [{'name': 'user'}, {'name': 'job'}], 'numeric': [{'name': 'age'}]}\n", + "generated_schema: {'text': [{'name': 'user'}, {'name': 'job'}, {'name': 'credit_score'}], 'numeric': [{'name': 'age'}], 'tag': []}\n", + "\n" + ] + } + ], + "source": [ + "# create a new index with the new schema defined above\n", + "index_schema = {\n", + " \"tag\": [{\"name\": \"credit_score\"}],\n", + " \"text\": [{\"name\": \"user\"}, {\"name\": \"job\"}],\n", + " \"numeric\": [{\"name\": \"age\"}],\n", + "}\n", "\n", "rds, keys = Redis.from_texts_return_keys(\n", - " texts, embeddings, redis_url=\"redis://localhost:6379\", index_name=\"link\"\n", - ")" + " texts,\n", + " embeddings,\n", + " metadatas=metadata,\n", + " redis_url=\"redis://localhost:6379\",\n", + " index_name=\"users_modified\",\n", + " index_schema=index_schema, # pass in the new index schema\n", + ")\n" ] }, { - "cell_type": "code", - "execution_count": null, + "cell_type": "markdown", "metadata": {}, - "outputs": [], "source": [ - "rds.index_name" + "The above warning is meant to notify users when they are overriding the default behavior. Ignore it if you are intentionally overriding the behavior." ] }, { - "cell_type": "code", - "execution_count": null, + "cell_type": "markdown", "metadata": {}, - "outputs": [], "source": [ - "query = \"What did the president say about Ketanji Brown Jackson\"\n", - "results = rds.similarity_search(query)\n", - "print(results[0].page_content)" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "print(rds.add_texts([\"Ankush went to Princeton\"]))" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "query = \"Princeton\"\n", - "results = rds.similarity_search(query)\n", - "print(results[0].page_content)" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "# Load from existing index\n", - "rds = Redis.from_existing_index(\n", - " embeddings, redis_url=\"redis://localhost:6379\", index_name=\"link\"\n", - ")\n", + "## Hybrid Filtering\n", "\n", - "query = \"What did the president say about Ketanji Brown Jackson\"\n", - "results = rds.similarity_search(query)\n", - "print(results[0].page_content)" + "With the Redis Filter Expression language built into langchain, you can create arbitrarily long chains of hybrid filters\n", + "that can be used to filter your search results. The expression language is derived from the [RedisVL Expression Syntax](https://redisvl.com)\n", + "and is designed to be easy to use and understand.\n", + "\n", + "The following are the available filter types:\n", + "- ``RedisText``: Filter by full-text search against metadata fields. Supports exact, fuzzy, and wildcard matching.\n", + "- ``RedisNum``: Filter by numeric range against metadata fields.\n", + "- ``RedisTag``: Filter by exact match against string based categorical metadata fields. Multiple tags can be specified like \"tag1,tag2,tag3\".\n", + "\n", + "The following are examples of utilizing these filters.\n", + "\n", + "```python\n", + "\n", + "from langchain.vectorstores.redis import RedisText, RedisNum, RedisTag\n", + "\n", + "# exact matching\n", + "has_high_credit = RedisTag(\"credit_score\") == \"high\"\n", + "does_not_have_high_credit = RedisTag(\"credit_score\") != \"low\"\n", + "\n", + "# fuzzy matching\n", + "job_starts_with_eng = RedisText(\"job\") % \"eng*\"\n", + "job_is_engineer = RedisText(\"job\") == \"engineer\"\n", + "job_is_not_engineer = RedisText(\"job\") != \"engineer\"\n", + "\n", + "# numeric filtering\n", + "age_is_18 = RedisNum(\"age\") == 18\n", + "age_is_not_18 = RedisNum(\"age\") != 18\n", + "age_is_greater_than_18 = RedisNum(\"age\") > 18\n", + "age_is_less_than_18 = RedisNum(\"age\") < 18\n", + "age_is_greater_than_or_equal_to_18 = RedisNum(\"age\") >= 18\n", + "age_is_less_than_or_equal_to_18 = RedisNum(\"age\") <= 18\n", + "\n", + "```\n", + "\n", + "The ``RedisFilter`` class can be used to simplify the import of these filters as follows\n", + "\n", + "```python\n", + "\n", + "from langchain.vectorstores.redis import RedisFilter\n", + "\n", + "# same examples as above\n", + "has_high_credit = RedisFilter.tag(\"credit_score\") == \"high\"\n", + "does_not_have_high_credit = RedisFilter.num(\"age\") > 8\n", + "job_starts_with_eng = RedisFilter.text(\"job\") % \"eng*\"\n", + "```\n", + "\n", + "The following are examples of using hybrid filter for search" + ] + }, + { + "cell_type": "code", + "execution_count": 22, + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Job: engineer\n", + "Engineers in the dataset: 2\n" + ] + } + ], + "source": [ + "from langchain.vectorstores.redis import RedisText\n", + "\n", + "is_engineer = RedisText(\"job\") == \"engineer\"\n", + "results = rds.similarity_search(\"foo\", k=3, filter=is_engineer)\n", + "\n", + "print(\"Job:\", results[0].metadata[\"job\"])\n", + "print(\"Engineers in the dataset:\", len(results))" + ] + }, + { + "cell_type": "code", + "execution_count": 23, + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Job: doctor\n", + "Job: doctor\n", + "Jobs in dataset that start with 'doc': 2\n" + ] + } + ], + "source": [ + "# fuzzy match\n", + "starts_with_doc = RedisText(\"job\") % \"doc*\"\n", + "results = rds.similarity_search(\"foo\", k=3, filter=starts_with_doc)\n", + "\n", + "for result in results:\n", + " print(\"Job:\", result.metadata[\"job\"])\n", + "print(\"Jobs in dataset that start with 'doc':\", len(results))" + ] + }, + { + "cell_type": "code", + "execution_count": 24, + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "User: derrick is 45\n", + "User: nancy is 94\n", + "User: joe is 35\n" + ] + } + ], + "source": [ + "from langchain.vectorstores.redis import RedisNum\n", + "\n", + "is_over_18 = RedisNum(\"age\") > 18\n", + "is_under_99 = RedisNum(\"age\") < 99\n", + "age_range = is_over_18 & is_under_99\n", + "results = rds.similarity_search(\"foo\", filter=age_range)\n", + "\n", + "for result in results:\n", + " print(\"User:\", result.metadata[\"user\"], \"is\", result.metadata[\"age\"])" + ] + }, + { + "cell_type": "code", + "execution_count": 25, + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "User: derrick is 45\n", + "User: nancy is 94\n", + "User: joe is 35\n" + ] + } + ], + "source": [ + "# make sure to use parenthesis around FilterExpressions\n", + "# if initializing them while constructing them\n", + "age_range = (RedisNum(\"age\") > 18) & (RedisNum(\"age\") < 99)\n", + "results = rds.similarity_search(\"foo\", filter=age_range)\n", + "\n", + "for result in results:\n", + " print(\"User:\", result.metadata[\"user\"], \"is\", result.metadata[\"age\"])" ] }, { @@ -196,46 +960,135 @@ }, { "cell_type": "code", - "execution_count": null, + "execution_count": 26, "metadata": {}, - "outputs": [], + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Content: foo --- Score: 0.0\n", + "Content: foo --- Score: 0.0\n", + "Content: foo --- Score: 0.0\n" + ] + } + ], "source": [ - "retriever = rds.as_retriever()" + "query = \"foo\"\n", + "results = rds.similarity_search_with_score(query, k=3, return_metadata=True)\n", + "\n", + "for result in results:\n", + " print(\"Content:\", result[0].page_content, \" --- Score: \", result[1])\n" ] }, { "cell_type": "code", - "execution_count": null, + "execution_count": 27, "metadata": {}, "outputs": [], "source": [ - "docs = retriever.get_relevant_documents(query)" + "retriever = rds.as_retriever(search_type=\"similarity\", search_kwargs={\"k\": 4})" + ] + }, + { + "cell_type": "code", + "execution_count": 28, + "metadata": {}, + "outputs": [ + { + "data": { + "text/plain": [ + "[Document(page_content='foo', metadata={'id': 'doc:users_modified:988ecca7574048e396756efc0e79aeca', 'user': 'john', 'job': 'engineer', 'credit_score': 'high', 'age': '18'}),\n", + " Document(page_content='foo', metadata={'id': 'doc:users_modified:009b1afeb4084cc6bdef858c7a99b48e', 'user': 'derrick', 'job': 'doctor', 'credit_score': 'low', 'age': '45'}),\n", + " Document(page_content='foo', metadata={'id': 'doc:users_modified:7087cee9be5b4eca93c30fbdd09a2731', 'user': 'nancy', 'job': 'doctor', 'credit_score': 'high', 'age': '94'}),\n", + " Document(page_content='bar', metadata={'id': 'doc:users_modified:01ef6caac12b42c28ad870aefe574253', 'user': 'tyler', 'job': 'engineer', 'credit_score': 'high', 'age': '100'})]" + ] + }, + "execution_count": 28, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "docs = retriever.get_relevant_documents(query)\n", + "docs" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ - "We can also use similarity_limit as a search method. This is only return documents if they are similar enough" + "There is also the `similarity_distance_threshold` retriever which allows the user to specify the vector distance" ] }, { "cell_type": "code", - "execution_count": null, + "execution_count": 29, "metadata": {}, "outputs": [], "source": [ - "retriever = rds.as_retriever(search_type=\"similarity_limit\")" + "retriever = rds.as_retriever(search_type=\"similarity_distance_threshold\", search_kwargs={\"k\": 4, \"distance_threshold\": 0.1})" ] }, { "cell_type": "code", - "execution_count": null, + "execution_count": 30, + "metadata": {}, + "outputs": [ + { + "data": { + "text/plain": [ + "[Document(page_content='foo', metadata={'id': 'doc:users_modified:988ecca7574048e396756efc0e79aeca', 'user': 'john', 'job': 'engineer', 'credit_score': 'high', 'age': '18'}),\n", + " Document(page_content='foo', metadata={'id': 'doc:users_modified:009b1afeb4084cc6bdef858c7a99b48e', 'user': 'derrick', 'job': 'doctor', 'credit_score': 'low', 'age': '45'}),\n", + " Document(page_content='foo', metadata={'id': 'doc:users_modified:7087cee9be5b4eca93c30fbdd09a2731', 'user': 'nancy', 'job': 'doctor', 'credit_score': 'high', 'age': '94'})]" + ] + }, + "execution_count": 30, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "docs = retriever.get_relevant_documents(query)\n", + "docs" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "Lastly, the ``similarity_score_threshold`` allows the user to define the minimum score for similar documents" + ] + }, + { + "cell_type": "code", + "execution_count": 31, "metadata": {}, "outputs": [], "source": [ - "# Here we can see it doesn't return any results because there are no relevant documents\n", - "retriever.get_relevant_documents(\"where did ankush go to college?\")" + "retriever = rds.as_retriever(search_type=\"similarity_score_threshold\", search_kwargs={\"score_threshold\": 0.9, \"k\": 10})" + ] + }, + { + "cell_type": "code", + "execution_count": 32, + "metadata": {}, + "outputs": [ + { + "data": { + "text/plain": [ + "[Document(page_content='foo', metadata={'id': 'doc:users_modified:988ecca7574048e396756efc0e79aeca', 'user': 'john', 'job': 'engineer', 'credit_score': 'high', 'age': '18'}),\n", + " Document(page_content='foo', metadata={'id': 'doc:users_modified:009b1afeb4084cc6bdef858c7a99b48e', 'user': 'derrick', 'job': 'doctor', 'credit_score': 'low', 'age': '45'}),\n", + " Document(page_content='foo', metadata={'id': 'doc:users_modified:7087cee9be5b4eca93c30fbdd09a2731', 'user': 'nancy', 'job': 'doctor', 'credit_score': 'high', 'age': '94'})]" + ] + }, + "execution_count": 32, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "retriever.get_relevant_documents(\"foo\")" ] }, { @@ -254,15 +1107,48 @@ }, { "cell_type": "code", - "execution_count": null, + "execution_count": 33, "metadata": { "scrolled": true }, - "outputs": [], + "outputs": [ + { + "data": { + "text/plain": [ + "True" + ] + }, + "execution_count": 33, + "metadata": {}, + "output_type": "execute_result" + } + ], "source": [ "Redis.delete(keys, redis_url=\"redis://localhost:6379\")" ] }, + { + "cell_type": "code", + "execution_count": 34, + "metadata": {}, + "outputs": [ + { + "data": { + "text/plain": [ + "True" + ] + }, + "execution_count": 34, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "# delete the indices too\n", + "Redis.drop_index(index_name=\"users\", delete_documents=True, redis_url=\"redis://localhost:6379\")\n", + "Redis.drop_index(index_name=\"users_modified\", delete_documents=True, redis_url=\"redis://localhost:6379\")" + ] + }, { "cell_type": "markdown", "metadata": {}, @@ -280,7 +1166,7 @@ }, { "cell_type": "code", - "execution_count": null, + "execution_count": 35, "metadata": {}, "outputs": [], "source": [ @@ -322,7 +1208,7 @@ "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", - "version": "3.11.3" + "version": "3.8.13" } }, "nbformat": 4, diff --git a/libs/langchain/langchain/cache.py b/libs/langchain/langchain/cache.py index a78c0608011..3160fbae6a5 100644 --- a/libs/langchain/langchain/cache.py +++ b/libs/langchain/langchain/cache.py @@ -33,6 +33,7 @@ from typing import ( Any, Callable, Dict, + List, Optional, Sequence, Tuple, @@ -302,6 +303,14 @@ class RedisSemanticCache(BaseCache): # TODO - implement a TTL policy in Redis + DEFAULT_SCHEMA = { + "content_key": "prompt", + "text": [ + {"name": "prompt"}, + ], + "extra": [{"name": "return_val"}, {"name": "llm_string"}], + } + def __init__( self, redis_url: str, embedding: Embeddings, score_threshold: float = 0.2 ): @@ -349,12 +358,14 @@ class RedisSemanticCache(BaseCache): embedding=self.embedding, index_name=index_name, redis_url=self.redis_url, + schema=cast(Dict, self.DEFAULT_SCHEMA), ) except ValueError: redis = RedisVectorstore( - embedding_function=self.embedding.embed_query, + embedding=self.embedding, index_name=index_name, redis_url=self.redis_url, + index_schema=cast(Dict, self.DEFAULT_SCHEMA), ) _embedding = self.embedding.embed_query(text="test") redis._create_index(dim=len(_embedding)) @@ -374,17 +385,18 @@ class RedisSemanticCache(BaseCache): def lookup(self, prompt: str, llm_string: str) -> Optional[RETURN_VAL_TYPE]: """Look up based on prompt and llm_string.""" llm_cache = self._get_llm_cache(llm_string) - generations = [] + generations: List = [] # Read from a Hash - results = llm_cache.similarity_search_limit_score( + results = llm_cache.similarity_search( query=prompt, k=1, - score_threshold=self.score_threshold, + distance_threshold=self.score_threshold, ) if results: for document in results: - for text in document.metadata["return_val"]: - generations.append(Generation(text=text)) + generations.extend( + _load_generations_from_json(document.metadata["return_val"]) + ) return generations if generations else None def update(self, prompt: str, llm_string: str, return_val: RETURN_VAL_TYPE) -> None: @@ -402,11 +414,11 @@ class RedisSemanticCache(BaseCache): ) return llm_cache = self._get_llm_cache(llm_string) - # Write to vectorstore + _dump_generations_to_json([g for g in return_val]) metadata = { "llm_string": llm_string, "prompt": prompt, - "return_val": [generation.text for generation in return_val], + "return_val": _dump_generations_to_json([g for g in return_val]), } llm_cache.add_texts(texts=[prompt], metadatas=[metadata]) diff --git a/libs/langchain/langchain/memory/utils.py b/libs/langchain/langchain/memory/utils.py index 2706f1fc7e0..eafb48904d4 100644 --- a/libs/langchain/langchain/memory/utils.py +++ b/libs/langchain/langchain/memory/utils.py @@ -1,7 +1,5 @@ from typing import Any, Dict, List -from langchain.schema.messages import get_buffer_string # noqa: 401 - def get_prompt_input_key(inputs: Dict[str, Any], memory_variables: List[str]) -> str: """ diff --git a/libs/langchain/langchain/utilities/redis.py b/libs/langchain/langchain/utilities/redis.py index a9535c44259..e6c5cb13883 100644 --- a/libs/langchain/langchain/utilities/redis.py +++ b/libs/langchain/langchain/utilities/redis.py @@ -1,16 +1,64 @@ from __future__ import annotations import logging -from typing import ( - TYPE_CHECKING, - Any, -) +import re +from typing import TYPE_CHECKING, Any, List, Optional, Pattern from urllib.parse import urlparse +import numpy as np + +logger = logging.getLogger(__name__) + if TYPE_CHECKING: from redis.client import Redis as RedisType -logger = logging.getLogger(__name__) + +def _array_to_buffer(array: List[float], dtype: Any = np.float32) -> bytes: + return np.array(array).astype(dtype).tobytes() + + +class TokenEscaper: + """ + Escape punctuation within an input string. + """ + + # Characters that RediSearch requires us to escape during queries. + # Source: https://redis.io/docs/stack/search/reference/escaping/#the-rules-of-text-field-tokenization + DEFAULT_ESCAPED_CHARS = r"[,.<>{}\[\]\\\"\':;!@#$%^&*()\-+=~\/]" + + def __init__(self, escape_chars_re: Optional[Pattern] = None): + if escape_chars_re: + self.escaped_chars_re = escape_chars_re + else: + self.escaped_chars_re = re.compile(self.DEFAULT_ESCAPED_CHARS) + + def escape(self, value: str) -> str: + def escape_symbol(match: re.Match) -> str: + value = match.group(0) + return f"\\{value}" + + return self.escaped_chars_re.sub(escape_symbol, value) + + +def check_redis_module_exist(client: RedisType, required_modules: List[dict]) -> None: + """Check if the correct Redis modules are installed.""" + installed_modules = client.module_list() + installed_modules = { + module[b"name"].decode("utf-8"): module for module in installed_modules + } + for module in required_modules: + if module["name"] in installed_modules and int( + installed_modules[module["name"]][b"ver"] + ) >= int(module["ver"]): + return + # otherwise raise error + error_message = ( + "Redis cannot be used as a vector database without RediSearch >=2.4" + "Please head to https://redis.io/docs/stack/search/quick_start/" + "to know more about installing the RediSearch module within Redis Stack." + ) + logger.error(error_message) + raise ValueError(error_message) def get_client(redis_url: str, **kwargs: Any) -> RedisType: diff --git a/libs/langchain/langchain/vectorstores/redis.py b/libs/langchain/langchain/vectorstores/redis.py deleted file mode 100644 index 56429bea86c..00000000000 --- a/libs/langchain/langchain/vectorstores/redis.py +++ /dev/null @@ -1,664 +0,0 @@ -from __future__ import annotations - -import json -import logging -import uuid -from typing import ( - TYPE_CHECKING, - Any, - Callable, - Dict, - Iterable, - List, - Literal, - Mapping, - Optional, - Tuple, - Type, -) - -import numpy as np - -from langchain.callbacks.manager import ( - AsyncCallbackManagerForRetrieverRun, - CallbackManagerForRetrieverRun, -) -from langchain.docstore.document import Document -from langchain.embeddings.base import Embeddings -from langchain.pydantic_v1 import root_validator -from langchain.utilities.redis import get_client -from langchain.utils import get_from_dict_or_env -from langchain.vectorstores.base import VectorStore, VectorStoreRetriever - -logger = logging.getLogger(__name__) - -if TYPE_CHECKING: - from redis.client import Redis as RedisType - from redis.commands.search.query import Query - - -# required modules -REDIS_REQUIRED_MODULES = [ - {"name": "search", "ver": 20400}, - {"name": "searchlight", "ver": 20400}, -] - -# distance mmetrics -REDIS_DISTANCE_METRICS = Literal["COSINE", "IP", "L2"] - - -def _check_redis_module_exist(client: RedisType, required_modules: List[dict]) -> None: - """Check if the correct Redis modules are installed.""" - installed_modules = client.module_list() - installed_modules = { - module[b"name"].decode("utf-8"): module for module in installed_modules - } - for module in required_modules: - if module["name"] in installed_modules and int( - installed_modules[module["name"]][b"ver"] - ) >= int(module["ver"]): - return - # otherwise raise error - error_message = ( - "Redis cannot be used as a vector database without RediSearch >=2.4" - "Please head to https://redis.io/docs/stack/search/quick_start/" - "to know more about installing the RediSearch module within Redis Stack." - ) - logger.error(error_message) - raise ValueError(error_message) - - -def _check_index_exists(client: RedisType, index_name: str) -> bool: - """Check if Redis index exists.""" - try: - client.ft(index_name).info() - except: # noqa: E722 - logger.info("Index does not exist") - return False - logger.info("Index already exists") - return True - - -def _redis_key(prefix: str) -> str: - """Redis key schema for a given prefix.""" - return f"{prefix}:{uuid.uuid4().hex}" - - -def _redis_prefix(index_name: str) -> str: - """Redis key prefix for a given index.""" - return f"doc:{index_name}" - - -def _default_relevance_score(val: float) -> float: - return 1 - val - - -class Redis(VectorStore): - """`Redis` vector store. - - To use, you should have the ``redis`` python package installed. - - Example: - .. code-block:: python - - from langchain.vectorstores import Redis - from langchain.embeddings import OpenAIEmbeddings - - embeddings = OpenAIEmbeddings() - vectorstore = Redis( - redis_url="redis://username:password@localhost:6379" - index_name="my-index", - embedding_function=embeddings.embed_query, - ) - - To use a redis replication setup with multiple redis server and redis sentinels - set "redis_url" to "redis+sentinel://" scheme. With this url format a path is - needed holding the name of the redis service within the sentinels to get the - correct redis server connection. The default service name is "mymaster". - - An optional username or password is used for booth connections to the rediserver - and the sentinel, different passwords for server and sentinel are not supported. - And as another constraint only one sentinel instance can be given: - - Example: - .. code-block:: python - - vectorstore = Redis( - redis_url="redis+sentinel://username:password@sentinelhost:26379/mymaster/0" - index_name="my-index", - embedding_function=embeddings.embed_query, - ) - """ - - def __init__( - self, - redis_url: str, - index_name: str, - embedding_function: Callable, - content_key: str = "content", - metadata_key: str = "metadata", - vector_key: str = "content_vector", - relevance_score_fn: Optional[Callable[[float], float]] = None, - distance_metric: REDIS_DISTANCE_METRICS = "COSINE", - **kwargs: Any, - ): - """Initialize with necessary components.""" - self.embedding_function = embedding_function - self.index_name = index_name - try: - redis_client = get_client(redis_url=redis_url, **kwargs) - # check if redis has redisearch module installed - _check_redis_module_exist(redis_client, REDIS_REQUIRED_MODULES) - except ValueError as e: - raise ValueError(f"Redis failed to connect: {e}") - - self.client = redis_client - self.content_key = content_key - self.metadata_key = metadata_key - self.vector_key = vector_key - self.distance_metric = distance_metric - self.relevance_score_fn = relevance_score_fn - - @property - def embeddings(self) -> Optional[Embeddings]: - # TODO: Accept embedding object directly - return None - - def _select_relevance_score_fn(self) -> Callable[[float], float]: - if self.relevance_score_fn: - return self.relevance_score_fn - - if self.distance_metric == "COSINE": - return self._cosine_relevance_score_fn - elif self.distance_metric == "IP": - return self._max_inner_product_relevance_score_fn - elif self.distance_metric == "L2": - return self._euclidean_relevance_score_fn - else: - return _default_relevance_score - - def _create_index(self, dim: int = 1536) -> None: - try: - from redis.commands.search.field import TextField, VectorField - from redis.commands.search.indexDefinition import IndexDefinition, IndexType - except ImportError: - raise ImportError( - "Could not import redis python package. " - "Please install it with `pip install redis`." - ) - - # Check if index exists - if not _check_index_exists(self.client, self.index_name): - # Define schema - schema = ( - TextField(name=self.content_key), - TextField(name=self.metadata_key), - VectorField( - self.vector_key, - "FLAT", - { - "TYPE": "FLOAT32", - "DIM": dim, - "DISTANCE_METRIC": self.distance_metric, - }, - ), - ) - prefix = _redis_prefix(self.index_name) - - # Create Redis Index - self.client.ft(self.index_name).create_index( - fields=schema, - definition=IndexDefinition(prefix=[prefix], index_type=IndexType.HASH), - ) - - def add_texts( - self, - texts: Iterable[str], - metadatas: Optional[List[dict]] = None, - embeddings: Optional[List[List[float]]] = None, - batch_size: int = 1000, - **kwargs: Any, - ) -> List[str]: - """Add more texts to the vectorstore. - - Args: - texts (Iterable[str]): Iterable of strings/text to add to the vectorstore. - metadatas (Optional[List[dict]], optional): Optional list of metadatas. - Defaults to None. - embeddings (Optional[List[List[float]]], optional): Optional pre-generated - embeddings. Defaults to None. - keys (List[str]) or ids (List[str]): Identifiers of entries. - Defaults to None. - batch_size (int, optional): Batch size to use for writes. Defaults to 1000. - - Returns: - List[str]: List of ids added to the vectorstore - """ - ids = [] - prefix = _redis_prefix(self.index_name) - - # Get keys or ids from kwargs - # Other vectorstores use ids - keys_or_ids = kwargs.get("keys", kwargs.get("ids")) - - # Write data to redis - pipeline = self.client.pipeline(transaction=False) - for i, text in enumerate(texts): - # Use provided values by default or fallback - key = keys_or_ids[i] if keys_or_ids else _redis_key(prefix) - metadata = metadatas[i] if metadatas else {} - embedding = embeddings[i] if embeddings else self.embedding_function(text) - pipeline.hset( - key, - mapping={ - self.content_key: text, - self.vector_key: np.array(embedding, dtype=np.float32).tobytes(), - self.metadata_key: json.dumps(metadata), - }, - ) - ids.append(key) - - # Write batch - if i % batch_size == 0: - pipeline.execute() - - # Cleanup final batch - pipeline.execute() - return ids - - def similarity_search( - self, query: str, k: int = 4, **kwargs: Any - ) -> List[Document]: - """ - Returns the most similar indexed documents to the query text. - - Args: - query (str): The query text for which to find similar documents. - k (int): The number of documents to return. Default is 4. - - Returns: - List[Document]: A list of documents that are most similar to the query text. - """ - docs_and_scores = self.similarity_search_with_score(query, k=k) - return [doc for doc, _ in docs_and_scores] - - def similarity_search_limit_score( - self, query: str, k: int = 4, score_threshold: float = 0.2, **kwargs: Any - ) -> List[Document]: - """ - Returns the most similar indexed documents to the query text within the - score_threshold range. - - Args: - query (str): The query text for which to find similar documents. - k (int): The number of documents to return. Default is 4. - score_threshold (float): The minimum matching score required for a document - to be considered a match. Defaults to 0.2. - Because the similarity calculation algorithm is based on cosine - similarity, the smaller the angle, the higher the similarity. - - Returns: - List[Document]: A list of documents that are most similar to the query text, - including the match score for each document. - - Note: - If there are no documents that satisfy the score_threshold value, - an empty list is returned. - - """ - docs_and_scores = self.similarity_search_with_score(query, k=k) - return [doc for doc, score in docs_and_scores if score < score_threshold] - - def _prepare_query(self, k: int) -> Query: - try: - from redis.commands.search.query import Query - except ImportError: - raise ValueError( - "Could not import redis python package. " - "Please install it with `pip install redis`." - ) - # Prepare the Query - hybrid_fields = "*" - base_query = ( - f"{hybrid_fields}=>[KNN {k} @{self.vector_key} $vector AS vector_score]" - ) - return_fields = [self.metadata_key, self.content_key, "vector_score", "id"] - return ( - Query(base_query) - .return_fields(*return_fields) - .sort_by("vector_score") - .paging(0, k) - .dialect(2) - ) - - def similarity_search_with_score( - self, query: str, k: int = 4 - ) -> List[Tuple[Document, float]]: - """Return docs most similar to query. - - Args: - query: Text to look up documents similar to. - k: Number of Documents to return. Defaults to 4. - - Returns: - List of Documents most similar to the query and score for each - """ - # Creates embedding vector from user query - embedding = self.embedding_function(query) - - # Creates Redis query - redis_query = self._prepare_query(k) - - params_dict: Mapping[str, str] = { - "vector": np.array(embedding) # type: ignore - .astype(dtype=np.float32) - .tobytes() - } - - # Perform vector search - results = self.client.ft(self.index_name).search(redis_query, params_dict) - - # Prepare document results - docs_and_scores: List[Tuple[Document, float]] = [] - for result in results.docs: - metadata = {**json.loads(result.metadata), "id": result.id} - doc = Document(page_content=result.content, metadata=metadata) - docs_and_scores.append((doc, float(result.vector_score))) - return docs_and_scores - - @classmethod - def from_texts_return_keys( - cls, - texts: List[str], - embedding: Embeddings, - metadatas: Optional[List[dict]] = None, - index_name: Optional[str] = None, - content_key: str = "content", - metadata_key: str = "metadata", - vector_key: str = "content_vector", - distance_metric: REDIS_DISTANCE_METRICS = "COSINE", - **kwargs: Any, - ) -> Tuple[Redis, List[str]]: - """Create a Redis vectorstore from raw documents. - This is a user-friendly interface that: - 1. Embeds documents. - 2. Creates a new index for the embeddings in Redis. - 3. Adds the documents to the newly created Redis index. - 4. Returns the keys of the newly created documents. - - This is intended to be a quick way to get started. - - Example: - .. code-block:: python - - from langchain.vectorstores import Redis - from langchain.embeddings import OpenAIEmbeddings - embeddings = OpenAIEmbeddings() - redisearch, keys = RediSearch.from_texts_return_keys( - texts, - embeddings, - redis_url="redis://username:password@localhost:6379" - ) - """ - redis_url = get_from_dict_or_env(kwargs, "redis_url", "REDIS_URL") - - if "redis_url" in kwargs: - kwargs.pop("redis_url") - - # Name of the search index if not given - if not index_name: - index_name = uuid.uuid4().hex - - # Create instance - instance = cls( - redis_url, - index_name, - embedding.embed_query, - content_key=content_key, - metadata_key=metadata_key, - vector_key=vector_key, - distance_metric=distance_metric, - **kwargs, - ) - - # Create embeddings over documents - embeddings = embedding.embed_documents(texts) - - # Create the search index - instance._create_index(dim=len(embeddings[0])) - - # Add data to Redis - keys = instance.add_texts(texts, metadatas, embeddings) - return instance, keys - - @classmethod - def from_texts( - cls: Type[Redis], - texts: List[str], - embedding: Embeddings, - metadatas: Optional[List[dict]] = None, - index_name: Optional[str] = None, - content_key: str = "content", - metadata_key: str = "metadata", - vector_key: str = "content_vector", - **kwargs: Any, - ) -> Redis: - """Create a Redis vectorstore from raw documents. - This is a user-friendly interface that: - 1. Embeds documents. - 2. Creates a new index for the embeddings in Redis. - 3. Adds the documents to the newly created Redis index. - - This is intended to be a quick way to get started. - - Example: - .. code-block:: python - - from langchain.vectorstores import Redis - from langchain.embeddings import OpenAIEmbeddings - embeddings = OpenAIEmbeddings() - redisearch = RediSearch.from_texts( - texts, - embeddings, - redis_url="redis://username:password@localhost:6379" - ) - """ - instance, _ = cls.from_texts_return_keys( - texts, - embedding, - metadatas=metadatas, - index_name=index_name, - content_key=content_key, - metadata_key=metadata_key, - vector_key=vector_key, - **kwargs, - ) - return instance - - @staticmethod - def delete( - ids: Optional[List[str]] = None, - **kwargs: Any, - ) -> bool: - """ - Delete a Redis entry. - - Args: - ids: List of ids (keys) to delete. - - Returns: - bool: Whether or not the deletions were successful. - """ - redis_url = get_from_dict_or_env(kwargs, "redis_url", "REDIS_URL") - - if ids is None: - raise ValueError("'ids' (keys)() were not provided.") - - try: - import redis # noqa: F401 - except ImportError: - raise ValueError( - "Could not import redis python package. " - "Please install it with `pip install redis`." - ) - try: - # We need to first remove redis_url from kwargs, - # otherwise passing it to Redis will result in an error. - if "redis_url" in kwargs: - kwargs.pop("redis_url") - client = get_client(redis_url=redis_url, **kwargs) - except ValueError as e: - raise ValueError(f"Your redis connected error: {e}") - # Check if index exists - try: - client.delete(*ids) - logger.info("Entries deleted") - return True - except: # noqa: E722 - # ids does not exist - return False - - @staticmethod - def drop_index( - index_name: str, - delete_documents: bool, - **kwargs: Any, - ) -> bool: - """ - Drop a Redis search index. - - Args: - index_name (str): Name of the index to drop. - delete_documents (bool): Whether to drop the associated documents. - - Returns: - bool: Whether or not the drop was successful. - """ - redis_url = get_from_dict_or_env(kwargs, "redis_url", "REDIS_URL") - try: - import redis # noqa: F401 - except ImportError: - raise ValueError( - "Could not import redis python package. " - "Please install it with `pip install redis`." - ) - try: - # We need to first remove redis_url from kwargs, - # otherwise passing it to Redis will result in an error. - if "redis_url" in kwargs: - kwargs.pop("redis_url") - client = get_client(redis_url=redis_url, **kwargs) - except ValueError as e: - raise ValueError(f"Your redis connected error: {e}") - # Check if index exists - try: - client.ft(index_name).dropindex(delete_documents) - logger.info("Drop index") - return True - except: # noqa: E722 - # Index not exist - return False - - @classmethod - def from_existing_index( - cls, - embedding: Embeddings, - index_name: str, - content_key: str = "content", - metadata_key: str = "metadata", - vector_key: str = "content_vector", - **kwargs: Any, - ) -> Redis: - """Connect to an existing Redis index.""" - redis_url = get_from_dict_or_env(kwargs, "redis_url", "REDIS_URL") - try: - import redis # noqa: F401 - except ImportError: - raise ValueError( - "Could not import redis python package. " - "Please install it with `pip install redis`." - ) - try: - # We need to first remove redis_url from kwargs, - # otherwise passing it to Redis will result in an error. - if "redis_url" in kwargs: - kwargs.pop("redis_url") - client = get_client(redis_url=redis_url, **kwargs) - # check if redis has redisearch module installed - _check_redis_module_exist(client, REDIS_REQUIRED_MODULES) - # ensure that the index already exists - assert _check_index_exists( - client, index_name - ), f"Index {index_name} does not exist" - except Exception as e: - raise ValueError(f"Redis failed to connect: {e}") - - return cls( - redis_url, - index_name, - embedding.embed_query, - content_key=content_key, - metadata_key=metadata_key, - vector_key=vector_key, - **kwargs, - ) - - def as_retriever(self, **kwargs: Any) -> RedisVectorStoreRetriever: - tags = kwargs.pop("tags", None) or [] - tags.extend(self._get_retriever_tags()) - return RedisVectorStoreRetriever(vectorstore=self, **kwargs, tags=tags) - - -class RedisVectorStoreRetriever(VectorStoreRetriever): - """Retriever for `Redis` vector store.""" - - vectorstore: Redis - """Redis VectorStore.""" - search_type: str = "similarity" - """Type of search to perform. Can be either 'similarity' or 'similarity_limit'.""" - k: int = 4 - """Number of documents to return.""" - score_threshold: float = 0.4 - """Score threshold for similarity_limit search.""" - - class Config: - """Configuration for this pydantic object.""" - - arbitrary_types_allowed = True - - @root_validator() - def validate_search_type(cls, values: Dict) -> Dict: - """Validate search type.""" - if "search_type" in values: - search_type = values["search_type"] - if search_type not in ("similarity", "similarity_limit"): - raise ValueError(f"search_type of {search_type} not allowed.") - return values - - def _get_relevant_documents( - self, query: str, *, run_manager: CallbackManagerForRetrieverRun - ) -> List[Document]: - if self.search_type == "similarity": - docs = self.vectorstore.similarity_search(query, k=self.k) - elif self.search_type == "similarity_limit": - docs = self.vectorstore.similarity_search_limit_score( - query, k=self.k, score_threshold=self.score_threshold - ) - else: - raise ValueError(f"search_type of {self.search_type} not allowed.") - return docs - - async def _aget_relevant_documents( - self, query: str, *, run_manager: AsyncCallbackManagerForRetrieverRun - ) -> List[Document]: - raise NotImplementedError("RedisVectorStoreRetriever does not support async") - - def add_documents(self, documents: List[Document], **kwargs: Any) -> List[str]: - """Add documents to vectorstore.""" - return self.vectorstore.add_documents(documents, **kwargs) - - async def aadd_documents( - self, documents: List[Document], **kwargs: Any - ) -> List[str]: - """Add documents to vectorstore.""" - return await self.vectorstore.aadd_documents(documents, **kwargs) diff --git a/libs/langchain/langchain/vectorstores/redis/__init__.py b/libs/langchain/langchain/vectorstores/redis/__init__.py new file mode 100644 index 00000000000..6f05acb4ab7 --- /dev/null +++ b/libs/langchain/langchain/vectorstores/redis/__init__.py @@ -0,0 +1,9 @@ +from .base import Redis +from .filters import ( + RedisFilter, + RedisNum, + RedisTag, + RedisText, +) + +__all__ = ["Redis", "RedisFilter", "RedisTag", "RedisText", "RedisNum"] diff --git a/libs/langchain/langchain/vectorstores/redis/base.py b/libs/langchain/langchain/vectorstores/redis/base.py new file mode 100644 index 00000000000..f3e966d3c11 --- /dev/null +++ b/libs/langchain/langchain/vectorstores/redis/base.py @@ -0,0 +1,1361 @@ +"""Wrapper around Redis vector database.""" + +from __future__ import annotations + +import logging +import os +import uuid +from typing import ( + TYPE_CHECKING, + Any, + Callable, + Dict, + Iterable, + List, + Mapping, + Optional, + Tuple, + Type, + Union, +) + +import yaml + +from langchain._api import deprecated +from langchain.callbacks.manager import ( + AsyncCallbackManagerForRetrieverRun, + CallbackManagerForRetrieverRun, +) +from langchain.docstore.document import Document +from langchain.embeddings.base import Embeddings +from langchain.utilities.redis import ( + _array_to_buffer, + check_redis_module_exist, + get_client, +) +from langchain.utils import get_from_dict_or_env +from langchain.vectorstores.base import VectorStore, VectorStoreRetriever +from langchain.vectorstores.redis.constants import ( + REDIS_REQUIRED_MODULES, + REDIS_TAG_SEPARATOR, +) + +logger = logging.getLogger(__name__) + +if TYPE_CHECKING: + from redis.client import Redis as RedisType + from redis.commands.search.query import Query + + from langchain.vectorstores.redis.filters import RedisFilterExpression + from langchain.vectorstores.redis.schema import RedisModel + + +def _redis_key(prefix: str) -> str: + """Redis key schema for a given prefix.""" + return f"{prefix}:{uuid.uuid4().hex}" + + +def _redis_prefix(index_name: str) -> str: + """Redis key prefix for a given index.""" + return f"doc:{index_name}" + + +def _default_relevance_score(val: float) -> float: + return 1 - val + + +def check_index_exists(client: RedisType, index_name: str) -> bool: + """Check if Redis index exists.""" + try: + client.ft(index_name).info() + except: # noqa: E722 + logger.info("Index does not exist") + return False + logger.info("Index already exists") + return True + + +class Redis(VectorStore): + """Wrapper around Redis vector database. + + To use, you should have the ``redis`` python package installed + and have a running Redis Enterprise or Redis-Stack server + + For production use cases, it is recommended to use Redis Enterprise + as the scaling, performance, stability and availability is much + better than Redis-Stack. + + For testing and prototyping, however, this is not required. + Redis-Stack is available as a docker container the full vector + search API available. + + .. code-block:: bash + # to run redis stack in docker locally + docker run -d -p 6379:6379 -p 8001:8001 redis/redis-stack:latest + + Once running, you can connect to the redis server with the following url schemas: + - redis://: # simple connection + - redis://:@: # connection with authentication + - rediss://: # connection with SSL + - rediss://:@: # connection with SSL and auth + + + Examples: + + The following examples show various ways to use the Redis VectorStore with + LangChain. + + For all the following examples assume we have the following imports: + + .. code-block:: python + + from langchain.vectorstores import Redis + from langchain.embeddings import OpenAIEmbeddings + + Initialize, create index, and load Documents + .. code-block:: python + + from langchain.vectorstores import Redis + from langchain.embeddings import OpenAIEmbeddings + + rds = Redis.from_documents( + documents, # a list of Document objects from loaders or created + embeddings, # an Embeddings object + redis_url="redis://localhost:6379", + ) + + Initialize, create index, and load Documents with metadata + .. code-block:: python + + + rds = Redis.from_texts( + texts, # a list of strings + metadata, # a list of metadata dicts + embeddings, # an Embeddings object + redis_url="redis://localhost:6379", + ) + + Initialize, create index, and load Documents with metadata and return keys + + .. code-block:: python + + rds, keys = Redis.from_texts_return_keys( + texts, # a list of strings + metadata, # a list of metadata dicts + embeddings, # an Embeddings object + redis_url="redis://localhost:6379", + ) + + For use cases where the index needs to stay alive, you can initialize + with an index name such that it's easier to reference later + + .. code-block:: python + + rds = Redis.from_texts( + texts, # a list of strings + metadata, # a list of metadata dicts + embeddings, # an Embeddings object + index_name="my-index", + redis_url="redis://localhost:6379", + ) + + Initialize and connect to an existing index (from above) + + .. code-block:: python + + rds = Redis.from_existing_index( + embeddings, # an Embeddings object + index_name="my-index", + redis_url="redis://localhost:6379", + ) + + + Advanced examples: + + Custom vector schema can be supplied to change the way that + Redis creates the underlying vector schema. This is useful + for production use cases where you want to optimize the + vector schema for your use case. ex. using HNSW instead of + FLAT (knn) which is the default + + .. code-block:: python + + vector_schema = { + "algorithm": "HNSW" + } + + rds = Redis.from_texts( + texts, # a list of strings + metadata, # a list of metadata dicts + embeddings, # an Embeddings object + vector_schema=vector_schema, + redis_url="redis://localhost:6379", + ) + + Custom index schema can be supplied to change the way that the + metadata is indexed. This is useful for you would like to use the + hybrid querying (filtering) capability of Redis. + + By default, this implementation will automatically generate the index + schema according to the following rules: + - All strings are indexed as text fields + - All numbers are indexed as numeric fields + - All lists of strings are indexed as tag fields (joined by + langchain.vectorstores.redis.constants.REDIS_TAG_SEPARATOR) + - All None values are not indexed but still stored in Redis these are + not retrievable through the interface here, but the raw Redis client + can be used to retrieve them. + - All other types are not indexed + + To override these rules, you can pass in a custom index schema like the following + + .. code-block:: yaml + + tag: + - name: credit_score + text: + - name: user + - name: job + + Typically, the ``credit_score`` field would be a text field since it's a string, + however, we can override this behavior by specifying the field type as shown with + the yaml config (can also be a dictionary) above and the code below. + + .. code-block:: python + + rds = Redis.from_texts( + texts, # a list of strings + metadata, # a list of metadata dicts + embeddings, # an Embeddings object + index_schema="path/to/index_schema.yaml", # can also be a dictionary + redis_url="redis://localhost:6379", + ) + + When connecting to an existing index where a custom schema has been applied, it's + important to pass in the same schema to the ``from_existing_index`` method. + Otherwise, the schema for newly added samples will be incorrect and metadata + will not be returned. + + """ + + DEFAULT_VECTOR_SCHEMA = { + "name": "content_vector", + "algorithm": "FLAT", + "dims": 1536, + "distance_metric": "COSINE", + "datatype": "FLOAT32", + } + + def __init__( + self, + redis_url: str, + index_name: str, + embedding: Embeddings, + index_schema: Optional[Union[Dict[str, str], str, os.PathLike]] = None, + vector_schema: Optional[Dict[str, Union[str, int]]] = None, + relevance_score_fn: Optional[Callable[[float], float]] = None, + **kwargs: Any, + ): + """Initialize with necessary components.""" + self._check_deprecated_kwargs(kwargs) + try: + # TODO use importlib to check if redis is installed + import redis # noqa: F401 + + except ImportError as e: + raise ImportError( + "Could not import redis python package. " + "Please install it with `pip install redis`." + ) from e + + self.index_name = index_name + self._embeddings = embedding + try: + redis_client = get_client(redis_url=redis_url, **kwargs) + # check if redis has redisearch module installed + check_redis_module_exist(redis_client, REDIS_REQUIRED_MODULES) + except ValueError as e: + raise ValueError(f"Redis failed to connect: {e}") + + self.client = redis_client + self.relevance_score_fn = relevance_score_fn + self._schema = self._get_schema_with_defaults(index_schema, vector_schema) + + @property + def embeddings(self) -> Optional[Embeddings]: + """Access the query embedding object if available.""" + return self._embeddings + + @classmethod + def from_texts_return_keys( + cls, + texts: List[str], + embedding: Embeddings, + metadatas: Optional[List[dict]] = None, + index_name: Optional[str] = None, + index_schema: Optional[Union[Dict[str, str], str, os.PathLike]] = None, + vector_schema: Optional[Dict[str, Union[str, int]]] = None, + **kwargs: Any, + ) -> Tuple[Redis, List[str]]: + """Create a Redis vectorstore from raw documents. + + This is a user-friendly interface that: + 1. Embeds documents. + 2. Creates a new Redis index if it doesn't already exist + 3. Adds the documents to the newly created Redis index. + 4. Returns the keys of the newly created documents once stored. + + This method will generate schema based on the metadata passed in + if the `index_schema` is not defined. If the `index_schema` is defined, + it will compare against the generated schema and warn if there are + differences. If you are purposefully defining the schema for the + metadata, then you can ignore that warning. + + To examine the schema options, initialize an instance of this class + and print out the schema using the `Redis.schema`` property. This + will include the content and content_vector classes which are + always present in the langchain schema. + + Example: + .. code-block:: python + + from langchain.vectorstores import Redis + from langchain.embeddings import OpenAIEmbeddings + embeddings = OpenAIEmbeddings() + redis, keys = Redis.from_texts_return_keys( + texts, + embeddings, + redis_url="redis://localhost:6379" + ) + + Args: + texts (List[str]): List of texts to add to the vectorstore. + embedding (Embeddings): Embeddings to use for the vectorstore. + metadatas (Optional[List[dict]], optional): Optional list of metadata + dicts to add to the vectorstore. Defaults to None. + index_name (Optional[str], optional): Optional name of the index to + create or add to. Defaults to None. + index_schema (Optional[Union[Dict[str, str], str, os.PathLike]], optional): + Optional fields to index within the metadata. Overrides generated + schema. Defaults to None. + vector_schema (Optional[Dict[str, Union[str, int]]], optional): Optional + vector schema to use. Defaults to None. + **kwargs (Any): Additional keyword arguments to pass to the Redis client. + + Returns: + Tuple[Redis, List[str]]: Tuple of the Redis instance and the keys of + the newly created documents. + + Raises: + ValueError: If the number of metadatas does not match the number of texts. + """ + try: + # TODO use importlib to check if redis is installed + import redis # noqa: F401 + + from langchain.vectorstores.redis.schema import read_schema + + except ImportError as e: + raise ImportError( + "Could not import redis python package. " + "Please install it with `pip install redis`." + ) from e + + redis_url = get_from_dict_or_env(kwargs, "redis_url", "REDIS_URL") + + if "redis_url" in kwargs: + kwargs.pop("redis_url") + + # flag to use generated schema + if "generate" in kwargs: + kwargs.pop("generate") + + # Name of the search index if not given + if not index_name: + index_name = uuid.uuid4().hex + + # type check for metadata + if metadatas: + if isinstance(metadatas, list) and len(metadatas) != len(texts): # type: ignore # noqa: E501 + raise ValueError("Number of metadatas must match number of texts") + if not (isinstance(metadatas, list) and isinstance(metadatas[0], dict)): + raise ValueError("Metadatas must be a list of dicts") + + generated_schema = _generate_field_schema(metadatas[0]) + if index_schema: + # read in the schema solely to compare to the generated schema + user_schema = read_schema(index_schema) + + # the very rare case where a super user decides to pass the index + # schema and a document loader is used that has metadata which + # we need to map into fields. + if user_schema != generated_schema: + logger.warning( + "`index_schema` does not match generated metadata schema.\n" + + "If you meant to manually override the schema, please " + + "ignore this message.\n" + + f"index_schema: {user_schema}\n" + + f"generated_schema: {generated_schema}\n" + ) + else: + # use the generated schema + index_schema = generated_schema + + # Create instance + instance = cls( + redis_url, + index_name, + embedding, + index_schema=index_schema, + vector_schema=vector_schema, + **kwargs, + ) + + # Create embeddings over documents + embeddings = embedding.embed_documents(texts) + + # Create the search index + instance._create_index(dim=len(embeddings[0])) + + # Add data to Redis + keys = instance.add_texts(texts, metadatas, embeddings) + return instance, keys + + @classmethod + def from_texts( + cls: Type[Redis], + texts: List[str], + embedding: Embeddings, + metadatas: Optional[List[dict]] = None, + index_name: Optional[str] = None, + index_schema: Optional[Union[Dict[str, str], str, os.PathLike]] = None, + vector_schema: Optional[Dict[str, Union[str, int]]] = None, + **kwargs: Any, + ) -> Redis: + """Create a Redis vectorstore from a list of texts. + + This is a user-friendly interface that: + 1. Embeds documents. + 2. Creates a new Redis index if it doesn't already exist + 3. Adds the documents to the newly created Redis index. + + This method will generate schema based on the metadata passed in + if the `index_schema` is not defined. If the `index_schema` is defined, + it will compare against the generated schema and warn if there are + differences. If you are purposefully defining the schema for the + metadata, then you can ignore that warning. + + To examine the schema options, initialize an instance of this class + and print out the schema using the `Redis.schema`` property. This + will include the content and content_vector classes which are + always present in the langchain schema. + + + Example: + .. code-block:: python + + from langchain.vectorstores import Redis + from langchain.embeddings import OpenAIEmbeddings + embeddings = OpenAIEmbeddings() + redisearch = RediSearch.from_texts( + texts, + embeddings, + redis_url="redis://username:password@localhost:6379" + ) + + Args: + texts (List[str]): List of texts to add to the vectorstore. + embedding (Embeddings): Embedding model class (i.e. OpenAIEmbeddings) + for embedding queries. + metadatas (Optional[List[dict]], optional): Optional list of metadata dicts + to add to the vectorstore. Defaults to None. + index_name (Optional[str], optional): Optional name of the index to create + or add to. Defaults to None. + index_schema (Optional[Union[Dict[str, str], str, os.PathLike]], optional): + Optional fields to index within the metadata. Overrides generated + schema. Defaults to None. + vector_schema (Optional[Dict[str, Union[str, int]]], optional): Optional + vector schema to use. Defaults to None. + **kwargs (Any): Additional keyword arguments to pass to the Redis client. + + Returns: + Redis: Redis VectorStore instance. + + Raises: + ValueError: If the number of metadatas does not match the number of texts. + ImportError: If the redis python package is not installed. + """ + instance, _ = cls.from_texts_return_keys( + texts, + embedding, + metadatas=metadatas, + index_name=index_name, + index_schema=index_schema, + vector_schema=vector_schema, + **kwargs, + ) + return instance + + @classmethod + def from_existing_index( + cls, + embedding: Embeddings, + index_name: str, + schema: Union[Dict[str, str], str, os.PathLike], + **kwargs: Any, + ) -> Redis: + """Connect to an existing Redis index. + + Example: + .. code-block:: python + + from langchain.vectorstores import Redis + from langchain.embeddings import OpenAIEmbeddings + embeddings = OpenAIEmbeddings() + redisearch = Redis.from_existing_index( + embeddings, + index_name="my-index", + redis_url="redis://username:password@localhost:6379" + ) + + Args: + embedding (Embeddings): Embedding model class (i.e. OpenAIEmbeddings) + for embedding queries. + index_name (str): Name of the index to connect to. + schema (Union[Dict[str, str], str, os.PathLike]): Schema of the index + and the vector schema. Can be a dict, or path to yaml file + + **kwargs (Any): Additional keyword arguments to pass to the Redis client. + + Returns: + Redis: Redis VectorStore instance. + + Raises: + ValueError: If the index does not exist. + ImportError: If the redis python package is not installed. + """ + redis_url = get_from_dict_or_env(kwargs, "redis_url", "REDIS_URL") + try: + # We need to first remove redis_url from kwargs, + # otherwise passing it to Redis will result in an error. + if "redis_url" in kwargs: + kwargs.pop("redis_url") + client = get_client(redis_url=redis_url, **kwargs) + # check if redis has redisearch module installed + check_redis_module_exist(client, REDIS_REQUIRED_MODULES) + # ensure that the index already exists + assert check_index_exists( + client, index_name + ), f"Index {index_name} does not exist" + except Exception as e: + raise ValueError(f"Redis failed to connect: {e}") + + return cls( + redis_url, + index_name, + embedding, + index_schema=schema, + **kwargs, + ) + + @property + def schema(self) -> Dict[str, List[Any]]: + """Return the schema of the index.""" + return self._schema.as_dict() + + def write_schema(self, path: Union[str, os.PathLike]) -> None: + """Write the schema to a yaml file.""" + with open(path, "w+") as f: + yaml.dump(self.schema, f) + + @staticmethod + def delete( + ids: Optional[List[str]] = None, + **kwargs: Any, + ) -> bool: + """ + Delete a Redis entry. + + Args: + ids: List of ids (keys in redis) to delete. + redis_url: Redis connection url. This should be passed in the kwargs + or set as an environment variable: REDIS_URL. + + Returns: + bool: Whether or not the deletions were successful. + + Raises: + ValueError: If the redis python package is not installed. + ValueError: If the ids (keys in redis) are not provided + """ + redis_url = get_from_dict_or_env(kwargs, "redis_url", "REDIS_URL") + + if ids is None: + raise ValueError("'ids' (keys)() were not provided.") + + try: + import redis # noqa: F401 + except ImportError: + raise ValueError( + "Could not import redis python package. " + "Please install it with `pip install redis`." + ) + try: + # We need to first remove redis_url from kwargs, + # otherwise passing it to Redis will result in an error. + if "redis_url" in kwargs: + kwargs.pop("redis_url") + client = get_client(redis_url=redis_url, **kwargs) + except ValueError as e: + raise ValueError(f"Your redis connected error: {e}") + # Check if index exists + try: + client.delete(*ids) + logger.info("Entries deleted") + return True + except: # noqa: E722 + # ids does not exist + return False + + @staticmethod + def drop_index( + index_name: str, + delete_documents: bool, + **kwargs: Any, + ) -> bool: + """ + Drop a Redis search index. + + Args: + index_name (str): Name of the index to drop. + delete_documents (bool): Whether to drop the associated documents. + + Returns: + bool: Whether or not the drop was successful. + """ + redis_url = get_from_dict_or_env(kwargs, "redis_url", "REDIS_URL") + try: + import redis # noqa: F401 + except ImportError: + raise ValueError( + "Could not import redis python package. " + "Please install it with `pip install redis`." + ) + try: + # We need to first remove redis_url from kwargs, + # otherwise passing it to Redis will result in an error. + if "redis_url" in kwargs: + kwargs.pop("redis_url") + client = get_client(redis_url=redis_url, **kwargs) + except ValueError as e: + raise ValueError(f"Your redis connected error: {e}") + # Check if index exists + try: + client.ft(index_name).dropindex(delete_documents) + logger.info("Drop index") + return True + except: # noqa: E722 + # Index not exist + return False + + def add_texts( + self, + texts: Iterable[str], + metadatas: Optional[List[dict]] = None, + embeddings: Optional[List[List[float]]] = None, + batch_size: int = 1000, + clean_metadata: bool = True, + **kwargs: Any, + ) -> List[str]: + """Add more texts to the vectorstore. + + Args: + texts (Iterable[str]): Iterable of strings/text to add to the vectorstore. + metadatas (Optional[List[dict]], optional): Optional list of metadatas. + Defaults to None. + embeddings (Optional[List[List[float]]], optional): Optional pre-generated + embeddings. Defaults to None. + keys (List[str]) or ids (List[str]): Identifiers of entries. + Defaults to None. + batch_size (int, optional): Batch size to use for writes. Defaults to 1000. + + Returns: + List[str]: List of ids added to the vectorstore + """ + ids = [] + prefix = _redis_prefix(self.index_name) + + # Get keys or ids from kwargs + # Other vectorstores use ids + keys_or_ids = kwargs.get("keys", kwargs.get("ids")) + + # type check for metadata + if metadatas: + if isinstance(metadatas, list) and len(metadatas) != len(texts): # type: ignore # noqa: E501 + raise ValueError("Number of metadatas must match number of texts") + if not (isinstance(metadatas, list) and isinstance(metadatas[0], dict)): + raise ValueError("Metadatas must be a list of dicts") + + # Write data to redis + pipeline = self.client.pipeline(transaction=False) + for i, text in enumerate(texts): + # Use provided values by default or fallback + key = keys_or_ids[i] if keys_or_ids else _redis_key(prefix) + metadata = metadatas[i] if metadatas else {} + metadata = _prepare_metadata(metadata) if clean_metadata else metadata + embedding = ( + embeddings[i] if embeddings else self._embeddings.embed_query(text) + ) + pipeline.hset( + key, + mapping={ + self._schema.content_key: text, + self._schema.content_vector_key: _array_to_buffer( + embedding, self._schema.vector_dtype + ), + **metadata, + }, + ) + ids.append(key) + + # Write batch + if i % batch_size == 0: + pipeline.execute() + + # Cleanup final batch + pipeline.execute() + return ids + + def as_retriever(self, **kwargs: Any) -> RedisVectorStoreRetriever: + tags = kwargs.pop("tags", None) or [] + tags.extend(self._get_retriever_tags()) + return RedisVectorStoreRetriever(vectorstore=self, **kwargs, tags=tags) + + @deprecated("0.0.272", alternative="similarity_search(distance_threshold=0.1)") + def similarity_search_limit_score( + self, query: str, k: int = 4, score_threshold: float = 0.2, **kwargs: Any + ) -> List[Document]: + """ + Returns the most similar indexed documents to the query text within the + score_threshold range. + + Deprecated: Use similarity_search with distance_threshold instead. + + Args: + query (str): The query text for which to find similar documents. + k (int): The number of documents to return. Default is 4. + score_threshold (float): The minimum matching *distance* required + for a document to be considered a match. Defaults to 0.2. + + Returns: + List[Document]: A list of documents that are most similar to the query text + including the match score for each document. + + Note: + If there are no documents that satisfy the score_threshold value, + an empty list is returned. + + """ + return self.similarity_search( + query, k=k, distance_threshold=score_threshold, **kwargs + ) + + def similarity_search_with_score( + self, + query: str, + k: int = 4, + filter: Optional[RedisFilterExpression] = None, + return_metadata: bool = True, + **kwargs: Any, + ) -> List[Tuple[Document, float]]: + """Run similarity search with **vector distance**. + + The "scores" returned from this function are the raw vector + distances from the query vector. For similarity scores, use + ``similarity_search_with_relevance_scores``. + + Args: + query (str): The query text for which to find similar documents. + k (int): The number of documents to return. Default is 4. + filter (RedisFilterExpression, optional): Optional metadata filter. + Defaults to None. + return_metadata (bool, optional): Whether to return metadata. + Defaults to True. + + Returns: + List[Tuple[Document, float]]: A list of documents that are + most similar to the query with the distance for each document. + """ + try: + import redis + + except ImportError as e: + raise ImportError( + "Could not import redis python package. " + "Please install it with `pip install redis`." + ) from e + + if "score_threshold" in kwargs: + logger.warning( + "score_threshold is deprecated. Use distance_threshold instead." + + "score_threshold should only be used in " + + "similarity_search_with_relevance_scores." + + "score_threshold will be removed in a future release.", + ) + + redis_query, params_dict = self._prepare_query( + query, + k=k, + filter=filter, + with_metadata=return_metadata, + with_distance=True, + **kwargs, + ) + + # Perform vector search + # ignore type because redis-py is wrong about bytes + try: + results = self.client.ft(self.index_name).search(redis_query, params_dict) # type: ignore # noqa: E501 + except redis.exceptions.ResponseError as e: + # split error message and see if it starts with "Syntax" + if str(e).split(" ")[0] == "Syntax": + raise ValueError( + "Query failed with syntax error. " + + "This is likely due to malformation of " + + "filter, vector, or query argument" + ) from e + raise e + + # Prepare document results + docs_with_scores: List[Tuple[Document, float]] = [] + for result in results.docs: + metadata = {} + if return_metadata: + metadata = {"id": result.id} + metadata.update(self._collect_metadata(result)) + + doc = Document(page_content=result.content, metadata=metadata) + distance = self._calculate_fp_distance(result.distance) + docs_with_scores.append((doc, distance)) + + return docs_with_scores + + def similarity_search( + self, + query: str, + k: int = 4, + filter: Optional[RedisFilterExpression] = None, + return_metadata: bool = True, + distance_threshold: Optional[float] = None, + **kwargs: Any, + ) -> List[Document]: + """Run similarity search + + Args: + query (str): The query text for which to find similar documents. + k (int): The number of documents to return. Default is 4. + filter (RedisFilterExpression, optional): Optional metadata filter. + Defaults to None. + return_metadata (bool, optional): Whether to return metadata. + Defaults to True. + distance_threshold (Optional[float], optional): Distance threshold + for vector distance from query vector. Defaults to None. + + Returns: + List[Document]: A list of documents that are most similar to the query + text. + + """ + try: + import redis + + except ImportError as e: + raise ImportError( + "Could not import redis python package. " + "Please install it with `pip install redis`." + ) from e + + if "score_threshold" in kwargs: + logger.warning( + "score_threshold is deprecated. Use distance_threshold instead." + + "score_threshold should only be used in " + + "similarity_search_with_relevance_scores." + + "score_threshold will be removed in a future release.", + ) + + redis_query, params_dict = self._prepare_query( + query, + k=k, + filter=filter, + distance_threshold=distance_threshold, + with_metadata=return_metadata, + with_distance=False, + ) + + # Perform vector search + # ignore type because redis-py is wrong about bytes + try: + results = self.client.ft(self.index_name).search(redis_query, params_dict) # type: ignore # noqa: E501 + except redis.exceptions.ResponseError as e: + # split error message and see if it starts with "Syntax" + if str(e).split(" ")[0] == "Syntax": + raise ValueError( + "Query failed with syntax error. " + + "This is likely due to malformation of " + + "filter, vector, or query argument" + ) from e + raise e + + # Prepare document results + docs = [] + for result in results.docs: + metadata = {} + if return_metadata: + metadata = {"id": result.id} + metadata.update(self._collect_metadata(result)) + + content_key = self._schema.content_key + docs.append( + Document(page_content=getattr(result, content_key), metadata=metadata) + ) + return docs + + def _collect_metadata(self, result: "Document") -> Dict[str, Any]: + """Collect metadata from Redis. + + Method ensures that there isn't a mismatch between the metadata + and the index schema passed to this class by the user or generated + by this class. + + Args: + result (Document): redis.commands.search.Document object returned + from Redis. + + Returns: + Dict[str, Any]: Collected metadata. + """ + # new metadata dict as modified by this method + meta = {} + for key in self._schema.metadata_keys: + try: + meta[key] = getattr(result, key) + except AttributeError: + # warning about attribute missing + logger.warning( + f"Metadata key {key} not found in metadata. " + + "Setting to None. \n" + + "Metadata fields defined for this instance: " + + f"{self._schema.metadata_keys}" + ) + meta[key] = None + return meta + + def _prepare_query( + self, + query: str, + k: int = 4, + filter: Optional[RedisFilterExpression] = None, + distance_threshold: Optional[float] = None, + with_metadata: bool = True, + with_distance: bool = False, + ) -> Tuple["Query", Dict[str, Any]]: + # Creates embedding vector from user query + embedding = self._embeddings.embed_query(query) + + # Creates Redis query + params_dict: Dict[str, Union[str, bytes, float]] = { + "vector": _array_to_buffer(embedding, self._schema.vector_dtype), + } + + # prepare return fields including score + return_fields = [self._schema.content_key] + if with_distance: + return_fields.append("distance") + if with_metadata: + return_fields.extend(self._schema.metadata_keys) + + if distance_threshold: + params_dict["distance_threshold"] = distance_threshold + return ( + self._prepare_range_query( + k, filter=filter, return_fields=return_fields + ), + params_dict, + ) + return ( + self._prepare_vector_query(k, filter=filter, return_fields=return_fields), + params_dict, + ) + + def _prepare_range_query( + self, + k: int, + filter: Optional[RedisFilterExpression] = None, + return_fields: List[str] = [], + ) -> "Query": + try: + from redis.commands.search.query import Query + except ImportError as e: + raise ImportError( + "Could not import redis python package. " + "Please install it with `pip install redis`." + ) from e + vector_key = self._schema.content_vector_key + base_query = f"@{vector_key}:[VECTOR_RANGE $distance_threshold $vector]" + + if filter: + base_query = "(" + base_query + " " + str(filter) + ")" + + query_string = base_query + "=>{$yield_distance_as: distance}" + + return ( + Query(query_string) + .return_fields(*return_fields) + .sort_by("distance") + .paging(0, k) + .dialect(2) + ) + + def _prepare_vector_query( + self, + k: int, + filter: Optional[RedisFilterExpression] = None, + return_fields: List[str] = [], + ) -> "Query": + """Prepare query for vector search. + + Args: + k: Number of results to return. + filter: Optional metadata filter. + + Returns: + query: Query object. + """ + try: + from redis.commands.search.query import Query + except ImportError as e: + raise ImportError( + "Could not import redis python package. " + "Please install it with `pip install redis`." + ) from e + query_prefix = "*" + if filter: + query_prefix = f"{str(filter)}" + vector_key = self._schema.content_vector_key + base_query = f"({query_prefix})=>[KNN {k} @{vector_key} $vector AS distance]" + + query = ( + Query(base_query) + .return_fields(*return_fields) + .sort_by("distance") + .paging(0, k) + .dialect(2) + ) + return query + + def _get_schema_with_defaults( + self, + index_schema: Optional[Union[Dict[str, str], str, os.PathLike]] = None, + vector_schema: Optional[Dict[str, Union[str, int]]] = None, + ) -> "RedisModel": + # should only be called after init of Redis (so Import handled) + from langchain.vectorstores.redis.schema import RedisModel, read_schema + + schema = RedisModel() + # read in schema (yaml file or dict) and + # pass to the Pydantic validators + if index_schema: + schema_values = read_schema(index_schema) + schema = RedisModel(**schema_values) + + # ensure user did not exclude the content field + # no modifications if content field found + schema.add_content_field() + + # if no content_vector field, add vector field to schema + # this makes adding a vector field to the schema optional when + # the user just wants additional metadata + try: + # see if user overrode the content vector + schema.content_vector + # if user overrode the content vector, check if they + # also passed vector schema. This won't be used since + # the index schema overrode the content vector + if vector_schema: + logger.warning( + "`vector_schema` is ignored since content_vector is " + + "overridden in `index_schema`." + ) + + # user did not override content vector + except ValueError: + # set default vector schema and update with user provided schema + # if the user provided any + vector_field = self.DEFAULT_VECTOR_SCHEMA.copy() + if vector_schema: + vector_field.update(vector_schema) + + # add the vector field either way + schema.add_vector_field(vector_field) + return schema + + def _create_index(self, dim: int = 1536) -> None: + try: + from redis.commands.search.indexDefinition import ( # type: ignore + IndexDefinition, + IndexType, + ) + + except ImportError: + raise ValueError( + "Could not import redis python package. " + "Please install it with `pip install redis`." + ) + + # Set vector dimension + # can't obtain beforehand because we don't + # know which embedding model is being used. + self._schema.content_vector.dims = dim + + # Check if index exists + if not check_index_exists(self.client, self.index_name): + prefix = _redis_prefix(self.index_name) + + # Create Redis Index + self.client.ft(self.index_name).create_index( + fields=self._schema.get_fields(), + definition=IndexDefinition(prefix=[prefix], index_type=IndexType.HASH), + ) + + def _calculate_fp_distance(self, distance: str) -> float: + """Calculate the distance based on the vector datatype + + Two datatypes supported: + - FLOAT32 + - FLOAT64 + + if it's FLOAT32, we need to round the distance to 4 decimal places + otherwise, round to 7 decimal places. + """ + if self._schema.content_vector.datatype == "FLOAT32": + return round(float(distance), 4) + return round(float(distance), 7) + + def _check_deprecated_kwargs(self, kwargs: Mapping[str, Any]) -> None: + """Check for deprecated kwargs.""" + + deprecated_kwargs = { + "redis_host": "redis_url", + "redis_port": "redis_url", + "redis_password": "redis_url", + "content_key": "index_schema", + "vector_key": "vector_schema", + "distance_metric": "vector_schema", + } + for key, value in kwargs.items(): + if key in deprecated_kwargs: + raise ValueError( + f"Keyword argument '{key}' is deprecated. " + f"Please use '{deprecated_kwargs[key]}' instead." + ) + + def _select_relevance_score_fn(self) -> Callable[[float], float]: + if self.relevance_score_fn: + return self.relevance_score_fn + + metric_map = { + "COSINE": self._cosine_relevance_score_fn, + "IP": self._max_inner_product_relevance_score_fn, + "L2": self._euclidean_relevance_score_fn, + } + try: + return metric_map[self._schema.content_vector.distance_metric] + except KeyError: + return _default_relevance_score + + +def _generate_field_schema(data: Dict[str, Any]) -> Dict[str, Any]: + """ + Generate a schema for the search index in Redis based on the input metadata. + + Given a dictionary of metadata, this function categorizes each metadata + field into one of the three categories: + - text: The field contains textual data. + - numeric: The field contains numeric data (either integer or float). + - tag: The field contains list of tags (strings). + + Args + data (Dict[str, Any]): A dictionary where keys are metadata field names + and values are the metadata values. + + Returns: + Dict[str, Any]: A dictionary with three keys "text", "numeric", and "tag". + Each key maps to a list of fields that belong to that category. + + Raises: + ValueError: If a metadata field cannot be categorized into any of + the three known types. + """ + result: Dict[str, Any] = { + "text": [], + "numeric": [], + "tag": [], + } + + for key, value in data.items(): + # Numeric fields + try: + int(value) + result["numeric"].append({"name": key}) + continue + except (ValueError, TypeError): + pass + + # None values are not indexed as of now + if value is None: + continue + + # if it's a list of strings, we assume it's a tag + if isinstance(value, (list, tuple)): + if not value or isinstance(value[0], str): + result["tag"].append({"name": key}) + else: + name = type(value[0]).__name__ + raise ValueError( + f"List/tuple values should contain strings: '{key}': {name}" + ) + continue + + # Check if value is string before processing further + if isinstance(value, str): + result["text"].append({"name": key}) + continue + + # Unable to classify the field value + name = type(value).__name__ + raise ValueError( + "Could not generate Redis index field type mapping " + + f"for metadata: '{key}': {name}" + ) + + return result + + +def _prepare_metadata(metadata: Dict[str, Any]) -> Dict[str, Any]: + """ + Prepare metadata for indexing in Redis by sanitizing its values. + + - String, integer, and float values remain unchanged. + - None or empty values are replaced with empty strings. + - Lists/tuples of strings are joined into a single string with a comma separator. + + Args: + metadata (Dict[str, Any]): A dictionary where keys are metadata + field names and values are the metadata values. + + Returns: + Dict[str, Any]: A sanitized dictionary ready for indexing in Redis. + + Raises: + ValueError: If any metadata value is not one of the known + types (string, int, float, or list of strings). + """ + + def raise_error(key: str, value: Any) -> None: + raise ValueError( + f"Metadata value for key '{key}' must be a string, int, " + + f"float, or list of strings. Got {type(value).__name__}" + ) + + clean_meta: Dict[str, Union[str, float, int]] = {} + for key, value in metadata.items(): + if not value: + clean_meta[key] = "" + continue + + # No transformation needed + if isinstance(value, (str, int, float)): + clean_meta[key] = value + + # if it's a list/tuple of strings, we join it + elif isinstance(value, (list, tuple)): + if not value or isinstance(value[0], str): + clean_meta[key] = REDIS_TAG_SEPARATOR.join(value) + else: + raise_error(key, value) + else: + raise_error(key, value) + return clean_meta + + +class RedisVectorStoreRetriever(VectorStoreRetriever): + """Retriever for Redis VectorStore.""" + + vectorstore: Redis + """Redis VectorStore.""" + search_type: str = "similarity" + """Type of search to perform. Can be either + 'similarity', + 'similarity_distance_threshold', + 'similarity_score_threshold' + """ + + search_kwargs: Dict[str, Any] = { + "k": 4, + "score_threshold": 0.9, + # set to None to avoid distance used in score_threshold search + "distance_threshold": None, + } + """Default search kwargs.""" + + allowed_search_types = [ + "similarity", + "similarity_distance_threshold", + "similarity_score_threshold", + ] + """Allowed search types.""" + + class Config: + """Configuration for this pydantic object.""" + + arbitrary_types_allowed = True + + def _get_relevant_documents( + self, query: str, *, run_manager: CallbackManagerForRetrieverRun + ) -> List[Document]: + if self.search_type == "similarity": + docs = self.vectorstore.similarity_search(query, **self.search_kwargs) + + elif self.search_type == "similarity_distance_threshold": + if self.search_kwargs["distance_threshold"] is None: + raise ValueError( + "distance_threshold must be provided for " + + "similarity_distance_threshold retriever" + ) + docs = self.vectorstore.similarity_search(query, **self.search_kwargs) + + elif self.search_type == "similarity_score_threshold": + docs_and_similarities = ( + self.vectorstore.similarity_search_with_relevance_scores( + query, **self.search_kwargs + ) + ) + docs = [doc for doc, _ in docs_and_similarities] + else: + raise ValueError(f"search_type of {self.search_type} not allowed.") + return docs + + async def _aget_relevant_documents( + self, query: str, *, run_manager: AsyncCallbackManagerForRetrieverRun + ) -> List[Document]: + raise NotImplementedError("RedisVectorStoreRetriever does not support async") + + def add_documents(self, documents: List[Document], **kwargs: Any) -> List[str]: + """Add documents to vectorstore.""" + return self.vectorstore.add_documents(documents, **kwargs) + + async def aadd_documents( + self, documents: List[Document], **kwargs: Any + ) -> List[str]: + """Add documents to vectorstore.""" + return await self.vectorstore.aadd_documents(documents, **kwargs) diff --git a/libs/langchain/langchain/vectorstores/redis/constants.py b/libs/langchain/langchain/vectorstores/redis/constants.py new file mode 100644 index 00000000000..ddbfe4c5847 --- /dev/null +++ b/libs/langchain/langchain/vectorstores/redis/constants.py @@ -0,0 +1,20 @@ +from typing import Any, Dict, List + +import numpy as np + +# required modules +REDIS_REQUIRED_MODULES = [ + {"name": "search", "ver": 20600}, + {"name": "searchlight", "ver": 20600}, +] + +# distance metrics +REDIS_DISTANCE_METRICS: List[str] = ["COSINE", "IP", "L2"] + +# supported vector datatypes +REDIS_VECTOR_DTYPE_MAP: Dict[str, Any] = { + "FLOAT32": np.float32, + "FLOAT64": np.float64, +} + +REDIS_TAG_SEPARATOR = "," diff --git a/libs/langchain/langchain/vectorstores/redis/filters.py b/libs/langchain/langchain/vectorstores/redis/filters.py new file mode 100644 index 00000000000..0f6608bae81 --- /dev/null +++ b/libs/langchain/langchain/vectorstores/redis/filters.py @@ -0,0 +1,420 @@ +from enum import Enum +from functools import wraps +from typing import Any, Callable, Dict, List, Optional, Union + +from langchain.utilities.redis import TokenEscaper + +# disable mypy error for dunder method overrides +# mypy: disable-error-code="override" + + +class RedisFilterOperator(Enum): + EQ = 1 + NE = 2 + LT = 3 + GT = 4 + LE = 5 + GE = 6 + OR = 7 + AND = 8 + LIKE = 9 + IN = 10 + + +class RedisFilter: + @staticmethod + def text(field: str) -> "RedisText": + return RedisText(field) + + @staticmethod + def num(field: str) -> "RedisNum": + return RedisNum(field) + + @staticmethod + def tag(field: str) -> "RedisTag": + return RedisTag(field) + + +class RedisFilterField: + escaper: "TokenEscaper" = TokenEscaper() + OPERATORS: Dict[RedisFilterOperator, str] = {} + + def __init__(self, field: str): + self._field = field + self._value: Any = None + self._operator: RedisFilterOperator = RedisFilterOperator.EQ + + def equals(self, other: "RedisFilterField") -> bool: + if not isinstance(other, type(self)): + return False + return self._field == other._field and self._value == other._value + + def _set_value( + self, val: Any, val_type: type, operator: RedisFilterOperator + ) -> None: + # check that the operator is supported by this class + if operator not in self.OPERATORS: + raise ValueError( + f"Operator {operator} not supported by {self.__class__.__name__}. " + + f"Supported operators are {self.OPERATORS.values()}" + ) + + if not isinstance(val, val_type): + raise TypeError( + f"Right side argument passed to operator {self.OPERATORS[operator]} " + f"with left side " + f"argument {self.__class__.__name__} must be of type {val_type}" + ) + self._value = val + self._operator = operator + + +def check_operator_misuse(func: Callable) -> Callable: + @wraps(func) + def wrapper(instance: Any, *args: List[Any], **kwargs: Dict[str, Any]) -> Any: + # Extracting 'other' from positional arguments or keyword arguments + other = kwargs.get("other") if "other" in kwargs else None + if not other: + for arg in args: + if isinstance(arg, type(instance)): + other = arg + break + + if isinstance(other, type(instance)): + raise ValueError( + "Equality operators are overridden for FilterExpression creation. Use " + ".equals() for equality checks" + ) + return func(instance, *args, **kwargs) + + return wrapper + + +class RedisTag(RedisFilterField): + """A RedisTag is a RedisFilterField representing a tag in a Redis index.""" + + OPERATORS: Dict[RedisFilterOperator, str] = { + RedisFilterOperator.EQ: "==", + RedisFilterOperator.NE: "!=", + RedisFilterOperator.IN: "==", + } + + OPERATOR_MAP: Dict[RedisFilterOperator, str] = { + RedisFilterOperator.EQ: "@%s:{%s}", + RedisFilterOperator.NE: "(-@%s:{%s})", + RedisFilterOperator.IN: "@%s:{%s}", + } + + def __init__(self, field: str): + """Create a RedisTag FilterField + + Args: + field (str): The name of the RedisTag field in the index to be queried + against. + """ + super().__init__(field) + + def _set_tag_value( + self, other: Union[List[str], str], operator: RedisFilterOperator + ) -> None: + if isinstance(other, list): + if not all(isinstance(tag, str) for tag in other): + raise ValueError("All tags must be strings") + else: + other = [other] + self._set_value(other, list, operator) + + @check_operator_misuse + def __eq__(self, other: Union[List[str], str]) -> "RedisFilterExpression": + """Create a RedisTag equality filter expression + + Args: + other (Union[List[str], str]): The tag(s) to filter on. + + Example: + >>> from langchain.vectorstores.redis import RedisTag + >>> filter = RedisTag("brand") == "nike" + """ + self._set_tag_value(other, RedisFilterOperator.EQ) + return RedisFilterExpression(str(self)) + + @check_operator_misuse + def __ne__(self, other: Union[List[str], str]) -> "RedisFilterExpression": + """Create a RedisTag inequality filter expression + + Args: + other (Union[List[str], str]): The tag(s) to filter on. + + Example: + >>> from langchain.vectorstores.redis import RedisTag + >>> filter = RedisTag("brand") != "nike" + """ + self._set_tag_value(other, RedisFilterOperator.NE) + return RedisFilterExpression(str(self)) + + @property + def _formatted_tag_value(self) -> str: + return "|".join([self.escaper.escape(tag) for tag in self._value]) + + def __str__(self) -> str: + if not self._value: + raise ValueError( + f"Operator must be used before calling __str__. Operators are " + f"{self.OPERATORS.values()}" + ) + """Return the Redis Query syntax for a RedisTag filter expression""" + return self.OPERATOR_MAP[self._operator] % ( + self._field, + self._formatted_tag_value, + ) + + +class RedisNum(RedisFilterField): + """A RedisFilterField representing a numeric field in a Redis index.""" + + OPERATORS: Dict[RedisFilterOperator, str] = { + RedisFilterOperator.EQ: "==", + RedisFilterOperator.NE: "!=", + RedisFilterOperator.LT: "<", + RedisFilterOperator.GT: ">", + RedisFilterOperator.LE: "<=", + RedisFilterOperator.GE: ">=", + } + OPERATOR_MAP: Dict[RedisFilterOperator, str] = { + RedisFilterOperator.EQ: "@%s:[%i %i]", + RedisFilterOperator.NE: "(-@%s:[%i %i])", + RedisFilterOperator.GT: "@%s:[(%i +inf]", + RedisFilterOperator.LT: "@%s:[-inf (%i]", + RedisFilterOperator.GE: "@%s:[%i +inf]", + RedisFilterOperator.LE: "@%s:[-inf %i]", + } + + def __str__(self) -> str: + """Return the Redis Query syntax for a Numeric filter expression""" + if not self._value: + raise ValueError( + f"Operator must be used before calling __str__. Operators are " + f"{self.OPERATORS.values()}" + ) + + if ( + self._operator == RedisFilterOperator.EQ + or self._operator == RedisFilterOperator.NE + ): + return self.OPERATOR_MAP[self._operator] % ( + self._field, + self._value, + self._value, + ) + else: + return self.OPERATOR_MAP[self._operator] % (self._field, self._value) + + @check_operator_misuse + def __eq__(self, other: int) -> "RedisFilterExpression": + """Create a Numeric equality filter expression + + Args: + other (int): The value to filter on. + + Example: + >>> from langchain.vectorstores.redis import RedisNum + >>> filter = RedisNum("zipcode") == 90210 + """ + self._set_value(other, int, RedisFilterOperator.EQ) + return RedisFilterExpression(str(self)) + + @check_operator_misuse + def __ne__(self, other: int) -> "RedisFilterExpression": + """Create a Numeric inequality filter expression + + Args: + other (int): The value to filter on. + + Example: + >>> from langchain.vectorstores.redis import RedisNum + >>> filter = RedisNum("zipcode") != 90210 + """ + self._set_value(other, int, RedisFilterOperator.NE) + return RedisFilterExpression(str(self)) + + def __gt__(self, other: int) -> "RedisFilterExpression": + """Create a RedisNumeric greater than filter expression + + Args: + other (int): The value to filter on. + + Example: + >>> from langchain.vectorstores.redis import RedisNum + >>> filter = RedisNum("age") > 18 + """ + self._set_value(other, int, RedisFilterOperator.GT) + return RedisFilterExpression(str(self)) + + def __lt__(self, other: int) -> "RedisFilterExpression": + """Create a Numeric less than filter expression + + Args: + other (int): The value to filter on. + + Example: + >>> from langchain.vectorstores.redis import RedisNum + >>> filter = RedisNum("age") < 18 + """ + self._set_value(other, int, RedisFilterOperator.LT) + return RedisFilterExpression(str(self)) + + def __ge__(self, other: int) -> "RedisFilterExpression": + """Create a Numeric greater than or equal to filter expression + + Args: + other (int): The value to filter on. + + Example: + >>> from langchain.vectorstores.redis import RedisNum + >>> filter = RedisNum("age") >= 18 + """ + self._set_value(other, int, RedisFilterOperator.GE) + return RedisFilterExpression(str(self)) + + def __le__(self, other: int) -> "RedisFilterExpression": + """Create a Numeric less than or equal to filter expression + + Args: + other (int): The value to filter on. + + Example: + >>> from langchain.vectorstores.redis import RedisNum + >>> filter = RedisNum("age") <= 18 + """ + self._set_value(other, int, RedisFilterOperator.LE) + return RedisFilterExpression(str(self)) + + +class RedisText(RedisFilterField): + """A RedisText is a RedisFilterField representing a text field in a Redis index.""" + + OPERATORS = { + RedisFilterOperator.EQ: "==", + RedisFilterOperator.NE: "!=", + RedisFilterOperator.LIKE: "%", + } + OPERATOR_MAP = { + RedisFilterOperator.EQ: '@%s:"%s"', + RedisFilterOperator.NE: '(-@%s:"%s")', + RedisFilterOperator.LIKE: "@%s:%s", + } + + @check_operator_misuse + def __eq__(self, other: str) -> "RedisFilterExpression": + """Create a RedisText equality filter expression + + Args: + other (str): The text value to filter on. + + Example: + >>> from langchain.vectorstores.redis import RedisText + >>> filter = RedisText("job") == "engineer" + """ + self._set_value(other, str, RedisFilterOperator.EQ) + return RedisFilterExpression(str(self)) + + @check_operator_misuse + def __ne__(self, other: str) -> "RedisFilterExpression": + """Create a RedisText inequality filter expression + + Args: + other (str): The text value to filter on. + + Example: + >>> from langchain.vectorstores.redis import RedisText + >>> filter = RedisText("job") != "engineer" + """ + self._set_value(other, str, RedisFilterOperator.NE) + return RedisFilterExpression(str(self)) + + def __mod__(self, other: str) -> "RedisFilterExpression": + """Create a RedisText like filter expression + + Args: + other (str): The text value to filter on. + + Example: + >>> from langchain.vectorstores.redis import RedisText + >>> filter = RedisText("job") % "engineer" + """ + self._set_value(other, str, RedisFilterOperator.LIKE) + return RedisFilterExpression(str(self)) + + def __str__(self) -> str: + if not self._value: + raise ValueError( + f"Operator must be used before calling __str__. Operators are " + f"{self.OPERATORS.values()}" + ) + + try: + return self.OPERATOR_MAP[self._operator] % (self._field, self._value) + except KeyError: + raise Exception("Invalid operator") + + +class RedisFilterExpression: + """A RedisFilterExpression is a logical expression of RedisFilterFields. + + RedisFilterExpressions can be combined using the & and | operators to create + complex logical expressions that evaluate to the Redis Query language. + + This presents an interface by which users can create complex queries + without having to know the Redis Query language. + + Filter expressions are not initialized directly. Instead they are built + by combining RedisFilterFields using the & and | operators. + + Examples: + + >>> from langchain.vectorstores.redis import RedisTag, RedisNum + >>> brand_is_nike = RedisTag("brand") == "nike" + >>> price_is_under_100 = RedisNum("price") < 100 + >>> filter = brand_is_nike & price_is_under_100 + >>> print(str(filter)) + (@brand:{nike} @price:[-inf (100)]) + + """ + + def __init__( + self, + _filter: Optional[str] = None, + operator: Optional[RedisFilterOperator] = None, + left: Optional["RedisFilterExpression"] = None, + right: Optional["RedisFilterExpression"] = None, + ): + self._filter = _filter + self._operator = operator + self._left = left + self._right = right + + def __and__(self, other: "RedisFilterExpression") -> "RedisFilterExpression": + return RedisFilterExpression( + operator=RedisFilterOperator.AND, left=self, right=other + ) + + def __or__(self, other: "RedisFilterExpression") -> "RedisFilterExpression": + return RedisFilterExpression( + operator=RedisFilterOperator.OR, left=self, right=other + ) + + def __str__(self) -> str: + # top level check that allows recursive calls to __str__ + if not self._filter and not self._operator: + raise ValueError("Improperly initialized RedisFilterExpression") + + # allow for single filter expression without operators as last + # expression in the chain might not have an operator + if self._operator: + operator_str = " | " if self._operator == RedisFilterOperator.OR else " " + return f"({str(self._left)}{operator_str}{str(self._right)})" + + # check that base case, the filter is set + if not self._filter: + raise ValueError("Improperly initialized RedisFilterExpression") + return self._filter diff --git a/libs/langchain/langchain/vectorstores/redis/schema.py b/libs/langchain/langchain/vectorstores/redis/schema.py new file mode 100644 index 00000000000..1ecd921928e --- /dev/null +++ b/libs/langchain/langchain/vectorstores/redis/schema.py @@ -0,0 +1,276 @@ +import os +from enum import Enum +from pathlib import Path +from typing import Any, Dict, List, Optional, Union + +import numpy as np +import yaml + +# ignore type error here as it's a redis-py type problem +from redis.commands.search.field import ( # type: ignore + NumericField, + TagField, + TextField, + VectorField, +) +from typing_extensions import Literal + +from langchain.pydantic_v1 import BaseModel, Field, validator +from langchain.vectorstores.redis.constants import REDIS_VECTOR_DTYPE_MAP + + +class RedisDistanceMetric(str, Enum): + l2 = "L2" + cosine = "COSINE" + ip = "IP" + + +class RedisField(BaseModel): + name: str = Field(...) + + +class TextFieldSchema(RedisField): + weight: float = 1 + no_stem: bool = False + phonetic_matcher: Optional[str] = None + withsuffixtrie: bool = False + no_index: bool = False + sortable: Optional[bool] = False + + def as_field(self) -> TextField: + return TextField( + self.name, + weight=self.weight, + no_stem=self.no_stem, + phonetic_matcher=self.phonetic_matcher, + sortable=self.sortable, + no_index=self.no_index, + ) + + +class TagFieldSchema(RedisField): + separator: str = "," + case_sensitive: bool = False + no_index: bool = False + sortable: Optional[bool] = False + + def as_field(self) -> TagField: + return TagField( + self.name, + separator=self.separator, + case_sensitive=self.case_sensitive, + sortable=self.sortable, + no_index=self.no_index, + ) + + +class NumericFieldSchema(RedisField): + no_index: bool = False + sortable: Optional[bool] = False + + def as_field(self) -> NumericField: + return NumericField(self.name, sortable=self.sortable, no_index=self.no_index) + + +class RedisVectorField(RedisField): + dims: int = Field(...) + algorithm: object = Field(...) + datatype: str = Field(default="FLOAT32") + distance_metric: RedisDistanceMetric = Field(default="COSINE") + initial_cap: int = Field(default=20000) + + @validator("distance_metric", pre=True) + def uppercase_strings(cls, v: str) -> str: + return v.upper() + + @validator("datatype", pre=True) + def uppercase_and_check_dtype(cls, v: str) -> str: + if v.upper() not in REDIS_VECTOR_DTYPE_MAP: + raise ValueError( + f"datatype must be one of {REDIS_VECTOR_DTYPE_MAP.keys()}. Got {v}" + ) + return v.upper() + + +class FlatVectorField(RedisVectorField): + algorithm: Literal["FLAT"] = "FLAT" + block_size: int = Field(default=1000) + + def as_field(self) -> VectorField: + return VectorField( + self.name, + self.algorithm, + { + "TYPE": self.datatype, + "DIM": self.dims, + "DISTANCE_METRIC": self.distance_metric, + "INITIAL_CAP": self.initial_cap, + "BLOCK_SIZE": self.block_size, + }, + ) + + +class HNSWVectorField(RedisVectorField): + algorithm: Literal["HNSW"] = "HNSW" + m: int = Field(default=16) + ef_construction: int = Field(default=200) + ef_runtime: int = Field(default=10) + epsilon: float = Field(default=0.8) + + def as_field(self) -> VectorField: + return VectorField( + self.name, + self.algorithm, + { + "TYPE": self.datatype, + "DIM": self.dims, + "DISTANCE_METRIC": self.distance_metric, + "INITIAL_CAP": self.initial_cap, + "M": self.m, + "EF_CONSTRUCTION": self.ef_construction, + "EF_RUNTIME": self.ef_runtime, + "EPSILON": self.epsilon, + }, + ) + + +class RedisModel(BaseModel): + # always have a content field for text + text: List[TextFieldSchema] = [TextFieldSchema(name="content")] + tag: Optional[List[TagFieldSchema]] = None + numeric: Optional[List[NumericFieldSchema]] = None + extra: Optional[List[RedisField]] = None + + # filled by default_vector_schema + vector: Optional[List[Union[FlatVectorField, HNSWVectorField]]] = None + content_key: str = "content" + content_vector_key: str = "content_vector" + + def add_content_field(self) -> None: + if self.text is None: + self.text = [] + for field in self.text: + if field.name == self.content_key: + return + self.text.append(TextFieldSchema(name=self.content_key)) + + def add_vector_field(self, vector_field: Dict[str, Any]) -> None: + # catch case where user inputted no vector field spec + # in the index schema + if self.vector is None: + self.vector = [] + + # ignore types as pydantic is handling type validation and conversion + if vector_field["algorithm"] == "FLAT": + self.vector.append(FlatVectorField(**vector_field)) # type: ignore + elif vector_field["algorithm"] == "HNSW": + self.vector.append(HNSWVectorField(**vector_field)) # type: ignore + else: + raise ValueError( + f"algorithm must be either FLAT or HNSW. Got " + f"{vector_field['algorithm']}" + ) + + def as_dict(self) -> Dict[str, List[Any]]: + schemas: Dict[str, List[Any]] = {"text": [], "tag": [], "numeric": []} + # iter over all class attributes + for attr, attr_value in self.__dict__.items(): + # only non-empty lists + if isinstance(attr_value, list) and len(attr_value) > 0: + field_values: List[Dict[str, Any]] = [] + # iterate over all fields in each category (tag, text, etc) + for val in attr_value: + value: Dict[str, Any] = {} + # iterate over values within each field to extract + # settings for that field (i.e. name, weight, etc) + for field, field_value in val.__dict__.items(): + # make enums into strings + if isinstance(field_value, Enum): + value[field] = field_value.value + # don't write null values + elif field_value is not None: + value[field] = field_value + field_values.append(value) + + schemas[attr] = field_values + + schema: Dict[str, List[Any]] = {} + # only write non-empty lists from defaults + for k, v in schemas.items(): + if len(v) > 0: + schema[k] = v + return schema + + @property + def content_vector(self) -> Union[FlatVectorField, HNSWVectorField]: + if not self.vector: + raise ValueError("No vector fields found") + for field in self.vector: + if field.name == self.content_vector_key: + return field + raise ValueError("No content_vector field found") + + @property + def vector_dtype(self) -> np.dtype: + # should only ever be called after pydantic has validated the schema + return REDIS_VECTOR_DTYPE_MAP[self.content_vector.datatype] + + @property + def is_empty(self) -> bool: + return all( + field is None for field in [self.tag, self.text, self.numeric, self.vector] + ) + + def get_fields(self) -> List["RedisField"]: + redis_fields: List["RedisField"] = [] + if self.is_empty: + return redis_fields + + for field_name in self.__fields__.keys(): + if field_name not in ["content_key", "content_vector_key", "extra"]: + field_group = getattr(self, field_name) + if field_group is not None: + for field in field_group: + redis_fields.append(field.as_field()) + return redis_fields + + @property + def metadata_keys(self) -> List[str]: + keys: List[str] = [] + if self.is_empty: + return keys + + for field_name in self.__fields__.keys(): + field_group = getattr(self, field_name) + if field_group is not None: + for field in field_group: + # check if it's a metadata field. exclude vector and content key + if not isinstance(field, str) and field.name not in [ + self.content_key, + self.content_vector_key, + ]: + keys.append(field.name) + return keys + + +def read_schema( + index_schema: Optional[Union[Dict[str, str], str, os.PathLike]] +) -> Dict[str, Any]: + # check if its a dict and return RedisModel otherwise, check if it's a path and + # read in the file assuming it's a yaml file and return a RedisModel + if isinstance(index_schema, dict): + return index_schema + elif isinstance(index_schema, Path): + with open(index_schema, "rb") as f: + return yaml.safe_load(f) + elif isinstance(index_schema, str): + if Path(index_schema).resolve().is_file(): + with open(index_schema, "rb") as f: + return yaml.safe_load(f) + else: + raise FileNotFoundError(f"index_schema file {index_schema} does not exist") + else: + raise TypeError( + f"index_schema must be a dict, or path to a yaml file " + f"Got {type(index_schema)}" + ) diff --git a/libs/langchain/tests/integration_tests/cache/test_redis_cache.py b/libs/langchain/tests/integration_tests/cache/test_redis_cache.py index 5d51a12e8e1..77b019eff0b 100644 --- a/libs/langchain/tests/integration_tests/cache/test_redis_cache.py +++ b/libs/langchain/tests/integration_tests/cache/test_redis_cache.py @@ -1,16 +1,27 @@ """Test Redis cache functionality.""" +import uuid +from typing import List + import pytest import langchain from langchain.cache import RedisCache, RedisSemanticCache +from langchain.embeddings.base import Embeddings from langchain.schema import Generation, LLMResult -from tests.integration_tests.vectorstores.fake_embeddings import FakeEmbeddings +from tests.integration_tests.vectorstores.fake_embeddings import ( + ConsistentFakeEmbeddings, + FakeEmbeddings, +) from tests.unit_tests.llms.fake_chat_model import FakeChatModel from tests.unit_tests.llms.fake_llm import FakeLLM REDIS_TEST_URL = "redis://localhost:6379" +def random_string() -> str: + return str(uuid.uuid4()) + + def test_redis_cache_ttl() -> None: import redis @@ -30,12 +41,10 @@ def test_redis_cache() -> None: llm_string = str(sorted([(k, v) for k, v in params.items()])) langchain.llm_cache.update("foo", llm_string, [Generation(text="fizz")]) output = llm.generate(["foo"]) - print(output) expected_output = LLMResult( generations=[[Generation(text="fizz")]], llm_output={}, ) - print(expected_output) assert output == expected_output langchain.llm_cache.redis.flushall() @@ -80,14 +89,90 @@ def test_redis_semantic_cache() -> None: langchain.llm_cache.clear(llm_string=llm_string) -def test_redis_semantic_cache_chat() -> None: - import redis +def test_redis_semantic_cache_multi() -> None: + langchain.llm_cache = RedisSemanticCache( + embedding=FakeEmbeddings(), redis_url=REDIS_TEST_URL, score_threshold=0.1 + ) + llm = FakeLLM() + params = llm.dict() + params["stop"] = None + llm_string = str(sorted([(k, v) for k, v in params.items()])) + langchain.llm_cache.update( + "foo", llm_string, [Generation(text="fizz"), Generation(text="Buzz")] + ) + output = llm.generate( + ["bar"] + ) # foo and bar will have the same embedding produced by FakeEmbeddings + expected_output = LLMResult( + generations=[[Generation(text="fizz"), Generation(text="Buzz")]], + llm_output={}, + ) + assert output == expected_output + # clear the cache + langchain.llm_cache.clear(llm_string=llm_string) - langchain.llm_cache = RedisCache(redis_=redis.Redis.from_url(REDIS_TEST_URL)) + +def test_redis_semantic_cache_chat() -> None: + langchain.llm_cache = RedisSemanticCache( + embedding=FakeEmbeddings(), redis_url=REDIS_TEST_URL, score_threshold=0.1 + ) llm = FakeChatModel() params = llm.dict() params["stop"] = None + llm_string = str(sorted([(k, v) for k, v in params.items()])) with pytest.warns(): llm.predict("foo") llm.predict("foo") - langchain.llm_cache.redis.flushall() + langchain.llm_cache.clear(llm_string=llm_string) + + +@pytest.mark.parametrize("embedding", [ConsistentFakeEmbeddings()]) +@pytest.mark.parametrize( + "prompts, generations", + [ + # Single prompt, single generation + ([random_string()], [[random_string()]]), + # Single prompt, multiple generations + ([random_string()], [[random_string(), random_string()]]), + # Single prompt, multiple generations + ([random_string()], [[random_string(), random_string(), random_string()]]), + # Multiple prompts, multiple generations + ( + [random_string(), random_string()], + [[random_string()], [random_string(), random_string()]], + ), + ], + ids=[ + "single_prompt_single_generation", + "single_prompt_multiple_generations", + "single_prompt_multiple_generations", + "multiple_prompts_multiple_generations", + ], +) +def test_redis_semantic_cache_hit( + embedding: Embeddings, prompts: List[str], generations: List[List[str]] +) -> None: + langchain.llm_cache = RedisSemanticCache( + embedding=embedding, redis_url=REDIS_TEST_URL + ) + + llm = FakeLLM() + params = llm.dict() + params["stop"] = None + llm_string = str(sorted([(k, v) for k, v in params.items()])) + + llm_generations = [ + [ + Generation(text=generation, generation_info=params) + for generation in prompt_i_generations + ] + for prompt_i_generations in generations + ] + for prompt_i, llm_generations_i in zip(prompts, llm_generations): + print(prompt_i) + print(llm_generations_i) + langchain.llm_cache.update(prompt_i, llm_string, llm_generations_i) + llm.generate(prompts) + assert llm.generate(prompts) == LLMResult( + generations=llm_generations, llm_output={} + ) diff --git a/libs/langchain/tests/integration_tests/vectorstores/fake_embeddings.py b/libs/langchain/tests/integration_tests/vectorstores/fake_embeddings.py index 550174e2e52..d202813f20f 100644 --- a/libs/langchain/tests/integration_tests/vectorstores/fake_embeddings.py +++ b/libs/langchain/tests/integration_tests/vectorstores/fake_embeddings.py @@ -52,6 +52,7 @@ class ConsistentFakeEmbeddings(FakeEmbeddings): def embed_query(self, text: str) -> List[float]: """Return consistent embeddings for the text, if seen before, or a constant one if the text is unknown.""" + return self.embed_documents([text])[0] if text not in self.known_texts: return [float(1.0)] * (self.dimensionality - 1) + [float(0.0)] return [float(1.0)] * (self.dimensionality - 1) + [ diff --git a/libs/langchain/tests/integration_tests/vectorstores/test_redis.py b/libs/langchain/tests/integration_tests/vectorstores/test_redis.py index aef3138e84a..3b7a4c7acc5 100644 --- a/libs/langchain/tests/integration_tests/vectorstores/test_redis.py +++ b/libs/langchain/tests/integration_tests/vectorstores/test_redis.py @@ -1,17 +1,28 @@ """Test Redis functionality.""" -from typing import List +import os +from typing import Any, Dict, List, Optional import pytest from langchain.docstore.document import Document -from langchain.vectorstores.redis import Redis -from tests.integration_tests.vectorstores.fake_embeddings import FakeEmbeddings +from langchain.vectorstores.redis import ( + Redis, + RedisFilter, + RedisNum, + RedisText, +) +from langchain.vectorstores.redis.filters import RedisFilterExpression +from tests.integration_tests.vectorstores.fake_embeddings import ( + ConsistentFakeEmbeddings, + FakeEmbeddings, +) TEST_INDEX_NAME = "test" TEST_REDIS_URL = "redis://localhost:6379" TEST_SINGLE_RESULT = [Document(page_content="foo")] -TEST_SINGLE_WITH_METADATA_RESULT = [Document(page_content="foo", metadata={"a": "b"})] +TEST_SINGLE_WITH_METADATA = {"a": "b"} TEST_RESULT = [Document(page_content="foo"), Document(page_content="foo")] +RANGE_SCORE = pytest.approx(0.0513, abs=0.002) COSINE_SCORE = pytest.approx(0.05, abs=0.002) IP_SCORE = -8.0 EUCLIDEAN_SCORE = 1.0 @@ -23,6 +34,27 @@ def drop(index_name: str) -> bool: ) +def convert_bytes(data: Any) -> Any: + if isinstance(data, bytes): + return data.decode("ascii") + if isinstance(data, dict): + return dict(map(convert_bytes, data.items())) + if isinstance(data, list): + return list(map(convert_bytes, data)) + if isinstance(data, tuple): + return map(convert_bytes, data) + return data + + +def make_dict(values: List[Any]) -> dict: + i = 0 + di = {} + while i < len(values) - 1: + di[values[i]] = values[i + 1] + i += 2 + return di + + @pytest.fixture def texts() -> List[str]: return ["foo", "bar", "baz"] @@ -31,7 +63,7 @@ def texts() -> List[str]: def test_redis(texts: List[str]) -> None: """Test end to end construction and search.""" docsearch = Redis.from_texts(texts, FakeEmbeddings(), redis_url=TEST_REDIS_URL) - output = docsearch.similarity_search("foo", k=1) + output = docsearch.similarity_search("foo", k=1, return_metadata=False) assert output == TEST_SINGLE_RESULT assert drop(docsearch.index_name) @@ -40,30 +72,55 @@ def test_redis_new_vector(texts: List[str]) -> None: """Test adding a new document""" docsearch = Redis.from_texts(texts, FakeEmbeddings(), redis_url=TEST_REDIS_URL) docsearch.add_texts(["foo"]) - output = docsearch.similarity_search("foo", k=2) + output = docsearch.similarity_search("foo", k=2, return_metadata=False) assert output == TEST_RESULT assert drop(docsearch.index_name) def test_redis_from_existing(texts: List[str]) -> None: """Test adding a new document""" - Redis.from_texts( + docsearch = Redis.from_texts( texts, FakeEmbeddings(), index_name=TEST_INDEX_NAME, redis_url=TEST_REDIS_URL ) + schema: Dict = docsearch.schema + + # write schema for the next test + docsearch.write_schema("test_schema.yml") + # Test creating from an existing docsearch2 = Redis.from_existing_index( - FakeEmbeddings(), index_name=TEST_INDEX_NAME, redis_url=TEST_REDIS_URL + FakeEmbeddings(), + index_name=TEST_INDEX_NAME, + redis_url=TEST_REDIS_URL, + schema=schema, ) - output = docsearch2.similarity_search("foo", k=1) + output = docsearch2.similarity_search("foo", k=1, return_metadata=False) assert output == TEST_SINGLE_RESULT +def test_redis_add_texts_to_existing() -> None: + """Test adding a new document""" + # Test creating from an existing with yaml from file + docsearch = Redis.from_existing_index( + FakeEmbeddings(), + index_name=TEST_INDEX_NAME, + redis_url=TEST_REDIS_URL, + schema="test_schema.yml", + ) + docsearch.add_texts(["foo"]) + output = docsearch.similarity_search("foo", k=2, return_metadata=False) + assert output == TEST_RESULT + assert drop(TEST_INDEX_NAME) + # remove the test_schema.yml file + os.remove("test_schema.yml") + + def test_redis_from_texts_return_keys(texts: List[str]) -> None: """Test from_texts_return_keys constructor.""" docsearch, keys = Redis.from_texts_return_keys( texts, FakeEmbeddings(), redis_url=TEST_REDIS_URL ) - output = docsearch.similarity_search("foo", k=1) + output = docsearch.similarity_search("foo", k=1, return_metadata=False) assert output == TEST_SINGLE_RESULT assert len(keys) == len(texts) assert drop(docsearch.index_name) @@ -73,21 +130,124 @@ def test_redis_from_documents(texts: List[str]) -> None: """Test from_documents constructor.""" docs = [Document(page_content=t, metadata={"a": "b"}) for t in texts] docsearch = Redis.from_documents(docs, FakeEmbeddings(), redis_url=TEST_REDIS_URL) - output = docsearch.similarity_search("foo", k=1) - assert output == TEST_SINGLE_WITH_METADATA_RESULT + output = docsearch.similarity_search("foo", k=1, return_metadata=True) + assert "a" in output[0].metadata.keys() + assert "b" in output[0].metadata.values() assert drop(docsearch.index_name) -def test_redis_add_texts_to_existing() -> None: - """Test adding a new document""" - # Test creating from an existing - docsearch = Redis.from_existing_index( - FakeEmbeddings(), index_name=TEST_INDEX_NAME, redis_url=TEST_REDIS_URL +# -- test filters -- # + + +@pytest.mark.parametrize( + "filter_expr, expected_length, expected_nums", + [ + (RedisText("text") == "foo", 1, None), + (RedisFilter.text("text") == "foo", 1, None), + (RedisText("text") % "ba*", 2, ["bar", "baz"]), + (RedisNum("num") > 2, 1, [3]), + (RedisNum("num") < 2, 1, [1]), + (RedisNum("num") >= 2, 2, [2, 3]), + (RedisNum("num") <= 2, 2, [1, 2]), + (RedisNum("num") != 2, 2, [1, 3]), + (RedisFilter.num("num") != 2, 2, [1, 3]), + (RedisFilter.tag("category") == "a", 3, None), + (RedisFilter.tag("category") == "b", 2, None), + (RedisFilter.tag("category") == "c", 2, None), + (RedisFilter.tag("category") == ["b", "c"], 3, None), + ], + ids=[ + "text-filter-equals-foo", + "alternative-text-equals-foo", + "text-filter-fuzzy-match-ba", + "number-filter-greater-than-2", + "number-filter-less-than-2", + "number-filter-greater-equals-2", + "number-filter-less-equals-2", + "number-filter-not-equals-2", + "alternative-number-not-equals-2", + "tag-filter-equals-a", + "tag-filter-equals-b", + "tag-filter-equals-c", + "tag-filter-equals-b-or-c", + ], +) +def test_redis_filters_1( + filter_expr: RedisFilterExpression, + expected_length: int, + expected_nums: Optional[list], +) -> None: + metadata = [ + {"name": "joe", "num": 1, "text": "foo", "category": ["a", "b"]}, + {"name": "john", "num": 2, "text": "bar", "category": ["a", "c"]}, + {"name": "jane", "num": 3, "text": "baz", "category": ["b", "c", "a"]}, + ] + documents = [Document(page_content="foo", metadata=m) for m in metadata] + docsearch = Redis.from_documents( + documents, FakeEmbeddings(), redis_url=TEST_REDIS_URL ) - docsearch.add_texts(["foo"]) - output = docsearch.similarity_search("foo", k=2) - assert output == TEST_RESULT - assert drop(TEST_INDEX_NAME) + + output = docsearch.similarity_search("foo", k=3, filter=filter_expr) + + assert len(output) == expected_length + + if expected_nums is not None: + for out in output: + assert ( + out.metadata["text"] in expected_nums + or int(out.metadata["num"]) in expected_nums + ) + + assert drop(docsearch.index_name) + + +# -- test index specification -- # + + +def test_index_specification_generation() -> None: + index_schema = { + "text": [{"name": "job"}, {"name": "title"}], + "numeric": [{"name": "salary"}], + } + + text = ["foo"] + meta = {"job": "engineer", "title": "principal engineer", "salary": 100000} + docs = [Document(page_content=t, metadata=meta) for t in text] + r = Redis.from_documents( + docs, FakeEmbeddings(), redis_url=TEST_REDIS_URL, index_schema=index_schema + ) + + output = r.similarity_search("foo", k=1, return_metadata=True) + assert output[0].metadata["job"] == "engineer" + assert output[0].metadata["title"] == "principal engineer" + assert int(output[0].metadata["salary"]) == 100000 + + info = convert_bytes(r.client.ft(r.index_name).info()) + attributes = info["attributes"] + assert len(attributes) == 5 + for attr in attributes: + d = make_dict(attr) + if d["identifier"] == "job": + assert d["type"] == "TEXT" + elif d["identifier"] == "title": + assert d["type"] == "TEXT" + elif d["identifier"] == "salary": + assert d["type"] == "NUMERIC" + elif d["identifier"] == "content": + assert d["type"] == "TEXT" + elif d["identifier"] == "content_vector": + assert d["type"] == "VECTOR" + else: + raise ValueError("Unexpected attribute in index schema") + + assert drop(r.index_name) + + +# -- test distance metrics -- # + +cosine_schema: Dict = {"distance_metric": "cosine"} +ip_schema: Dict = {"distance_metric": "IP"} +l2_schema: Dict = {"distance_metric": "L2"} def test_cosine(texts: List[str]) -> None: @@ -96,7 +256,7 @@ def test_cosine(texts: List[str]) -> None: texts, FakeEmbeddings(), redis_url=TEST_REDIS_URL, - distance_metric="COSINE", + vector_schema=cosine_schema, ) output = docsearch.similarity_search_with_score("far", k=2) _, score = output[1] @@ -107,7 +267,7 @@ def test_cosine(texts: List[str]) -> None: def test_l2(texts: List[str]) -> None: """Test Flat L2 distance.""" docsearch = Redis.from_texts( - texts, FakeEmbeddings(), redis_url=TEST_REDIS_URL, distance_metric="L2" + texts, FakeEmbeddings(), redis_url=TEST_REDIS_URL, vector_schema=l2_schema ) output = docsearch.similarity_search_with_score("far", k=2) _, score = output[1] @@ -118,7 +278,7 @@ def test_l2(texts: List[str]) -> None: def test_ip(texts: List[str]) -> None: """Test inner product distance.""" docsearch = Redis.from_texts( - texts, FakeEmbeddings(), redis_url=TEST_REDIS_URL, distance_metric="IP" + texts, FakeEmbeddings(), redis_url=TEST_REDIS_URL, vector_schema=ip_schema ) output = docsearch.similarity_search_with_score("far", k=2) _, score = output[1] @@ -126,29 +286,34 @@ def test_ip(texts: List[str]) -> None: assert drop(docsearch.index_name) -def test_similarity_search_limit_score(texts: List[str]) -> None: +def test_similarity_search_limit_distance(texts: List[str]) -> None: """Test similarity search limit score.""" docsearch = Redis.from_texts( - texts, FakeEmbeddings(), redis_url=TEST_REDIS_URL, distance_metric="COSINE" + texts, + FakeEmbeddings(), + redis_url=TEST_REDIS_URL, ) - output = docsearch.similarity_search_limit_score("far", k=2, score_threshold=0.1) - assert len(output) == 1 - _, score = output[0] - assert score == COSINE_SCORE + output = docsearch.similarity_search(texts[0], k=3, distance_threshold=0.1) + + # can't check score but length of output should be 2 + assert len(output) == 2 assert drop(docsearch.index_name) -def test_similarity_search_with_score_with_limit_score(texts: List[str]) -> None: +def test_similarity_search_with_score_with_limit_distance(texts: List[str]) -> None: """Test similarity search with score with limit score.""" + docsearch = Redis.from_texts( - texts, FakeEmbeddings(), redis_url=TEST_REDIS_URL, distance_metric="COSINE" + texts, ConsistentFakeEmbeddings(), redis_url=TEST_REDIS_URL ) - output = docsearch.similarity_search_with_relevance_scores( - "far", k=2, score_threshold=0.1 + output = docsearch.similarity_search_with_score( + texts[0], k=3, distance_threshold=0.1, return_metadata=True ) - assert len(output) == 1 - _, score = output[0] - assert score == COSINE_SCORE + + assert len(output) == 2 + for out, score in output: + if out.page_content == texts[1]: + score == COSINE_SCORE assert drop(docsearch.index_name) @@ -156,6 +321,48 @@ def test_delete(texts: List[str]) -> None: """Test deleting a new document""" docsearch = Redis.from_texts(texts, FakeEmbeddings(), redis_url=TEST_REDIS_URL) ids = docsearch.add_texts(["foo"]) - got = docsearch.delete(ids=ids) + got = docsearch.delete(ids=ids, redis_url=TEST_REDIS_URL) assert got assert drop(docsearch.index_name) + + +def test_redis_as_retriever() -> None: + texts = ["foo", "foo", "foo", "foo", "bar"] + docsearch = Redis.from_texts( + texts, ConsistentFakeEmbeddings(), redis_url=TEST_REDIS_URL + ) + + retriever = docsearch.as_retriever(search_type="similarity", search_kwargs={"k": 3}) + results = retriever.get_relevant_documents("foo") + assert len(results) == 3 + assert all([d.page_content == "foo" for d in results]) + + assert drop(docsearch.index_name) + + +def test_redis_retriever_distance_threshold() -> None: + texts = ["foo", "bar", "baz"] + docsearch = Redis.from_texts(texts, FakeEmbeddings(), redis_url=TEST_REDIS_URL) + + retriever = docsearch.as_retriever( + search_type="similarity_distance_threshold", + search_kwargs={"k": 3, "distance_threshold": 0.1}, + ) + results = retriever.get_relevant_documents("foo") + assert len(results) == 2 + + assert drop(docsearch.index_name) + + +def test_redis_retriever_score_threshold() -> None: + texts = ["foo", "bar", "baz"] + docsearch = Redis.from_texts(texts, FakeEmbeddings(), redis_url=TEST_REDIS_URL) + + retriever = docsearch.as_retriever( + search_type="similarity_score_threshold", + search_kwargs={"k": 3, "score_threshold": 0.91}, + ) + results = retriever.get_relevant_documents("foo") + assert len(results) == 2 + + assert drop(docsearch.index_name)