mirror of
https://github.com/hwchase17/langchain.git
synced 2025-07-01 10:54:15 +00:00
Feature/csharp text splitter doc (#10571)
- **Description:** Just docs related to csharp code splitter - **Issue:** It's related to a request made by @baskaryan in a comment on my previous PR #10350 - **Dependencies:** None - **Twitter handle:** @ather19 --------- Co-authored-by: Bagatur <baskaryan@gmail.com>
This commit is contained in:
parent
2c11302598
commit
bfd48925e5
File diff suppressed because it is too large
Load Diff
@ -1,439 +1,440 @@
|
|||||||
{
|
{
|
||||||
"cells": [
|
"cells": [
|
||||||
{
|
|
||||||
"cell_type": "markdown",
|
|
||||||
"id": "13afcae7",
|
|
||||||
"metadata": {},
|
|
||||||
"source": [
|
|
||||||
"# OpenSearch\n",
|
|
||||||
"\n",
|
|
||||||
"> [OpenSearch](https://opensearch.org/) is a scalable, flexible, and extensible open-source software suite for search, analytics, and observability applications licensed under Apache 2.0. `OpenSearch` is a distributed search and analytics engine based on `Apache Lucene`.\n",
|
|
||||||
"\n",
|
|
||||||
"In this notebook, we'll demo the `SelfQueryRetriever` with an `OpenSearch` vector store."
|
|
||||||
]
|
|
||||||
},
|
|
||||||
{
|
|
||||||
"cell_type": "markdown",
|
|
||||||
"id": "68e75fb9",
|
|
||||||
"metadata": {},
|
|
||||||
"source": [
|
|
||||||
"## Creating an OpenSearch vector store\n",
|
|
||||||
"\n",
|
|
||||||
"First, we'll want to create an `OpenSearch` vector store and seed it with some data. We've created a small demo set of documents that contain summaries of movies.\n",
|
|
||||||
"\n",
|
|
||||||
"**Note:** The self-query retriever requires you to have `lark` installed (`pip install lark`). We also need the `opensearch-py` package."
|
|
||||||
]
|
|
||||||
},
|
|
||||||
{
|
|
||||||
"cell_type": "code",
|
|
||||||
"execution_count": null,
|
|
||||||
"outputs": [],
|
|
||||||
"source": [
|
|
||||||
"!pip install lark opensearch-py"
|
|
||||||
],
|
|
||||||
"metadata": {
|
|
||||||
"collapsed": false,
|
|
||||||
"pycharm": {
|
|
||||||
"name": "#%%\n"
|
|
||||||
}
|
|
||||||
}
|
|
||||||
},
|
|
||||||
{
|
|
||||||
"cell_type": "code",
|
|
||||||
"execution_count": 3,
|
|
||||||
"id": "cb4a5787",
|
|
||||||
"metadata": {
|
|
||||||
"tags": []
|
|
||||||
},
|
|
||||||
"outputs": [
|
|
||||||
{
|
{
|
||||||
"name": "stdin",
|
"cell_type": "markdown",
|
||||||
"output_type": "stream",
|
"id": "13afcae7",
|
||||||
"text": [
|
"metadata": {},
|
||||||
"OpenAI API Key: ········\n"
|
"source": [
|
||||||
]
|
"# OpenSearch\n",
|
||||||
}
|
"\n",
|
||||||
],
|
"> [OpenSearch](https://opensearch.org/) is a scalable, flexible, and extensible open-source software suite for search, analytics, and observability applications licensed under Apache 2.0. `OpenSearch` is a distributed search and analytics engine based on `Apache Lucene`.\n",
|
||||||
"source": [
|
"\n",
|
||||||
"from langchain.schema import Document\n",
|
"In this notebook, we'll demo the `SelfQueryRetriever` with an `OpenSearch` vector store."
|
||||||
"from langchain.embeddings.openai import OpenAIEmbeddings\n",
|
]
|
||||||
"from langchain.vectorstores import OpenSearchVectorSearch\n",
|
|
||||||
"import os\n",
|
|
||||||
"import getpass\n",
|
|
||||||
"\n",
|
|
||||||
"os.environ[\"OPENAI_API_KEY\"] = getpass.getpass(\"OpenAI API Key:\")\n",
|
|
||||||
"\n",
|
|
||||||
"embeddings = OpenAIEmbeddings()"
|
|
||||||
]
|
|
||||||
},
|
|
||||||
{
|
|
||||||
"cell_type": "code",
|
|
||||||
"execution_count": 8,
|
|
||||||
"id": "bcbe04d9",
|
|
||||||
"metadata": {
|
|
||||||
"tags": []
|
|
||||||
},
|
|
||||||
"outputs": [],
|
|
||||||
"source": [
|
|
||||||
"docs = [\n",
|
|
||||||
" Document(\n",
|
|
||||||
" page_content=\"A bunch of scientists bring back dinosaurs and mayhem breaks loose\",\n",
|
|
||||||
" metadata={\"year\": 1993, \"rating\": 7.7, \"genre\": \"science fiction\"},\n",
|
|
||||||
" ),\n",
|
|
||||||
" Document(\n",
|
|
||||||
" page_content=\"Leo DiCaprio gets lost in a dream within a dream within a dream within a ...\",\n",
|
|
||||||
" metadata={\"year\": 2010, \"director\": \"Christopher Nolan\", \"rating\": 8.2},\n",
|
|
||||||
" ),\n",
|
|
||||||
" Document(\n",
|
|
||||||
" page_content=\"A psychologist / detective gets lost in a series of dreams within dreams within dreams and Inception reused the idea\",\n",
|
|
||||||
" metadata={\"year\": 2006, \"director\": \"Satoshi Kon\", \"rating\": 8.6},\n",
|
|
||||||
" ),\n",
|
|
||||||
" Document(\n",
|
|
||||||
" page_content=\"A bunch of normal-sized women are supremely wholesome and some men pine after them\",\n",
|
|
||||||
" metadata={\"year\": 2019, \"director\": \"Greta Gerwig\", \"rating\": 8.3},\n",
|
|
||||||
" ),\n",
|
|
||||||
" Document(\n",
|
|
||||||
" page_content=\"Toys come alive and have a blast doing so\",\n",
|
|
||||||
" metadata={\"year\": 1995, \"genre\": \"animated\"},\n",
|
|
||||||
" ),\n",
|
|
||||||
" Document(\n",
|
|
||||||
" page_content=\"Three men walk into the Zone, three men walk out of the Zone\",\n",
|
|
||||||
" metadata={\n",
|
|
||||||
" \"year\": 1979,\n",
|
|
||||||
" \"rating\": 9.9,\n",
|
|
||||||
" \"director\": \"Andrei Tarkovsky\",\n",
|
|
||||||
" \"genre\": \"science fiction\",\n",
|
|
||||||
" },\n",
|
|
||||||
" ),\n",
|
|
||||||
"]\n",
|
|
||||||
"vectorstore = OpenSearchVectorSearch.from_documents(\n",
|
|
||||||
" docs, embeddings, index_name=\"opensearch-self-query-demo\", opensearch_url=\"http://localhost:9200\"\n",
|
|
||||||
")"
|
|
||||||
]
|
|
||||||
},
|
|
||||||
{
|
|
||||||
"cell_type": "markdown",
|
|
||||||
"id": "5ecaab6d",
|
|
||||||
"metadata": {},
|
|
||||||
"source": [
|
|
||||||
"## Creating our self-querying retriever\n",
|
|
||||||
"Now we can instantiate our retriever. To do this we'll need to provide some information upfront about the metadata fields that our documents support and a short description of the document contents."
|
|
||||||
]
|
|
||||||
},
|
|
||||||
{
|
|
||||||
"cell_type": "code",
|
|
||||||
"execution_count": 9,
|
|
||||||
"id": "86e34dbf",
|
|
||||||
"metadata": {
|
|
||||||
"tags": []
|
|
||||||
},
|
|
||||||
"outputs": [],
|
|
||||||
"source": [
|
|
||||||
"from langchain.llms import OpenAI\n",
|
|
||||||
"from langchain.retrievers.self_query.base import SelfQueryRetriever\n",
|
|
||||||
"from langchain.chains.query_constructor.base import AttributeInfo\n",
|
|
||||||
"\n",
|
|
||||||
"metadata_field_info = [\n",
|
|
||||||
" AttributeInfo(\n",
|
|
||||||
" name=\"genre\",\n",
|
|
||||||
" description=\"The genre of the movie\",\n",
|
|
||||||
" type=\"string or list[string]\",\n",
|
|
||||||
" ),\n",
|
|
||||||
" AttributeInfo(\n",
|
|
||||||
" name=\"year\",\n",
|
|
||||||
" description=\"The year the movie was released\",\n",
|
|
||||||
" type=\"integer\",\n",
|
|
||||||
" ),\n",
|
|
||||||
" AttributeInfo(\n",
|
|
||||||
" name=\"director\",\n",
|
|
||||||
" description=\"The name of the movie director\",\n",
|
|
||||||
" type=\"string\",\n",
|
|
||||||
" ),\n",
|
|
||||||
" AttributeInfo(\n",
|
|
||||||
" name=\"rating\", description=\"A 1-10 rating for the movie\", type=\"float\"\n",
|
|
||||||
" ),\n",
|
|
||||||
"]\n",
|
|
||||||
"document_content_description = \"Brief summary of a movie\"\n",
|
|
||||||
"llm = OpenAI(temperature=0)\n",
|
|
||||||
"retriever = SelfQueryRetriever.from_llm(\n",
|
|
||||||
" llm, vectorstore, document_content_description, metadata_field_info, verbose=True\n",
|
|
||||||
")"
|
|
||||||
]
|
|
||||||
},
|
|
||||||
{
|
|
||||||
"cell_type": "markdown",
|
|
||||||
"id": "ea9df8d4",
|
|
||||||
"metadata": {},
|
|
||||||
"source": [
|
|
||||||
"## Testing it out\n",
|
|
||||||
"And now we can try actually using our retriever!"
|
|
||||||
]
|
|
||||||
},
|
|
||||||
{
|
|
||||||
"cell_type": "code",
|
|
||||||
"execution_count": 10,
|
|
||||||
"id": "38a126e9",
|
|
||||||
"metadata": {},
|
|
||||||
"outputs": [
|
|
||||||
{
|
|
||||||
"name": "stdout",
|
|
||||||
"output_type": "stream",
|
|
||||||
"text": [
|
|
||||||
"query='dinosaur' filter=None limit=None\n"
|
|
||||||
]
|
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"data": {
|
"cell_type": "markdown",
|
||||||
"text/plain": [
|
"id": "68e75fb9",
|
||||||
"[Document(page_content='A bunch of scientists bring back dinosaurs and mayhem breaks loose', metadata={'year': 1993, 'rating': 7.7, 'genre': 'science fiction'}),\n",
|
"metadata": {},
|
||||||
" Document(page_content='Toys come alive and have a blast doing so', metadata={'year': 1995, 'genre': 'animated'}),\n",
|
"source": [
|
||||||
" Document(page_content='Leo DiCaprio gets lost in a dream within a dream within a dream within a ...', metadata={'year': 2010, 'director': 'Christopher Nolan', 'rating': 8.2}),\n",
|
"## Creating an OpenSearch vector store\n",
|
||||||
" Document(page_content='Three men walk into the Zone, three men walk out of the Zone', metadata={'year': 1979, 'rating': 9.9, 'director': 'Andrei Tarkovsky', 'genre': 'science fiction'})]"
|
"\n",
|
||||||
|
"First, we'll want to create an `OpenSearch` vector store and seed it with some data. We've created a small demo set of documents that contain summaries of movies.\n",
|
||||||
|
"\n",
|
||||||
|
"**Note:** The self-query retriever requires you to have `lark` installed (`pip install lark`). We also need the `opensearch-py` package."
|
||||||
]
|
]
|
||||||
},
|
|
||||||
"execution_count": 10,
|
|
||||||
"metadata": {},
|
|
||||||
"output_type": "execute_result"
|
|
||||||
}
|
|
||||||
],
|
|
||||||
"source": [
|
|
||||||
"# This example only specifies a relevant query\n",
|
|
||||||
"retriever.get_relevant_documents(\"What are some movies about dinosaurs\")"
|
|
||||||
]
|
|
||||||
},
|
|
||||||
{
|
|
||||||
"cell_type": "code",
|
|
||||||
"execution_count": 11,
|
|
||||||
"id": "60bf0074-e65e-4558-a4f2-8190f3e4e2f9",
|
|
||||||
"metadata": {},
|
|
||||||
"outputs": [
|
|
||||||
{
|
|
||||||
"name": "stdout",
|
|
||||||
"output_type": "stream",
|
|
||||||
"text": [
|
|
||||||
"query=' ' filter=Comparison(comparator=<Comparator.GT: 'gt'>, attribute='rating', value=8.5) limit=None\n"
|
|
||||||
]
|
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"data": {
|
"cell_type": "code",
|
||||||
"text/plain": [
|
"execution_count": null,
|
||||||
"[Document(page_content='Three men walk into the Zone, three men walk out of the Zone', metadata={'year': 1979, 'rating': 9.9, 'director': 'Andrei Tarkovsky', 'genre': 'science fiction'}),\n",
|
"outputs": [],
|
||||||
" Document(page_content='A psychologist / detective gets lost in a series of dreams within dreams within dreams and Inception reused the idea', metadata={'year': 2006, 'director': 'Satoshi Kon', 'rating': 8.6})]"
|
"source": [
|
||||||
]
|
"!pip install lark opensearch-py"
|
||||||
},
|
],
|
||||||
"execution_count": 11,
|
"metadata": {
|
||||||
"metadata": {},
|
"collapsed": false,
|
||||||
"output_type": "execute_result"
|
"pycharm": {
|
||||||
}
|
"name": "#%%\n"
|
||||||
],
|
}
|
||||||
"source": [
|
},
|
||||||
"# This example only specifies a filter\n",
|
"id": "6078a74d"
|
||||||
"retriever.get_relevant_documents(\"I want to watch a movie rated higher than 8.5\")\n"
|
|
||||||
]
|
|
||||||
},
|
|
||||||
{
|
|
||||||
"cell_type": "code",
|
|
||||||
"execution_count": 12,
|
|
||||||
"id": "b19d4da0",
|
|
||||||
"metadata": {},
|
|
||||||
"outputs": [
|
|
||||||
{
|
|
||||||
"name": "stdout",
|
|
||||||
"output_type": "stream",
|
|
||||||
"text": [
|
|
||||||
"query='women' filter=Comparison(comparator=<Comparator.EQ: 'eq'>, attribute='director', value='Greta Gerwig') limit=None\n"
|
|
||||||
]
|
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"data": {
|
"cell_type": "code",
|
||||||
"text/plain": [
|
"execution_count": 3,
|
||||||
"[Document(page_content='A bunch of normal-sized women are supremely wholesome and some men pine after them', metadata={'year': 2019, 'director': 'Greta Gerwig', 'rating': 8.3})]"
|
"id": "cb4a5787",
|
||||||
|
"metadata": {
|
||||||
|
"tags": []
|
||||||
|
},
|
||||||
|
"outputs": [
|
||||||
|
{
|
||||||
|
"name": "stdin",
|
||||||
|
"output_type": "stream",
|
||||||
|
"text": [
|
||||||
|
"OpenAI API Key: \u00b7\u00b7\u00b7\u00b7\u00b7\u00b7\u00b7\u00b7\n"
|
||||||
|
]
|
||||||
|
}
|
||||||
|
],
|
||||||
|
"source": [
|
||||||
|
"from langchain.schema import Document\n",
|
||||||
|
"from langchain.embeddings.openai import OpenAIEmbeddings\n",
|
||||||
|
"from langchain.vectorstores import OpenSearchVectorSearch\n",
|
||||||
|
"import os\n",
|
||||||
|
"import getpass\n",
|
||||||
|
"\n",
|
||||||
|
"os.environ[\"OPENAI_API_KEY\"] = getpass.getpass(\"OpenAI API Key:\")\n",
|
||||||
|
"\n",
|
||||||
|
"embeddings = OpenAIEmbeddings()"
|
||||||
]
|
]
|
||||||
},
|
|
||||||
"execution_count": 12,
|
|
||||||
"metadata": {},
|
|
||||||
"output_type": "execute_result"
|
|
||||||
}
|
|
||||||
],
|
|
||||||
"source": [
|
|
||||||
"# This example specifies a query and a filter\n",
|
|
||||||
"retriever.get_relevant_documents(\"Has Greta Gerwig directed any movies about women\")"
|
|
||||||
]
|
|
||||||
},
|
|
||||||
{
|
|
||||||
"cell_type": "code",
|
|
||||||
"execution_count": 13,
|
|
||||||
"id": "a59f946b-78a1-4d3e-9942-63834c7d7589",
|
|
||||||
"metadata": {},
|
|
||||||
"outputs": [
|
|
||||||
{
|
|
||||||
"name": "stdout",
|
|
||||||
"output_type": "stream",
|
|
||||||
"text": [
|
|
||||||
"query=' ' filter=Operation(operator=<Operator.AND: 'and'>, arguments=[Comparison(comparator=<Comparator.GTE: 'gte'>, attribute='rating', value=8.5), Comparison(comparator=<Comparator.CONTAIN: 'contain'>, attribute='genre', value='science fiction')]) limit=None\n"
|
|
||||||
]
|
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"data": {
|
"cell_type": "code",
|
||||||
"text/plain": [
|
"execution_count": 8,
|
||||||
"[Document(page_content='Three men walk into the Zone, three men walk out of the Zone', metadata={'year': 1979, 'rating': 9.9, 'director': 'Andrei Tarkovsky', 'genre': 'science fiction'})]"
|
"id": "bcbe04d9",
|
||||||
|
"metadata": {
|
||||||
|
"tags": []
|
||||||
|
},
|
||||||
|
"outputs": [],
|
||||||
|
"source": [
|
||||||
|
"docs = [\n",
|
||||||
|
" Document(\n",
|
||||||
|
" page_content=\"A bunch of scientists bring back dinosaurs and mayhem breaks loose\",\n",
|
||||||
|
" metadata={\"year\": 1993, \"rating\": 7.7, \"genre\": \"science fiction\"},\n",
|
||||||
|
" ),\n",
|
||||||
|
" Document(\n",
|
||||||
|
" page_content=\"Leo DiCaprio gets lost in a dream within a dream within a dream within a ...\",\n",
|
||||||
|
" metadata={\"year\": 2010, \"director\": \"Christopher Nolan\", \"rating\": 8.2},\n",
|
||||||
|
" ),\n",
|
||||||
|
" Document(\n",
|
||||||
|
" page_content=\"A psychologist / detective gets lost in a series of dreams within dreams within dreams and Inception reused the idea\",\n",
|
||||||
|
" metadata={\"year\": 2006, \"director\": \"Satoshi Kon\", \"rating\": 8.6},\n",
|
||||||
|
" ),\n",
|
||||||
|
" Document(\n",
|
||||||
|
" page_content=\"A bunch of normal-sized women are supremely wholesome and some men pine after them\",\n",
|
||||||
|
" metadata={\"year\": 2019, \"director\": \"Greta Gerwig\", \"rating\": 8.3},\n",
|
||||||
|
" ),\n",
|
||||||
|
" Document(\n",
|
||||||
|
" page_content=\"Toys come alive and have a blast doing so\",\n",
|
||||||
|
" metadata={\"year\": 1995, \"genre\": \"animated\"},\n",
|
||||||
|
" ),\n",
|
||||||
|
" Document(\n",
|
||||||
|
" page_content=\"Three men walk into the Zone, three men walk out of the Zone\",\n",
|
||||||
|
" metadata={\n",
|
||||||
|
" \"year\": 1979,\n",
|
||||||
|
" \"rating\": 9.9,\n",
|
||||||
|
" \"director\": \"Andrei Tarkovsky\",\n",
|
||||||
|
" \"genre\": \"science fiction\",\n",
|
||||||
|
" },\n",
|
||||||
|
" ),\n",
|
||||||
|
"]\n",
|
||||||
|
"vectorstore = OpenSearchVectorSearch.from_documents(\n",
|
||||||
|
" docs, embeddings, index_name=\"opensearch-self-query-demo\", opensearch_url=\"http://localhost:9200\"\n",
|
||||||
|
")"
|
||||||
]
|
]
|
||||||
},
|
|
||||||
"execution_count": 13,
|
|
||||||
"metadata": {},
|
|
||||||
"output_type": "execute_result"
|
|
||||||
}
|
|
||||||
],
|
|
||||||
"source": [
|
|
||||||
"# This example specifies a composite filter\n",
|
|
||||||
"retriever.get_relevant_documents(\"What's a highly rated (above 8.5) science fiction film?\")"
|
|
||||||
]
|
|
||||||
},
|
|
||||||
{
|
|
||||||
"cell_type": "markdown",
|
|
||||||
"id": "39bd1de1-b9fe-4a98-89da-58d8a7a6ae51",
|
|
||||||
"metadata": {},
|
|
||||||
"source": [
|
|
||||||
"## Filter k\n",
|
|
||||||
"\n",
|
|
||||||
"We can also use the self query retriever to specify `k`: the number of documents to fetch.\n",
|
|
||||||
"\n",
|
|
||||||
"We can do this by passing `enable_limit=True` to the constructor."
|
|
||||||
]
|
|
||||||
},
|
|
||||||
{
|
|
||||||
"cell_type": "code",
|
|
||||||
"execution_count": 14,
|
|
||||||
"id": "bff36b88-b506-4877-9c63-e5a1a8d78e64",
|
|
||||||
"metadata": {
|
|
||||||
"tags": []
|
|
||||||
},
|
|
||||||
"outputs": [],
|
|
||||||
"source": [
|
|
||||||
"retriever = SelfQueryRetriever.from_llm(\n",
|
|
||||||
" llm,\n",
|
|
||||||
" vectorstore,\n",
|
|
||||||
" document_content_description,\n",
|
|
||||||
" metadata_field_info,\n",
|
|
||||||
" enable_limit=True,\n",
|
|
||||||
" verbose=True,\n",
|
|
||||||
")"
|
|
||||||
]
|
|
||||||
},
|
|
||||||
{
|
|
||||||
"cell_type": "code",
|
|
||||||
"execution_count": 15,
|
|
||||||
"id": "2758d229-4f97-499c-819f-888acaf8ee10",
|
|
||||||
"metadata": {
|
|
||||||
"tags": []
|
|
||||||
},
|
|
||||||
"outputs": [
|
|
||||||
{
|
|
||||||
"name": "stdout",
|
|
||||||
"output_type": "stream",
|
|
||||||
"text": [
|
|
||||||
"query='dinosaur' filter=None limit=2\n"
|
|
||||||
]
|
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"data": {
|
"cell_type": "markdown",
|
||||||
"text/plain": [
|
"id": "5ecaab6d",
|
||||||
"[Document(page_content='A bunch of scientists bring back dinosaurs and mayhem breaks loose', metadata={'year': 1993, 'rating': 7.7, 'genre': 'science fiction'}),\n",
|
"metadata": {},
|
||||||
" Document(page_content='Toys come alive and have a blast doing so', metadata={'year': 1995, 'genre': 'animated'})]"
|
"source": [
|
||||||
|
"## Creating our self-querying retriever\n",
|
||||||
|
"Now we can instantiate our retriever. To do this we'll need to provide some information upfront about the metadata fields that our documents support and a short description of the document contents."
|
||||||
]
|
]
|
||||||
},
|
|
||||||
"execution_count": 15,
|
|
||||||
"metadata": {},
|
|
||||||
"output_type": "execute_result"
|
|
||||||
}
|
|
||||||
],
|
|
||||||
"source": [
|
|
||||||
"# This example only specifies a relevant query\n",
|
|
||||||
"retriever.get_relevant_documents(\"what are two movies about dinosaurs\")"
|
|
||||||
]
|
|
||||||
},
|
|
||||||
{
|
|
||||||
"cell_type": "markdown",
|
|
||||||
"id": "61a10294",
|
|
||||||
"metadata": {},
|
|
||||||
"source": [
|
|
||||||
"## Complex queries in Action!\n",
|
|
||||||
"We've tried out some simple queries, but what about more complex ones? Let's try out a few more complex queries that utilize the full power of OpenSearch."
|
|
||||||
]
|
|
||||||
},
|
|
||||||
{
|
|
||||||
"cell_type": "code",
|
|
||||||
"execution_count": 16,
|
|
||||||
"id": "e460da93",
|
|
||||||
"metadata": {},
|
|
||||||
"outputs": [
|
|
||||||
{
|
|
||||||
"name": "stdout",
|
|
||||||
"output_type": "stream",
|
|
||||||
"text": [
|
|
||||||
"query='animated toys' filter=Operation(operator=<Operator.AND: 'and'>, arguments=[Operation(operator=<Operator.OR: 'or'>, arguments=[Comparison(comparator=<Comparator.EQ: 'eq'>, attribute='genre', value='animated'), Comparison(comparator=<Comparator.EQ: 'eq'>, attribute='genre', value='comedy')]), Comparison(comparator=<Comparator.GTE: 'gte'>, attribute='year', value=1990)]) limit=None\n"
|
|
||||||
]
|
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"data": {
|
"cell_type": "code",
|
||||||
"text/plain": [
|
"execution_count": 9,
|
||||||
"[Document(page_content='Toys come alive and have a blast doing so', metadata={'year': 1995, 'genre': 'animated'})]"
|
"id": "86e34dbf",
|
||||||
|
"metadata": {
|
||||||
|
"tags": []
|
||||||
|
},
|
||||||
|
"outputs": [],
|
||||||
|
"source": [
|
||||||
|
"from langchain.llms import OpenAI\n",
|
||||||
|
"from langchain.retrievers.self_query.base import SelfQueryRetriever\n",
|
||||||
|
"from langchain.chains.query_constructor.base import AttributeInfo\n",
|
||||||
|
"\n",
|
||||||
|
"metadata_field_info = [\n",
|
||||||
|
" AttributeInfo(\n",
|
||||||
|
" name=\"genre\",\n",
|
||||||
|
" description=\"The genre of the movie\",\n",
|
||||||
|
" type=\"string or list[string]\",\n",
|
||||||
|
" ),\n",
|
||||||
|
" AttributeInfo(\n",
|
||||||
|
" name=\"year\",\n",
|
||||||
|
" description=\"The year the movie was released\",\n",
|
||||||
|
" type=\"integer\",\n",
|
||||||
|
" ),\n",
|
||||||
|
" AttributeInfo(\n",
|
||||||
|
" name=\"director\",\n",
|
||||||
|
" description=\"The name of the movie director\",\n",
|
||||||
|
" type=\"string\",\n",
|
||||||
|
" ),\n",
|
||||||
|
" AttributeInfo(\n",
|
||||||
|
" name=\"rating\", description=\"A 1-10 rating for the movie\", type=\"float\"\n",
|
||||||
|
" ),\n",
|
||||||
|
"]\n",
|
||||||
|
"document_content_description = \"Brief summary of a movie\"\n",
|
||||||
|
"llm = OpenAI(temperature=0)\n",
|
||||||
|
"retriever = SelfQueryRetriever.from_llm(\n",
|
||||||
|
" llm, vectorstore, document_content_description, metadata_field_info, verbose=True\n",
|
||||||
|
")"
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
"execution_count": 16,
|
|
||||||
"metadata": {},
|
|
||||||
"output_type": "execute_result"
|
|
||||||
}
|
|
||||||
],
|
|
||||||
"source": [
|
|
||||||
"retriever.get_relevant_documents(\"what animated or comedy movies have been released in the last 30 years about animated toys?\")"
|
|
||||||
]
|
|
||||||
},
|
|
||||||
{
|
|
||||||
"cell_type": "code",
|
|
||||||
"execution_count": 17,
|
|
||||||
"id": "0851fc42",
|
|
||||||
"metadata": {
|
|
||||||
"pycharm": {
|
|
||||||
"name": "#%%\n"
|
|
||||||
}
|
|
||||||
},
|
|
||||||
"outputs": [
|
|
||||||
{
|
{
|
||||||
"data": {
|
"cell_type": "markdown",
|
||||||
"text/plain": [
|
"id": "ea9df8d4",
|
||||||
"{'acknowledged': True}"
|
"metadata": {},
|
||||||
|
"source": [
|
||||||
|
"## Testing it out\n",
|
||||||
|
"And now we can try actually using our retriever!"
|
||||||
|
]
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"cell_type": "code",
|
||||||
|
"execution_count": 10,
|
||||||
|
"id": "38a126e9",
|
||||||
|
"metadata": {},
|
||||||
|
"outputs": [
|
||||||
|
{
|
||||||
|
"name": "stdout",
|
||||||
|
"output_type": "stream",
|
||||||
|
"text": [
|
||||||
|
"query='dinosaur' filter=None limit=None\n"
|
||||||
|
]
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"data": {
|
||||||
|
"text/plain": [
|
||||||
|
"[Document(page_content='A bunch of scientists bring back dinosaurs and mayhem breaks loose', metadata={'year': 1993, 'rating': 7.7, 'genre': 'science fiction'}),\n",
|
||||||
|
" Document(page_content='Toys come alive and have a blast doing so', metadata={'year': 1995, 'genre': 'animated'}),\n",
|
||||||
|
" Document(page_content='Leo DiCaprio gets lost in a dream within a dream within a dream within a ...', metadata={'year': 2010, 'director': 'Christopher Nolan', 'rating': 8.2}),\n",
|
||||||
|
" Document(page_content='Three men walk into the Zone, three men walk out of the Zone', metadata={'year': 1979, 'rating': 9.9, 'director': 'Andrei Tarkovsky', 'genre': 'science fiction'})]"
|
||||||
|
]
|
||||||
|
},
|
||||||
|
"execution_count": 10,
|
||||||
|
"metadata": {},
|
||||||
|
"output_type": "execute_result"
|
||||||
|
}
|
||||||
|
],
|
||||||
|
"source": [
|
||||||
|
"# This example only specifies a relevant query\n",
|
||||||
|
"retriever.get_relevant_documents(\"What are some movies about dinosaurs\")"
|
||||||
|
]
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"cell_type": "code",
|
||||||
|
"execution_count": 11,
|
||||||
|
"id": "60bf0074-e65e-4558-a4f2-8190f3e4e2f9",
|
||||||
|
"metadata": {},
|
||||||
|
"outputs": [
|
||||||
|
{
|
||||||
|
"name": "stdout",
|
||||||
|
"output_type": "stream",
|
||||||
|
"text": [
|
||||||
|
"query=' ' filter=Comparison(comparator=<Comparator.GT: 'gt'>, attribute='rating', value=8.5) limit=None\n"
|
||||||
|
]
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"data": {
|
||||||
|
"text/plain": [
|
||||||
|
"[Document(page_content='Three men walk into the Zone, three men walk out of the Zone', metadata={'year': 1979, 'rating': 9.9, 'director': 'Andrei Tarkovsky', 'genre': 'science fiction'}),\n",
|
||||||
|
" Document(page_content='A psychologist / detective gets lost in a series of dreams within dreams within dreams and Inception reused the idea', metadata={'year': 2006, 'director': 'Satoshi Kon', 'rating': 8.6})]"
|
||||||
|
]
|
||||||
|
},
|
||||||
|
"execution_count": 11,
|
||||||
|
"metadata": {},
|
||||||
|
"output_type": "execute_result"
|
||||||
|
}
|
||||||
|
],
|
||||||
|
"source": [
|
||||||
|
"# This example only specifies a filter\n",
|
||||||
|
"retriever.get_relevant_documents(\"I want to watch a movie rated higher than 8.5\")\n"
|
||||||
|
]
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"cell_type": "code",
|
||||||
|
"execution_count": 12,
|
||||||
|
"id": "b19d4da0",
|
||||||
|
"metadata": {},
|
||||||
|
"outputs": [
|
||||||
|
{
|
||||||
|
"name": "stdout",
|
||||||
|
"output_type": "stream",
|
||||||
|
"text": [
|
||||||
|
"query='women' filter=Comparison(comparator=<Comparator.EQ: 'eq'>, attribute='director', value='Greta Gerwig') limit=None\n"
|
||||||
|
]
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"data": {
|
||||||
|
"text/plain": [
|
||||||
|
"[Document(page_content='A bunch of normal-sized women are supremely wholesome and some men pine after them', metadata={'year': 2019, 'director': 'Greta Gerwig', 'rating': 8.3})]"
|
||||||
|
]
|
||||||
|
},
|
||||||
|
"execution_count": 12,
|
||||||
|
"metadata": {},
|
||||||
|
"output_type": "execute_result"
|
||||||
|
}
|
||||||
|
],
|
||||||
|
"source": [
|
||||||
|
"# This example specifies a query and a filter\n",
|
||||||
|
"retriever.get_relevant_documents(\"Has Greta Gerwig directed any movies about women\")"
|
||||||
|
]
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"cell_type": "code",
|
||||||
|
"execution_count": 13,
|
||||||
|
"id": "a59f946b-78a1-4d3e-9942-63834c7d7589",
|
||||||
|
"metadata": {},
|
||||||
|
"outputs": [
|
||||||
|
{
|
||||||
|
"name": "stdout",
|
||||||
|
"output_type": "stream",
|
||||||
|
"text": [
|
||||||
|
"query=' ' filter=Operation(operator=<Operator.AND: 'and'>, arguments=[Comparison(comparator=<Comparator.GTE: 'gte'>, attribute='rating', value=8.5), Comparison(comparator=<Comparator.CONTAIN: 'contain'>, attribute='genre', value='science fiction')]) limit=None\n"
|
||||||
|
]
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"data": {
|
||||||
|
"text/plain": [
|
||||||
|
"[Document(page_content='Three men walk into the Zone, three men walk out of the Zone', metadata={'year': 1979, 'rating': 9.9, 'director': 'Andrei Tarkovsky', 'genre': 'science fiction'})]"
|
||||||
|
]
|
||||||
|
},
|
||||||
|
"execution_count": 13,
|
||||||
|
"metadata": {},
|
||||||
|
"output_type": "execute_result"
|
||||||
|
}
|
||||||
|
],
|
||||||
|
"source": [
|
||||||
|
"# This example specifies a composite filter\n",
|
||||||
|
"retriever.get_relevant_documents(\"What's a highly rated (above 8.5) science fiction film?\")"
|
||||||
|
]
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"cell_type": "markdown",
|
||||||
|
"id": "39bd1de1-b9fe-4a98-89da-58d8a7a6ae51",
|
||||||
|
"metadata": {},
|
||||||
|
"source": [
|
||||||
|
"## Filter k\n",
|
||||||
|
"\n",
|
||||||
|
"We can also use the self query retriever to specify `k`: the number of documents to fetch.\n",
|
||||||
|
"\n",
|
||||||
|
"We can do this by passing `enable_limit=True` to the constructor."
|
||||||
|
]
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"cell_type": "code",
|
||||||
|
"execution_count": 14,
|
||||||
|
"id": "bff36b88-b506-4877-9c63-e5a1a8d78e64",
|
||||||
|
"metadata": {
|
||||||
|
"tags": []
|
||||||
|
},
|
||||||
|
"outputs": [],
|
||||||
|
"source": [
|
||||||
|
"retriever = SelfQueryRetriever.from_llm(\n",
|
||||||
|
" llm,\n",
|
||||||
|
" vectorstore,\n",
|
||||||
|
" document_content_description,\n",
|
||||||
|
" metadata_field_info,\n",
|
||||||
|
" enable_limit=True,\n",
|
||||||
|
" verbose=True,\n",
|
||||||
|
")"
|
||||||
|
]
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"cell_type": "code",
|
||||||
|
"execution_count": 15,
|
||||||
|
"id": "2758d229-4f97-499c-819f-888acaf8ee10",
|
||||||
|
"metadata": {
|
||||||
|
"tags": []
|
||||||
|
},
|
||||||
|
"outputs": [
|
||||||
|
{
|
||||||
|
"name": "stdout",
|
||||||
|
"output_type": "stream",
|
||||||
|
"text": [
|
||||||
|
"query='dinosaur' filter=None limit=2\n"
|
||||||
|
]
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"data": {
|
||||||
|
"text/plain": [
|
||||||
|
"[Document(page_content='A bunch of scientists bring back dinosaurs and mayhem breaks loose', metadata={'year': 1993, 'rating': 7.7, 'genre': 'science fiction'}),\n",
|
||||||
|
" Document(page_content='Toys come alive and have a blast doing so', metadata={'year': 1995, 'genre': 'animated'})]"
|
||||||
|
]
|
||||||
|
},
|
||||||
|
"execution_count": 15,
|
||||||
|
"metadata": {},
|
||||||
|
"output_type": "execute_result"
|
||||||
|
}
|
||||||
|
],
|
||||||
|
"source": [
|
||||||
|
"# This example only specifies a relevant query\n",
|
||||||
|
"retriever.get_relevant_documents(\"what are two movies about dinosaurs\")"
|
||||||
|
]
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"cell_type": "markdown",
|
||||||
|
"id": "61a10294",
|
||||||
|
"metadata": {},
|
||||||
|
"source": [
|
||||||
|
"## Complex queries in Action!\n",
|
||||||
|
"We've tried out some simple queries, but what about more complex ones? Let's try out a few more complex queries that utilize the full power of OpenSearch."
|
||||||
|
]
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"cell_type": "code",
|
||||||
|
"execution_count": 16,
|
||||||
|
"id": "e460da93",
|
||||||
|
"metadata": {},
|
||||||
|
"outputs": [
|
||||||
|
{
|
||||||
|
"name": "stdout",
|
||||||
|
"output_type": "stream",
|
||||||
|
"text": [
|
||||||
|
"query='animated toys' filter=Operation(operator=<Operator.AND: 'and'>, arguments=[Operation(operator=<Operator.OR: 'or'>, arguments=[Comparison(comparator=<Comparator.EQ: 'eq'>, attribute='genre', value='animated'), Comparison(comparator=<Comparator.EQ: 'eq'>, attribute='genre', value='comedy')]), Comparison(comparator=<Comparator.GTE: 'gte'>, attribute='year', value=1990)]) limit=None\n"
|
||||||
|
]
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"data": {
|
||||||
|
"text/plain": [
|
||||||
|
"[Document(page_content='Toys come alive and have a blast doing so', metadata={'year': 1995, 'genre': 'animated'})]"
|
||||||
|
]
|
||||||
|
},
|
||||||
|
"execution_count": 16,
|
||||||
|
"metadata": {},
|
||||||
|
"output_type": "execute_result"
|
||||||
|
}
|
||||||
|
],
|
||||||
|
"source": [
|
||||||
|
"retriever.get_relevant_documents(\"what animated or comedy movies have been released in the last 30 years about animated toys?\")"
|
||||||
|
]
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"cell_type": "code",
|
||||||
|
"execution_count": 17,
|
||||||
|
"id": "0851fc42",
|
||||||
|
"metadata": {
|
||||||
|
"pycharm": {
|
||||||
|
"name": "#%%\n"
|
||||||
|
}
|
||||||
|
},
|
||||||
|
"outputs": [
|
||||||
|
{
|
||||||
|
"data": {
|
||||||
|
"text/plain": [
|
||||||
|
"{'acknowledged': True}"
|
||||||
|
]
|
||||||
|
},
|
||||||
|
"execution_count": 17,
|
||||||
|
"metadata": {},
|
||||||
|
"output_type": "execute_result"
|
||||||
|
}
|
||||||
|
],
|
||||||
|
"source": [
|
||||||
|
"vectorstore.client.indices.delete(index=\"opensearch-self-query-demo\")\n"
|
||||||
]
|
]
|
||||||
},
|
|
||||||
"execution_count": 17,
|
|
||||||
"metadata": {},
|
|
||||||
"output_type": "execute_result"
|
|
||||||
}
|
}
|
||||||
],
|
],
|
||||||
"source": [
|
"metadata": {
|
||||||
"vectorstore.client.indices.delete(index=\"opensearch-self-query-demo\")\n"
|
"kernelspec": {
|
||||||
]
|
"display_name": "Python 3 (ipykernel)",
|
||||||
}
|
"language": "python",
|
||||||
],
|
"name": "python3"
|
||||||
"metadata": {
|
},
|
||||||
"kernelspec": {
|
"language_info": {
|
||||||
"display_name": "Python 3 (ipykernel)",
|
"codemirror_mode": {
|
||||||
"language": "python",
|
"name": "ipython",
|
||||||
"name": "python3"
|
"version": 3
|
||||||
|
},
|
||||||
|
"file_extension": ".py",
|
||||||
|
"mimetype": "text/x-python",
|
||||||
|
"name": "python",
|
||||||
|
"nbconvert_exporter": "python",
|
||||||
|
"pygments_lexer": "ipython3",
|
||||||
|
"version": "3.9.18"
|
||||||
|
}
|
||||||
},
|
},
|
||||||
"language_info": {
|
"nbformat": 4,
|
||||||
"codemirror_mode": {
|
"nbformat_minor": 5
|
||||||
"name": "ipython",
|
|
||||||
"version": 3
|
|
||||||
},
|
|
||||||
"file_extension": ".py",
|
|
||||||
"mimetype": "text/x-python",
|
|
||||||
"name": "python",
|
|
||||||
"nbconvert_exporter": "python",
|
|
||||||
"pygments_lexer": "ipython3",
|
|
||||||
"version": "3.9.18"
|
|
||||||
}
|
|
||||||
},
|
|
||||||
"nbformat": 4,
|
|
||||||
"nbformat_minor": 5
|
|
||||||
}
|
}
|
@ -31,7 +31,8 @@ from langchain.text_splitter import (
|
|||||||
'markdown',
|
'markdown',
|
||||||
'latex',
|
'latex',
|
||||||
'html',
|
'html',
|
||||||
'sol',]
|
'sol',
|
||||||
|
'csharp']
|
||||||
```
|
```
|
||||||
|
|
||||||
</CodeOutputBlock>
|
</CodeOutputBlock>
|
||||||
@ -342,3 +343,72 @@ sol_docs
|
|||||||
```
|
```
|
||||||
|
|
||||||
</CodeOutputBlock>
|
</CodeOutputBlock>
|
||||||
|
|
||||||
|
|
||||||
|
## C#
|
||||||
|
Here's an example using the C# text splitter:
|
||||||
|
|
||||||
|
```csharp
|
||||||
|
using System;
|
||||||
|
class Program
|
||||||
|
{
|
||||||
|
static void Main()
|
||||||
|
{
|
||||||
|
int age = 30; // Change the age value as needed
|
||||||
|
|
||||||
|
// Categorize the age without any console output
|
||||||
|
if (age < 18)
|
||||||
|
{
|
||||||
|
// Age is under 18
|
||||||
|
}
|
||||||
|
else if (age >= 18 && age < 65)
|
||||||
|
{
|
||||||
|
// Age is an adult
|
||||||
|
}
|
||||||
|
else
|
||||||
|
{
|
||||||
|
// Age is a senior citizen
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
<CodeOutputBlock lang="python">
|
||||||
|
|
||||||
|
```
|
||||||
|
[Document(page_content='using System;', metadata={}),
|
||||||
|
Document(page_content='class Program\n{', metadata={}),
|
||||||
|
Document(page_content='static void', metadata={}),
|
||||||
|
Document(page_content='Main()', metadata={}),
|
||||||
|
Document(page_content='{', metadata={}),
|
||||||
|
Document(page_content='int age', metadata={}),
|
||||||
|
Document(page_content='= 30; // Change', metadata={}),
|
||||||
|
Document(page_content='the age value', metadata={}),
|
||||||
|
Document(page_content='as needed', metadata={}),
|
||||||
|
Document(page_content='//', metadata={}),
|
||||||
|
Document(page_content='Categorize the', metadata={}),
|
||||||
|
Document(page_content='age without any', metadata={}),
|
||||||
|
Document(page_content='console output', metadata={}),
|
||||||
|
Document(page_content='if (age', metadata={}),
|
||||||
|
Document(page_content='< 18)', metadata={}),
|
||||||
|
Document(page_content='{', metadata={}),
|
||||||
|
Document(page_content='//', metadata={}),
|
||||||
|
Document(page_content='Age is under 18', metadata={}),
|
||||||
|
Document(page_content='}', metadata={}),
|
||||||
|
Document(page_content='else if', metadata={}),
|
||||||
|
Document(page_content='(age >= 18 &&', metadata={}),
|
||||||
|
Document(page_content='age < 65)', metadata={}),
|
||||||
|
Document(page_content='{', metadata={}),
|
||||||
|
Document(page_content='//', metadata={}),
|
||||||
|
Document(page_content='Age is an adult', metadata={}),
|
||||||
|
Document(page_content='}', metadata={}),
|
||||||
|
Document(page_content='else', metadata={}),
|
||||||
|
Document(page_content='{', metadata={}),
|
||||||
|
Document(page_content='//', metadata={}),
|
||||||
|
Document(page_content='Age is a senior', metadata={}),
|
||||||
|
Document(page_content='citizen', metadata={}),
|
||||||
|
Document(page_content='}\n }', metadata={}),
|
||||||
|
Document(page_content='}', metadata={})]
|
||||||
|
```
|
||||||
|
|
||||||
|
</CodeOutputBlock>
|
||||||
|
Loading…
Reference in New Issue
Block a user