mirror of
https://github.com/hwchase17/langchain.git
synced 2025-09-29 15:28:54 +00:00
58d2bfe310
This is a simple change for a variable name in the Embeddings section of this document: https://python.langchain.com/docs/tutorials/retrievers/#embeddings . The variable name `embeddings_model` seems to be wrong and doesn't match what follows in the template: `embeddings`.
691 lines
31 KiB
Plaintext
691 lines
31 KiB
Plaintext
{
|
|
"cells": [
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "bf37a837-7a6a-447b-8779-38f26c585887",
|
|
"metadata": {},
|
|
"source": [
|
|
"# Build a semantic search engine\n",
|
|
"\n",
|
|
"This tutorial will familiarize you with LangChain's [document loader](/docs/concepts/document_loaders), [embedding](/docs/concepts/embedding_models), and [vector store](/docs/concepts/vectorstores) abstractions. These abstractions are designed to support retrieval of data-- from (vector) databases and other sources-- for integration with LLM workflows. They are important for applications that fetch data to be reasoned over as part of model inference, as in the case of retrieval-augmented generation, or [RAG](/docs/concepts/rag) (see our RAG tutorial [here](/docs/tutorials/rag)).\n",
|
|
"\n",
|
|
"Here we will build a search engine over a PDF document. This will allow us to retrieve passages in the PDF that are similar to an input query.\n",
|
|
"\n",
|
|
"## Concepts\n",
|
|
"\n",
|
|
"This guide focuses on retrieval of text data. We will cover the following concepts:\n",
|
|
"\n",
|
|
"- Documents and document loaders;\n",
|
|
"- Text splitters;\n",
|
|
"- Embeddings;\n",
|
|
"- Vector stores and retrievers.\n",
|
|
"\n",
|
|
"## Setup\n",
|
|
"\n",
|
|
"### Jupyter Notebook\n",
|
|
"\n",
|
|
"This and other tutorials are perhaps most conveniently run in a Jupyter notebook. See [here](https://jupyter.org/install) for instructions on how to install.\n",
|
|
"\n",
|
|
"### Installation\n",
|
|
"\n",
|
|
"This tutorial requires the `langchain-community` and `pypdf` packages:\n",
|
|
"\n",
|
|
"import Tabs from '@theme/Tabs';\n",
|
|
"import TabItem from '@theme/TabItem';\n",
|
|
"import CodeBlock from \"@theme/CodeBlock\";\n",
|
|
"\n",
|
|
"<Tabs>\n",
|
|
" <TabItem value=\"pip\" label=\"Pip\" default>\n",
|
|
" <CodeBlock language=\"bash\">pip install langchain-community pypdf</CodeBlock>\n",
|
|
" </TabItem>\n",
|
|
" <TabItem value=\"conda\" label=\"Conda\">\n",
|
|
" <CodeBlock language=\"bash\">conda install langchain-community pypdf -c conda-forge</CodeBlock>\n",
|
|
" </TabItem>\n",
|
|
"</Tabs>\n",
|
|
"\n",
|
|
"\n",
|
|
"For more details, see our [Installation guide](/docs/how_to/installation).\n",
|
|
"\n",
|
|
"### LangSmith\n",
|
|
"\n",
|
|
"Many of the applications you build with LangChain will contain multiple steps with multiple invocations of LLM calls.\n",
|
|
"As these applications get more and more complex, it becomes crucial to be able to inspect what exactly is going on inside your chain or agent.\n",
|
|
"The best way to do this is with [LangSmith](https://smith.langchain.com).\n",
|
|
"\n",
|
|
"After you sign up at the link above, make sure to set your environment variables to start logging traces:\n",
|
|
"\n",
|
|
"```shell\n",
|
|
"export LANGCHAIN_TRACING_V2=\"true\"\n",
|
|
"export LANGCHAIN_API_KEY=\"...\"\n",
|
|
"```\n",
|
|
"\n",
|
|
"Or, if in a notebook, you can set them with:\n",
|
|
"\n",
|
|
"```python\n",
|
|
"import getpass\n",
|
|
"import os\n",
|
|
"\n",
|
|
"os.environ[\"LANGCHAIN_TRACING_V2\"] = \"true\"\n",
|
|
"os.environ[\"LANGCHAIN_API_KEY\"] = getpass.getpass()\n",
|
|
"```\n",
|
|
"\n",
|
|
"\n",
|
|
"## Documents and Document Loaders\n",
|
|
"\n",
|
|
"LangChain implements a [Document](https://python.langchain.com/api_reference/core/documents/langchain_core.documents.base.Document.html) abstraction, which is intended to represent a unit of text and associated metadata. It has three attributes:\n",
|
|
"\n",
|
|
"- `page_content`: a string representing the content;\n",
|
|
"- `metadata`: a dict containing arbitrary metadata;\n",
|
|
"- `id`: (optional) a string identifier for the document.\n",
|
|
"\n",
|
|
"The `metadata` attribute can capture information about the source of the document, its relationship to other documents, and other information. Note that an individual `Document` object often represents a chunk of a larger document.\n",
|
|
"\n",
|
|
"We can generate sample documents when desired:\n",
|
|
"```python\n",
|
|
"from langchain_core.documents import Document\n",
|
|
"\n",
|
|
"documents = [\n",
|
|
" Document(\n",
|
|
" page_content=\"Dogs are great companions, known for their loyalty and friendliness.\",\n",
|
|
" metadata={\"source\": \"mammal-pets-doc\"},\n",
|
|
" ),\n",
|
|
" Document(\n",
|
|
" page_content=\"Cats are independent pets that often enjoy their own space.\",\n",
|
|
" metadata={\"source\": \"mammal-pets-doc\"},\n",
|
|
" ),\n",
|
|
"]\n",
|
|
"```"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "f8593578-5699-4b19-96c4-7c990d37a2ec",
|
|
"metadata": {},
|
|
"source": [
|
|
"However, the LangChain ecosystem implements [document loaders](/docs/concepts/document_loaders) that [integrate with hundreds of common sources](/docs/integrations/document_loaders/). This makes it easy to incorporate data from these sources into your AI application.\n",
|
|
"\n",
|
|
"### Loading documents\n",
|
|
"\n",
|
|
"Let's load a PDF into a sequence of `Document` objects. There is a sample PDF in the LangChain repo [here](https://github.com/langchain-ai/langchain/tree/master/docs/docs/example_data) -- a 10-k filing for Nike from 2023. We can consult the LangChain documentation for [available PDF document loaders](/docs/integrations/document_loaders/#pdfs). Let's select [PyPDFLoader](/docs/integrations/document_loaders/pypdfloader/), which is fairly lightweight."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 1,
|
|
"id": "a2ac32c4-1036-42d8-8a3d-f7f57e3a0df7",
|
|
"metadata": {},
|
|
"outputs": [
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"107\n"
|
|
]
|
|
}
|
|
],
|
|
"source": [
|
|
"from langchain_community.document_loaders import PyPDFLoader\n",
|
|
"\n",
|
|
"file_path = \"../example_data/nke-10k-2023.pdf\"\n",
|
|
"loader = PyPDFLoader(file_path)\n",
|
|
"\n",
|
|
"docs = loader.load()\n",
|
|
"\n",
|
|
"print(len(docs))"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "b90f4800-bb82-416b-beba-f42ae88a5c66",
|
|
"metadata": {},
|
|
"source": [
|
|
":::tip\n",
|
|
"\n",
|
|
"See [this guide](/docs/how_to/document_loader_pdf/) for more detail on PDF document loaders.\n",
|
|
"\n",
|
|
":::\n",
|
|
"\n",
|
|
"`PyPDFLoader` loads one `Document` object per PDF page. For each, we can easily access:\n",
|
|
"\n",
|
|
"- The string content of the page;\n",
|
|
"- Metadata containing the file name and page number."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 2,
|
|
"id": "850e2ca5-6b20-4e58-ad99-b19786358a3e",
|
|
"metadata": {},
|
|
"outputs": [
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"Table of Contents\n",
|
|
"UNITED STATES\n",
|
|
"SECURITIES AND EXCHANGE COMMISSION\n",
|
|
"Washington, D.C. 20549\n",
|
|
"FORM 10-K\n",
|
|
"(Mark One)\n",
|
|
"☑ ANNUAL REPORT PURSUANT TO SECTION 13 OR 15(D) OF THE SECURITIES EXCHANGE ACT OF 1934\n",
|
|
"FO\n",
|
|
"\n",
|
|
"{'source': '../example_data/nke-10k-2023.pdf', 'page': 0}\n"
|
|
]
|
|
}
|
|
],
|
|
"source": [
|
|
"print(f\"{docs[0].page_content[:200]}\\n\")\n",
|
|
"print(docs[0].metadata)"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "2ca6980f-4870-490a-9fe6-8caeead3c1d1",
|
|
"metadata": {},
|
|
"source": [
|
|
"### Splitting\n",
|
|
"\n",
|
|
"For both information retrieval and downstream question-answering purposes, a page may be too coarse a representation. Our goal in the end will be to retrieve `Document` objects that answer an input query, and further splitting our PDF will help ensure that the meanings of relevant portions of the document are not \"washed out\" by surrounding text.\n",
|
|
"\n",
|
|
"We can use [text splitters](/docs/concepts/text_splitters) for this purpose. Here we will use a simple text splitter that partitions based on characters. We will split our documents into chunks of 1000 characters\n",
|
|
"with 200 characters of overlap between chunks. The overlap helps\n",
|
|
"mitigate the possibility of separating a statement from important\n",
|
|
"context related to it. We use the\n",
|
|
"[RecursiveCharacterTextSplitter](/docs/how_to/recursive_text_splitter),\n",
|
|
"which will recursively split the document using common separators like\n",
|
|
"new lines until each chunk is the appropriate size. This is the\n",
|
|
"recommended text splitter for generic text use cases.\n",
|
|
"\n",
|
|
"We set `add_start_index=True` so that the character index where each\n",
|
|
"split Document starts within the initial Document is preserved as\n",
|
|
"metadata attribute “start_index”.\n",
|
|
"\n",
|
|
"See [this guide](/docs/how_to/document_loader_pdf/) for more detail about working with PDFs, including how to extract text from specific sections and images. "
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 3,
|
|
"id": "11c16e79-c8af-4949-9363-9a93a911a0e1",
|
|
"metadata": {},
|
|
"outputs": [
|
|
{
|
|
"data": {
|
|
"text/plain": [
|
|
"514"
|
|
]
|
|
},
|
|
"execution_count": 3,
|
|
"metadata": {},
|
|
"output_type": "execute_result"
|
|
}
|
|
],
|
|
"source": [
|
|
"from langchain_text_splitters import RecursiveCharacterTextSplitter\n",
|
|
"\n",
|
|
"text_splitter = RecursiveCharacterTextSplitter(\n",
|
|
" chunk_size=1000, chunk_overlap=200, add_start_index=True\n",
|
|
")\n",
|
|
"all_splits = text_splitter.split_documents(docs)\n",
|
|
"\n",
|
|
"len(all_splits)"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "5c066d46-9187-4d5a-98d3-974c37610276",
|
|
"metadata": {},
|
|
"source": [
|
|
"## Embeddings\n",
|
|
"\n",
|
|
"Vector search is a common way to store and search over unstructured data (such as unstructured text). The idea is to store numeric vectors that are associated with the text. Given a query, we can [embed](/docs/concepts/embedding_models) it as a vector of the same dimension and use vector similarity metrics (such as cosine similarity) to identify related text.\n",
|
|
"\n",
|
|
"LangChain supports embeddings from [dozens of providers](/docs/integrations/text_embedding/). These models specify how text should be converted into a numeric vector. Let's select a model:\n",
|
|
"\n",
|
|
"import EmbeddingTabs from \"@theme/EmbeddingTabs\";\n",
|
|
"\n",
|
|
"<EmbeddingTabs customVarName=\"embeddings\" />"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 3,
|
|
"id": "4d238a38-b9b3-494a-9cea-8694a1b03bc7",
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"# | output: false\n",
|
|
"# | echo: false\n",
|
|
"\n",
|
|
"from langchain_openai import OpenAIEmbeddings\n",
|
|
"\n",
|
|
"embeddings = OpenAIEmbeddings()"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 5,
|
|
"id": "c5f3b0ac-4e18-4c6b-84e7-e8822c59ce17",
|
|
"metadata": {},
|
|
"outputs": [
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"Generated vectors of length 1536\n",
|
|
"\n",
|
|
"[-0.008586574345827103, -0.03341241180896759, -0.008936782367527485, -0.0036674530711025, 0.010564599186182022, 0.009598285891115665, -0.028587326407432556, -0.015824200585484505, 0.0030416189692914486, -0.012899317778646946]\n"
|
|
]
|
|
}
|
|
],
|
|
"source": [
|
|
"vector_1 = embeddings.embed_query(all_splits[0].page_content)\n",
|
|
"vector_2 = embeddings.embed_query(all_splits[1].page_content)\n",
|
|
"\n",
|
|
"assert len(vector_1) == len(vector_2)\n",
|
|
"print(f\"Generated vectors of length {len(vector_1)}\\n\")\n",
|
|
"print(vector_1[:10])"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "1cac19bd-27d1-40f1-9c27-7a586b685b4e",
|
|
"metadata": {},
|
|
"source": [
|
|
"Armed with a model for generating text embeddings, we can next store them in a special data structure that supports efficient similarity search.\n",
|
|
"\n",
|
|
"## Vector stores\n",
|
|
"\n",
|
|
"LangChain [VectorStore](https://python.langchain.com/api_reference/core/vectorstores/langchain_core.vectorstores.base.VectorStore.html) objects contain methods for adding text and `Document` objects to the store, and querying them using various similarity metrics. They are often initialized with [embedding](/docs/how_to/embed_text) models, which determine how text data is translated to numeric vectors.\n",
|
|
"\n",
|
|
"LangChain includes a suite of [integrations](/docs/integrations/vectorstores) with different vector store technologies. Some vector stores are hosted by a provider (e.g., various cloud providers) and require specific credentials to use; some (such as [Postgres](/docs/integrations/vectorstores/pgvector)) run in separate infrastructure that can be run locally or via a third-party; others can run in-memory for lightweight workloads. Let's select a vector store:\n",
|
|
"\n",
|
|
"import VectorStoreTabs from \"@theme/VectorStoreTabs\";\n",
|
|
"\n",
|
|
"<VectorStoreTabs/>"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 6,
|
|
"id": "5b0d3730-c008-4246-8b03-dd3058513e1c",
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"# | output: false\n",
|
|
"# | echo: false\n",
|
|
"\n",
|
|
"from langchain_chroma import Chroma\n",
|
|
"\n",
|
|
"vector_store = Chroma(embedding_function=embeddings)"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "e3b3035f-1371-4965-ab7a-04eae25e47f3",
|
|
"metadata": {},
|
|
"source": [
|
|
"Having instantiated our vector store, we can now index the documents."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 7,
|
|
"id": "2ea92e04-331c-4f83-aa2a-508322bdfbfc",
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"ids = vector_store.add_documents(documents=all_splits)"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "ff0f0b43-e5b8-4c79-b782-a02f17345487",
|
|
"metadata": {},
|
|
"source": [
|
|
"Note that most vector store implementations will allow you to connect to an existing vector store-- e.g., by providing a client, index name, or other information. See the documentation for a specific [integration](/docs/integrations/vectorstores) for more detail.\n",
|
|
"\n",
|
|
"Once we've instantiated a `VectorStore` that contains documents, we can query it. [VectorStore](https://python.langchain.com/api_reference/core/vectorstores/langchain_core.vectorstores.base.VectorStore.html) includes methods for querying:\n",
|
|
"- Synchronously and asynchronously;\n",
|
|
"- By string query and by vector;\n",
|
|
"- With and without returning similarity scores;\n",
|
|
"- By similarity and [maximum marginal relevance](https://python.langchain.com/api_reference/core/vectorstores/langchain_core.vectorstores.base.VectorStore.html#langchain_core.vectorstores.base.VectorStore.max_marginal_relevance_search) (to balance similarity with query to diversity in retrieved results).\n",
|
|
"\n",
|
|
"The methods will generally include a list of [Document](https://python.langchain.com/api_reference/core/documents/langchain_core.documents.base.Document.html#langchain_core.documents.base.Document) objects in their outputs.\n",
|
|
"\n",
|
|
"### Usage\n",
|
|
"\n",
|
|
"Embeddings typically represent text as a \"dense\" vector such that texts with similar meanings are gemoetrically close. This lets us retrieve relevant information just by passing in a question, without knowledge of any specific key-terms used in the document.\n",
|
|
"\n",
|
|
"Return documents based on similarity to a string query:"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 8,
|
|
"id": "7e01ed91-1a98-4221-960a-bd7a2541a548",
|
|
"metadata": {},
|
|
"outputs": [
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"page_content='direct to consumer operations sell products through the following number of retail stores in the United States:\n",
|
|
"U.S. RETAIL STORES NUMBER\n",
|
|
"NIKE Brand factory stores 213 \n",
|
|
"NIKE Brand in-line stores (including employee-only stores) 74 \n",
|
|
"Converse stores (including factory stores) 82 \n",
|
|
"TOTAL 369 \n",
|
|
"In the United States, NIKE has eight significant distribution centers. Refer to Item 2. Properties for further information.\n",
|
|
"2023 FORM 10-K 2' metadata={'page': 4, 'source': '../example_data/nke-10k-2023.pdf', 'start_index': 3125}\n"
|
|
]
|
|
}
|
|
],
|
|
"source": [
|
|
"results = vector_store.similarity_search(\n",
|
|
" \"How many distribution centers does Nike have in the US?\"\n",
|
|
")\n",
|
|
"\n",
|
|
"print(results[0])"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "4d4f9857-5a7d-4b5f-82b8-ff76539143c2",
|
|
"metadata": {},
|
|
"source": [
|
|
"Async query:"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 9,
|
|
"id": "7ff9e061-7710-40b2-93dc-1ca2b71ef96d",
|
|
"metadata": {},
|
|
"outputs": [
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"page_content='Table of Contents\n",
|
|
"PART I\n",
|
|
"ITEM 1. BUSINESS\n",
|
|
"GENERAL\n",
|
|
"NIKE, Inc. was incorporated in 1967 under the laws of the State of Oregon. As used in this Annual Report on Form 10-K (this \"Annual Report\"), the terms \"we,\" \"us,\" \"our,\"\n",
|
|
"\"NIKE\" and the \"Company\" refer to NIKE, Inc. and its predecessors, subsidiaries and affiliates, collectively, unless the context indicates otherwise.\n",
|
|
"Our principal business activity is the design, development and worldwide marketing and selling of athletic footwear, apparel, equipment, accessories and services. NIKE is\n",
|
|
"the largest seller of athletic footwear and apparel in the world. We sell our products through NIKE Direct operations, which are comprised of both NIKE-owned retail stores\n",
|
|
"and sales through our digital platforms (also referred to as \"NIKE Brand Digital\"), to retail accounts and to a mix of independent distributors, licensees and sales' metadata={'page': 3, 'source': '../example_data/nke-10k-2023.pdf', 'start_index': 0}\n"
|
|
]
|
|
}
|
|
],
|
|
"source": [
|
|
"results = await vector_store.asimilarity_search(\"When was Nike incorporated?\")\n",
|
|
"\n",
|
|
"print(results[0])"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "d4172698-9ad7-4422-99b2-bdc268e99c75",
|
|
"metadata": {},
|
|
"source": [
|
|
"Return scores:"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 11,
|
|
"id": "52dfc576-40a7-4030-aeb5-bb4d3a493e3e",
|
|
"metadata": {},
|
|
"outputs": [
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"Score: 0.23699893057346344\n",
|
|
"\n",
|
|
"page_content='Table of Contents\n",
|
|
"FISCAL 2023 NIKE BRAND REVENUE HIGHLIGHTS\n",
|
|
"The following tables present NIKE Brand revenues disaggregated by reportable operating segment, distribution channel and major product line:\n",
|
|
"FISCAL 2023 COMPARED TO FISCAL 2022\n",
|
|
"•NIKE, Inc. Revenues were $51.2 billion in fiscal 2023, which increased 10% and 16% compared to fiscal 2022 on a reported and currency-neutral basis, respectively.\n",
|
|
"The increase was due to higher revenues in North America, Europe, Middle East & Africa (\"EMEA\"), APLA and Greater China, which contributed approximately 7, 6,\n",
|
|
"2 and 1 percentage points to NIKE, Inc. Revenues, respectively.\n",
|
|
"•NIKE Brand revenues, which represented over 90% of NIKE, Inc. Revenues, increased 10% and 16% on a reported and currency-neutral basis, respectively. This\n",
|
|
"increase was primarily due to higher revenues in Men's, the Jordan Brand, Women's and Kids' which grew 17%, 35%,11% and 10%, respectively, on a wholesale\n",
|
|
"equivalent basis.' metadata={'page': 35, 'source': '../example_data/nke-10k-2023.pdf', 'start_index': 0}\n"
|
|
]
|
|
}
|
|
],
|
|
"source": [
|
|
"# Note that providers implement different scores; the score here\n",
|
|
"# is a distance metric that varies inversely with similarity.\n",
|
|
"\n",
|
|
"results = vector_store.similarity_search_with_score(\"What was Nike's revenue in 2023?\")\n",
|
|
"doc, score = results[0]\n",
|
|
"print(f\"Score: {score}\\n\")\n",
|
|
"print(doc)"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "b4991642-7275-40a9-b11a-e3beccbf2614",
|
|
"metadata": {},
|
|
"source": [
|
|
"Return documents based on similarity to an embedded query:"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 11,
|
|
"id": "7be726c1-b24c-414a-9862-d412b94784b2",
|
|
"metadata": {},
|
|
"outputs": [
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"page_content='Table of Contents\n",
|
|
"GROSS MARGIN\n",
|
|
"FISCAL 2023 COMPARED TO FISCAL 2022\n",
|
|
"For fiscal 2023, our consolidated gross profit increased 4% to $22,292 million compared to $21,479 million for fiscal 2022. Gross margin decreased 250 basis points to\n",
|
|
"43.5% for fiscal 2023 compared to 46.0% for fiscal 2022 due to the following:\n",
|
|
"*Wholesale equivalent\n",
|
|
"The decrease in gross margin for fiscal 2023 was primarily due to:\n",
|
|
"•Higher NIKE Brand product costs, on a wholesale equivalent basis, primarily due to higher input costs and elevated inbound freight and logistics costs as well as\n",
|
|
"product mix;\n",
|
|
"•Lower margin in our NIKE Direct business, driven by higher promotional activity to liquidate inventory in the current period compared to lower promotional activity in\n",
|
|
"the prior period resulting from lower available inventory supply;\n",
|
|
"•Unfavorable changes in net foreign currency exchange rates, including hedges; and\n",
|
|
"•Lower off-price margin, on a wholesale equivalent basis.\n",
|
|
"This was partially offset by:' metadata={'page': 36, 'source': '../example_data/nke-10k-2023.pdf', 'start_index': 0}\n"
|
|
]
|
|
}
|
|
],
|
|
"source": [
|
|
"embedding = embeddings.embed_query(\"How were Nike's margins impacted in 2023?\")\n",
|
|
"\n",
|
|
"results = vector_store.similarity_search_by_vector(embedding)\n",
|
|
"print(results[0])"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "168dbbec-ea97-4cc9-bb1a-75519c2d08af",
|
|
"metadata": {},
|
|
"source": [
|
|
"Learn more:\n",
|
|
"\n",
|
|
"- [API reference](https://python.langchain.com/api_reference/core/vectorstores/langchain_core.vectorstores.base.VectorStore.html)\n",
|
|
"- [How-to guide](/docs/how_to/vectorstores)\n",
|
|
"- [Integration-specific docs](/docs/integrations/vectorstores)\n",
|
|
"\n",
|
|
"## Retrievers\n",
|
|
"\n",
|
|
"LangChain `VectorStore` objects do not subclass [Runnable](https://python.langchain.com/api_reference/core/index.html#langchain-core-runnables). LangChain [Retrievers](https://python.langchain.com/api_reference/core/index.html#langchain-core-retrievers) are Runnables, so they implement a standard set of methods (e.g., synchronous and asynchronous `invoke` and `batch` operations). Although we can construct retrievers from vector stores, retrievers can interface with non-vector store sources of data, as well (such as external APIs).\n",
|
|
"\n",
|
|
"We can create a simple version of this ourselves, without subclassing `Retriever`. If we choose what method we wish to use to retrieve documents, we can create a runnable easily. Below we will build one around the `similarity_search` method:"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 14,
|
|
"id": "58b8e826-1556-489c-b27b-a1efbc4cd689",
|
|
"metadata": {},
|
|
"outputs": [
|
|
{
|
|
"data": {
|
|
"text/plain": [
|
|
"[[Document(metadata={'page': 4, 'source': '../example_data/nke-10k-2023.pdf', 'start_index': 3125}, page_content='direct to consumer operations sell products through the following number of retail stores in the United States:\\nU.S. RETAIL STORES NUMBER\\nNIKE Brand factory stores 213 \\nNIKE Brand in-line stores (including employee-only stores) 74 \\nConverse stores (including factory stores) 82 \\nTOTAL 369 \\nIn the United States, NIKE has eight significant distribution centers. Refer to Item 2. Properties for further information.\\n2023 FORM 10-K 2')],\n",
|
|
" [Document(metadata={'page': 3, 'source': '../example_data/nke-10k-2023.pdf', 'start_index': 0}, page_content='Table of Contents\\nPART I\\nITEM 1. BUSINESS\\nGENERAL\\nNIKE, Inc. was incorporated in 1967 under the laws of the State of Oregon. As used in this Annual Report on Form 10-K (this \"Annual Report\"), the terms \"we,\" \"us,\" \"our,\"\\n\"NIKE\" and the \"Company\" refer to NIKE, Inc. and its predecessors, subsidiaries and affiliates, collectively, unless the context indicates otherwise.\\nOur principal business activity is the design, development and worldwide marketing and selling of athletic footwear, apparel, equipment, accessories and services. NIKE is\\nthe largest seller of athletic footwear and apparel in the world. We sell our products through NIKE Direct operations, which are comprised of both NIKE-owned retail stores\\nand sales through our digital platforms (also referred to as \"NIKE Brand Digital\"), to retail accounts and to a mix of independent distributors, licensees and sales')]]"
|
|
]
|
|
},
|
|
"execution_count": 14,
|
|
"metadata": {},
|
|
"output_type": "execute_result"
|
|
}
|
|
],
|
|
"source": [
|
|
"from typing import List\n",
|
|
"\n",
|
|
"from langchain_core.documents import Document\n",
|
|
"from langchain_core.runnables import chain\n",
|
|
"\n",
|
|
"\n",
|
|
"@chain\n",
|
|
"def retriever(query: str) -> List[Document]:\n",
|
|
" return vector_store.similarity_search(query, k=1)\n",
|
|
"\n",
|
|
"\n",
|
|
"retriever.batch(\n",
|
|
" [\n",
|
|
" \"How many distribution centers does Nike have in the US?\",\n",
|
|
" \"When was Nike incorporated?\",\n",
|
|
" ],\n",
|
|
")"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "a36d3f64-a8bc-4baa-b2ea-07e324a0143e",
|
|
"metadata": {},
|
|
"source": [
|
|
"Vectorstores implement an `as_retriever` method that will generate a Retriever, specifically a [VectorStoreRetriever](https://python.langchain.com/api_reference/core/vectorstores/langchain_core.vectorstores.base.VectorStoreRetriever.html). These retrievers include specific `search_type` and `search_kwargs` attributes that identify what methods of the underlying vector store to call, and how to parameterize them. For instance, we can replicate the above with the following:"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 13,
|
|
"id": "4989fe5e-ac58-4751-bc35-f53ff885860c",
|
|
"metadata": {},
|
|
"outputs": [
|
|
{
|
|
"data": {
|
|
"text/plain": [
|
|
"[[Document(metadata={'page': 4, 'source': '../example_data/nke-10k-2023.pdf', 'start_index': 3125}, page_content='direct to consumer operations sell products through the following number of retail stores in the United States:\\nU.S. RETAIL STORES NUMBER\\nNIKE Brand factory stores 213 \\nNIKE Brand in-line stores (including employee-only stores) 74 \\nConverse stores (including factory stores) 82 \\nTOTAL 369 \\nIn the United States, NIKE has eight significant distribution centers. Refer to Item 2. Properties for further information.\\n2023 FORM 10-K 2')],\n",
|
|
" [Document(metadata={'page': 3, 'source': '../example_data/nke-10k-2023.pdf', 'start_index': 0}, page_content='Table of Contents\\nPART I\\nITEM 1. BUSINESS\\nGENERAL\\nNIKE, Inc. was incorporated in 1967 under the laws of the State of Oregon. As used in this Annual Report on Form 10-K (this \"Annual Report\"), the terms \"we,\" \"us,\" \"our,\"\\n\"NIKE\" and the \"Company\" refer to NIKE, Inc. and its predecessors, subsidiaries and affiliates, collectively, unless the context indicates otherwise.\\nOur principal business activity is the design, development and worldwide marketing and selling of athletic footwear, apparel, equipment, accessories and services. NIKE is\\nthe largest seller of athletic footwear and apparel in the world. We sell our products through NIKE Direct operations, which are comprised of both NIKE-owned retail stores\\nand sales through our digital platforms (also referred to as \"NIKE Brand Digital\"), to retail accounts and to a mix of independent distributors, licensees and sales')]]"
|
|
]
|
|
},
|
|
"execution_count": 13,
|
|
"metadata": {},
|
|
"output_type": "execute_result"
|
|
}
|
|
],
|
|
"source": [
|
|
"retriever = vector_store.as_retriever(\n",
|
|
" search_type=\"similarity\",\n",
|
|
" search_kwargs={\"k\": 1},\n",
|
|
")\n",
|
|
"\n",
|
|
"retriever.batch(\n",
|
|
" [\n",
|
|
" \"How many distribution centers does Nike have in the US?\",\n",
|
|
" \"When was Nike incorporated?\",\n",
|
|
" ],\n",
|
|
")"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "6b79ded3-39ed-4aeb-8b70-cd36795ae239",
|
|
"metadata": {},
|
|
"source": [
|
|
"`VectorStoreRetriever` supports search types of `\"similarity\"` (default), `\"mmr\"` (maximum marginal relevance, described above), and `\"similarity_score_threshold\"`. We can use the latter to threshold documents output by the retriever by similarity score.\n",
|
|
"\n",
|
|
"Retrievers can easily be incorporated into more complex applications, such as [retrieval-augmented generation (RAG)](/docs/concepts/rag) applications that combine a given question with retrieved context into a prompt for a LLM. To learn more about building such an application, check out the [RAG tutorial](/docs/tutorials/rag) tutorial."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "3d9be7cb-2081-48a4-b6e4-d5e2d562ffd4",
|
|
"metadata": {},
|
|
"source": [
|
|
"### Learn more:\n",
|
|
"\n",
|
|
"Retrieval strategies can be rich and complex. For example:\n",
|
|
"\n",
|
|
"- We can [infer hard rules and filters](/docs/how_to/self_query/) from a query (e.g., \"using documents published after 2020\");\n",
|
|
"- We can [return documents that are linked](/docs/how_to/parent_document_retriever/) to the retrieved context in some way (e.g., via some document taxonomy);\n",
|
|
"- We can generate [multiple embeddings](/docs/how_to/multi_vector) for each unit of context;\n",
|
|
"- We can [ensemble results](/docs/how_to/ensemble_retriever) from multiple retrievers;\n",
|
|
"- We can assign weights to documents, e.g., to weigh [recent documents](/docs/how_to/time_weighted_vectorstore/) higher.\n",
|
|
"\n",
|
|
"The [retrievers](/docs/how_to#retrievers) section of the how-to guides covers these and other built-in retrieval strategies.\n",
|
|
"\n",
|
|
"It is also straightforward to extend the [BaseRetriever](https://python.langchain.com/api_reference/core/retrievers/langchain_core.retrievers.BaseRetriever.html) class in order to implement custom retrievers. See our how-to guide [here](/docs/how_to/custom_retriever).\n",
|
|
"\n",
|
|
"\n",
|
|
"## Next steps\n",
|
|
"\n",
|
|
"You've now seen how to build a semantic search engine over a PDF document.\n",
|
|
"\n",
|
|
"For more on document loaders:\n",
|
|
"\n",
|
|
"- [Conceptual guide](/docs/concepts/document_loaders)\n",
|
|
"- [How-to guides](/docs/how_to/#document-loaders)\n",
|
|
"- [Available integrations](/docs/integrations/document_loaders/)\n",
|
|
"\n",
|
|
"For more on embeddings:\n",
|
|
"\n",
|
|
"- [Conceptual guide](/docs/concepts/embedding_models/)\n",
|
|
"- [How-to guides](/docs/how_to/#embedding-models)\n",
|
|
"- [Available integrations](/docs/integrations/text_embedding/)\n",
|
|
"\n",
|
|
"For more on vector stores:\n",
|
|
"\n",
|
|
"- [Conceptual guide](/docs/concepts/vectorstores/)\n",
|
|
"- [How-to guides](/docs/how_to/#vector-stores)\n",
|
|
"- [Available integrations](/docs/integrations/vectorstores/)\n",
|
|
"\n",
|
|
"For more on RAG, see:\n",
|
|
"\n",
|
|
"- [Build a Retrieval Augmented Generation (RAG) App](/docs/tutorials/rag/)\n",
|
|
"- [Related how-to guides](/docs/how_to/#qa-with-rag)"
|
|
]
|
|
}
|
|
],
|
|
"metadata": {
|
|
"kernelspec": {
|
|
"display_name": "Python 3 (ipykernel)",
|
|
"language": "python",
|
|
"name": "python3"
|
|
},
|
|
"language_info": {
|
|
"codemirror_mode": {
|
|
"name": "ipython",
|
|
"version": 3
|
|
},
|
|
"file_extension": ".py",
|
|
"mimetype": "text/x-python",
|
|
"name": "python",
|
|
"nbconvert_exporter": "python",
|
|
"pygments_lexer": "ipython3",
|
|
"version": "3.10.4"
|
|
}
|
|
},
|
|
"nbformat": 4,
|
|
"nbformat_minor": 5
|
|
}
|