github/langchain
1
0
Fork 0
mirror of https://github.com/hwchase17/langchain.git synced 2025-09-06 13:33:37 +00:00
Code Issues Packages Projects Releases Wiki Activity

Docs refactor (#480)

Browse Source 
Big docs refactor! Motivation is to make it easier for people to find
resources they are looking for. To accomplish this, there are now three
main sections:

- Getting Started: steps for getting started, walking through most core
functionality
- Modules: these are different modules of functionality that langchain
provides. Each part here has a "getting started", "how to", "key
concepts" and "reference" section (except in a few select cases where it
didnt easily fit).
- Use Cases: this is to separate use cases (like summarization, question
answering, evaluation, etc) from the modules, and provide a different
entry point to the code base.

There is also a full reference section, as well as extra resources
(glossary, gallery, etc)

Co-authored-by: Shreya Rajpal <ShreyaR@users.noreply.github.com>
This commit is contained in:
Harrison Chase
2023-01-02 08:24:09 -08:00
committed by GitHub
parent c5f0af9398
commit 985496f4be
164 changed files with 4326 additions and 2586 deletions
Download Patch File Download Diff File Expand all files Collapse all files

128
docs/modules/utils/combine_docs_examples/embeddings.ipynb Normal file
View File

@@ -0,0 +1,128 @@
{
"cells": [
{
"cell_type": "markdown",
"id": "249b4058",
"metadata": {},
"source": [
"# Embeddings\n",
"\n",
"This notebook goes over how to use the Embedding class in LangChain.\n",
"\n",
"The Embedding class is a class designed for interfacing with embeddings. There are lots of Embedding providers (OpenAI, Cohere, Hugging Face, etc) - this class is designed to provide a standard interface for all of them.\n",
"\n",
"Embeddings create a vector representation of a piece of text. This is useful because it means we can think about text in the vector space, and do things like semantic search where we look for pieces of text that are most similar in the vector space.\n",
"\n",
"The base Embedding class in LangChain exposes two methods: `embed_documents` and `embed_query`. The largest difference is that these two methods have different interfaces: one works over multiple documents, while the other works over a single document. Besides this, another reason for having these as two separate methods is that some embedding providers have different embedding methods for documents (to be searched over) vs queries (the search query itself)."
]
},
{
"cell_type": "markdown",
"id": "278b6c63",
"metadata": {},
"source": [
"## OpenAI\n",
"\n",
"Let's load the OpenAI Embedding class."
]
},
{
"cell_type": "code",
"execution_count": 1,
"id": "0be1af71",
"metadata": {},
"outputs": [],
"source": [
"from langchain.embeddings import OpenAIEmbeddings"
]
},
{
"cell_type": "code",
"execution_count": 2,
"id": "2c66e5da",
"metadata": {},
"outputs": [],
"source": [
"embeddings = OpenAIEmbeddings()"
]
},
{
"cell_type": "code",
"execution_count": 3,
"id": "01370375",
"metadata": {},
"outputs": [],
"source": [
"text = \"This is a test document.\""
]
},
{
"cell_type": "code",
"execution_count": 4,
"id": "bfb6142c",
"metadata": {},
"outputs": [],
"source": [
"query_result = embeddings.embed_query(text)"
]
},
{
"cell_type": "code",
"execution_count": 5,
"id": "0356c3b7",
"metadata": {},
"outputs": [],
"source": [
"doc_result = embeddings.embed_documents([text])"
]
},
{
"cell_type": "markdown",
"id": "42f76e43",
"metadata": {},
"source": [
"## Cohere\n",
"\n",
"TODO: add documentation for Cohere embeddings."
]
},
{
"cell_type": "markdown",
"id": "ed47bb62",
"metadata": {},
"source": [
"## Hugging Face Hub\n",
"TODO: add documentation for Hugging Face Hub embeddings."
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "ff9be586",
"metadata": {},
"outputs": [],
"source": []
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3 (ipykernel)",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.10.9"
}
},
"nbformat": 4,
"nbformat_minor": 5
}

242
docs/modules/utils/combine_docs_examples/hyde.ipynb Normal file
View File

@@ -0,0 +1,242 @@
{
"cells": [
{
"cell_type": "markdown",
"id": "ccb74c9b",
"metadata": {},
"source": [
"# Hypothetical Document Embeddings\n",
"This notebook goes over how to use Hypothetical Document Embeddings (HyDE), as described in [this paper](https://arxiv.org/abs/2212.10496). \n",
"\n",
"At a high level, HyDE is an embedding technique that takes queries, generates a hypothetical answer, and then embeds that generated document and uses that as the final example. \n",
"\n",
"In order to use HyDE, we therefor need to provide a base embedding model, as well as an LLMChain that can be used to generate those documents. By default, the HyDE class comes with some default prompts to use (see the paper for more details on them), but we can also create our own."
]
},
{
"cell_type": "code",
"execution_count": 1,
"id": "546e87ee",
"metadata": {},
"outputs": [],
"source": [
"from langchain.llms import OpenAI\n",
"from langchain.embeddings import OpenAIEmbeddings, HypotheticalDocumentEmbedder\n",
"from langchain.chains import LLMChain\n",
"from langchain.prompts import PromptTemplate"
]
},
{
"cell_type": "code",
"execution_count": 2,
"id": "c0ea895f",
"metadata": {},
"outputs": [],
"source": [
"base_embeddings = OpenAIEmbeddings()\n",
"llm = OpenAI()"
]
},
{
"cell_type": "code",
"execution_count": 3,
"id": "50729989",
"metadata": {},
"outputs": [],
"source": [
"# Load with `web_search` prompt\n",
"embeddings = HypotheticalDocumentEmbedder.from_llm(llm, base_embeddings, \"web_search\")"
]
},
{
"cell_type": "code",
"execution_count": 4,
"id": "3aa573d6",
"metadata": {},
"outputs": [],
"source": [
"# Now we can use it as any embedding class!\n",
"result = embeddings.embed_query(\"Where is the Taj Mahal?\")"
]
},
{
"cell_type": "markdown",
"id": "c7a0b556",
"metadata": {},
"source": [
"## Multiple generations\n",
"We can also generate multiple documents and then combine the embeddings for those. By default, we combine those by taking the average. We can do this by changing the LLM we use to generate documents to return multiple things."
]
},
{
"cell_type": "code",
"execution_count": 5,
"id": "05da7060",
"metadata": {},
"outputs": [],
"source": [
"multi_llm = OpenAI(n=4, best_of=4)"
]
},
{
"cell_type": "code",
"execution_count": 6,
"id": "9b1e12bd",
"metadata": {},
"outputs": [],
"source": [
"embeddings = HypotheticalDocumentEmbedder.from_llm(multi_llm, base_embeddings, \"web_search\")"
]
},
{
"cell_type": "code",
"execution_count": 7,
"id": "a60cd343",
"metadata": {},
"outputs": [],
"source": [
"result = embeddings.embed_query(\"Where is the Taj Mahal?\")"
]
},
{
"cell_type": "markdown",
"id": "1da90437",
"metadata": {},
"source": [
"## Using our own prompts\n",
"Besides using preconfigured prompts, we can also easily construct our own prompts and use those in the LLMChain that is generating the documents. This can be useful if we know the domain our queries will be in, as we can condition the prompt to generate text more similar to that.\n",
"\n",
"In the example below, let's condition it generate text about a state of the union address (because we will use that in the next example)."
]
},
{
"cell_type": "code",
"execution_count": 8,
"id": "0b4a650f",
"metadata": {},
"outputs": [],
"source": [
"prompt_template = \"\"\"Please answer the user's question about the most recent state of the union address\n",
"Question: {question}\n",
"Answer:\"\"\"\n",
"prompt = PromptTemplate(input_variables=[\"question\"], template=prompt_template)\n",
"llm_chain = LLMChain(llm=llm, prompt=prompt)"
]
},
{
"cell_type": "code",
"execution_count": 9,
"id": "7f7e2b86",
"metadata": {},
"outputs": [],
"source": [
"embeddings = HypotheticalDocumentEmbedder(llm_chain=llm_chain, base_embeddings=base_embeddings)"
]
},
{
"cell_type": "code",
"execution_count": 10,
"id": "6dd83424",
"metadata": {},
"outputs": [],
"source": [
"result = embeddings.embed_query(\"What did the president say about Ketanji Brown Jackson\")"
]
},
{
"cell_type": "markdown",
"id": "31388123",
"metadata": {},
"source": [
"## Using HyDE\n",
"Now that we have HyDE, we can use it as we would any other embedding class! Here is using it to find similar passages in the state of the union example."
]
},
{
"cell_type": "code",
"execution_count": 11,
"id": "97719b29",
"metadata": {},
"outputs": [],
"source": [
"from langchain.text_splitter import CharacterTextSplitter\n",
"from langchain.vectorstores import FAISS\n",
"\n",
"with open('../../state_of_the_union.txt') as f:\n",
" state_of_the_union = f.read()\n",
"text_splitter = CharacterTextSplitter(chunk_size=1000, chunk_overlap=0)\n",
"texts = text_splitter.split_text(state_of_the_union)"
]
},
{
"cell_type": "code",
"execution_count": 12,
"id": "bfcfc039",
"metadata": {},
"outputs": [],
"source": [
"docsearch = FAISS.from_texts(texts, embeddings)\n",
"\n",
"query = \"What did the president say about Ketanji Brown Jackson\"\n",
"docs = docsearch.similarity_search(query)"
]
},
{
"cell_type": "code",
"execution_count": 13,
"id": "632af7f2",
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"In state after state, new laws have been passed, not only to suppress the vote, but to subvert entire elections. \n",
"\n",
"We cannot let this happen. \n",
"\n",
"Tonight. I call on the Senate to: Pass the Freedom to Vote Act. Pass the John Lewis Voting Rights Act. And while you’re at it, pass the Disclose Act so Americans can know who is funding our elections. \n",
"\n",
"Tonight, I’d like to honor someone who has dedicated his life to serve this country: Justice Stephen Breyer—an Army veteran, Constitutional scholar, and retiring Justice of the United States Supreme Court. Justice Breyer, thank you for your service. \n",
"\n",
"One of the most serious constitutional responsibilities a President has is nominating someone to serve on the United States Supreme Court. \n",
"\n",
"And I did that 4 days ago, when I nominated Circuit Court of Appeals Judge Ketanji Brown Jackson. One of our nation’s top legal minds, who will continue Justice Breyer’s legacy of excellence. \n"
]
}
],
"source": [
"print(docs[0].page_content)"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "b9e57b93",
"metadata": {},
"outputs": [],
"source": []
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3 (ipykernel)",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.10.9"
}
},
"nbformat": 4,
"nbformat_minor": 5
}

453
docs/modules/utils/combine_docs_examples/textsplitter.ipynb Normal file
View File

@@ -0,0 +1,453 @@
{
"cells": [
{
"cell_type": "markdown",
"id": "b118c9dc",
"metadata": {},
"source": [
"# Text Splitter\n",
"\n",
"When you want to deal wit long pieces of text, it is necessary to split up that text into chunks.\n",
"This notebook showcases several ways to do that.\n",
"\n",
"At a high level, text splitters work as following:\n",
"\n",
"1. Split the text up into small, semantically meaningful chunks (often sentences).\n",
"2. Start combining these small chunks into a larger chunk until you reach a certain size (as measured by some function).\n",
"3. Once you reach that size, make that chunk its own piece of text and then start creating a new chunk of text with some overlap (to keep context between chunks)."
]
},
{
"cell_type": "code",
"execution_count": 1,
"id": "e82c4685",
"metadata": {},
"outputs": [],
"source": [
"from langchain.text_splitter import CharacterTextSplitter, NLTKTextSplitter, SpacyTextSplitter\n",
"# This is a long document we can split up.\n",
"with open('../../state_of_the_union.txt') as f:\n",
" state_of_the_union = f.read()"
]
},
{
"cell_type": "markdown",
"id": "5c461b26",
"metadata": {},
"source": [
"## Character Text Splitting\n",
"\n",
"Let's start with the most simple method: let's split based on characters (by default \"\\n\\n\") and measure chunk length by number of characters."
]
},
{
"cell_type": "code",
"execution_count": 2,
"id": "79ff6737",
"metadata": {},
"outputs": [],
"source": [
"text_splitter = CharacterTextSplitter( \n",
" separator = \"\\n\\n\",\n",
" chunk_size = 1000,\n",
" chunk_overlap = 200,\n",
" length_function = len,\n",
")"
]
},
{
"cell_type": "code",
"execution_count": 3,
"id": "38547666",
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Madam Speaker, Madam Vice President, our First Lady and Second Gentleman. Members of Congress and the Cabinet. Justices of the Supreme Court. My fellow Americans. \n",
"\n",
"Last year COVID-19 kept us apart. This year we are finally together again. \n",
"\n",
"Tonight, we meet as Democrats Republicans and Independents. But most importantly as Americans. \n",
"\n",
"With a duty to one another to the American people to the Constitution. \n",
"\n",
"And with an unwavering resolve that freedom will always triumph over tyranny. \n",
"\n",
"Six days ago, Russia’s Vladimir Putin sought to shake the foundations of the free world thinking he could make it bend to his menacing ways. But he badly miscalculated. \n",
"\n",
"He thought he could roll into Ukraine and the world would roll over. Instead he met a wall of strength he never imagined. \n",
"\n",
"He met the Ukrainian people. \n",
"\n",
"From President Zelenskyy to every Ukrainian, their fearlessness, their courage, their determination, inspires the world. \n"
]
}
],
"source": [
"texts = text_splitter.split_text(state_of_the_union)\n",
"print(texts[0])"
]
},
{
"cell_type": "markdown",
"id": "87a71115",
"metadata": {},
"source": [
"## Document creation\n",
"We can also use the text splitter to create \"Documents\" directly. Documents a way of bundling pieces of text with associated metadata so that chains can interact with them. We can also create documents with empty metadata though!\n",
"\n",
"In the below example, we pass two pieces of text to get split up (we pass two just to show off the interface of splitting multiple pieces of text)."
]
},
{
"cell_type": "code",
"execution_count": 4,
"id": "4cd16222",
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"page_content='Madam Speaker, Madam Vice President, our First Lady and Second Gentleman. Members of Congress and the Cabinet. Justices of the Supreme Court. My fellow Americans. \\n\\nLast year COVID-19 kept us apart. This year we are finally together again. \\n\\nTonight, we meet as Democrats Republicans and Independents. But most importantly as Americans. \\n\\nWith a duty to one another to the American people to the Constitution. \\n\\nAnd with an unwavering resolve that freedom will always triumph over tyranny. \\n\\nSix days ago, Russia’s Vladimir Putin sought to shake the foundations of the free world thinking he could make it bend to his menacing ways. But he badly miscalculated. \\n\\nHe thought he could roll into Ukraine and the world would roll over. Instead he met a wall of strength he never imagined. \\n\\nHe met the Ukrainian people. \\n\\nFrom President Zelenskyy to every Ukrainian, their fearlessness, their courage, their determination, inspires the world. ' lookup_str='' metadata={} lookup_index=0\n"
]
}
],
"source": [
"documents = text_splitter.create_documents([state_of_the_union, state_of_the_union])\n",
"print(documents[0])"
]
},
{
"cell_type": "markdown",
"id": "2cede1b1",
"metadata": {},
"source": [
"Here's an example of passing metadata along with the documents, notice that it is split along with the documents."
]
},
{
"cell_type": "code",
"execution_count": 5,
"id": "4a47515a",
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"page_content='Madam Speaker, Madam Vice President, our First Lady and Second Gentleman. Members of Congress and the Cabinet. Justices of the Supreme Court. My fellow Americans. \\n\\nLast year COVID-19 kept us apart. This year we are finally together again. \\n\\nTonight, we meet as Democrats Republicans and Independents. But most importantly as Americans. \\n\\nWith a duty to one another to the American people to the Constitution. \\n\\nAnd with an unwavering resolve that freedom will always triumph over tyranny. \\n\\nSix days ago, Russia’s Vladimir Putin sought to shake the foundations of the free world thinking he could make it bend to his menacing ways. But he badly miscalculated. \\n\\nHe thought he could roll into Ukraine and the world would roll over. Instead he met a wall of strength he never imagined. \\n\\nHe met the Ukrainian people. \\n\\nFrom President Zelenskyy to every Ukrainian, their fearlessness, their courage, their determination, inspires the world. ' lookup_str='' metadata={'document': 1} lookup_index=0\n"
]
}
],
"source": [
"metadatas = [{\"document\": 1}, {\"document\": 2}]\n",
"documents = text_splitter.create_documents([state_of_the_union, state_of_the_union], metadatas=metadatas)\n",
"print(documents[0])"
]
},
{
"cell_type": "markdown",
"id": "13dc0983",
"metadata": {},
"source": [
"## HuggingFace Length Function\n",
"Most LLMs are constrained by the number of tokens that you can pass in, which is not the same as the number of characters. In order to get a more accurate estimate, we can use HuggingFace tokenizers to count the text length."
]
},
{
"cell_type": "code",
"execution_count": 6,
"id": "a8ce51d5",
"metadata": {},
"outputs": [],
"source": [
"from transformers import GPT2TokenizerFast\n",
"\n",
"tokenizer = GPT2TokenizerFast.from_pretrained(\"gpt2\")"
]
},
{
"cell_type": "code",
"execution_count": 7,
"id": "ca5e72c0",
"metadata": {},
"outputs": [],
"source": [
"text_splitter = CharacterTextSplitter.from_huggingface_tokenizer(tokenizer, chunk_size=100, chunk_overlap=0)\n",
"texts = text_splitter.split_text(state_of_the_union)"
]
},
{
"cell_type": "code",
"execution_count": 8,
"id": "37cdfbeb",
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Madam Speaker, Madam Vice President, our First Lady and Second Gentleman. Members of Congress and the Cabinet. Justices of the Supreme Court. My fellow Americans. \n",
"\n",
"Last year COVID-19 kept us apart. This year we are finally together again. \n",
"\n",
"Tonight, we meet as Democrats Republicans and Independents. But most importantly as Americans. \n",
"\n",
"With a duty to one another to the American people to the Constitution. \n"
]
}
],
"source": [
"print(texts[0])"
]
},
{
"cell_type": "markdown",
"id": "7683b36a",
"metadata": {},
"source": [
"## tiktoken (OpenAI) Length Function\n",
"You can also use tiktoken, a open source tokenizer package from OpenAI to estimate tokens used. Will probably be more accurate for their models."
]
},
{
"cell_type": "code",
"execution_count": 9,
"id": "825f7c0a",
"metadata": {},
"outputs": [],
"source": [
"text_splitter = CharacterTextSplitter.from_tiktoken_encoder(chunk_size=100, chunk_overlap=0)\n",
"texts = text_splitter.split_text(state_of_the_union)"
]
},
{
"cell_type": "code",
"execution_count": 10,
"id": "ae35d165",
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Madam Speaker, Madam Vice President, our First Lady and Second Gentleman. Members of Congress and the Cabinet. Justices of the Supreme Court. My fellow Americans. \n",
"\n",
"Last year COVID-19 kept us apart. This year we are finally together again. \n",
"\n",
"Tonight, we meet as Democrats Republicans and Independents. But most importantly as Americans. \n",
"\n",
"With a duty to one another to the American people to the Constitution. \n"
]
}
],
"source": [
"print(texts[0])"
]
},
{
"cell_type": "markdown",
"id": "ea2973ac",
"metadata": {},
"source": [
"## NLTK Text Splitter\n",
"Rather than just splitting on \"\\n\\n\", we can use NLTK to split based on tokenizers."
]
},
{
"cell_type": "code",
"execution_count": 11,
"id": "20fa9c23",
"metadata": {},
"outputs": [],
"source": [
"text_splitter = NLTKTextSplitter(chunk_size=1000)"
]
},
{
"cell_type": "code",
"execution_count": 12,
"id": "5ea10835",
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Madam Speaker, Madam Vice President, our First Lady and Second Gentleman.\n",
"\n",
"Members of Congress and the Cabinet.\n",
"\n",
"Justices of the Supreme Court.\n",
"\n",
"My fellow Americans.\n",
"\n",
"Last year COVID-19 kept us apart.\n",
"\n",
"This year we are finally together again.\n",
"\n",
"Tonight, we meet as Democrats Republicans and Independents.\n",
"\n",
"But most importantly as Americans.\n",
"\n",
"With a duty to one another to the American people to the Constitution.\n",
"\n",
"And with an unwavering resolve that freedom will always triumph over tyranny.\n",
"\n",
"Six days ago, Russia’s Vladimir Putin sought to shake the foundations of the free world thinking he could make it bend to his menacing ways.\n",
"\n",
"But he badly miscalculated.\n",
"\n",
"He thought he could roll into Ukraine and the world would roll over.\n",
"\n",
"Instead he met a wall of strength he never imagined.\n",
"\n",
"He met the Ukrainian people.\n",
"\n",
"From President Zelenskyy to every Ukrainian, their fearlessness, their courage, their determination, inspires the world.\n",
"\n",
"Groups of citizens blocking tanks with their bodies.\n"
]
}
],
"source": [
"texts = text_splitter.split_text(state_of_the_union)\n",
"print(texts[0])"
]
},
{
"cell_type": "markdown",
"id": "dab86b60",
"metadata": {},
"source": [
"## Spacy Text Splitter\n",
"Another alternative to NLTK is to use Spacy."
]
},
{
"cell_type": "code",
"execution_count": 13,
"id": "f9cc9dfc",
"metadata": {},
"outputs": [],
"source": [
"text_splitter = SpacyTextSplitter(chunk_size=1000)"
]
},
{
"cell_type": "code",
"execution_count": 14,
"id": "cef2b29e",
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Madam Speaker, Madam Vice President, our First Lady and Second Gentleman.\n",
"\n",
"Members of Congress and the Cabinet.\n",
"\n",
"Justices of the Supreme Court.\n",
"\n",
"My fellow Americans. \n",
"\n",
"\n",
"\n",
"Last year COVID-19 kept us apart.\n",
"\n",
"This year we are finally together again.\n",
"\n",
"\n",
"\n",
"\n",
"\n",
"Tonight, we meet as Democrats Republicans and Independents.\n",
"\n",
"But most importantly as Americans.\n",
"\n",
"\n",
"\n",
"\n",
"\n",
"With a duty to one another to the American people to the Constitution. \n",
"\n",
"\n",
"\n",
"And with an unwavering resolve that freedom will always triumph over tyranny.\n",
"\n",
"\n",
"\n",
"\n",
"\n",
"Six days ago, Russia’s Vladimir Putin sought to shake the foundations of the free world thinking he could make it bend to his menacing ways.\n",
"\n",
"But he badly miscalculated.\n",
"\n",
"\n",
"\n",
"\n",
"\n",
"He thought he could roll into Ukraine and the world would roll over.\n",
"\n",
"Instead he met a wall of strength he never imagined.\n",
"\n",
"\n",
"\n",
"\n",
"\n",
"He met the Ukrainian people.\n",
"\n",
"\n",
"\n",
"\n",
"\n",
"From President Zelenskyy to every Ukrainian, their fearlessness, their courage, their determination, inspires the world.\n",
"\n",
"\n",
"\n",
"\n",
"\n",
"Groups of citizens blocking tanks with their bodies.\n"
]
}
],
"source": [
"texts = text_splitter.split_text(state_of_the_union)\n",
"print(texts[0])"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "a1a118b1",
"metadata": {},
"outputs": [],
"source": []
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3 (ipykernel)",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.10.9"
}
},
"nbformat": 4,
"nbformat_minor": 5
}

434
docs/modules/utils/combine_docs_examples/vectorstores.ipynb Normal file
View File

@@ -0,0 +1,434 @@
{
"cells": [
{
"cell_type": "markdown",
"id": "7ef4d402-6662-4a26-b612-35b542066487",
"metadata": {
"pycharm": {
"name": "#%% md\n"
}
},
"source": [
"# VectorStores\n",
"\n",
"This notebook show cases how to use VectorStores. A key part of working with vectorstores is creating the vector to put in them, which is usually created via embeddings. Therefor, it is recommended that you familiarize yourself with the [embedding notebook](embeddings.ipynb) before diving into this."
]
},
{
"cell_type": "code",
"execution_count": 1,
"id": "965eecee",
"metadata": {
"pycharm": {
"name": "#%%\n"
}
},
"outputs": [],
"source": [
"from langchain.embeddings.openai import OpenAIEmbeddings\n",
"from langchain.text_splitter import CharacterTextSplitter\n",
"from langchain.vectorstores import ElasticVectorSearch, Pinecone, Weaviate, FAISS"
]
},
{
"cell_type": "code",
"execution_count": 2,
"id": "68481687",
"metadata": {
"pycharm": {
"name": "#%%\n"
}
},
"outputs": [],
"source": [
"with open('../../state_of_the_union.txt') as f:\n",
" state_of_the_union = f.read()\n",
"text_splitter = CharacterTextSplitter(chunk_size=1000, chunk_overlap=0)\n",
"texts = text_splitter.split_text(state_of_the_union)\n",
"\n",
"embeddings = OpenAIEmbeddings()"
]
},
{
"cell_type": "code",
"execution_count": 3,
"id": "015f4ff5",
"metadata": {
"pycharm": {
"name": "#%%\n"
}
},
"outputs": [],
"source": [
"docsearch = FAISS.from_texts(texts, embeddings)\n",
"\n",
"query = \"What did the president say about Ketanji Brown Jackson\"\n",
"docs = docsearch.similarity_search(query)"
]
},
{
"cell_type": "code",
"execution_count": 4,
"id": "67baf32e",
"metadata": {
"pycharm": {
"name": "#%%\n"
}
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"In state after state, new laws have been passed, not only to suppress the vote, but to subvert entire elections. \n",
"\n",
"We cannot let this happen. \n",
"\n",
"Tonight. I call on the Senate to: Pass the Freedom to Vote Act. Pass the John Lewis Voting Rights Act. And while you’re at it, pass the Disclose Act so Americans can know who is funding our elections. \n",
"\n",
"Tonight, I’d like to honor someone who has dedicated his life to serve this country: Justice Stephen Breyer—an Army veteran, Constitutional scholar, and retiring Justice of the United States Supreme Court. Justice Breyer, thank you for your service. \n",
"\n",
"One of the most serious constitutional responsibilities a President has is nominating someone to serve on the United States Supreme Court. \n",
"\n",
"And I did that 4 days ago, when I nominated Circuit Court of Appeals Judge Ketanji Brown Jackson. One of our nation’s top legal minds, who will continue Justice Breyer’s legacy of excellence. \n"
]
}
],
"source": [
"print(docs[0].page_content)"
]
},
{
"cell_type": "markdown",
"id": "bbf5ec44",
"metadata": {},
"source": [
"## From Documents\n",
"We can also initialize a vectorstore from documents directly. This is useful when we use the method on the text splitter to get documents directly (handy when the original documents have associated metadata)."
]
},
{
"cell_type": "code",
"execution_count": 5,
"id": "df4a459c",
"metadata": {},
"outputs": [],
"source": [
"documents = text_splitter.create_documents([state_of_the_union], metadatas=[{\"source\": \"State of the Union\"}])"
]
},
{
"cell_type": "code",
"execution_count": 6,
"id": "4b480245",
"metadata": {},
"outputs": [],
"source": [
"docsearch = FAISS.from_documents(documents, embeddings)\n",
"\n",
"query = \"What did the president say about Ketanji Brown Jackson\"\n",
"docs = docsearch.similarity_search(query)"
]
},
{
"cell_type": "code",
"execution_count": 7,
"id": "86aa4cda",
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"In state after state, new laws have been passed, not only to suppress the vote, but to subvert entire elections. \n",
"\n",
"We cannot let this happen. \n",
"\n",
"Tonight. I call on the Senate to: Pass the Freedom to Vote Act. Pass the John Lewis Voting Rights Act. And while you’re at it, pass the Disclose Act so Americans can know who is funding our elections. \n",
"\n",
"Tonight, I’d like to honor someone who has dedicated his life to serve this country: Justice Stephen Breyer—an Army veteran, Constitutional scholar, and retiring Justice of the United States Supreme Court. Justice Breyer, thank you for your service. \n",
"\n",
"One of the most serious constitutional responsibilities a President has is nominating someone to serve on the United States Supreme Court. \n",
"\n",
"And I did that 4 days ago, when I nominated Circuit Court of Appeals Judge Ketanji Brown Jackson. One of our nation’s top legal minds, who will continue Justice Breyer’s legacy of excellence. \n"
]
}
],
"source": [
"print(docs[0].page_content)"
]
},
{
"cell_type": "markdown",
"id": "eea6e627",
"metadata": {},
"source": [
"## Requires having ElasticSearch setup"
]
},
{
"cell_type": "code",
"execution_count": 5,
"id": "4906b8a3",
"metadata": {
"pycharm": {
"name": "#%%\n"
}
},
"outputs": [],
"source": [
"docsearch = ElasticVectorSearch.from_texts(texts, embeddings, elasticsearch_url=\"http://localhost:9200\")\n",
"\n",
"query = \"What did the president say about Ketanji Brown Jackson\"\n",
"docs = docsearch.similarity_search(query)"
]
},
{
"cell_type": "code",
"execution_count": 6,
"id": "95f9eee9",
"metadata": {
"pycharm": {
"name": "#%%\n"
}
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Tonight, I’d like to honor someone who has dedicated his life to serve this country: Justice Stephen Breyer—an Army veteran, Constitutional scholar, and retiring Justice of the United States Supreme Court. Justice Breyer, thank you for your service. \n",
"\n",
"One of the most serious constitutional responsibilities a President has is nominating someone to serve on the United States Supreme Court. \n",
"\n",
"And I did that 4 days ago, when I nominated Circuit Court of Appeals Judge Ketanji Brown Jackson. One of our nation’s top legal minds, who will continue Justice Breyer’s legacy of excellence. \n",
"\n",
"A former top litigator in private practice. A former federal public defender. And from a family of public school educators and police officers. A consensus builder. Since she’s been nominated, she’s received a broad range of support—from the Fraternal Order of Police to former judges appointed by Democrats and Republicans. \n",
"\n",
"And if we are to advance liberty and justice, we need to secure the Border and fix the immigration system. \n"
]
}
],
"source": [
"print(docs[0].page_content)"
]
},
{
"cell_type": "markdown",
"id": "7f9cb9e7",
"metadata": {},
"source": [
"## Weaviate"
]
},
{
"cell_type": "code",
"execution_count": 11,
"id": "1037a85e",
"metadata": {},
"outputs": [],
"source": [
"import weaviate\n",
"import os\n",
"\n",
"WEAVIATE_URL = \"\"\n",
"client = weaviate.Client(\n",
" url=WEAVIATE_URL,\n",
" additional_headers={\n",
" 'X-OpenAI-Api-Key': os.environ[\"OPENAI_API_KEY\"]\n",
" }\n",
")"
]
},
{
"cell_type": "code",
"execution_count": 12,
"id": "b9043766",
"metadata": {},
"outputs": [],
"source": [
"client.schema.delete_all()\n",
"client.schema.get()\n",
"schema = {\n",
" \"classes\": [\n",
" {\n",
" \"class\": \"Paragraph\",\n",
" \"description\": \"A written paragraph\",\n",
" \"vectorizer\": \"text2vec-openai\",\n",
" \"moduleConfig\": {\n",
" \"text2vec-openai\": {\n",
" \"model\": \"babbage\",\n",
" \"type\": \"text\"\n",
" }\n",
" },\n",
" \"properties\": [\n",
" {\n",
" \"dataType\": [\"text\"],\n",
" \"description\": \"The content of the paragraph\",\n",
" \"moduleConfig\": {\n",
" \"text2vec-openai\": {\n",
" \"skip\": False,\n",
" \"vectorizePropertyName\": False\n",
" }\n",
" },\n",
" \"name\": \"content\",\n",
" },\n",
" ],\n",
" },\n",
" ]\n",
"}\n",
"\n",
"client.schema.create(schema)"
]
},
{
"cell_type": "code",
"execution_count": 13,
"id": "ac20d99c",
"metadata": {},
"outputs": [],
"source": [
"with client.batch as batch:\n",
" for text in texts:\n",
" batch.add_data_object({\"content\": text}, \"Paragraph\")"
]
},
{
"cell_type": "code",
"execution_count": 14,
"id": "01645d61",
"metadata": {},
"outputs": [],
"source": [
"from langchain.vectorstores.weaviate import Weaviate"
]
},
{
"cell_type": "code",
"execution_count": 15,
"id": "bdd97d29",
"metadata": {},
"outputs": [],
"source": [
"vectorstore = Weaviate(client, \"Paragraph\", \"content\")"
]
},
{
"cell_type": "code",
"execution_count": 16,
"id": "b70c0f98",
"metadata": {},
"outputs": [],
"source": [
"query = \"What did the president say about Ketanji Brown Jackson\"\n",
"docs = vectorstore.similarity_search(query)"
]
},
{
"cell_type": "code",
"execution_count": 17,
"id": "07533e40",
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"In state after state, new laws have been passed, not only to suppress the vote, but to subvert entire elections. \n",
"\n",
"We cannot let this happen. \n",
"\n",
"Tonight. I call on the Senate to: Pass the Freedom to Vote Act. Pass the John Lewis Voting Rights Act. And while you’re at it, pass the Disclose Act so Americans can know who is funding our elections. \n",
"\n",
"Tonight, I’d like to honor someone who has dedicated his life to serve this country: Justice Stephen Breyer—an Army veteran, Constitutional scholar, and retiring Justice of the United States Supreme Court. Justice Breyer, thank you for your service. \n",
"\n",
"One of the most serious constitutional responsibilities a President has is nominating someone to serve on the United States Supreme Court. \n",
"\n",
"And I did that 4 days ago, when I nominated Circuit Court of Appeals Judge Ketanji Brown Jackson. One of our nation’s top legal minds, who will continue Justice Breyer’s legacy of excellence. \n"
]
}
],
"source": [
"print(docs[0].page_content)"
]
},
{
"cell_type": "markdown",
"id": "007f3102",
"metadata": {},
"source": [
"## Pinecone"
]
},
{
"cell_type": "code",
"execution_count": 5,
"id": "7f6047e5",
"metadata": {},
"outputs": [],
"source": [
"import pinecone \n",
"\n",
"# initialize pinecone\n",
"pinecone.init(api_key=\"\", environment=\"us-west1-gcp\")\n",
"\n",
"index_name = \"langchain-demo\"\n",
"\n",
"docsearch = Pinecone.from_texts(texts, embeddings, index_name=index_name)\n",
"\n",
"query = \"What did the president say about Ketanji Brown Jackson\"\n",
"docs = docsearch.similarity_search(query)"
]
},
{
"cell_type": "code",
"execution_count": 7,
"id": "8e81f1f0",
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"Document(page_content='A former top litigator in private practice. A former federal public defender. And from a family of public school educators and police officers. A consensus builder. Since she’s been nominated, she’s received a broad range of support—from the Fraternal Order of Police to former judges appointed by Democrats and Republicans. \\n\\nAnd if we are to advance liberty and justice, we need to secure the Border and fix the immigration system. \\n\\nWe can do both. At our border, we’ve installed new technology like cutting-edge scanners to better detect drug smuggling. \\n\\nWe’ve set up joint patrols with Mexico and Guatemala to catch more human traffickers. \\n\\nWe’re putting in place dedicated immigration judges so families fleeing persecution and violence can have their cases heard faster. \\n\\nWe’re securing commitments and supporting partners in South and Central America to host more refugees and secure their own borders. ', lookup_str='', metadata={}, lookup_index=0)"
]
},
"execution_count": 7,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"docs[0]"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "e7d74bd2",
"metadata": {},
"outputs": [],
"source": []
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3 (ipykernel)",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.10.9"
}
},
"nbformat": 4,
"nbformat_minor": 5
}

21
docs/modules/utils/combine_docs_how_to.rst Normal file
View File

@@ -0,0 +1,21 @@
Utilities for working with Documents
====================================
There are a lot of different utilities that LangChain provides integrations for
These guides go over how to use them.
The utilities here are all utilities that make it easier to work with documents.
`Text Splitters <combine_docs_examples/textsplitter.html>`_: A walkthrough of how to split large documents up into smaller, more manageable pieces of text.
`VectorStores <combine_docs_examples/vectorstores.html>`_: A walkthrough of vectorstore functionalities, and different types of vectorstores, that LangChain supports.
`Embeddings <combine_docs_examples/embeddings.html>`_: A walkthrough of embedding functionalities, and different types of embeddings, that LangChain supports.
`HyDE <combine_docs_examples/hyde.html>`_: How to use Hypothetical Document Embeddings, a novel way of constructing embeddings for document retrieval systems.
.. toctree::
:maxdepth: 1
:glob:
:hidden:
combine_docs_examples/*

85
docs/modules/utils/examples/bash.ipynb Normal file
View File

@@ -0,0 +1,85 @@
{
"cells": [
{
"cell_type": "markdown",
"id": "8f210ec3",
"metadata": {},
"source": [
"# Bash\n",
"It can often be useful to have an LLM generate bash commands, and then run them. A common use case this is for letting it interact with your local file system. We provide an easy util to execute bash commands."
]
},
{
"cell_type": "code",
"execution_count": 1,
"id": "f7b3767b",
"metadata": {},
"outputs": [],
"source": [
"from langchain.utilities import BashProcess"
]
},
{
"cell_type": "code",
"execution_count": 2,
"id": "cf1c92f0",
"metadata": {},
"outputs": [],
"source": [
"bash = BashProcess()"
]
},
{
"cell_type": "code",
"execution_count": 3,
"id": "2fa952fc",
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"bash.ipynb\n",
"google_search.ipynb\n",
"python.ipynb\n",
"requests.ipynb\n",
"serpapi.ipynb\n",
"\n"
]
}
],
"source": [
"print(bash.run(\"ls\"))"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "851fee9f",
"metadata": {},
"outputs": [],
"source": []
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3 (ipykernel)",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.10.9"
}
},
"nbformat": 4,
"nbformat_minor": 5
}

100
docs/modules/utils/examples/google_search.ipynb Normal file
View File

@@ -0,0 +1,100 @@
{
"cells": [
{
"cell_type": "markdown",
"id": "245a954a",
"metadata": {},
"source": [
"# Google Search\n",
"\n",
"This notebook goes over how to use the google search component.\n",
"\n",
"First, you need to set up the proper API keys and environment variables. To set it up, follow the instructions found [here](https://stackoverflow.com/questions/37083058/programmatically-searching-google-in-python-using-custom-search).\n",
"\n",
"Then we will need to set some environment variables."
]
},
{
"cell_type": "code",
"execution_count": 3,
"id": "34bb5968",
"metadata": {},
"outputs": [],
"source": [
"import os\n",
"os.environ[\"GOOGLE_CSE_ID\"] = \n",
"os.environ[\"GOOGLE_API_KEY\"] = "
]
},
{
"cell_type": "code",
"execution_count": 4,
"id": "ac4910f8",
"metadata": {},
"outputs": [],
"source": [
"from langchain.utilities import GoogleSearchAPIWrapper"
]
},
{
"cell_type": "code",
"execution_count": 5,
"id": "84b8f773",
"metadata": {},
"outputs": [],
"source": [
"search = GoogleSearchAPIWrapper()"
]
},
{
"cell_type": "code",
"execution_count": 7,
"id": "068991a6",
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"'STATE OF HAWAII. 1 Child\\'s First Name. (Type or print). 2. Sex. BARACK. 3. This Birth. CERTIFICATE OF LIVE BIRTH. FILE. NUMBER 151 le. lb. Middle Name. Barack Hussein Obama II is an American politician who served as the 44th president of the United States from 2009 to 2017. A member of the Democratic Party,\\xa0... First Lady Michelle LaVaughn Robinson Obama is a lawyer, writer, and the wife of the 44th President, Barack Obama. She is the first African-American First\\xa0... Barack Obama, in full Barack Hussein Obama II, (born August 4, 1961, Honolulu, Hawaii, U.S.), 44th president of the United States (2009–17) and the first\\xa0... Aug 18, 2017 ... It took him several seconds and multiple clues to remember former President Barack Obama\\'s first name. Miller knew that every answer had to\\xa0... Feb 9, 2015 ... Michael Jordan misspelled Barack Obama\\'s first name on 50th-birthday gift ... Knowing Obama is a Chicagoan and huge basketball fan,\\xa0... His full name is Barack Hussein Obama II. Since the “II” is simply because he was named for his father, his last name is Obama. Jan 16, 2007 ... 4, 1961, in Honolulu. His first name means \"one who is blessed\" in Swahili. While Obama\\'s father, Barack Hussein Obama Sr., was from Kenya, his\\xa0... Jan 19, 2017 ... Hopeful parents named their sons for the first Black president, whose name is a variation of the Hebrew name Baruch, which means “blessed”\\xa0... Feb 27, 2020 ... President Barack Obama was born Barack Hussein Obama, II, as shown here on his birth certificate here . As reported by Reuters here , his\\xa0...'"
]
},
"execution_count": 7,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"search.run(\"Obama's first name?\")"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "028f4cba",
"metadata": {},
"outputs": [],
"source": []
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3 (ipykernel)",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.10.9"
}
},
"nbformat": 4,
"nbformat_minor": 5
}

86
docs/modules/utils/examples/python.ipynb Normal file
View File

@@ -0,0 +1,86 @@
{
"cells": [
{
"cell_type": "markdown",
"id": "984a8fca",
"metadata": {},
"source": [
"# Python REPL\n",
"\n",
"Sometimes, for complex calculations, rather than have an LLM generate the answer directly, it can be better to have the LLM generate code to calculate the answer, and then run that code to get the answer. In order to easily do that, we provide a simple Python REPL to execute commands in.\n",
"\n",
"This interface will only return things that are printed - therefor, if you want to use it to calculate an answer, make sure to have it print out the answer."
]
},
{
"cell_type": "code",
"execution_count": 1,
"id": "f6593089",
"metadata": {},
"outputs": [],
"source": [
"from langchain.utilities import PythonREPL"
]
},
{
"cell_type": "code",
"execution_count": 2,
"id": "6f21f0a4",
"metadata": {},
"outputs": [],
"source": [
"python_repl = PythonREPL()"
]
},
{
"cell_type": "code",
"execution_count": 3,
"id": "7ebbbaea",
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"'2\\n'"
]
},
"execution_count": 3,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"python_repl.run(\"print(1+1)\")"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "54fc1f03",
"metadata": {},
"outputs": [],
"source": []
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3 (ipykernel)",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.10.9"
}
},
"nbformat": 4,
"nbformat_minor": 5
}

84
docs/modules/utils/examples/requests.ipynb Normal file
View File

File diff suppressed because one or more lines are too long

84
docs/modules/utils/examples/serpapi.ipynb Normal file
View File

@@ -0,0 +1,84 @@
{
"cells": [
{
"cell_type": "markdown",
"id": "dc23c48e",
"metadata": {},
"source": [
"# SerpAPI\n",
"\n",
"This notebook goes over how to use the SerpAPI component to search the web."
]
},
{
"cell_type": "code",
"execution_count": 1,
"id": "54bf5afd",
"metadata": {},
"outputs": [],
"source": [
"from langchain.utilities import SerpAPIWrapper"
]
},
{
"cell_type": "code",
"execution_count": 2,
"id": "31f8f382",
"metadata": {},
"outputs": [],
"source": [
"search = SerpAPIWrapper()"
]
},
{
"cell_type": "code",
"execution_count": 3,
"id": "25ce0225",
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"'Barack Hussein Obama II'"
]
},
"execution_count": 3,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"search.run(\"Obama's first name?\")"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "27f5959a",
"metadata": {},
"outputs": [],
"source": []
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3 (ipykernel)",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.10.9"
}
},
"nbformat": 4,
"nbformat_minor": 5
}

24
docs/modules/utils/generic_how_to.rst Normal file
View File

@@ -0,0 +1,24 @@
Generic Utilities
=================
There are a lot of different utilities that LangChain provides integrations for
These guides go over how to use them.
The utilities listed here are all generic utilities.
`Bash <examples/bash.html>`_: How to use a bash wrapper to execute bash commands.
`Python REPL <examples/python.html>`_: How to use a Python wrapper to execute python commands.
`Requests <examples/requests.html>`_: How to use a requests wrapper to interact with the web.
`Google Search <examples/google_search.html>`_: How to use the google search wrapper to search the web.
`SerpAPI <examples/serpapi.html>`_: How to use the SerpAPI wrapper to search the web.
.. toctree::
:maxdepth: 1
:glob:
:hidden:
examples/*

17
docs/modules/utils/how_to_guides.rst Normal file
View File

@@ -0,0 +1,17 @@
How-To Guides
=============
There are a lot of different utilities that LangChain provides integrations for
These guides go over how to use them.
These can largely be grouped into two categories:
1. `Generic Utilities <generic_how_to.html>`_: Generic utilities, including search, python REPLs, etc.
2. `Utilities for working with Documents <combine_docs_how_to.html>`_: Utilities aimed at making it easy to work with documents (text splitting, embeddings, vectorstores, etc).
.. toctree::
:maxdepth: 1
:glob:
:hidden:
generic_how_to.rst
combine_docs_how_to.rst

38
docs/modules/utils/key_concepts.md Normal file
View File

@@ -0,0 +1,38 @@
# Key Concepts
## Text Splitter
This class is responsible for splitting long pieces of text into smaller components.
It contains different ways for splitting text (on characters, using Spacy, etc)
as well as different ways for measuring length (token based, character based, etc).
## Embeddings
These classes are very similar to the LLM classes in that they are wrappers around models,
but rather than return a string they return an embedding (list of floats). These are particularly useful when
implementing semantic search functionality. They expose separate methods for embedding queries versus embedding documents.
## Vectorstores
These are datastores that store embeddings of documents in vector form.
They expose a method for passing in a string and finding similar documents.
## Python REPL
Sometimes, for complex calculations, rather than have an LLM generate the answer directly,
it can be better to have the LLM generate code to calculate the answer, and then run that code to get the answer.
In order to easily do that, we provide a simple Python REPL to execute commands in.
This interface will only return things that are printed -
therefor, if you want to use it to calculate an answer, make sure to have it print out the answer.
## Bash
It can often be useful to have an LLM generate bash commands, and then run them.
A common use case this is for letting it interact with your local file system.
We provide an easy component to execute bash commands.
## Requests Wrapper
The web contains a lot of information that LLMs do not have access to.
In order to easily let LLMs interact with that information,
we provide a wrapper around the Python Requests module that takes in a URL and fetches data from that URL.
## Google Search
This uses the official Google Search API to look up information on the web.
## SerpAPI
This uses SerpAPI, a third party search API engine, to interact with Google Search.