github/langchain
1
0
Fork 0
mirror of https://github.com/hwchase17/langchain.git synced 2025-09-07 22:11:51 +00:00
Code Issues Packages Projects Releases Wiki Activity

openai[patch]: Explicitly support embedding dimensions (#16596)

Browse Source
This commit is contained in:
Bagatur
2024-01-25 15:16:04 -08:00
committed by GitHub
parent 5df8ab574e
commit 61e876aad8
9 changed files with 208 additions and 176 deletions
Download Patch File Download Diff File Expand all files Collapse all files

300
docs/docs/integrations/text_embedding/openai.ipynb
View File

@@ -10,9 +10,42 @@
"Let's load the OpenAI Embedding class."
]
},
{
"cell_type": "markdown",
"id": "40ff98ff-58e9-4716-8788-227a5c3f473d",
"metadata": {},
"source": [
"## Setup\n",
"\n",
"First we install langchain-openai and set the required env vars"
]
},
{
"cell_type": "code",
"execution_count": 6,
"execution_count": null,
"id": "c66c4613-6c67-40ca-b3b1-c026750d1742",
"metadata": {},
"outputs": [],
"source": [
"%pip install -qU langchain-openai"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "62e3710e-55a0-44fb-ba51-2f1d520dfc38",
"metadata": {},
"outputs": [],
"source": [
"import getpass\n",
"import os\n",
"\n",
"os.environ[\"OPENAI_API_KEY\"] = getpass.getpass()"
]
},
{
"cell_type": "code",
"execution_count": 1,
"id": "0be1af71",
"metadata": {},
"outputs": [],
@@ -22,17 +55,17 @@
},
{
"cell_type": "code",
"execution_count": 29,
"execution_count": 5,
"id": "2c66e5da",
"metadata": {},
"outputs": [],
"source": [
"embeddings = OpenAIEmbeddings()"
"embeddings = OpenAIEmbeddings(model=\"text-embedding-3-large\")"
]
},
{
"cell_type": "code",
"execution_count": 30,
"execution_count": 6,
"id": "01370375",
"metadata": {},
"outputs": [],
@@ -40,33 +73,50 @@
"text = \"This is a test document.\""
]
},
{
"cell_type": "markdown",
"id": "f012c222-3fa9-470a-935c-758b2048d9af",
"metadata": {},
"source": [
"## Usage\n",
"### Embed query"
]
},
{
"cell_type": "code",
"execution_count": 31,
"execution_count": 7,
"id": "bfb6142c",
"metadata": {},
"outputs": [],
"outputs": [
{
"name": "stderr",
"output_type": "stream",
"text": [
"Warning: model not found. Using cl100k_base encoding.\n"
]
}
],
"source": [
"query_result = embeddings.embed_query(text)"
]
},
{
"cell_type": "code",
"execution_count": 32,
"execution_count": 8,
"id": "91bc875d-829b-4c3d-8e6f-fc2dda30a3bd",
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"[-0.003186025367556387,\n",
" 0.011071979803637493,\n",
" -0.004020420763285827,\n",
" -0.011658221276953042,\n",
" -0.0010534035786864363]"
"[-0.014380056377383358,\n",
" -0.027191711627651764,\n",
" -0.020042716111860304,\n",
" 0.057301379620345545,\n",
" -0.022267658631828974]"
]
},
"execution_count": 32,
"execution_count": 8,
"metadata": {},
"output_type": "execute_result"
}
@@ -75,33 +125,49 @@
"query_result[:5]"
]
},
{
"cell_type": "markdown",
"id": "6b733391-1e23-438b-a6bc-0d77eed9426e",
"metadata": {},
"source": [
"## Embed documents"
]
},
{
"cell_type": "code",
"execution_count": 33,
"execution_count": 9,
"id": "0356c3b7",
"metadata": {},
"outputs": [],
"outputs": [
{
"name": "stderr",
"output_type": "stream",
"text": [
"Warning: model not found. Using cl100k_base encoding.\n"
]
}
],
"source": [
"doc_result = embeddings.embed_documents([text])"
]
},
{
"cell_type": "code",
"execution_count": 34,
"execution_count": 10,
"id": "a4b0d49e-0c73-44b6-aed5-5b426564e085",
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"[-0.003186025367556387,\n",
" 0.011071979803637493,\n",
" -0.004020420763285827,\n",
" -0.011658221276953042,\n",
" -0.0010534035786864363]"
"[-0.014380056377383358,\n",
" -0.027191711627651764,\n",
" -0.020042716111860304,\n",
" 0.057301379620345545,\n",
" -0.022267658631828974]"
]
},
"execution_count": 34,
"execution_count": 10,
"metadata": {},
"output_type": "execute_result"
}
@@ -112,131 +178,87 @@
},
{
"cell_type": "markdown",
"id": "bb61bbeb",
"id": "e7dc464a-6fa2-4cff-ab2e-49a0566d819b",
"metadata": {},
"source": [
"Let's load the OpenAI Embedding class with first generation models (e.g. text-search-ada-doc-001/text-search-ada-query-001). Note: These are not recommended models - see [here](https://platform.openai.com/docs/guides/embeddings/what-are-embeddings)"
]
},
{
"cell_type": "code",
"execution_count": 1,
"id": "c0b072cc",
"metadata": {},
"outputs": [],
"source": [
"from langchain_openai import OpenAIEmbeddings"
]
},
{
"cell_type": "code",
"execution_count": 23,
"id": "a56b70f5",
"metadata": {},
"outputs": [],
"source": [
"embeddings = OpenAIEmbeddings(model=\"text-embedding-ada-002\")"
]
},
{
"cell_type": "code",
"execution_count": 24,
"id": "14aefb64",
"metadata": {},
"outputs": [],
"source": [
"text = \"This is a test document.\""
]
},
{
"cell_type": "code",
"execution_count": 25,
"id": "3c39ed33",
"metadata": {},
"outputs": [],
"source": [
"query_result = embeddings.embed_query(text)"
]
},
{
"cell_type": "code",
"execution_count": 26,
"id": "2ee7ce9f-d506-4810-8897-e44334412714",
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"[0.004452846988523035,\n",
" 0.034550655976098514,\n",
" -0.015029939040690051,\n",
" 0.03827273883655212,\n",
" 0.005785414075152477]"
]
},
"execution_count": 26,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"query_result[:5]"
]
},
{
"cell_type": "code",
"execution_count": 27,
"id": "e3221db6",
"metadata": {},
"outputs": [],
"source": [
"doc_result = embeddings.embed_documents([text])"
]
},
{
"cell_type": "code",
"execution_count": 28,
"id": "a0865409-3a6d-468f-939f-abde17c7cac3",
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"[0.004452846988523035,\n",
" 0.034550655976098514,\n",
" -0.015029939040690051,\n",
" 0.03827273883655212,\n",
" 0.005785414075152477]"
]
},
"execution_count": 28,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"doc_result[0][:5]"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "aaad49f8",
"metadata": {},
"outputs": [],
"source": [
"import os\n",
"## Specify dimensions\n",
"\n",
"# if you are behind an explicit proxy, you can use the OPENAI_PROXY environment variable to pass through\n",
"os.environ[\"OPENAI_PROXY\"] = \"http://proxy.yourcompany.com:8080\""
"With the `text-embedding-3` class of models, you can specify the size of the embeddings you want returned. For example by default `text-embedding-3-large` returned embeddings of dimension 3072:"
]
},
{
"cell_type": "code",
"execution_count": 11,
"id": "f7be1e7b-54c6-4893-b8ad-b872e6705735",
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"3072"
]
},
"execution_count": 11,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"len(doc_result[0])"
]
},
{
"cell_type": "markdown",
"id": "33287142-0835-4958-962f-385ae4447431",
"metadata": {},
"source": [
"But by passing in `dimensions=1024` we can reduce the size of our embeddings to 1024:"
]
},
{
"cell_type": "code",
"execution_count": 15,
"id": "854ee772-2de9-4a83-84e0-908033d98e4e",
"metadata": {},
"outputs": [],
"source": [
"embeddings_1024 = OpenAIEmbeddings(model=\"text-embedding-3-large\", dimensions=1024)"
]
},
{
"cell_type": "code",
"execution_count": 16,
"id": "3b464396-8d94-478b-8329-849b56e1ae23",
"metadata": {},
"outputs": [
{
"name": "stderr",
"output_type": "stream",
"text": [
"Warning: model not found. Using cl100k_base encoding.\n"
]
},
{
"data": {
"text/plain": [
"1024"
]
},
"execution_count": 16,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"len(embeddings_1024.embed_documents([text])[0])"
]
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3 (ipykernel)",
"display_name": "poetry-venv",
"language": "python",
"name": "python3"
"name": "poetry-venv"
},
"language_info": {
"codemirror_mode": {
@@ -248,7 +270,7 @@
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.10.12"
"version": "3.9.1"
},
"vscode": {
"interpreter": {