mirror of
https://github.com/hwchase17/langchain.git
synced 2025-09-29 23:38:51 +00:00
354 lines
11 KiB
Plaintext
354 lines
11 KiB
Plaintext
{
|
|
"cells": [
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "4f7e423b",
|
|
"metadata": {},
|
|
"source": [
|
|
"# How to select examples from a LangSmith dataset\n",
|
|
"\n",
|
|
"import Prerequisites from \"@theme/Prerequisites\";\n",
|
|
"import Compatibility from \"@theme/Compatibility\";\n",
|
|
"\n",
|
|
"<Prerequisites titlesAndLinks={[\n",
|
|
" [\"Chat models\", \"/docs/concepts/chat_models\"],\n",
|
|
" [\"Few-shot-prompting\", \"/docs/concepts/few-shot-prompting\"],\n",
|
|
" [\"LangSmith\", \"https://docs.smith.langchain.com/\"],\n",
|
|
"]} />\n",
|
|
"\n",
|
|
"\n",
|
|
"<Compatibility packagesAndVersions={[\n",
|
|
" [\"langsmith\", \"0.1.101\"],\n",
|
|
" [\"langchain-core\", \"0.2.34\"],\n",
|
|
"]} />\n",
|
|
"\n",
|
|
"\n",
|
|
"[LangSmith](https://docs.smith.langchain.com/) datasets have built-in support for similarity search, making them a great tool for building and querying few-shot examples.\n",
|
|
"\n",
|
|
"In this guide we'll see how to use an indexed LangSmith dataset as a few-shot example selector.\n",
|
|
"\n",
|
|
"## Setup\n",
|
|
"\n",
|
|
"Before getting started make sure you've [created a LangSmith account](https://smith.langchain.com/) and set your credentials:"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 2,
|
|
"id": "85445e0e",
|
|
"metadata": {},
|
|
"outputs": [
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"Set LangSmith API key:\n",
|
|
"\n",
|
|
"········\n"
|
|
]
|
|
}
|
|
],
|
|
"source": [
|
|
"import getpass\n",
|
|
"import os\n",
|
|
"\n",
|
|
"if not os.environ.get(\"LANGSMITH_API_KEY\"):\n",
|
|
" os.environ[\"LANGSMITH_API_KEY\"] = getpass.getpass(\"Set LangSmith API key:\\n\\n\")\n",
|
|
"\n",
|
|
"os.environ[\"LANGSMITH_TRACING\"] = \"true\""
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "ca899e29",
|
|
"metadata": {},
|
|
"source": [
|
|
"We'll need to install the `langsmith` SDK. In this example we'll also make use of `langchain`, `langchain-openai`, and `langchain-benchmarks`:"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"id": "b4fa7810",
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"%pip install -qU \"langsmith>=0.1.101\" \"langchain-core>=0.2.34\" langchain langchain-openai langchain-benchmarks"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "fc716e12",
|
|
"metadata": {},
|
|
"source": [
|
|
"Now we'll clone a public dataset and turn on indexing for the dataset. We can also turn on indexing via the [LangSmith UI](https://docs.smith.langchain.com/how_to_guides/datasets/index_datasets_for_dynamic_few_shot_example_selection).\n",
|
|
"\n",
|
|
"We'll clone the [Multiverse math few shot example dataset](https://blog.langchain.dev/few-shot-prompting-to-improve-tool-calling-performance/).\n",
|
|
"\n",
|
|
"This enables searching over the dataset and will make sure that anytime we update/add examples they are also indexed."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"id": "cf53d280",
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"from langsmith import Client as LangSmith\n",
|
|
"\n",
|
|
"ls_client = LangSmith()\n",
|
|
"\n",
|
|
"dataset_name = \"multiverse-math-few-shot-examples-v2\"\n",
|
|
"dataset_public_url = (\n",
|
|
" \"https://smith.langchain.com/public/620596ee-570b-4d2b-8c8f-f828adbe5242/d\"\n",
|
|
")\n",
|
|
"\n",
|
|
"ls_client.clone_public_dataset(dataset_public_url)\n",
|
|
"\n",
|
|
"dataset_id = ls_client.read_dataset(dataset_name=dataset_name).id\n",
|
|
"\n",
|
|
"ls_client.index_dataset(dataset_id=dataset_id)"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "5767d171",
|
|
"metadata": {},
|
|
"source": [
|
|
"## Querying dataset\n",
|
|
"\n",
|
|
"Indexing can take a few seconds. Once the dataset is indexed, we can search for similar examples. Note that the input to the `similar_examples` method must have the same schema as the examples inputs. In this case our example inputs are a dictionary with a \"question\" key:"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 12,
|
|
"id": "5013a56f",
|
|
"metadata": {},
|
|
"outputs": [
|
|
{
|
|
"data": {
|
|
"text/plain": [
|
|
"3"
|
|
]
|
|
},
|
|
"execution_count": 12,
|
|
"metadata": {},
|
|
"output_type": "execute_result"
|
|
}
|
|
],
|
|
"source": [
|
|
"examples = ls_client.similar_examples(\n",
|
|
" {\"question\": \"whats the negation of the negation of the negation of 3\"},\n",
|
|
" limit=3,\n",
|
|
" dataset_id=dataset_id,\n",
|
|
")\n",
|
|
"len(examples)"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 13,
|
|
"id": "a142db06",
|
|
"metadata": {},
|
|
"outputs": [
|
|
{
|
|
"data": {
|
|
"text/plain": [
|
|
"'evaluate the negation of -100'"
|
|
]
|
|
},
|
|
"execution_count": 13,
|
|
"metadata": {},
|
|
"output_type": "execute_result"
|
|
}
|
|
],
|
|
"source": [
|
|
"examples[0].inputs[\"question\"]"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "d2627125",
|
|
"metadata": {},
|
|
"source": [
|
|
"For this dataset, the outputs are the conversation that followed the question in OpenAI message format:"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 14,
|
|
"id": "af5b9191",
|
|
"metadata": {},
|
|
"outputs": [
|
|
{
|
|
"data": {
|
|
"text/plain": [
|
|
"[{'role': 'assistant',\n",
|
|
" 'content': None,\n",
|
|
" 'tool_calls': [{'id': 'toolu_01HTpq4cYNUac6F7omUc2Wz3',\n",
|
|
" 'type': 'function',\n",
|
|
" 'function': {'name': 'negate', 'arguments': '{\"a\": -100}'}}]},\n",
|
|
" {'role': 'tool',\n",
|
|
" 'content': '-100.0',\n",
|
|
" 'tool_call_id': 'toolu_01HTpq4cYNUac6F7omUc2Wz3'},\n",
|
|
" {'role': 'assistant', 'content': 'So the answer is 100.'},\n",
|
|
" {'role': 'user',\n",
|
|
" 'content': '100 is incorrect. Please refer to the output of your tool call.'},\n",
|
|
" {'role': 'assistant',\n",
|
|
" 'content': [{'text': \"You're right, my previous answer was incorrect. Let me re-evaluate using the tool output:\",\n",
|
|
" 'type': 'text'}],\n",
|
|
" 'tool_calls': [{'id': 'toolu_01XsJQboYghGDygQpPjJkeRq',\n",
|
|
" 'type': 'function',\n",
|
|
" 'function': {'name': 'negate', 'arguments': '{\"a\": -100}'}}]},\n",
|
|
" {'role': 'tool',\n",
|
|
" 'content': '-100.0',\n",
|
|
" 'tool_call_id': 'toolu_01XsJQboYghGDygQpPjJkeRq'},\n",
|
|
" {'role': 'assistant', 'content': 'The answer is -100.0'},\n",
|
|
" {'role': 'user',\n",
|
|
" 'content': 'You have the correct numerical answer but are returning additional text. Please only respond with the numerical answer.'},\n",
|
|
" {'role': 'assistant', 'content': '-100.0'}]"
|
|
]
|
|
},
|
|
"execution_count": 14,
|
|
"metadata": {},
|
|
"output_type": "execute_result"
|
|
}
|
|
],
|
|
"source": [
|
|
"examples[0].outputs[\"conversation\"]"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "e852c8ef",
|
|
"metadata": {},
|
|
"source": [
|
|
"## Creating dynamic few-shot prompts\n",
|
|
"\n",
|
|
"The search returns the examples whose inputs are most similar to the query input. We can use this for few-shot prompting a model like so:"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 20,
|
|
"id": "12cba1e1",
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"from langchain.chat_models import init_chat_model\n",
|
|
"from langchain_benchmarks.tool_usage.tasks.multiverse_math import (\n",
|
|
" add,\n",
|
|
" cos,\n",
|
|
" divide,\n",
|
|
" log,\n",
|
|
" multiply,\n",
|
|
" negate,\n",
|
|
" pi,\n",
|
|
" power,\n",
|
|
" sin,\n",
|
|
" subtract,\n",
|
|
")\n",
|
|
"from langchain_core.runnables import RunnableLambda\n",
|
|
"from langsmith import AsyncClient as AsyncLangSmith\n",
|
|
"\n",
|
|
"async_ls_client = AsyncLangSmith()\n",
|
|
"\n",
|
|
"\n",
|
|
"def similar_examples(input_: dict) -> dict:\n",
|
|
" examples = ls_client.similar_examples(input_, limit=5, dataset_id=dataset_id)\n",
|
|
" return {**input_, \"examples\": examples}\n",
|
|
"\n",
|
|
"\n",
|
|
"async def asimilar_examples(input_: dict) -> dict:\n",
|
|
" examples = await async_ls_client.similar_examples(\n",
|
|
" input_, limit=5, dataset_id=dataset_id\n",
|
|
" )\n",
|
|
" return {**input_, \"examples\": examples}\n",
|
|
"\n",
|
|
"\n",
|
|
"def construct_prompt(input_: dict) -> list:\n",
|
|
" instructions = \"\"\"You are great at using mathematical tools.\"\"\"\n",
|
|
" examples = []\n",
|
|
" for ex in input_[\"examples\"]:\n",
|
|
" examples.append({\"role\": \"user\", \"content\": ex.inputs[\"question\"]})\n",
|
|
" for msg in ex.outputs[\"conversation\"]:\n",
|
|
" if msg[\"role\"] == \"assistant\":\n",
|
|
" msg[\"name\"] = \"example_assistant\"\n",
|
|
" if msg[\"role\"] == \"user\":\n",
|
|
" msg[\"name\"] = \"example_user\"\n",
|
|
" examples.append(msg)\n",
|
|
" return [\n",
|
|
" {\"role\": \"system\", \"content\": instructions},\n",
|
|
" *examples,\n",
|
|
" {\"role\": \"user\", \"content\": input_[\"question\"]},\n",
|
|
" ]\n",
|
|
"\n",
|
|
"\n",
|
|
"tools = [add, cos, divide, log, multiply, negate, pi, power, sin, subtract]\n",
|
|
"llm = init_chat_model(\"gpt-4o-2024-08-06\")\n",
|
|
"llm_with_tools = llm.bind_tools(tools)\n",
|
|
"\n",
|
|
"example_selector = RunnableLambda(func=similar_examples, afunc=asimilar_examples)\n",
|
|
"\n",
|
|
"chain = example_selector | construct_prompt | llm_with_tools"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 21,
|
|
"id": "c423b367",
|
|
"metadata": {},
|
|
"outputs": [
|
|
{
|
|
"data": {
|
|
"text/plain": [
|
|
"[{'name': 'negate',\n",
|
|
" 'args': {'a': 3},\n",
|
|
" 'id': 'call_uMSdoTl6ehfHh5a6JQUb2NoZ',\n",
|
|
" 'type': 'tool_call'}]"
|
|
]
|
|
},
|
|
"execution_count": 21,
|
|
"metadata": {},
|
|
"output_type": "execute_result"
|
|
}
|
|
],
|
|
"source": [
|
|
"ai_msg = await chain.ainvoke({\"question\": \"whats the negation of the negation of 3\"})\n",
|
|
"ai_msg.tool_calls"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "94489b4a",
|
|
"metadata": {},
|
|
"source": [
|
|
"Looking at the LangSmith trace, we can see that relevant examples were pulled in in the `similar_examples` step and passed as messages to ChatOpenAI: https://smith.langchain.com/public/9585e30f-765a-4ed9-b964-2211420cd2f8/r/fdea98d6-e90f-49d4-ac22-dfd012e9e0d9."
|
|
]
|
|
}
|
|
],
|
|
"metadata": {
|
|
"kernelspec": {
|
|
"display_name": "poetry-venv-311",
|
|
"language": "python",
|
|
"name": "poetry-venv-311"
|
|
},
|
|
"language_info": {
|
|
"codemirror_mode": {
|
|
"name": "ipython",
|
|
"version": 3
|
|
},
|
|
"file_extension": ".py",
|
|
"mimetype": "text/x-python",
|
|
"name": "python",
|
|
"nbconvert_exporter": "python",
|
|
"pygments_lexer": "ipython3",
|
|
"version": "3.11.9"
|
|
}
|
|
},
|
|
"nbformat": 4,
|
|
"nbformat_minor": 5
|
|
}
|