{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "1a4596ea-a631-416d-a2a4-3577c140493d",
   "metadata": {
    "tags": []
   },
   "source": [
    "# Debug, Evaluate, and Monitor LLMs with LangSmith\n",
    "\n",
    "LangChain makes it easy to get started with Agents and other LLM applications. Even so, delivering a high-quality agent to production can be deceptively difficult. To aid the development process, we've designed tracing and callbacks at the core of LangChain. In this notebook, you will get started prototyping, testing, and monitoring an LLM agent.\n",
    "\n",
    "When might you want to use tracing? Some situations we've found it useful include:\n",
    "- Quickly debugging a new chain, agent, or set of tools\n",
    "- Evaluating a given chain across different LLMs or Chat Models to compare results or improve prompts\n",
    "- Running a given chain multiple time on a dataset to ensure it consistently meets a quality bar.\n",
    "- Capturing production traces and using LangChain summarizers to analyze app usage"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "138fbb8f-960d-4d26-9dd5-6d6acab3ee55",
   "metadata": {},
   "source": [
    "## Prerequisites\n",
    "\n",
    "**Either [create a hosted LangSmith account](https://www.langchain.plus/) and connect with an API key OR\n",
    "run the server locally.**\n",
    "\n",
    "\n",
    "To run the local server, execute the following comand in your terminal:\n",
    "```\n",
    "pip install --upgrade langchain\n",
    "langchain plus start\n",
    "```\n",
    "\n",
    "Now, let's get started by creating a client to connect to LangChain+."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2d77d064-41b4-41fb-82e6-2d16461269ec",
   "metadata": {
    "tags": []
   },
   "source": [
    "## Debug your Agent\n",
    "\n",
    "First, configure your environment variables to tell LangChain to log traces. This is done by setting the `LANGCHAIN_TRACING_V2` environment variable to true.\n",
    "You can tell LangChain which project to log to by setting the `LANGCHAIN_PROJECT` environment variable. This will automatically create a debug project for you.\n",
    "\n",
    "For more information on other ways to set up tracing, please reference the [LangSmith documentation](https://docs.langchain.plus/docs/)\n",
    "\n",
    "**NOTE:** You must also set your `OPENAI_API_KEY` and `SERPAPI_API_KEY` environment variables in order to run the following tutorial.\n",
    "\n",
    "**NOTE:** You can optionally set the `LANGCHAIN_ENDPOINT` and `LANGCHAIN_API_KEY` environment variables if using the hosted version which is in private beta."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "904db9a5-f387-4a57-914c-c8af8d39e249",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "You can click the link below to view the UI\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<a href=\"https://dev.langchain.plus\", target=\"_blank\" rel=\"noopener\">LangChain+ Client</a>"
      ],
      "text/plain": [
       "LangChainPlusClient (API URL: https://dev.api.langchain.plus)"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import os\n",
    "from uuid import uuid4\n",
    "from langchainplus_sdk import LangChainPlusClient\n",
    "\n",
    "unique_id = uuid4().hex[0:8]\n",
    "os.environ[\"LANGCHAIN_TRACING_V2\"] = \"true\"\n",
    "os.environ[\"LANGCHAIN_PROJECT\"] = f\"Tracing Walkthrough - {unique_id}\"\n",
    "# os.environ[\"LANGCHAIN_ENDPOINT\"] = \"https://api.langchain.plus\"  # Uncomment this line to use the hosted version\n",
    "# os.environ[\"LANGCHAIN_API_KEY\"] = \"<YOUR-LANGCHAINPLUS-API-KEY>\"  # Uncomment this line to use the hosted version.\n",
    "\n",
    "# Used by the agent below\n",
    "# os.environ[\"OPENAI_API_KEY\"] = \"<YOUR-OPENAI-API-KEY>\"\n",
    "# os.environ[\"SERPAPI_API_KEY\"] = \"<YOUR-SERPAPI-API-KEY>\"\n",
    "\n",
    "client = LangChainPlusClient()\n",
    "print(\"You can click the link below to view the UI\")\n",
    "client"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ca27fa11-ddce-4af0-971e-c5c37d5b92ef",
   "metadata": {},
   "source": [
    "Now, start prototyping your agent. We will use a straightforward math example."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "7c801853-8e96-404d-984c-51ace59cbbef",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "from langchain.chat_models import ChatOpenAI\n",
    "from langchain.agents import initialize_agent, load_tools\n",
    "from langchain.agents import AgentType\n",
    "\n",
    "llm = ChatOpenAI(temperature=0)\n",
    "tools = load_tools([\"serpapi\", \"llm-math\"], llm=llm)\n",
    "agent = initialize_agent(\n",
    "    tools, llm, agent=AgentType.ZERO_SHOT_REACT_DESCRIPTION, verbose=False\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "19537902-b95c-4390-80a4-f6c9a937081e",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "import asyncio\n",
    "\n",
    "inputs = [\n",
    "    \"How many people live in canada as of 2023?\",\n",
    "    \"who is dua lipa's boyfriend? what is his age raised to the .43 power?\",\n",
    "    \"what is dua lipa's boyfriend age raised to the .43 power?\",\n",
    "    \"how far is it from paris to boston in miles\",\n",
    "    \"what was the total number of points scored in the 2023 super bowl? what is that number raised to the .23 power?\",\n",
    "    \"what was the total number of points scored in the 2023 super bowl raised to the .23 power?\",\n",
    "    \"how many more points were scored in the 2023 super bowl than in the 2022 super bowl?\",\n",
    "    \"what is 153 raised to .1312 power?\",\n",
    "    \"who is kendall jenner's boyfriend? what is his height (in inches) raised to .13 power?\",\n",
    "    \"what is 1213 divided by 4345?\",\n",
    "]\n",
    "results = []\n",
    "\n",
    "\n",
    "async def arun(agent, input_example):\n",
    "    try:\n",
    "        return await agent.arun(input_example)\n",
    "    except Exception as e:\n",
    "        # The agent sometimes makes mistakes! These will be captured by the tracing.\n",
    "        return e\n",
    "\n",
    "\n",
    "for input_example in inputs:\n",
    "    results.append(arun(agent, input_example))\n",
    "results = await asyncio.gather(*results)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "0405ff30-21fe-413d-85cf-9fa3c649efec",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "from langchain.callbacks.tracers.langchain import wait_for_all_tracers\n",
    "\n",
    "# Logs are submitted in a background thread. Make sure they've been submitted before moving on.\n",
    "wait_for_all_tracers()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9decb964-be07-4b6c-9802-9825c8be7b64",
   "metadata": {},
   "source": [
    "Assuming you've successfully initiated the server as described earlier, your agent logs should show up in your server. You can check by clicking on the link below:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "b7bc3934-bb1a-452c-a723-f9cdb0b416f9",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<a href=\"https://dev.langchain.plus\", target=\"_blank\" rel=\"noopener\">LangChain+ Client</a>"
      ],
      "text/plain": [
       "LangChainPlusClient (API URL: https://dev.api.langchain.plus)"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "client"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6c43c311-4e09-4d57-9ef3-13afb96ff430",
   "metadata": {},
   "source": [
    "## Test\n",
    "\n",
    "Once you've debugged a prototype of your agent, you will want to create tests and benchmark evaluations as you think about putting it into a production environment.\n",
    "\n",
    "In this notebook, you will run evaluators to test an agent. You will do so in a few steps:\n",
    "\n",
    "1. Create a dataset\n",
    "2. Select or create evaluators to measure performance\n",
    "3. Define the LLM or Chain initializer to test\n",
    "4. Run the chain and evaluators using the helper functions"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "beab1a29-b79d-4a99-b5b1-0870c2d772b1",
   "metadata": {},
   "source": [
    "### 1. Create Dataset\n",
    "\n",
    "Below, use the client to create a dataset from the Agent runs you just logged while debugging above. You will use these later to measure performance.\n",
    "\n",
    "For more information on datasets, including how to create them from CSVs or other files or how to create them in the web app, please refer to the [LangSmith documentation](https://docs.langchain.plus/docs)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "d14a9881-2a01-404c-8c56-0b78565c3ff4",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "dataset_name = \"calculator-example-dataset\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "17580c4b-bd04-4dde-9d21-9d4edd25b00d",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "if dataset_name in set([dataset.name for dataset in client.list_datasets()]):\n",
    "    client.delete_dataset(dataset_name=dataset_name)\n",
    "dataset = client.create_dataset(\n",
    "    dataset_name, description=\"A calculator example dataset\"\n",
    ")\n",
    "\n",
    "runs = client.list_runs(\n",
    "    project_name=os.environ[\"LANGCHAIN_PROJECT\"],\n",
    "    execution_order=1,  # Only return the top-level runs\n",
    "    error=False,  # Only runs that succeed\n",
    ")\n",
    "for run in runs:\n",
    "    client.create_example(inputs=run.inputs, outputs=run.outputs, dataset_id=dataset.id)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "92e8944f-e6fc-4bdf-9611-b2db39698cbe",
   "metadata": {},
   "source": [
    "### 2. Select RunEvaluators\n",
    "\n",
    "Manually comparing the results of chains in the UI is effective, but it can be time consuming.\n",
    "It's easier to leverage AI-assisted feedback to evaluate your agent's performance.\n",
    "\n",
    "Below, we will create some pre-implemented run evaluators that do the following:\n",
    "- Compare results against ground truth labels. (You used the debug outputs above for this)\n",
    "- Evaluate the overall agent trajectory based on the tool usage and intermediate steps.\n",
    "- Evaluating 'aspects' of the agent's response in a reference-free manner using custom criteria\n",
    "- Evaluating performance based on 'context' such as retrieved documents or tool results.\n",
    "\n",
    "For a longer discussion of how to select an appropriate evaluator for your use case and how to create your own\n",
    "custom evaluators, please refer to the [LangSmith documentation](https://docs.langchain.plus/docs/).\n",
    "\n",
    "Below, create the run evaluators.\n",
    "\n",
    "**Note: the feedback API is currently experimental and subject to change.**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "56298faa-9ff2-43a2-b35a-ee306e3bf64d",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "from langchain.evaluation.run_evaluators import (\n",
    "    get_qa_evaluator,\n",
    "    get_criteria_evaluator,\n",
    "    get_trajectory_evaluator,\n",
    ")\n",
    "from langchain.chat_models import ChatOpenAI\n",
    "\n",
    "# You can use any model, but stronger llms tend to be more reliable\n",
    "eval_llm = ChatOpenAI(model=\"gpt-4\", temperature=0)\n",
    "\n",
    "# Measures accuracy against ground truth\n",
    "qa_evaluator = get_qa_evaluator(eval_llm)\n",
    "\n",
    "# Measures how effective and efficient the agent's actions are\n",
    "tools = load_tools([\"serpapi\", \"llm-math\"], llm=llm)\n",
    "trajectory_evaluator = get_trajectory_evaluator(eval_llm, agent_tools=tools)\n",
    "\n",
    "# Measure helpfulness. We have some pre-defined criteria you can select\n",
    "helpfulness_evaluator = get_criteria_evaluator(\n",
    "    eval_llm,\n",
    "    \"helpfulness\",\n",
    ")\n",
    "\n",
    "# Custom criteria are specified as a dictionary\n",
    "custom_criteria_evaluator = get_criteria_evaluator(\n",
    "    eval_llm,\n",
    "    {\n",
    "        \"fifth-grader-score\": \"Do you have to be smarter than a fifth grader to answer this question?\"\n",
    "    },\n",
    ")\n",
    "\n",
    "evaluators = [\n",
    "    qa_evaluator,\n",
    "    trajectory_evaluator,\n",
    "    helpfulness_evaluator,\n",
    "    custom_criteria_evaluator,\n",
    "]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8adfd29c-b258-49e5-94b4-74597a12ba16",
   "metadata": {
    "tags": []
   },
   "source": [
    "### 3. Define the Agent or LLM to Test\n",
    "\n",
    "You can evaluate any LLM or chain. Since chains can have memory, we need to pass an\n",
    "initializer function that returns a new chain for each row.\n",
    "\n",
    "In this case, you will test an agent that uses OpenAI's function calling endpoints, but it can be any simple chain."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "f42d8ecc-d46a-448b-a89c-04b0f6907f75",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "from langchain.chat_models import ChatOpenAI\n",
    "from langchain.agents import initialize_agent, load_tools\n",
    "from langchain.agents import AgentType\n",
    "\n",
    "llm = ChatOpenAI(model=\"gpt-3.5-turbo-0613\", temperature=0)\n",
    "tools = load_tools([\"serpapi\", \"llm-math\"], llm=llm)\n",
    "\n",
    "\n",
    "# Since chains can be stateful (e.g. they can have memory), we need provide\n",
    "# a way to initialize a new chain for each row in the dataset. This is done\n",
    "# by passing in a factory function that returns a new chain for each row.\n",
    "def agent_factory():\n",
    "    return initialize_agent(tools, llm, agent=AgentType.OPENAI_FUNCTIONS, verbose=False)\n",
    "\n",
    "\n",
    "# If your chain is NOT stateful, your factory can return the object directly\n",
    "# to improve runtime performance. For example:\n",
    "# chain_factory = lambda: agent"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "07885b10",
   "metadata": {
    "tags": []
   },
   "source": [
    "### 4. Run the Agent and Evaluators\n",
    "\n",
    "With the dataset, agent, and evaluators selected, you can use the helper function below to run them all.\n",
    "\n",
    "The run traces and evaluation feedback will automatically be associated with the dataset for easy attribution and analysis."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "3733269b-8085-4644-9d5d-baedcff13a2f",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "\u001b[0;31mSignature:\u001b[0m\n",
       "\u001b[0marun_on_dataset\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m    \u001b[0mdataset_name\u001b[0m\u001b[0;34m:\u001b[0m \u001b[0;34m'str'\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m    \u001b[0mllm_or_chain_factory\u001b[0m\u001b[0;34m:\u001b[0m \u001b[0;34m'MODEL_OR_CHAIN_FACTORY'\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m    \u001b[0;34m*\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m    \u001b[0mconcurrency_level\u001b[0m\u001b[0;34m:\u001b[0m \u001b[0;34m'int'\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;36m5\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m    \u001b[0mnum_repetitions\u001b[0m\u001b[0;34m:\u001b[0m \u001b[0;34m'int'\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;36m1\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m    \u001b[0mproject_name\u001b[0m\u001b[0;34m:\u001b[0m \u001b[0;34m'Optional[str]'\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;32mNone\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m    \u001b[0mverbose\u001b[0m\u001b[0;34m:\u001b[0m \u001b[0;34m'bool'\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;32mFalse\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m    \u001b[0mclient\u001b[0m\u001b[0;34m:\u001b[0m \u001b[0;34m'Optional[LangChainPlusClient]'\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;32mNone\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m    \u001b[0mtags\u001b[0m\u001b[0;34m:\u001b[0m \u001b[0;34m'Optional[List[str]]'\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;32mNone\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m    \u001b[0mrun_evaluators\u001b[0m\u001b[0;34m:\u001b[0m \u001b[0;34m'Optional[Sequence[RunEvaluator]]'\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;32mNone\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m\u001b[0;34m)\u001b[0m \u001b[0;34m->\u001b[0m \u001b[0;34m'Dict[str, Any]'\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
       "\u001b[0;31mDocstring:\u001b[0m\n",
       "Asynchronously run the Chain or language model on a dataset\n",
       "and store traces to the specified project name.\n",
       "\n",
       "Args:\n",
       "    dataset_name: Name of the dataset to run the chain on.\n",
       "    llm_or_chain_factory: Language model or Chain constructor to run\n",
       "        over the dataset. The Chain constructor is used to permit\n",
       "        independent calls on each example without carrying over state.\n",
       "    concurrency_level: The number of async tasks to run concurrently.\n",
       "    num_repetitions: Number of times to run the model on each example.\n",
       "        This is useful when testing success rates or generating confidence\n",
       "        intervals.\n",
       "    project_name: Name of the project to store the traces in.\n",
       "        Defaults to {dataset_name}-{chain class name}-{datetime}.\n",
       "    verbose: Whether to print progress.\n",
       "    client: Client to use to read the dataset. If not provided, a new\n",
       "        client will be created using the credentials in the environment.\n",
       "    tags: Tags to add to each run in the project.\n",
       "    run_evaluators: Evaluators to run on the results of the chain.\n",
       "\n",
       "Returns:\n",
       "    A dictionary containing the run's project name and the resulting model outputs.\n",
       "\u001b[0;31mFile:\u001b[0m      ~/code/lc/lckg/langchain/client/runner_utils.py\n",
       "\u001b[0;31mType:\u001b[0m      function"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "from langchain.client import (\n",
    "    arun_on_dataset,\n",
    "    run_on_dataset,  # Available if your chain doesn't support async calls.\n",
    ")\n",
    "\n",
    "?arun_on_dataset"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "a8088b7d-3ab6-4279-94c8-5116fe7cee33",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Processed examples: 6\r"
     ]
    }
   ],
   "source": [
    "chain_results = await arun_on_dataset(\n",
    "    dataset_name=dataset_name,\n",
    "    llm_or_chain_factory=agent_factory,\n",
    "    concurrency_level=5,  # Optional, sets the number of examples to run at a time\n",
    "    verbose=True,\n",
    "    client=client,\n",
    "    tags=[\n",
    "        \"testing-notebook\",\n",
    "    ],  # Optional, adds a tag to the resulting chain runs\n",
    "    run_evaluators=evaluators,\n",
    ")\n",
    "\n",
    "# Sometimes, the agent will error due to parsing issues, incompatible tool inputs, etc.\n",
    "# These are logged as warnings here and captured as errors in the tracing UI."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cdacd159-eb4d-49e9-bb2a-c55322c40ed4",
   "metadata": {
    "tags": []
   },
   "source": [
    "### Review the Test Results\n",
    "\n",
    "You can review the test results tracing UI below by navigating to the Testing project \n",
    "with the title that starts with **\"calculator-example-dataset-AgentExecutor-\"**\n",
    "\n",
    "This will show the new runs and the feedback logged from the selected evaluators."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "id": "136db492-d6ca-4215-96f9-439c23538241",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<a href=\"https://dev.langchain.plus\", target=\"_blank\" rel=\"noopener\">LangChain+ Client</a>"
      ],
      "text/plain": [
       "LangChainPlusClient (API URL: https://dev.api.langchain.plus)"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# You can navigate to the UI by clicking on the link below\n",
    "client"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5f2c0539-09c1-42f9-a2ee-6a88a378d479",
   "metadata": {
    "tags": []
   },
   "source": [
    "For a real production application, you will want to add many more test cases and\n",
    "incorporate larger datasets to run benchmark evaluations to measure aggregate performance\n",
    "across. For more information on recommended ways to do this, see [LangSmith Documentation](https://docs.langchain.plus/docs/)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cd67201c-8dc1-4689-981c-759800749e25",
   "metadata": {},
   "source": [
    "## Monitor\n",
    "\n",
    "Once your agent passed the selected quality bar, you can deploy it to production. For this notebook, you will simulate user interactions directly while logging your traces to LangSmith for monitoring.\n",
    "\n",
    "For more information on real production deployments, check out the [LangChain documentation](https://python.langchain.com/docs/guides/deployments/) or contact us at [support@langchain.dev](mailto:support@langchain.dev).\n",
    "\n",
    "**First, create a new project to use in your production deployment.**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "id": "3718710f-f719-4861-a351-0bb9d639d9fd",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "deployment_name = f\"Search + Calculator Deployment - {unique_id}\"\n",
    "project = client.create_project(deployment_name, mode=\"monitor\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3a993ae7-6d26-495a-8633-64936bf94127",
   "metadata": {
    "tags": []
   },
   "source": [
    "**Then, deploy your agent to production, making sure to configure the environment to log to the monitoring project.**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "id": "56dba20a-c07c-4b18-a4e7-834ab6dc87ef",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "agent = initialize_agent(tools, llm, agent=AgentType.OPENAI_FUNCTIONS, verbose=False)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "id": "569389d4-b613-47ce-99d3-e0031f308185",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "LLMMathChain._evaluate(\"\n",
      "US_GDP / average_lifespan\n",
      "\") raised error: 'US_GDP'. Please try again with a valid numerical expression\n"
     ]
    }
   ],
   "source": [
    "os.environ[\"LANGCHAIN_PROJECT\"] = deployment_name\n",
    "\n",
    "inputs = [\n",
    "    \"What's the ratio of the current US GDP to the average lifespan of a human?\",\n",
    "    \"What's sin of 180 degrees?\",\n",
    "    \"I need help on my homework\",\n",
    "    \"If the price of bushel of wheat increases by 10 cents, about how much will that impact the average cost of bread?\",\n",
    "    # etc.\n",
    "]\n",
    "for query in inputs:\n",
    "    try:\n",
    "        await agent.arun(query)\n",
    "    except Exception as e:\n",
    "        print(e)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2646f0fb-81d4-43ce-8a9b-54b8e19841e2",
   "metadata": {},
   "source": [
    "## Conclusion\n",
    "\n",
    "Congratulations! You have succesfully created connected an agent to LangSmith to trace and debug, evaluated it for accuracy, helpfulness, and trajectory efficiency over a dataset, and instrumented a monitoring project for a simulated \"production\" application!\n",
    "\n",
    "This was a quick guide to get started, but there are many more ways to use LangSmith to speed up your developer flow and produce better products.\n",
    "\n",
    "For more information on how you can get the most out of LangSmith, check out [LangSmith documentation](https://docs.langchain.plus/docs/),\n",
    "\n",
    "and please reach out with questions, feature requests, or feedback at [support@langchain.dev](mailto:support@langchain.dev)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "90b7fbff-162d-4c9c-b6fc-33bd5445745f",
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}
