langchain/docs/docs/how_to/qa_sources.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "4ef893cf-eac1-45e6-9eb6-72e9ca043200",
   "metadata": {},
   "source": [
    "# How to get your RAG application to return sources\n",
    "\n",
    "Often in [Q&A](/docs/concepts/rag/) applications it's important to show users the sources that were used to generate the answer. The simplest way to do this is for the chain to return the Documents that were retrieved in each generation.\n",
    "\n",
    "We'll work off of the Q&A app we built over the [LLM Powered Autonomous Agents](https://lilianweng.github.io/posts/2023-06-23-agent/) blog post by Lilian Weng in the [RAG tutorial](/docs/tutorials/rag).\n",
    "\n",
    "We will cover two approaches:\n",
    "\n",
    "1. Using the basic RAG chain covered in [Part 1](/docs/tutorials/rag) of the RAG tutorial;\n",
    "2. Using a conversational RAG chain as convered in [Part 2](/docs/tutorials/qa_chat_history) of the tutorial.\n",
    "\n",
    "We will also show how to structure sources into the model response, such that a model can report what specific sources it used in generating its answer."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "487d8d79-5ee9-4aa4-9fdf-cd5f4303e099",
   "metadata": {},
   "source": [
    "## Setup\n",
    "\n",
    "### Dependencies\n",
    "\n",
    "We'll use the following packages:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "28d272cd-4e31-40aa-bbb4-0be0a1f49a14",
   "metadata": {},
   "outputs": [],
   "source": [
    "%pip install --upgrade --quiet langchain langchain-community langchainhub beautifulsoup4"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1665e740-ce01-4f09-b9ed-516db0bd326f",
   "metadata": {},
   "source": [
    "### LangSmith\n",
    "\n",
    "Many of the applications you build with LangChain will contain multiple steps with multiple invocations of LLM calls. As these applications get more and more complex, it becomes crucial to be able to inspect what exactly is going on inside your chain or agent. The best way to do this is with [LangSmith](https://smith.langchain.com).\n",
    "\n",
    "Note that LangSmith is not needed, but it is helpful. If you do want to use LangSmith, after you sign up at the link above, make sure to set your environment variables to start logging traces:\n",
    "\n",
    "```python\n",
    "os.environ[\"LANGCHAIN_TRACING_V2\"] = \"true\"\n",
    "os.environ[\"LANGCHAIN_API_KEY\"] = getpass.getpass()\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "83814ea8-c214-4d9b-b360-a578a7ed5914",
   "metadata": {},
   "source": [
    "### Components\n",
    "\n",
    "We will need to select three components from LangChain's suite of integrations.\n",
    "\n",
    "A [chat model](/docs/integrations/chat/):\n",
    "\n",
    "import ChatModelTabs from \"@theme/ChatModelTabs\";\n",
    "\n",
    "<ChatModelTabs customVarName=\"llm\" />"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "7be1032f-ea27-4103-a535-86b8730796db",
   "metadata": {},
   "outputs": [],
   "source": [
    "# | output: false\n",
    "# | echo: false\n",
    "\n",
    "from langchain_openai import ChatOpenAI\n",
    "\n",
    "llm = ChatOpenAI(model=\"gpt-4o-mini\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "09f98df1-ed09-487d-aab1-b3a22ffc873c",
   "metadata": {},
   "source": [
    "An [embedding model](/docs/integrations/text_embedding/):\n",
    "\n",
    "import EmbeddingTabs from \"@theme/EmbeddingTabs\";\n",
    "\n",
    "<EmbeddingTabs/>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "02d000ac-0563-4da9-af48-761659a41f91",
   "metadata": {},
   "outputs": [],
   "source": [
    "# | output: false\n",
    "# | echo: false\n",
    "\n",
    "from langchain_openai import OpenAIEmbeddings\n",
    "\n",
    "embeddings = OpenAIEmbeddings()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c9fcc22d-12a3-4887-a7e8-6f651ffef9ae",
   "metadata": {},
   "source": [
    "And a [vector store](/docs/integrations/vectorstores/):\n",
    "\n",
    "import VectorStoreTabs from \"@theme/VectorStoreTabs\";\n",
    "\n",
    "<VectorStoreTabs/>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "d8d16b63-6866-49d3-96a3-8fdb20e4591b",
   "metadata": {},
   "outputs": [],
   "source": [
    "# | output: false\n",
    "# | echo: false\n",
    "\n",
    "from langchain_core.vectorstores import InMemoryVectorStore\n",
    "\n",
    "vector_store = InMemoryVectorStore(embeddings)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fa6ba684-26cf-4860-904e-a4d51380c134",
   "metadata": {},
   "source": [
    "## RAG application\n",
    "\n",
    "Let's reconstruct the Q&A app with sources we built over the [LLM Powered Autonomous Agents](https://lilianweng.github.io/posts/2023-06-23-agent/) blog post by Lilian Weng in the [RAG tutorial](/docs/tutorials/rag).\n",
    "\n",
    "First we index our documents:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "24a69b8c-024e-4e34-b827-9c9de46512a3",
   "metadata": {},
   "outputs": [],
   "source": [
    "import bs4\n",
    "from langchain import hub\n",
    "from langchain_community.document_loaders import WebBaseLoader\n",
    "from langchain_core.documents import Document\n",
    "from langchain_text_splitters import RecursiveCharacterTextSplitter\n",
    "from typing_extensions import List, TypedDict\n",
    "\n",
    "# Load and chunk contents of the blog\n",
    "loader = WebBaseLoader(\n",
    "    web_paths=(\"https://lilianweng.github.io/posts/2023-06-23-agent/\",),\n",
    "    bs_kwargs=dict(\n",
    "        parse_only=bs4.SoupStrainer(\n",
    "            class_=(\"post-content\", \"post-title\", \"post-header\")\n",
    "        )\n",
    "    ),\n",
    ")\n",
    "docs = loader.load()\n",
    "\n",
    "text_splitter = RecursiveCharacterTextSplitter(chunk_size=1000, chunk_overlap=200)\n",
    "all_splits = text_splitter.split_documents(docs)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "dfc82ead-8504-4ae8-b593-9b69f23e5a57",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Index chunks\n",
    "_ = vector_store.add_documents(documents=all_splits)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7cd9a4e6-8df7-48c8-879a-63660b963adf",
   "metadata": {},
   "source": [
    "Next we build the application:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "def66576-2309-45b5-8699-f0287df8ca9e",
   "metadata": {},
   "outputs": [],
   "source": [
    "from langchain import hub\n",
    "from langchain_core.documents import Document\n",
    "from langgraph.graph import START, StateGraph\n",
    "from typing_extensions import List, TypedDict\n",
    "\n",
    "# Define prompt for question-answering\n",
    "prompt = hub.pull(\"rlm/rag-prompt\")\n",
    "\n",
    "\n",
    "# Define state for application\n",
    "class State(TypedDict):\n",
    "    question: str\n",
    "    context: List[Document]\n",
    "    answer: str\n",
    "\n",
    "\n",
    "# Define application steps\n",
    "def retrieve(state: State):\n",
    "    retrieved_docs = vector_store.similarity_search(state[\"question\"])\n",
    "    return {\"context\": retrieved_docs}\n",
    "\n",
    "\n",
    "def generate(state: State):\n",
    "    docs_content = \"\\n\\n\".join(doc.page_content for doc in state[\"context\"])\n",
    "    messages = prompt.invoke({\"question\": state[\"question\"], \"context\": docs_content})\n",
    "    response = llm.invoke(messages)\n",
    "    return {\"answer\": response.content}\n",
    "\n",
    "\n",
    "# Compile application and test\n",
    "graph_builder = StateGraph(State).add_sequence([retrieve, generate])\n",
    "graph_builder.add_edge(START, \"retrieve\")\n",
    "graph = graph_builder.compile()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "53911d2e-f9b9-44b8-9baf-1c308201a7d0",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAGsAAADqCAIAAAAqMSwmAAAAAXNSR0IArs4c6QAAGfFJREFUeJztnXdAFFf+wN/2vgvLUnfpHUEsaDSioGIDFYkFCybRmJwXkivmd6neaeLF80zjciaaOzVFMLEkxmDHKCqiCFEUBKSLwALbe53d3x/roYm7MwuzuAPu5y+deW/2Ox9m5r157817OKvVCjygAO/uAIY9HoNo8RhEi8cgWjwG0eIxiBYiyvwqqUkhMWlVkFYJmU1Wi2UY1I0IREAk4ulsAp1F9A4g0ZmoJOAGVx+UCA0ttzRtNRoyHQesODqLQGcTaAyiBRoGBokknFpp1iohrcps0FlIZHxEEiMqmcn2IQ3iaAM2qJaby4vFVgC8eKTwJIafgDqIX8UUwjZda41G1mtkehOfns8jUwf2ZBuYwcoz0tpyxdMLeLHjWQMPFevUlCnKj4knZfkkT/VyPtcADB7d2RU1ljlqEmewEQ4PfjkrlfQYZ+cFOJne2St2z1/bxs7wHvH6AADjM7ihcYyjO7uczWB1gt0bW8XdemdSjhiaqlXffdjhTErku/jozq6xM7xDYuku+PsOK+orlF2tuowV/vDJEAxWlUhpTMKoySP/5rVL1VkpjYFw+nDPQbXcXHNZ8cTqAwCkZHDPHxTBp4EzWF4sfnoBz9VRDTMmz/cpLxbDJHBoUCI0WAEYkfW+ATF+pre426DXmB0lcGiw5ZbGizeYt5zBUVtbazAY3JUdHgab2FqrdbTXocG2Gk14EmOIYvoNxcXFzz//vE6nc0t2RCKSmK01akd77RtUSk0UOv6xvfMO+vKxVSSG7uqzEZ7IUMvMjpqdHBiUmIaoC+/u3bvr169PTU3NzMzcunWrxWIpLi7etm0bACAjIyMlJaW4uBgA0Nvbu2nTpoyMjEmTJuXm5p46dcqWXS6Xp6Sk7Nu3b+PGjampqS+++KLd7C7HbLIqxCa7u+w3jWlVEJ1FGIpQtmzZ0t7e/tprr2k0mqqqKjweP2XKlLy8vMLCwoKCAiaTGRISAgAwm823b99esmSJl5fXuXPnNm7cGBwcPGrUKNtB9uzZs3Tp0l27dhEIBH9//0ezuxw6m6BVQt5+dnY5MKiE6OwhMdjd3R0XF5eTkwMAyMvLAwBwuVyBQAAASExM9PK63yjC5/MPHTqEw+EAANnZ2RkZGaWlpf0Gk5KS8vPz+4/5aHaXw2ATNUr7xbHDkoREHpIOgMzMzKtXr27fvl0qlcKnbGxs3LBhw9y5c3NyciAIkkgk/bsmTpw4FLHBQKbiHb282ddEZeBVMoc1IDTk5+dv2LDhzJkzCxcuPHjwoKNklZWVzz33nNFo3LRp0/bt2zkcjsVi6d9Lo9GGIjYYFGITnWX/frW/lc4ialVDYhCHw61cuTI7O3vr1q3bt2+PiYkZM2aMbdfDf+Tdu3cLBIKCggIikeiksiEdvgJTMNi/BpneBAptSO5iW82DwWCsX78eANDQ0NAvSCR68AYql8tjYmJs+oxGo1arffga/A2PZnc5DA6B5W3//cL+Ncj1p4g6jXKR0cuX7NpQ3njjDSaTOWnSpLKyMgBAfHw8ACA5OZlAIHz44YcLFy40GAyLFy+21UuOHj3K4XCKioqUSmVLS4ujq+zR7K6NuatZZzEDR/0nhM2bN9vdoZKZNQpzYLiLnzidnZ1lZWWnTp3S6XSvvvpqeno6AIDNZvv7+5eUlFy6dEmpVM6fPz85Obm1tfW7776rqqqaNWtWbm7u6dOn4+LifHx8vvnmm9TU1ISEhP5jPprdtTHfvCD3D6MGhNl/v3DYPtjdqquvUM5Eal98Eji+R5iazeM4aCVw2NkcFEG7dkp6r1EbHGO/dVqpVC5cuNDuLoFA0NnZ+ej2tLS0d9991+nIB8m6deuam5sf3R4fH19fX//o9sTExB07djg6Wv01JYWGd6QPoY26757+/EFR7mvBdvdaLJaenh77B8XZPyyNRvP29nb0c65CJBKZTHbewBxFRSaTeTyHzaB7/tq24vVgR1UZ5Fb+i0dEITH0sFGPqZEGa9y+qtAqoQmzuTBpEKos03J8L/wgUkrsv1SPbLpbdA2VKnh9wJneToMe2vV6syt6EIcTOo3pizdbnEnpVH+x0QB98VazWmFCHdjwoK9Tv+dvrWazxZnEzo760Kmhb7d3zHnWnx81wjuOm2+qqs7Ilv/F2VaygY08On+gTykzTVnA4/Epg40Qu3S16K4US/xDKVNzfJ3PNeDRbx0N2svF4pA4un8wNTyRQSDiBh4qtjDqLa216p52vVRonLzAJzBsYK9hgxyB2XJL3Xhd1VariR3PIlHwDDaRwSFQ6YThMIQVEPA4rcqsUZo1SkitMHU26iISmTEpzNC4wVTaBmmwn44GrazPqFGaNQrIYrGaja5UCEFQTU1Nf/OXq6DQ8bZmZwab4BNIRvlkR2twSFGr1fPnzy8tLXV3IHB4xvKjxWMQLVg3aGuCxTJYN2i3PQpTYN3g0HUBuwqsG5TL5e4OAQGsGwwIcParBHeBdYOOmsGxA9YNJiUluTsEBLBusKamxt0hIIB1g3Q61psjsW5Qq3U4gBkjYN0g9sG6QU9JghZPSTLywbpBLhepw9vdYN0g4nBrt4N1g7Gxse4OAQGsG7xz5467Q0AA6waxD9YNelpY0eJpYR35eAyiBesGExMT3R0CAlg3WFtb6+4QEMC6QezjMYgWrBv01AfR4qkPjnywbjAsLMzdISCAdYPt7e3uDgEBrBvEPlg3SCAMyaQtLgTrBiEIcncICGDdoKe/GC2e/mK0YL+nCYtf5Lz44ovd3d1EItFisQiFwsDAQDwebzKZTpw44e7Q7IDFa3DVqlVKpbKrq0soFAIAhEJhV1cXZgtlLBpMT0+Pjo5+eIvVasVskYJFgwCA1atXPzz2MjAwcPny5W6NyCEYNTh9+vTw8PD+Z3RycvLo0aPdHZR9MGoQALBmzRpb4yCPx8PsBYhpg+np6REREbZKNWYfggNYp0mngSTdRqPB4RR2Q8Gi2b8zyA5kpq9prdU8zt+l0vA8PsXJxXKQ64OQ2XpmX29nkzY4lmHUP1aDbgMHhK3a8ETm7DzkidsQDBp00Pf/7powhxcQhvWvElxOW62qsUqR8wqfQICbjQPB4Dd/vztzZSDbx8XzOA4Xulu0t8tlz7zCh0kDd6vXlisiRjOfWH0AgKBIOtuHBDOlPILB3g4DzfGscU8IFBpB1GWESQBn0KS3cLhP7gVog+NL1mvgyk84gzotBD0ZZS8MFjMw6eHaybFbox4ueAyixWMQLR6DaPEYRIvHIFo8BtHiMYgWj0G0eAyixWMQLe40CEFQTU01fBqz2Zz3bM7OXQWPK6gB406DH3y05eOCrfBpcDgci8WmUh/T6o2DYAib/6xWq23BOUcYYVeLtGUnEAg7P/t6CKJzGa40qFDIFz2Tsf53f2xqvnP5cml0dNynBbsBAEd/OnzwUKFY3BcQEDRzxtzcZaspFMq27ZvPl5YAAKbPTAEA7C/6KTAgaM0Ly8LDIsPCIn848p3BoN/x6ZfrXloBAMhbtfaFtS8DAPR6/e49n/187pTRaAgWhC5btnrG9Nn1Dbdfzn/utQ3vzM/KsUXy1df/2f/tl4cOnORwvIQ93Z9//vEv1yvIZEpMdNzatS/HxSYgncoAcP01WFi4Jzt76Ucf7rKNFfrq6/8cOlz4TM7y0NCIe/faDxz8prOr4+0338tbuVbU1ysUdr315nsAAB/u/TVWKiuv6A36rX//RKvT8vnBW9778N333rTtslgs72z8c09P96qVa7y8uNXVVVv+/rZer8uclx0dFXum5Hi/wZKzJ9LSMjgcL4lE/Oof1vL5wa/k/x8Ohztz5vgf/7Tuy72HggLhuj4GhOsNJiQkrXvh/pKQYrGoaP/eje+8nzZtpm2Lj4/vJwX/eCX//wSCEA7HSyqTJCX9asJuApH413e29i9Qlzolvf9RcPHSuVs1N74tKubxfAEAGTPn6nTa73/4NnNedlZWTsG/tvX0CAMCAm/fvtXd3fnWG+8CAPYV7vb24n70wU7bwm2zMjLznl1UXn5hyeKVrjpf1xscN+7BkpC//FJhNpvf37rx/a0bbVtsXYNiUR+bxbabPT4+0dH6flevlpnN5pV5DxaHgiCIwWACAGbOmLvri4KzP5/MW7X2TMnxiIioxMRkAEBFxeU+UW/m/Kn9WUwmk0zmyhlYXG+QSn1w/hKpGACw9f0CP99fdV0HBQkcZadRHS4sIJNJfHx4H3+46+GNBCIRAMBkMmdMn3P255O5y1afLy2xPTQBAFKZZPLkqS+te/XhLByOK7/VG9quONb/LrSQEPufJg1oBC2LxZbLZf7+gRSKnbU9srJyTpw8uq9wt9lsypg5rz+LQiF39OsuYWjrg2PHTsDhcEd+PNC/5eG1wqlUmlQqgVlO8jeMGzcRgqCfig/bPVpCfGJUZExh0d6MmfMYDEZ/ltram3ca6+1mcQlDa1DAD34mZ3l5+cW3N/75xMmj+wr35D27qLGpwbY3efQ4lUr58SdbT58+Vl5+EfFoszIy4+JG7friX5/u+ODU6eIdn3205oWler2+P0FWVo7Val2w4MGqk889+xKLxf7L6/mFRXuPn/hx0+bX3//HRtee45B3qOe/vMHPz//IkQOVlVd8fHhTU6f78u4vRT1rVuadxrozJcevXL00d86Cp5+eBn8oEon0wT8/++/uf587d/rYsR8EgpCFC5bYClkbGTPnXbp0LjrqwfB/fpBgx6d7d35RULR/Lw6Hi46Oy1mU69oThBs3c+TzroTJ3KCIx71YMKZoqVaJO7UZqxwO4vK0zaDFYxAtHoNo8RhEi8cgWjwG0eIxiBaPQbR4DKLFYxAtHoNo8RhEi8cgWuAMsnkkADA3C8NjBocHDA5cGyCcQRqdIO7SwyR4Eujt0DG9BmswLIGuEMF9zvMkoFGYQ+LgWkjhDAZF0HwCyVeK+4YgsOFB6UFh9BgGhwf3YRfy98XXz8mE7YagSDqPTyWRn4iSx6iDRN365hvKseneMeOY8ImdmrHnboOm8Re1Tg1Jex7vTW21GoxGu32bQwrHh8TmkZJS2X4C5DFjWJzzqB/PKuRPBB6DaMG6QSzPk2ID6wY98w+iJSoqyt0hIIB1g83Nze4OAQGsG4yPj3d3CAhg3WB9fb0TqdwJ1g3GxcW5OwQEsG6woaHB3SEggHWD2AfrBnk8nrtDQADrBsVisbtDQADrBn8zKTAGwbrBpqYmd4eAANYNYh+sG4yJiXF3CAhg3WBjY6O7Q0AA6wZ9fX3dHQICWDcoEoncHQICWDeIfbBu0NPCihZPC+vIx2MQLVg3mJDgyplNhgKsG6yrq3N3CAhg3SD28RhEC9YNeuqDaPHUB0c+WDeYmJjo7hAQwLrB2tpad4eAANYNYh+sGwwODnZ3CAhg3eC9e/fcHQICWDfo6WlCi6enCS3Y72nC4hc5+fn5UqmURCJBENTQ0BAbG0skEiEIKioqcndodsDicnRpaWkfffQRBEG2Gb1tNzIG/9I2sHgXL1u27NFKzMSJEx0kdzNYNAgAyMvLe/iDRDabvWLFCrdG5BCMGly0aBGf/2DS7ejo6GnTEGbIdBcYNQgAWLFihe0y5HA4eXl57g7HIdg1mJOTY7sMIyMjp06d6kQO9+DislirhCDIZYVm7uLn9+zZk7v4eZXM7KpjEkk4GpPgqqO5oD7Y26Fvq9VIhKbuVp1BC3n7U/QauHVC3Q6BhFPLTFQGISiS5icghycyfAJRfUM/eIO3yuQNlWqd1srg0pk8OpFEIFJc+bcdOqxWq9kImQ2QWqxRi7VevqSEiazYFNbgjjYYg03Vqos/iFk8uneoF4mMxTr5gDDqTNK7MpPWlLaYFxI34OXqB2zw5Nd9GjXgBHFI1GHv7mH0KqNapPQLIk7L8RlQxoEZPPhJJ5nF8OLbXxhjBCBpl5GJpgUvBjqfZQAGj+wUkpgMJo8x2PCGB9IuBZsJZSx3tk3IWYNHd3UTGMwRr8+GQqhk0EwZK/ycSexUjfpysdhKoDwh+gAAnEC2TGy9dUnuTGJkg6IuQ3O11kvgynVlsI9vFO/KCalOjVy3RTZ46YiYG+btosCGEwHR3LKjyN9FIhjsbNLqdTgWb8C1pBEAJ5AlbDPI+hCmGkMwWH1RyRiejz+pTCiVdaM8CJ3HrClTwKdBMNhRp2b5DT+DYmnnPz7JudeFdpYLli+9pUYDnwbOYEeDlu1Hw+Ph1t58FLVGrtUqB5RlEMBXwiyQ2SX9KhQ6yWrFwc8ZCFcfrCyR3m228sKQS+GqG8d/vvi1XNET4BeJw+G9vQJW574PAJDKun86WdDYco1EpPCDYudlrA/mJwAAviz6iy8vlEAgVlT9aIZM8TFTnlnwOo16f67E8mvfX7i8X6Hs43oHjR09O31KHolE0Wjkm7bNmT/n1S5h4+36C/yguPx1X1y7XlxecVjY00yh0GOjJmVnbWAyvKWy7q0f5/THljI2a/kzfwMAGI36k2d33rh12mQy+PJC01NXjUmahXhqohbJqBRKwiSOowSEzZs3O9rXUKkymog0DkLjT239hcKDG5MSps+Y+ty9rrq7924tW/S2F8dfqRR/+p+1JCJ1+rRnY6Ke6hLeKSndOyo+jcXkVteUVN04zmH7LcraEMyPP3/xGwgyx0Q9BQA4c+6/Jef3TBy/8Knx2Uwm9+Ll/WLJvaSEdJNJX1pW2NFVFxP51LxZv4+LeZrD9i2/9gOVwkgZm+XHC6uqPiHsaRqXPIdIovj7hdfUnZ8z46W5M1+Ki57MoHMsFsvufX+613k7bcrKMaNnmc3Gk2d3cjj+gqBY+LPTyg10BuBHOZyKFa51QC2HiDTkSSDLKw77+0UszX4LABAsSNjywfz6O+WhwUklF/YyGdzfrdlBIBABAOOT520rWFxRdXRR1gYAgK9PyMol7+JwuBDBqFt15+80X50PXlUoRT9f/GrVki2jE2fYDs5h8b4v/md25gbbf0MFiZmzft//00sWvtm/qieeQPz5wpcmk4FEoggCYwEAfr5h4aH3FwWtqTvf1l799ms/cti+AIBxo+cYjNqyKweeGr/wkRP6FQQSQS03wSSAM0gk4/AU5AYYubKP53O/c5LD9iWTqFqdEgDQ0FguV/S+vSW9PyUEmeTKXtu/SSRq/8lzvQLbO24BAJparkGQuejw34oO/+1/mawAAIWqj83kAQCiIyc8/NNmyFR25cD1m6dkih4yiWq1WtQambdXwKNB1t+5DFnMD9/dFgvU/9yAk0AlWq1wLeRwgiCTFTKYaQDhLvbx5nd21ZvMRhKRLOxpNpr0/MAYAIBKLUmITc2anf9wYirFTtAEAsligQAASpUYAPBC3sdenF+9k/pwBXq9GgBAJj+4m6xW697CDfe66mdPXxcanFRTV1pats9qtb8Co0otYbN469d89vBGPB75+jDpzTgKXKEEdwgGh6BQIr/WTJ+6eteX+V/szY+OnPDLzZPB/ISUsVkAADqNrdEq/HwHsGYmjXa/3cyZXC3t15taKlcufW/c6DkAALEEbpwcncZWa2TeXoEk0sDa9M0GM2vQM3pzeESLE91GYSHJUycvt1gtYmlnemreyy/ssj34oiMmtHfcfLhSZjAirJkZHZGCw+HKKg46k0WrUQAA+IH3iwKNVm5bJdr2iAAAKFUPvu6OipxgsUDl1753PhgbeBxgcWGfdTD7AsNoddckIMxhQW7jYvn+5taqtNRVOIAj4IkiSUdQQDQAYNb0dfWNl//79R+mTVnJYnAbmq5YLNCaVR/AHIrnE5w6KffSle/2Fr42Kj5NpRJfrjj8wuqPBUF25i8LCU4kEsknSz5/KmWRsKfp3MWvAQA9vS08H4EXx9/Hm3/h8n4yiabRKaZOyh2fPK+i6sdjp/8tkwv5gbHdPU01daWv/+EAmYxQVCr7NAGwBuBqM2wuqbxYxA1mw1eqzZDpl+oTVTeO19Sdv3n75yuVPyhVkoS4VDqdPSpuWq+4/Xr1yTvNV2kU5lMp2QF+EQCA6poSvUEzecL953pjc0WX8M6Mac8BAGKjJlEp9Lo7ZdU1Z8SSewlx00bFTaWQabbaTHzsFFuNEgBApTL8/SIqrx+runEMgswrl76nUIna7t6cMDYLh8OFBic2NF29UXNGJhcmxqcxGJzRiTN1OtXN2rO36s7r9ZqJ4xeEh47B4+HuQr3aqJNpJ82Da/dHaGE9+VWPAaJ5BSGUWRAE2VZtN5mNx0/vuFxxaNumS7Z7eVgjapMHCqypC+Hm/kI4ybHTvU7vE8EbrLpx4uTZnWOSZnG9g1RqaU3d+QC/iBGgDwAg71LOW4kwFB7hPANCqd6+RGWvhu3vsH3B3y88PDT5+s1TWq2CxeKNipuWkbZmsDFjCOk9ReRoBvzSGk71k8j6jD/u6gmfwIdPNvK4c6F97eYwEhVhGAFyG7W3HzlxMkvUInVdbMMAYV3ftMW+iPqc7WmaMMubwYDk3UPeZoURJG0yQSQpfoJT3eID6C8+Xdin1ZO8R253u42+Fhk/FD9lAdfJ9AMYPzgnzw8P6aQdssHGNgzobRJzuRbn9Q1m3Ez5MUlnm4nlx6axH/fCK0OKRqrTSNQxY6hjpg2sX3cwY7c6GrQXj4jxJBI31IvKhFvDaFigUxrEbTIKxZq2mOcfgtwe+hsGP36w6Yaqplwl7TEyeXQmj04kE0gUAoE0DIYQ2gYPmoxmtUirEmkDI2ijp7BC4wfZoYZ2DKtSYmqr1fR0GHvv6nRqiMok6tQuG7E7FBCJOAtkpTKJAWHUoHBKeCKDwUb1+uTir8LMRqsLx1EPBSQSDk8cWO8jPFj8rm54gd2vIYYLHoNo8RhEi8cgWjwG0eIxiJb/B1sJjsMcn1hqAAAAAElFTkSuQmCC",
      "text/plain": [
       "<IPython.core.display.Image object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "from IPython.display import Image, display\n",
    "\n",
    "display(Image(graph.get_graph().draw_mermaid_png()))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e18645d6-b32d-4c8d-8c94-b8e8e7372c7f",
   "metadata": {},
   "source": [
    "Because we're tracking the retrieved context in our application's state, it is accessible after invoking the application:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "cd7dde97-25c6-49ce-9a0a-4cd36e36da2d",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Context: [Document(id='c8471b37-07d8-4d51-856e-4b2c22bca88d', metadata={'source': 'https://lilianweng.github.io/posts/2023-06-23-agent/'}, page_content='Fig. 1. Overview of a LLM-powered autonomous agent system.\\nComponent One: Planning#\\nA complicated task usually involves many steps. An agent needs to know what they are and plan ahead.\\nTask Decomposition#\\nChain of thought (CoT; Wei et al. 2022) has become a standard prompting technique for enhancing model performance on complex tasks. The model is instructed to “think step by step” to utilize more test-time computation to decompose hard tasks into smaller and simpler steps. CoT transforms big tasks into multiple manageable tasks and shed lights into an interpretation of the model’s thinking process.'), Document(id='acb7eb6f-f252-4353-aec2-f459135354ba', metadata={'source': 'https://lilianweng.github.io/posts/2023-06-23-agent/'}, page_content='Tree of Thoughts (Yao et al. 2023) extends CoT by exploring multiple reasoning possibilities at each step. It first decomposes the problem into multiple thought steps and generates multiple thoughts per step, creating a tree structure. The search process can be BFS (breadth-first search) or DFS (depth-first search) with each state evaluated by a classifier (via a prompt) or majority vote.\\nTask decomposition can be done (1) by LLM with simple prompting like \"Steps for XYZ.\\\\n1.\", \"What are the subgoals for achieving XYZ?\", (2) by using task-specific instructions; e.g. \"Write a story outline.\" for writing a novel, or (3) with human inputs.'), Document(id='4fae6668-7fec-4237-9b2d-78132f4f3f3f', metadata={'source': 'https://lilianweng.github.io/posts/2023-06-23-agent/'}, page_content='Resources:\\n1. Internet access for searches and information gathering.\\n2. Long Term memory management.\\n3. GPT-3.5 powered Agents for delegation of simple tasks.\\n4. File output.\\n\\nPerformance Evaluation:\\n1. Continuously review and analyze your actions to ensure you are performing to the best of your abilities.\\n2. Constructively self-criticize your big-picture behavior constantly.\\n3. Reflect on past decisions and strategies to refine your approach.\\n4. Every command has a cost, so be smart and efficient. Aim to complete tasks in the least number of steps.'), Document(id='3c79dd86-595e-42e8-b64d-404780f9e2d9', metadata={'source': 'https://lilianweng.github.io/posts/2023-06-23-agent/'}, page_content=\"(3) Task execution: Expert models execute on the specific tasks and log results.\\nInstruction:\\n\\nWith the input and the inference results, the AI assistant needs to describe the process and results. The previous stages can be formed as - User Input: {{ User Input }}, Task Planning: {{ Tasks }}, Model Selection: {{ Model Assignment }}, Task Execution: {{ Predictions }}. You must first answer the user's request in a straightforward manner. Then describe the task process and show your analysis and model inference results to the user in the first person. If inference results contain a file path, must tell the user the complete file path.\")]\n",
      "\n",
      "\n",
      "Answer: Task Decomposition is the process of breaking down a complex task into smaller, manageable steps to facilitate execution. This can be achieved through techniques like Chain of Thought, which encourages step-by-step reasoning, or Tree of Thoughts, which explores multiple reasoning paths for each step. It can be implemented using simple prompts, specific instructions, or human input to effectively tackle the original task.\n"
     ]
    }
   ],
   "source": [
    "result = graph.invoke({\"question\": \"What is Task Decomposition?\"})\n",
    "\n",
    "print(f'Context: {result[\"context\"]}\\n\\n')\n",
    "print(f'Answer: {result[\"answer\"]}')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "50cca598-8a10-4cf9-bafc-b56f966bf73d",
   "metadata": {},
   "source": [
    "Here, `\"context\"` contains the sources that the LLM used in generating the response in `\"answer\"`."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d74c8dac-54ed-4335-b6cc-4a7a97b4b5ce",
   "metadata": {},
   "source": [
    "## Structure sources in model response\n",
    "\n",
    "Up to this point, we've simply propagated the documents returned from the retrieval step through to the final response. But this may not illustrate what subset of information the model relied on when generating its answer. Below, we show how to structure sources into the model response, allowing the model to report what specific context it relied on for its answer.\n",
    "\n",
    "It is straightforward to extend the above LangGraph implementation. Below, we make a simple change: we use the model's tool-calling features to generate [structured output](/docs/how_to/structured_output/), consisting of an answer and list of sources. The schema for the response is represented in the `AnswerWithSources` TypedDict, below."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "ab70db7e-397b-4ffe-9346-b488b57527c9",
   "metadata": {},
   "outputs": [],
   "source": [
    "from typing import List\n",
    "\n",
    "from typing_extensions import Annotated, TypedDict\n",
    "\n",
    "\n",
    "# Desired schema for response\n",
    "class AnswerWithSources(TypedDict):\n",
    "    \"\"\"An answer to the question, with sources.\"\"\"\n",
    "\n",
    "    answer: str\n",
    "    sources: Annotated[\n",
    "        List[str],\n",
    "        ...,\n",
    "        \"List of sources (author + year) used to answer the question\",\n",
    "    ]\n",
    "\n",
    "\n",
    "class State(TypedDict):\n",
    "    question: str\n",
    "    context: List[Document]\n",
    "    # highlight-next-line\n",
    "    answer: AnswerWithSources\n",
    "\n",
    "\n",
    "def generate(state: State):\n",
    "    docs_content = \"\\n\\n\".join(doc.page_content for doc in state[\"context\"])\n",
    "    messages = prompt.invoke({\"question\": state[\"question\"], \"context\": docs_content})\n",
    "    # highlight-start\n",
    "    structured_llm = llm.with_structured_output(AnswerWithSources)\n",
    "    response = structured_llm.invoke(messages)\n",
    "    # highlight-end\n",
    "    return {\"answer\": response}\n",
    "\n",
    "\n",
    "graph_builder = StateGraph(State).add_sequence([retrieve, generate])\n",
    "graph_builder.add_edge(START, \"retrieve\")\n",
    "graph = graph_builder.compile()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "id": "95b6b784-8584-4b02-8307-8f2d93d4a166",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{\n",
      "  \"answer\": \"Chain of Thought (CoT) is a prompting technique that enhances model performance by instructing it to think step by step, allowing the decomposition of complex tasks into smaller, manageable steps. This method not only aids in task execution but also provides insights into the model's reasoning process. CoT has become a standard approach in improving how language models handle intricate problem-solving tasks.\",\n",
      "  \"sources\": [\n",
      "    \"Wei et al. 2022\"\n",
      "  ]\n",
      "}\n"
     ]
    }
   ],
   "source": [
    "import json\n",
    "\n",
    "result = graph.invoke({\"question\": \"What is Chain of Thought?\"})\n",
    "print(json.dumps(result[\"answer\"], indent=2))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "dd4108d3-f2e9-41fb-8528-fdac8c68b337",
   "metadata": {},
   "source": [
    ":::tip\n",
    "\n",
    "View [LangSmith trace](https://smith.langchain.com/public/51d543f7-bdf6-4d93-9ecd-2fc09bf6d666/r).\n",
    "\n",
    ":::"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a795ee54-f87c-4224-bdfc-6f037c1e3630",
   "metadata": {},
   "source": [
    "## Conversational RAG"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "82c8e1ec-6f77-468b-a06d-f032a9fe3488",
   "metadata": {},
   "source": [
    "[Part 2](/docs/tutorials/qa_chat_history) of the RAG tutorial implements a different architecture, in which steps in the RAG flow are represented via successive [message](/docs/concepts/messages/) objects. This leverages additional [tool-calling](/docs/concepts/tool_calling/) features of chat models, and more naturally accommodates a \"back-and-forth\" conversational user experience.\n",
    "\n",
    "In that tutorial (and below), we propagate the retrieved documents as [artifacts](/docs/how_to/tool_artifacts/) on the tool messages. That makes it easy to pluck out the retrieved documents. Below, we add them as an additional key in the state, for convenience.\n",
    "\n",
    "Note that we define the response format of the tool as `\"content_and_artifact\"`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "id": "cbbc4668-d9de-420f-9a4b-532b634f897e",
   "metadata": {},
   "outputs": [],
   "source": [
    "from langchain_core.tools import tool\n",
    "\n",
    "\n",
    "@tool(response_format=\"content_and_artifact\")\n",
    "def retrieve(query: str):\n",
    "    \"\"\"Retrieve information related to a query.\"\"\"\n",
    "    retrieved_docs = vector_store.similarity_search(query, k=2)\n",
    "    serialized = \"\\n\\n\".join(\n",
    "        (f\"Source: {doc.metadata}\\n\" f\"Content: {doc.page_content}\")\n",
    "        for doc in retrieved_docs\n",
    "    )\n",
    "    return serialized, retrieved_docs"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f0e4859f-efad-41d0-aca3-2a6f314464e1",
   "metadata": {},
   "source": [
    "We can now build and compile the exact same application as in [Part 2](/docs/tutorials/qa_chat_history) of the RAG tutorial, with two changes:\n",
    "\n",
    "1. We add a `context` key of the state to store retrieved documents;\n",
    "2. In the `generate` step, we pluck out the retrieved documents and populate them in the state.\n",
    "\n",
    "These changes are highlighted below."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "id": "e38542b3-9d10-41a4-9af5-7495835eb45b",
   "metadata": {},
   "outputs": [],
   "source": [
    "from langchain_core.messages import SystemMessage\n",
    "from langgraph.graph import END, MessagesState, StateGraph\n",
    "from langgraph.prebuilt import ToolNode, tools_condition\n",
    "\n",
    "\n",
    "class State(MessagesState):\n",
    "    # highlight-next-line\n",
    "    context: List[Document]\n",
    "\n",
    "\n",
    "# Step 1: Generate an AIMessage that may include a tool-call to be sent.\n",
    "def query_or_respond(state: State):\n",
    "    \"\"\"Generate tool call for retrieval or respond.\"\"\"\n",
    "    llm_with_tools = llm.bind_tools([retrieve])\n",
    "    response = llm_with_tools.invoke(state[\"messages\"])\n",
    "    # MessagesState appends messages to state instead of overwriting\n",
    "    return {\"messages\": [response]}\n",
    "\n",
    "\n",
    "# Step 2: Execute the retrieval.\n",
    "tools = ToolNode([retrieve])\n",
    "\n",
    "\n",
    "# Step 3: Generate a response using the retrieved content.\n",
    "def generate(state: MessagesState):\n",
    "    \"\"\"Generate answer.\"\"\"\n",
    "    # Get generated ToolMessages\n",
    "    recent_tool_messages = []\n",
    "    for message in reversed(state[\"messages\"]):\n",
    "        if message.type == \"tool\":\n",
    "            recent_tool_messages.append(message)\n",
    "        else:\n",
    "            break\n",
    "    tool_messages = recent_tool_messages[::-1]\n",
    "\n",
    "    # Format into prompt\n",
    "    docs_content = \"\\n\\n\".join(doc.content for doc in tool_messages)\n",
    "    system_message_content = (\n",
    "        \"You are an assistant for question-answering tasks. \"\n",
    "        \"Use the following pieces of retrieved context to answer \"\n",
    "        \"the question. If you don't know the answer, say that you \"\n",
    "        \"don't know. Use three sentences maximum and keep the \"\n",
    "        \"answer concise.\"\n",
    "        \"\\n\\n\"\n",
    "        f\"{docs_content}\"\n",
    "    )\n",
    "    conversation_messages = [\n",
    "        message\n",
    "        for message in state[\"messages\"]\n",
    "        if message.type in (\"human\", \"system\")\n",
    "        or (message.type == \"ai\" and not message.tool_calls)\n",
    "    ]\n",
    "    prompt = [SystemMessage(system_message_content)] + conversation_messages\n",
    "\n",
    "    # Run\n",
    "    response = llm.invoke(prompt)\n",
    "    context = []\n",
    "    # highlight-start\n",
    "    for tool_message in tool_messages:\n",
    "        context.extend(tool_message.artifact)\n",
    "    # highlight-end\n",
    "    return {\"messages\": [response], \"context\": context}"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0c7f4bcd-d1cc-48ef-9717-911ff6fc681e",
   "metadata": {},
   "source": [
    "We can compile the application as before:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "id": "2addebfd-4236-4eec-abb5-7d742c5b49c9",
   "metadata": {},
   "outputs": [],
   "source": [
    "graph_builder = StateGraph(MessagesState)\n",
    "\n",
    "graph_builder.add_node(query_or_respond)\n",
    "graph_builder.add_node(tools)\n",
    "graph_builder.add_node(generate)\n",
    "\n",
    "graph_builder.set_entry_point(\"query_or_respond\")\n",
    "graph_builder.add_conditional_edges(\n",
    "    \"query_or_respond\",\n",
    "    tools_condition,\n",
    "    {END: END, \"tools\": \"tools\"},\n",
    ")\n",
    "graph_builder.add_edge(\"tools\", \"generate\")\n",
    "graph_builder.add_edge(\"generate\", END)\n",
    "\n",
    "graph = graph_builder.compile()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "id": "dacfd0b1-fbd8-4cd2-b429-34c47e0bafb0",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAALcAAAGwCAIAAABkfmPEAAAAAXNSR0IArs4c6QAAIABJREFUeJztnWdAU9f7x092SMgiYW+UqigoCk7EvVBx1L2tVq2ita1trbXWX1tHh3XjqLN122q1ThTrFsE9UIYKgswkkJA9/y+u/0g1JKBJ7rlwPq+Sm3vP/ebmm3Oee+45zyGZzWaAQNiEjLcABAFALkHYB7kEYR/kEoR9kEsQ9kEuQdiHireAd6UsX6uQG1Ryg15v1qlNeMupFQw3Mo1BZnMpbB7NM4COtxz7ENUlObcVT+8rnj5QhkawjUYzm0v18KaTKXjLqjVlzzVKuYHOpDzPUoa1cA+LdA9pzsJbVI2QCNerlnldfvUfcXAzdkgzdmgkm0oj4a3ondAojU/vK4ueaUqeqTsOFIVFsvFWZAUiuaSyXH/69xKRP6PTQCGTTZx6o3ZUlOmv/iMmk0i9xnvDZn3CuCT3jiLthGTgND+eiIa3FidSVqD9a23h0Fn+3sFMvLW8ghguKcxWP7gq6zvJB28hLuLgyoJe43z4nrD8HwjgknuXZIU5qoQPfPEW4lIOrips28cjuBkUIS3s/SVFT9S5d6oamkUAAMPnBpzbV6qUGfEWAmB3iUZlunG2YujsALyF4MOYr0LO7i3FWwWA3SWX/y4Pj3bHWwVuMJgk7yDGjTMVeAuB2CUVpfrSfE2ztly8heBJ+wTh9VMSE95dyvC65P5lWechXq45l0KhePz4MV6H26bbcK9bqThXJ7C6xAzuXa4MaurmmrONGjXqyJEjeB1um4BwVmaazEmF1xJIXfL0gTK0hev6qnU63dsdiPUjvPXhtYErpNIYZEmxE09hF0hdUvRUHR7NcUbJO3bsSEhIiIuLmzJlSnp6OgBgwIABUqn04MGDMTExAwYMwH719evXJyYmtmvXrn///snJyUbjyzvSH3/8sXfv3hcvXhwyZEhMTExGRsabhzucJrHc51kqZ5RcSyB9Jlz6XBMW6fi7m/T09HXr1vXt27djx45Xr15VqVQAgJ9++ikpKalNmzZjx46l0+kAAAqFcv369fj4+ICAgKysrG3btnG53HHjxmGFKBSK5OTk+fPnq9Xq2NjYNw93OCwOpQC55E2UMiOb6/jneUVFRQCAESNGREVFJSQkYBsjIiKoVKpIJGrVqhW2hUKh7Ny5k0R6+citsLDw3LlzFpfodLqFCxe2aNGipsMdjjuXqpQZnFR4bYDUJSq5gc11vLa4uDgul/vNN998/vnncXFxNvaUSqW//fZbWlqaXC4HAHA4r5o/JpNpsYhrYHEpSjmenbBQxiVmQGeSyRTHPz0XiUTbtm0LDg6eO3fulClTysrKrO4mkUjGjh2bnp7+0UcfrV27tlmzZpa4BADAYrn62QqFQsJ3LAGULiEBCpXkpDo2JCRkzZo1GzZsyM3NXbx4sWV79aeef/31l1QqTU5O7tOnT/PmzX187D+LdupDU4XMQGPg+UtB6RIAWFyqUu4Ul2B3rbGxsZ07d7Z0hbm5uYnFYss+lZWVAoHAYo7KykrbJnjtcIejkjslSqs9kMYlviFMtdLxLfHDhw+//PLLESNGsFisq1evRkREYNujo6NPnTq1Y8cOLpcbFRUVExNz4MCBDRs2tGzZ8ty5c1euXDGZTJWVlXw+32qxrx3euHFjx8rWakwif4Zjy6wTlOq1LjyoFcZnD5SNohx8MyyTybKzs1NSUtLT01u3br1gwQJ3d3cAQFRUVFZW1okTJx4/fty8efPu3bubTKaDBw+mpqYGBgZ+8803t2/fVqlUMTExV65cefbs2fjx46sX+9rhoaGhjpV96XB5RHseR4DbXxrSUUg6tWnHd3nTloXhLQR/NErTrqV5U5fgeSkgbXHobuSwSPfSfI2N4Z+//PLLsWPH3tzerFmzR48eWT1k+/btDv+jv8bly5cXLlxo9aOAgIDCwsI3t2/bti0srEYTFOSoIzrwHKqxzkBalwAAXuSq009JhyT517RDZWUl1nn6GiRSjV/Ky8uLSnXuH0Oj0UilUqsf1STMtqrti/OGzw1w5+P5f4a0LgEA+Dd2o9BI+Y9UNY395PP5NYWTOMJkMv38/BxV2r1LsrBINr4WgfdOGKNToijrRhXeKvDk2UNlp4EivFXA7RKhLz3gPbfUfdZ7SOs9h9YWxvYSUOn4z+CC2iUAgIh2XDqDfO2YBG8hriblj9LGrTh+jVw0Dss28Eav1bl7oVKtNLVP8MBbiIs4s6s0vDUnJAKKyTgEqEswWnbhk0jgxPZivIU4HYPOfODXAv/GbvBYhDB1CcaTe8rzf5a16S5o1RW6WxuHkHZC8vyxquswL68gPPvj34RILgEAGI3g2j/irJtVrbrwQ5qzhb4ESBFjl9J8TWGOOu2kpF1fYUxPAcA/Wn0dgrkEQ1VlvH9Z9uSewqA3NY7ikCiAzaVyBFSjkRjfhUImy6Q6ldxIIoHM63KuB7VxK07LLnwyrO0/IV1iQS7RFz3TKir0qioDiUxSVDp4sEF+fj6dTvf1dfAsZTaXSiIBFpfC9aD5N3ZjcWDPxQJv32tt4AppXKET0zf8+usero9PvzHRzjsFIYC1jkPABHIJwj7IJbbgcDhublD0fuILcoktqqqq1Go13irwB7nEFnQ6nUKB/QbEBSCX2EKn01WfidNgQS6xBYPBoNFgSZSII8glttBqtXq9Hm8V+INcYgsej+f6+Z4QQuy+V2cjk8nQnTCqS+xAoVAs+SkaMsgltjAajYR+GuookEsQ9kEusQWPx0NxCXKJHWQyGeqhRy5B1ArkElvQ6XRnzysmBMglttDpdAYDnskRIQG5xBYcDofJhGhlNLxALrFFVVWVRqPBWwX+IJcg7INcYgs0CgkDucQWaBQSBnIJwj7IJQj7IJfYgsvlouc4yCV2kMvl6DkOcgmiViCXIOyDXGILGo2G+kuQS+yg1+tRfwlyCaJWIJfYgsViMRhwJcLDBeQSW6hUKq1Wi7cK/EEuQdgHucQWaMoWBnKJLdCULQzkElvw+Xw0mxzNJrdDZWUlGveK6hI7oNHRGMTOHe0kEhMTscsil8spFAqbzcYi2aNHj+ItDR9Qi2MFLy+vmzdvWp7gyOVys9nco0cPvHXhBmpxrDBhwgShUFh9i1AonDBhAn6KcAa5xArx8fEhISGWt2azuWXLli1atMBVFJ4gl1hn9OjRXC4Xey0UCj/44AO8FeEJcol1evToER4ebjabzWZzdHR0s2bN8FaEJ8glNTJq1Cg+n+/n5zd+/Hi8teDMO93jKCoNkmKdXmdynB6ICPSIjQjuLhAImMaQ3LsKvOU4BTqT7OnPcHO3Mx7vLftL5BL9xUPi8kJtcARbWYVGcxEVOoNckKX0b+TWc6w3reblrd/GJYpKw9/JRd1H+3E8UHdLfaDsueb6ifKhs/2ZLOsRSJ3jErMJ7Pwub9CsIGSReoNXELP7aN99Pz+vaYc61yVX/5GweIxGLd0dIQ8BEfcuSrkCSmQc782P6lyXFD1VcwSoFqmHsLjU0ufWU/q8RYtDchegxUDqIVwPul5jvWGps0uUMr3ZhB4j10NMRrNaaf12FfWqIeyDXIKwD3IJwj7IJQj7IJcg7INcgrAPcgnCPsglCPsglyDsg1yCsA9yCcI+yCX1k+Mn/u7WI0YiETukNOQShH2I7RKnTnKuU+H1e7q1K8YTFRW/2LRp9a3b6VQqrXev/lnZmd269h6UOGzrtuT9B/5IOXUN2+1xVuZHMycsX7amXduOAIDbd278tmXdkyfZAoFHdKvYqVNmCYUiAMDkKSNCQxqFhDQ6dHifVqsZOWLCnr3bDx44xeO+HGS1ZNk3mQ/v7d51xIakzEcPNm5alZWVyWS6dewQ/9FHn3A53DcLP7j/lLu79VF5Mlnl4KE9Z0z/OCc368qV8+HhTdes2gIAOHL0zwMHd4nFZT4+fj269x05YjyDwdBoNKvWLL969SIAICoqOmnmPB8f34GDujZt0lytUefmZvF4/D69B0wY/yG2mKTBYNi+Y+PplGMyWWVwcOikidPjOnUFAPz5155z/6YMHzZ269b1Eqk4PLzpvE8XBgW9nIaYk5u1dt3PWVmZQg9RYGCwA39Bp7tEKpXM+XiKVqMZMWK8t5fPhUupd+/e6ta1t+2jbt5Kn//VnF49E4YMHlkll/11aO+n82Zs2rALyxORkXFNo9Us/WGlSq0KDWn0x64t//6bMnjQcCxDa1rapcGDRtgoPC/v6WfzZoSENPri829llRXbd2wsKytZ8csG7NPqhddkEQu7dm0dNGj4il82YlPPd+zcfPDPXUOHjAoODisoyNt/4PfCF88XzP9uz97tp08fmzxphlAoOp1yzLICwvOCvI9mfCISel5Lu7R7z3aFomrO7C8AAL+s+OFs6slxYz8ICWl0NvXkN4vmrV75W1RUNADg0aMHBw788dlnCw0Gw6+/Lln247cb1u8EADx/nvfJp9N4XP6HU5MoFOrvf/xWxx/KFk53yb79v0sk4vXrdkQ0awEAaNeu0+ChPe0etXbdzwMHDMUuGQAgJqb9xMnDMm5c6xzXDQBAoVK/+Xqp5VrHxnY4nXIMc8mNG2kKhaJH9742Ct+1eyuZTP7px3Ucdw4AgMPhLl2+6O7dWy1btn6zcNtEREROnTILey0Wl+/es23h10u6xL/MTiAUeq5ctSxp1rzikiI3N7cxoydRqdT+CYMth3ft0qtrl54AgBYtWsrlsn+OHZo4cbqssuJ0yrEJ46dOmjgdANAlvse4CUN27Nz064qN2FFLfljp4SEEAAwdOip5w0qZXMbj8jZuXk0mkdev28HnCwAAZDJ51erltfkKtcHpLrl1O/298KaYRWpJSUlxfv6zFy8Kjh0/XH17WVkp9qJZsxbVf8W+fQb+77v5z5/nBQWFnL94tlGj8JCQMBvl37l7Mzo6FrMIZjIAQFZ2JuaS1wq3TevWbS2vb968bjAYlixduGTpQmwLFqyIy8t69uiXmnrqy/mzZ838LCyssdWi2rbteOz44Zycx8XFLwAAcXHdsO0kEik2pv2ZsycsezKZL+V5e/sCACTicgadkZFxLTFxGGYRAIBjl0F2ukuqquTh4U3rdEhFhQQAMHHCtPjO3atv9/AQYS/cmP/5FTt17MLl8k6nHJs0cfrVKxfGjJlsu3ylUsHnCSxvORwuVhNYLdw2zGo7S6RiAMDSJau8PL2r7+PnFxAW1njZ0tUbN62a8uGo/gmD5348/81f0d2dAwBQq1VKpQIAIOB7WD7icnkqlUqpVL52CI1KAwAYTUaJVGwwGHx9/GqvvE443SVCoafk/3+A16gpTyZ2vbRajSUusw2NRuvZs1/KmeMRzSIVSkX3bn1s7y8SecnlMsvbigqp5aTvAuY2AIBV2e3adoyNaf/Xob3JG1Z6e/uOHzfltR3E5WUAAE9PbywPsVwuE4k8sY+kUgmVSrWRuwszPfZFnIHT74SbvNfscVZmds7jNz/i8QR6vV72/z9YSUkR9iIgIMjb2+fkqaOWFYwMBoNer7dxlr59BorF5ckbV0ZGtvL29rEtqXnzqDt3b1oWCr54MRUAEBnZ6q2+3yuio2NJJNLhv/dbtlj063Q6LFYYPmysSOSZ88bVMJvNJ08d5bhzgoNCmzVrQSKR0q5fthybdv1y8+ZRNlbXYLPZ/v6B5y+ctX2V3hqn1yUjR0w4cfLIvM9nDh821tPTKz39quWjmDbtSCTSuvW/DHt/TN6zJ5t+W4NtJ5FIs2Z+tujbz2fNnpQ4cJjJaDydcqxXr4Rh74+p6SzhjZsEBYU8f543Yvg4u5LGjfng3LnTX341e+CA98vKSnb+vjm6VUyrlm3e8ZsG+AcOHTLqr0N7Fyz8JK5TV4lE/PeRA8uWrn4vvOmhw/uuXL3Qq2eCRFIuFpc3aRKBHfLv+RShUMRgMC9cOHv7zo3p0+a4ubn5uwX06T1gx85NRqPRzy/g+PHDUqlkwVff2z77xAnTli77Jmn25L59E8lk8l+H9r7j16mO013i4+P784/rN25e/ceuLRwOt13bTpaPgoND53+x+Pc/fvv40tSoyOjpH85Z/tNi7KPOcd2WLVm1fcfG9ckr2Gz3qMjoqKjWtk8U0SyyqKgQu2WwTUBA0E/L123esvann//n5sbq1TNhxvS5DkkTPWvmp15e3ocP78/IuCYUijrHdfMUeWGhiV6n27BxJZvtPnToqJEjXqa6EIm8TqccKyjI9/L0njH9Y8v2uR/PZ7PdD/+9v6pKHhrSaOkPK1tHx9o+da+e/RSKqgMH/ti0eXVIcFhERGRBQf67fyOMOs8A3bE4r+8HAWzeW9oL64ya+/H8QYnD3q6Emvhm0TyD0bBsySrHFus8Bg7qmtBv8Ecz5uIt5CUlz9T3L0mHzvZ/86P6MJfzzNmTZ1NPZmRcs/SMKRSK0WMHWN15+rSPB/QfUsuS58yd+uxZ7pvbO3bs8tWX/3sHyQSjPrjk5MkjeoP+x+Vro1vFYFtYLNbmTXus7szlWJktXROLFi7TG6zEg3W6W64HuLrFQUCLjRaH2M+EEa4BuQRhH+QShH2QSxD2QS5B2Ae5BGEf5BKEfZBLEPZBLkHYB7kEYZ86u8TDj16vp540XEgkEk9kPUdrnV1CpZIkxVpHqELARXmhmsm2Phyuzi4Ji3SXFlvPMIwgNDKJPiSCbfWjOrukaSxHpzHevVjhCGEIWEg7Xi7wpPo3tj4A+y3Xx0nZVcpgUQVedJE/WpSZwJgM5vIibckzlcifHttLUNNub7/qdNaNqrxMpdEAxC+gDlOqquSWORCuxGw2qZQqtr05pPgi8KG7scnvteYENWXZ2s9cr5k+ffqzZ8/wOvu1a9e+/fZbvM7uQNAK9gj71NtetYyMjCtXruCtAgAATpw4kZ2djbeKd6J+uiQjI+Pq1audOnWqxb5OJyEhYf/+/fn5Dpsd43pQi4OwTz2sS5YsWWIwGPBW8TolJSXr16/HW8VbUt9cMnny5IEDBzo2e4dD8PHx8fb2XrZsGd5C3oZ61eLodDoymQyhRSxoNBoqlQqzQqvUn7rk6dOn9+7dg/wHYDKZqamplqQYRKGeuCQnJ2fBggUxMTF4C7FPaGjo5Ml2sjXBRj1pcQoLCwMCAvBWUVvEYrHJZPLy8sJbSG2pD3XJs2fP+Hw+3irqgEgkcnd3NxqtL8sKIYR3yebNm1NSUuwmZoWNkpKSUaNG4a2ithC7xRGLxY8fP46Li8NbyNtw8uRJDodDCPHEdgnCNRC4xfn444/T0tLwVvFOpKen//nnn3irsA9RXZKWltanT5/27dvjLeSdaNu27d69e/Py8vAWYgfU4uCMXq/X6XRstvVhyZBAyLpkzZo1hYWFeKtwDDQajUQiQX5XTDyX7N27V6fTEagPzS4XL15ctGgR3ipsAfVTjzcxm80DBw4kXO+Ibfr27ZuamlpRUSEQ1DiKHV8IFpdkZWWJRCKhUIi3kIYFkVqchw8f7ty5s15axGAwHDhwAG8VNUIkl2RnZ8+cORNvFU6BSqXeunXrzJkzeAuxDsFanHpMSUlJdnZ2fHw83kKsQBiX7N+/v3nz5i1a1GFlN4SjIEaLo1ar165dW+8tcujQoRs3buCtwgrEcIlerz969CjeKpyOh4fHvn378FZhBWL0l3C5OEwHdz1dunSBc0gsMeqS4cOHv7kCZv2DRCL17WtrJWS8IIBLCgoK9Ho95M/DHMWhQ4f++ecfvFW8DgFc4uXltWvXLrxVuAgfH5+UlBS8VbwOYe6EGwhms7m0tNTHx85ity6GAC45dOiQyWQaNszBq0Eiag8BWpyioqKqqiq8VbiO//3vf5cuXcJbxX8gwJ3wpEmTyGQCuNlR+Pr6ZmZmdu7cGW8hryBAi9PQ0Ol0Wq2Ww+HgLeQVBHDJ5s2b+Xz+iBEj8BbScCFATa5Wq+HskXQSGo1m3LhxeKv4DwSISyZOnEihWM+PXi9hMpklJSWVlZXwTH4mQIvTAJHJZGw2G55cLPC6pGfPnlQq1WQyqdVqCoXCZDJNJhOTyWwID4dhA964xMPDo7y8XCqVqtVqhUIhFoslEkl9mmBhg1WrVkE1DBZel4wZM4bJ/M9SCHw+f+zYsfgpch18Pr+srAxvFa+At8UBAIwePTonJ8fytnXr1ps3b8ZVkYvAsr/D05cIiw6rjBo1ik6nY695PN748ePxVuQiSCQSPBaB3SWDBg0KCgrCXjdu3BiqTmuncv/+/RkzZuCt4hVQuwQAMHLkSDqdzuVyYetociocDqe8vBxvFa9wfFxSJTWYTI4sc+bMmUKh8Pvvv3dgmVQqmc2Ht6fObDYrlUp4pkM70iXnD5Zn36ryCXWrKNE5qkwnwRPRyl9omsRw44eI8NZCABzjEr3OvGPxs85DfTwDmXQm7K0YhlZlLHqizrxWMeLTQDJ81Ur//v2PHDkCSferY37R33/IGzgj2D+cRRSLAAAYLEpopHvrnqIDKwvw1mIFlUqlUqnwVvESB9QlGacrKAxKeDRRp8w8uFzJ4ZNbdIJLv06ns/QC4I4D/voFOSoO3/rK54SAxaO8eKLGW8XrkEgQdXg6wCVkCpnvyXCEGHzw8GaY4MtqNnHiRHhW+3OAS6TFGnhc/xaYjGZZOXRLIgsEAnhS8kERQiPeBKrl2whzS9LQUCgU8Kw+iFwCKYsWLYJkPWTkEnjhcrnwjPZFcQmkLF68GG8Jr0B1CaSoVCqdDpbHYcglkLJ8+XJ4Ensil0AKi8VCcQnCDvPnz8dbwitQXQIpBoMBnr5X5BJIWb58OTzz0/BxiUKhyM55/I6FTJ4y4rvvv3KQIuggk8kmkwlvFS/BJy6ZOm1Uh/ad3wtvisvZCcGCBQvwlvAKfOoSeHoCELUBh7pk1JgBFRXSv48c/PvIQW9vn317jmHB2vYdG0+nHJPJKoODQydNnB7XqSu2f+ajBxs3rcrKymQy3Tp2iP/oo0+4nNfHlWk0mlVrll+9ehEAEBUVnTRzno+Pr+u/mgP5+eefGzduPGTIELyFAHxcsvjbn774MqlVyzbDh42l/f+gvV9W/HA29eS4sR+EhDQ6m3rym0XzVq/8LSoqOi/v6WfzZoSENPri829llRXbd2wsKytZ8cuG18rcs3f76dPHJk+aIRSKTqccc3Nzc/33cixQ3ePg4JKmTSKoVKpQKIqMbIVtef4873TKsQnjp06aOB0A0CW+x7gJQ3bs3PTrio27dm8lk8k//biO484BAHA43KXLF929e6tly9bVyywuKXJzcxszehKVSu2fMNj1X8rhzJ07F55JoFDouHvvFgAgLq4b9pZEIsXGtM/KzgQA3Ll7Mzo6FrMIACA2tgMAAPuoOj179NNoNF/On/30aa7L5TsFNzc3BgOWcaJQuESpVAAABHwPyxYul6dSqZRKpVKp4PNerYzJ4XABAGLx67Mj27XtuGzpammFZMqHo35Z8QM843femvXr1584cQJvFS/BzSXVh8qKRF4AALlcZtkilUqoVCqTyRSJvKpvr6iQAgDc3a1kuWzXtuPW3/bN/OiT4yf+3rtvp/O/gXORy+XwzMfBxyVuTDeJRGx526xZCxKJlHb9MvZWp9OlXb/cvHkUhUJp3jzqzt2blhyNFy+mAgCwgIZOo1dVyS2HYD1Rw4eNFYk8c965yw53kpKS+vfvj7eKl+DTqxYZGZ167tSevTs4HG7ziKiwsMZ9eg/YsXOT0Wj08ws4fvywVCpZ8NX3AIBxYz44d+70l1/NHjjg/bKykp2/b45uFdOqZRsAQOPGTU6cPLI++ddpH84+dHjflasXevVMkEjKxeLyJk0icPleDgSqrMCUdx8Tdfvfyvfa8GiMOlRLzZtH5eZmnTl7IifncdOmzYODQmNjOiiVipOnjpw7d5rNYs/7bCEWqHK5vMgW0Rk3rv1z7K+s7Efduvb+fN4iLKyLaBZZVFR4+fK/gwePVCir7t65eTb1ZF7+0379EidNnF77GwS1wliYpWzRife2F8AprF+/XiwWh4eH4y0EOGYG6LZFzwZMC3LjwDIYoq5Ii7XXjpaO+iIIbyH/YdmyZeHh4ZCs5IHGl0BKUlISJAkHkEvgBaq4BIr+EsSboP4ShH2g6i9BLQ6koLgEYR8UlyDsg+IShH1QXIKwD4pLEPZBcQnCPiguQdgHxSUI+9S3uETkzyB0lUSikPlesIwwtVDf4hKT0VxRAl0mzNojKdJQ4EtqXN/ikqCm7Cqp3hFi8EElNwSEs/BW8TpQxSWOSWO9a1l+uwQvnxDizZXKvV2V91A+ZJY/3kJep6qqikqlQjL9zDEuMZvBnuXPW3QWCH2ZPBF81bc1Kst0JXmq4qeqxGl+gIS3GrhxZEr8tJPS3DtVLA61vEDjqDIBACaTmQQAiezIX9LDh6HXmd5rw4npKajF7jiwfv360NDQhIQEvIUAB98Jt+/n0b6fh8EAzEZHpqVPTk7m8/ljxoxxYJkUKgnClZOqA1Vc4vg7cioVAKpDa3CygUQx0hgNq1Wob/0lCGdQ3/pLnA2bzYYk1HclUPWXEKAuUSqV8KxN5jLqeVzicHg8HpvNxluFq0FxSd2QyWTw5HtxGSguqRsNsy5BcUndaJh1CYpL6gaDwaDRiNHr70BQXFI3tFqtXk/gZ85vB4pLEPaBKi4hgEvc3d1ZLOjGfzgbFJfUDYVCgeISfIFFhw1oNBo818tloLikbuj1+nqQv7WuoLgEYR8Ul9SNhtn3iuKSutEw+15RXIKwD4pL6gaZTG6AdQmKS+qGyWSCZ51Dl4HikrrB5XIbYPSK4pK6IZfLlUol3ipcDYpLEPahUCgOnFD3jhCgxWmYjB07FsUldYDFYjGZTLxVuBoUl9QNlUplWWur4YDiEoR9UH8Jwj6ov6RuNMynfSguqRsymQz1l+ALAVzSMEFxSd3gcDgNcHQ0ikvqBpaHDm8VrgbFJQj7oLikbnA4nAZ4j4PikrrR78h4AAAVFElEQVTRMFscFJfUDTqd3gBzIaG4pG7odDqdToe3CleD4hKEfVBcUjca5uhoqOISR2YYdyzDhw9/+vQpifQfhWFhYQcPHsRVV0ME3v/ogAEDsFQDpP+HwWCMGzcOb10uAsUltWLYsGEBAQHVtwQHBw8aNAg/RS4FqrgEXpew2ezExEQKhWJ5O3LkSLxFuY6kpKT+/fvjreIl8LrkteokODh48ODBeCtyHRwOB5686lC7hMViJSYmUqlUFos1bNgwvOW4FBSX1IH333/f398/ICAgMTERby0uBaq4xM6dcHmh9ta5ytJ8jVqBWzYio9FIIpHw6jJhsqgUGvANdYvtLeAKXZfejTDr9uVlqq4dk7Ts6sH3pLu5w9LD42JIJKCQGeQSfcapsr4Tfb2DoVt52AXU6JJH6VWPb1T1HOvncknwcmJLYYf+HkFNXTFwDqp1+6xX4xqVKQtZ5A36Tg64cabCNZ3VUMUl1tuR4mdqx665WT8gU4BOayov0HoFOb3dgeo5jvW6RC42+ARDETfBhl8jVkWp1gUnIkB/iVZt1GkbXPqh2qBVmXQ6VzQ5qL8EYR8CxCUI3IEqLoFFB+I10LhXhH1QXIKwD4pLEPZBcQnCPiguQdgHxSUI+6C4BGEfFJcg7IPiEoR9UFziIoxG4/37d/BW8ZaguMRF/Lzi+6yszO1bD+At5G2AKi5xVl1SWPjcSSVXx/bQbp3WFQNBnARU40sc5laJRLx23c83b16n0mht2rS7eDF104ZdoaGNAABHjv554OAusbjMx8evR/e+I0eMZzAYOblZs+d8sHzpms1b1j55ku3t7Tv9wzmdOnXBSisuKUpO/vXmret0OuO98KYffDCzaZMIAMDqNT9euJg679OFyRtXvnhR8MvPyYEBwVu3J1+/fkWpVAQGBo8ZPblnj74AgOU/Lf73/BkAQLceMQCAPbuP+vr4AQBu37nx25Z1T55kCwQe0a1ip06ZJRSKHHURHAhU414d4xKj0bjg67nSCsnHH8+XSsW/bVkX3SoGs8iOnZsP/rlr6JBRwcFhBQV5+w/8Xvji+YL53wEAtFrt/76fPzvpc18fv+07Nv6w9Ot9e47xeHyJRDx7zgf+/oFJs+aRSKSUlOMfz526MfkPrEClUrF1e/Lcj+drNOrW0bHFJUWPHz8clDiMx+VfvHxuydKF/v6BzZo2Hzfmg/Ky0uLiF1/N/w4AIPQQAQBu3kqf/9WcXj0ThgweWSWX/XVo76fzZmzasAvCNTPqYVzy6NGD7JzH3y5a3rVLTwDA8+d5J08d1el0crls955tC79e0iW+B7anUOi5ctWypFnzsLezkz7v3q03AGDq1KTpM8bdvXcrvnP3P3ZtEfA9Vvy8AWuYe/VMGDdh8LETh2fPmoelRpr36cJmzVpgJfj5+u/YdpBEIgEA+vUbNOT9nleunG/WtHlAQBCPx5dWSCIjW1l0rl3388ABQ+fM/gJ7GxPTfuLkYRk3rnWO6+aQ6+BAoIpLHKOjrLwUAODn93JOb0BAkMlkUqtVN29eNxgMS5YuXLJ0IfYRFkmIy8uwt27Ml02vt7cvAEAsLgcAXL9+pay8NGFAZ0v5er2+vKwUe81kMi0Wwch9kr1j56asrEysVpNKJVZFlpQU5+c/e/Gi4Njxw/8R//8lQwVU/SWOcYm/fyAA4P79O++FN8WqFpHIk8fjS6RiAMDSJau8PL2r7+/nF/As70n1LTQqDQBgMhkBANIKSYcOnadNnV19BzbbHXvh5vaf6TC3bmd8OX92dKuYLz7/ls1iL1r8uclsfcRuRYUEADBxwrT4zt2rb/fwQHGJHRzjkibvNYuNab/5tzWlpcWVsoorVy8s/HoJAIDD4WI7BAWF1L40Docrk1XW8pA//tji5xewdMkqrH62VE4Y1W+C3N05AACtVlMnMXgBVVzisDvh2UmfBwQEFRTm83mCdWu3YwFKdHQsiUQ6/Pd+y25qtdpuUa1bt33w4G5W9qPaHCWTVzZu9B5mEZ1Op1KrLIsPM5luUqnE8jYgIMjb2+fkqaOW0gwGg16vf4cv7URmzpzZr18/vFW8hLJ48eI3t77IVRsNwCe0tvfrBoNhwqShCf0Gt2rZxtPTCwDA4/LpdDqXy6uqqkpJOZ6d80ir1aZdv7J0+TfR0bFCoUgqlfxz7FCP7n0DA4OxyGPP3u1tYztERESGhYWfOXvizJkTRqOxoDB/9+5tFy6ldu/WBwtZ8vOfjRwx3nLq/Od5Fy6cFQg8SktLVq1Z/uJFAQmAAQOGkkgkhaLq3L+nJZLyqip5WVlJUFCIt7fviRNHrl67aDaDzMz7a9b+pDfoIyIia3+9CrNV7nyKd5DT74mYTCY8WW4d0+JQqdSYNu3/2LXFYHiZmoDjzlmzemtISNismZ96eXkfPrw/I+OaUCjqHNfNU+RluzR/v4B1a7Zt2LRq955tJBIpPLzpkME1ZkH6YNJHUol47bqfORzugP5DRwwb9+uqpbfv3GgdHdurV0JWdmbKmePX0i717TOwY8f4znHdli1ZtX3HxvXJK9hs96jI6Kio1g65Ag5n06ZNwcHBffv2xVsIqHE2efopqVYDWnXzqH1BRqMRy25lNpuLil9M/XDUiOHjJk+a4VC1+JN2rNwnhB7ZiefsEy1btiw8PByS3D6OqUu0Wu3MpIleXj4to1rTaPT7929rNJpGjd5zSOENkxkzZmApKmHAMS4hkUi9e/U/d+709h0b6XR6aGjjbxctf+2GE1EnBAIB3hJe4RiX0On0kSPGVw8qEe9IcnJyeHh4r1698BYC6vn4EkIjlUoVCgXeKl4Cy5MCxGskJSXVtzthhMPh8/l4S3gFanEgZdWqVampqXireAlyCaRIpVKNRoO3ipegFgdSZs2aBc+alsglkOLt7V2LvVwEanEgZc2aNTdu3MBbxUuQSyDl6dOntRll4RqstzhUOtkMIF2pDV8YLDKF6opMuHPnzhWJYBlEZ70uYfMokuIGtzhrbSgv1HAFrngIFxIS4u7u7oIT1QbrLhH6MMwmVJdYgUIhefi4YsGC77777vHjxy44UW2w7hKRP91dQLl7QepyPVBz/US5f7gbi+uKYC43N9doNLrgRLXB1son5/8sN5vJrbp6UOkNPSe9TmPKOC0W+tJje7mo47ykpEQoFEIyxMTOKko3z1bcvyIjkUlu7hQXqvoPZpMJAEDCaRUlOoNcUaZzc6e06MBt4fwhanBif9VpsxnIJXqlHLe1tg4ePMjhcPAaAUoCwF1Ac+dRSa516fTp05cuXSoUCl161hqw3/dKIgGeiMYT4Vf1MSuo7sAvDJYJ+K4hKysLnpEDqFcNUlavXg3PJFACuIRGo8Ezr9pltGzZEm8JryCAS/R6vWWaTwNBqVTOnz8fbxWvIIBLOBwOi+WKBRXhoaKiAp4uNWKMHFAqlXgtJowXAoHghx9+wFvFKwjgEh6PB0+07xrYbHaLFi1qsaOLIMB/VKfTVVZW4q3CpaSlpa1duxZvFa8ggEsYDAY8I0BdQ15eHlRfmQAtDovFevHiBd4qXEr//v2xTHGQQACXsNlseGa5uQZ4+tMwCNDiCAQC2K6as1m3bl1mZibeKl5BAJfweLz79+/jrcKlnD9/Hp7E0cRwiUgkEovFeKtwKZ988klwcDDeKl5BDJfodDpLEr2GQKdOnaDqSIRIig34fH5BQQHeKlzEo0ePVqxYgbeK/0AMl7Rp06akpARvFS7i7t27doeGuRgC3AljAWxmZma7du3wFuIKEhMTYRspQYy6pGnTplA9I3UqLBYLtudWxHBJZGRkRUUF3ipcgUajGTFiBN4qXocYLvH29q6oqHj69CneQpzO9evXAwIC8FbxOvbH0EPC1q1bPTw8hgwZgrcQ51JZWUmlUuGZ+4lBjLoEANCrV6/ff/8dbxVOh81mw5PcxgJhXBIUFBQQEHD16lW8hTiRe/fuTZs2DaqnwRiEcQkAYNSoUfv27cNbhRO5d+/exIkT8VZhBcLEJRjDhw9fuXIlhPFd/YZIdQmWk27lypV4q3AKhYWFubm5eKuwDsFc0rVrVxqNdubMGbyFOJ7Ro0f7+fnhrcI6BGtxsIV4OnTokJ6ejrcQR5KVlaXX66EaN18d4rkEAJCSkvL48eM5c+bgLaShQLAWB6N37956vX7Pnj14C3EMn3322YMHD/BWYQtC1iUYSUlJ48aNa9++Pd5C3omTJ0+azWZI1g2uCQK7BAAwduzYVatWeXp64i2knkPIFsfC7t27BwwYQNCMBEVFRV9//TXeKmoFsV0CALh48WJ8fDzeKupMVVXVhg0blixZgreQWkHsFgejsrJy3rx5W7ZswVtIvYXwdQk2dnrJkiWQB4AWDAYDJKsE1wFzfaGgoCAxMRFvFfb58ssvTSYT3irqRn1ocSzk5+cnJyf/+OOPeAupb9SHFsdCcHDwlClTRo8ejbcQ6wwfPlypVOKt4q3AuzJzPA8fPvzss8+qbxk5cqTrZbx20oMHD6rVatfLcAj1qi7BiIiIGDt27NSpU7G3/fv3z8/PP3z4sCs1rF27Njc3d/jw4ZYtw4YNYzKZrtTgQOpVXFKdGzduHD16NCcnJycnBwDQpUsXV06rHD16dHZ2NolE8vHxcXd3X7Fihb+/v8vO7nDqYV2CERMTk5GRgVkEW+CsrKzMNad++PChTCbDhq+WlJTo9XpCW6Q+uyQxMbG8vNzyViKR3Lx50zWnvnz5cmlpqeVtfn4+UfpyaqJ+uqRfv36vpWJTqVSXLl1yzdmvXLny2paysrL+/fu75uzOoH665OTJkx06dAgMDKRQKJbA6+HDh3K53NmnzsnJqaiosMyWoFKp/v7+nTt3Pn78uLNP7TzgmtvuQNatW5ebm3v58mWs/i8uLpZKpTdv3uzWrZtTz5uenl5SUkIikby9vQMDA+Pj4+Pi4gIDA516UmdTT+5xzGbw7KGq7LlGITMoZUYKlayU6S2farQapVKpqKpisVienl5OVVJUXGQ0GNju7u5sNp3+ah1IrpCm15nYXCpPRPUKZAQ1IVJmfcK75Mld5b3Lshe5KoG/O41JozIoVDqFRqeazHBl2CKTyHqtwaAzGPRmrVxdJdEENWW3jOcFvgdRlr2aILBLnj9WXTgkZnLcmDwmx5NIf00AgNlklpeplBIljWbqMlTkFeiK1WffGqK65MSOMnGR3quxB5MDV0KYuqKQqMufSIObsbuPgGKJPqsQzyUmE/hjSb4gyIPrRbD6wwbSArlJoxo2B9LON4K5xGgw/77kuV9zbwYbioV2HYhCotZI5cPmwDi9j2D9Jb99/SyotV/9swgAwF3o5ibi7f4RxoSlRKpL9v9ayPEVsPhEfbJaG2QlVQyKtu8Eb7yF/AfC1CXpKRVuAvf6bREAAM+Ho1FTHmc4vY+4ThDDJVq16VZqBdenQaxswfHlXTwEV959Yrjk0mGxd2MPvFW4CAqNzPfj3DgLUeZSArhEVWUsLdAJAmCsSK7fODLvm3ZyuYP/+qJQQdZNiEbIEsAleQ+VZFo9vKmxAZlCMhrAi1w13kJeQgCX5NxRsoX1pwOtlrCErCf3YFmHjgAjB9RKk0+wU1yi02lOnt1w+95pvV7rKQruGje2VWQvAMDFq3vv3D8b33H0ybMbqqrE/n5Nhw/6ysszBDvqRVHW3yd+LXiRyeWIPIVBzhAGAOB6sqWlUicVXldgd4laYZSJtT5OKNlkMm3b/VlFRXH3+Inu7h5Pnt7cdWChVqdu1yYRAPC88MGFK7uHD1pgNBr+PLps36Hv5kzfBgAoLc/bsO0jNouf0GsmhUw9c36rE6QBAACVQSl+qnJS4XUFdpco5UY60yki72f++yzvzoLP/uZxPQEAraP6aHWqy9f2Yy4BAEwe+wuXIwQAxLUf8c+p1UqVjM3iHT+9lkQiz56+1Z0tAACQyORD//zkDHlkColEAjqNic7EPyqA3SVqucFJT30fZV0xmgxLf32V2N5kMroxXyWAZ9BfjvwQ8H0BAHJ5OY3KyMpN6xD7PmYRAACF7MQLyBExlXIjcol9yFSSXuOUJDZVCgmXI5oxef1/TmftV6dSaJiH5FVio9HgIfB1hp43Ucv1VCoU2cZhdwmbR9VrjM4omeXGVSgrBHxfGq22I4CwKkShcFF/l1ZlYPOg+IHwr81sw+ZSdWqn1CWNG8WaTMar6X9Ztmh1dvonmEy2SBh492GqwaC3vee7Y9SbaAwymeLs89QKKKxqAxqDxPdkGLRGKsPBF6xNy37Xb/x97PTaispif98mRSU59zPPfzFnP51u64Fi725T9/z57drNU9u2HkAiky9d2+9YVRa0Sr1PCCxDYmF3CQDAN5QhLVN6BHIdWyyVSvtw4poTKetv30u5lnHYUxjUse1QCsXOBWndsq9aXXX+yu5jKWu9PcOCA1uUi/MdKwxDIVaGt4DlATgBxpc8z1Jd/LsiIMoZnSbw8uRawbA5/jwRFI8mCFCXBDVhUakVJoOZXEPAbzabv1na0+pH7iy+QlX55vbmTeNHv/+tA0Wu3zK9uNTKihR8rnelvPTN7Tyu1+ez99ZUmkah8wpgQmIRYtQlAIAHV2SZNzVe4aKadpBWFFndbjDoqVQr15pOd7P0eTgEmbzcaLQS0tYkgEym8Hk1DkgrvFvSdahHADRTdQhQlwAAWnTiZZyp0KkNdDfrgj0EOA8qxjpwHYJConZjA3gsQoA7YQs9RnlVlcjwVuEKVOKqHiOdO021rhDGJUFNWcHhNPFTWB6TOomih6Wtu3L4XrBEJBiEcQkAIKaXgMc3lz2BaKifYynKFDeOdGvcCq7FhAkTvVbn3AGxpBx4hjky9oSBoszyiFhWq3gHdws5BOK5BACQdkKan60XhnpQ6USqC2tCq9SXZIljenCbt4fRIkR1CQDg6X3l2b2lAj+OZ5gHgOK56dtg1JnKn0q0Cm3CZF/PAHinxRPVJRi3/63MvF5FZdAYPBbXi0WmEMMvBp1JUa5UV6oMOkNsT0HTtjBOD6gOsV2CZUHKva14cl9RmK0mU8lUBoVCo9CYdIPeKeMN3hoqg6JX6Yw6o9ls0qkMYVHuYZHs0OZsvHXVCsK7pDqV5XqlzKCSG/U6k0EPVy4kGoNCo5PYPCqLS+UJidGZaaFeuQThJOrDPQLC2SCXIOyDXIKwD3IJwj7IJQj7IJcg7PN/PioelnZIG1UAAAAASUVORK5CYII=",
      "text/plain": [
       "<IPython.core.display.Image object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "display(Image(graph.get_graph().draw_mermaid_png()))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "60608d11-aeb4-4be4-a8c4-5dacaff2e377",
   "metadata": {},
   "source": [
    "Invoking our application, we see that the retrieved [Document](https://python.langchain.com/api_reference/core/documents/langchain_core.documents.base.Document.html) objects are accessible from the application state."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "id": "a21bad79-3773-48b4-9d77-406e29b0cbf2",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "================================\u001b[1m Human Message \u001b[0m=================================\n",
      "\n",
      "What is Task Decomposition?\n",
      "==================================\u001b[1m Ai Message \u001b[0m==================================\n",
      "Tool Calls:\n",
      "  retrieve (call_oA0XZ5hF70X0oW4ccNUFCFxX)\n",
      " Call ID: call_oA0XZ5hF70X0oW4ccNUFCFxX\n",
      "  Args:\n",
      "    query: Task Decomposition\n",
      "=================================\u001b[1m Tool Message \u001b[0m=================================\n",
      "Name: retrieve\n",
      "\n",
      "Source: {'source': 'https://lilianweng.github.io/posts/2023-06-23-agent/'}\n",
      "Content: Fig. 1. Overview of a LLM-powered autonomous agent system.\n",
      "Component One: Planning#\n",
      "A complicated task usually involves many steps. An agent needs to know what they are and plan ahead.\n",
      "Task Decomposition#\n",
      "Chain of thought (CoT; Wei et al. 2022) has become a standard prompting technique for enhancing model performance on complex tasks. The model is instructed to “think step by step” to utilize more test-time computation to decompose hard tasks into smaller and simpler steps. CoT transforms big tasks into multiple manageable tasks and shed lights into an interpretation of the model’s thinking process.\n",
      "\n",
      "Source: {'source': 'https://lilianweng.github.io/posts/2023-06-23-agent/'}\n",
      "Content: Tree of Thoughts (Yao et al. 2023) extends CoT by exploring multiple reasoning possibilities at each step. It first decomposes the problem into multiple thought steps and generates multiple thoughts per step, creating a tree structure. The search process can be BFS (breadth-first search) or DFS (depth-first search) with each state evaluated by a classifier (via a prompt) or majority vote.\n",
      "Task decomposition can be done (1) by LLM with simple prompting like \"Steps for XYZ.\\n1.\", \"What are the subgoals for achieving XYZ?\", (2) by using task-specific instructions; e.g. \"Write a story outline.\" for writing a novel, or (3) with human inputs.\n",
      "==================================\u001b[1m Ai Message \u001b[0m==================================\n",
      "\n",
      "Task Decomposition is the process of breaking down a complicated task into smaller, manageable steps. It often utilizes techniques like Chain of Thought (CoT) prompting, which encourages models to think step by step, enhancing performance on complex tasks. This approach helps clarify the model's reasoning and makes it easier to tackle difficult problems.\n"
     ]
    }
   ],
   "source": [
    "input_message = \"What is Task Decomposition?\"\n",
    "\n",
    "for step in graph.stream(\n",
    "    {\"messages\": [{\"role\": \"user\", \"content\": input_message}]},\n",
    "    stream_mode=\"values\",\n",
    "):\n",
    "    step[\"messages\"][-1].pretty_print()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "id": "6c528829-a4f8-4a2c-8e3e-700d63a0daa2",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[Document(id='c8471b37-07d8-4d51-856e-4b2c22bca88d', metadata={'source': 'https://lilianweng.github.io/posts/2023-06-23-agent/'}, page_content='Fig. 1. Overview of a LLM-powered autonomous agent system.\\nComponent One: Planning#\\nA complicated task usually involves many steps. An agent needs to know what they are and plan ahead.\\nTask Decomposition#\\nChain of thought (CoT; Wei et al. 2022) has become a standard prompting technique for enhancing model performance on complex tasks. The model is instructed to “think step by step” to utilize more test-time computation to decompose hard tasks into smaller and simpler steps. CoT transforms big tasks into multiple manageable tasks and shed lights into an interpretation of the model’s thinking process.'),\n",
       " Document(id='acb7eb6f-f252-4353-aec2-f459135354ba', metadata={'source': 'https://lilianweng.github.io/posts/2023-06-23-agent/'}, page_content='Tree of Thoughts (Yao et al. 2023) extends CoT by exploring multiple reasoning possibilities at each step. It first decomposes the problem into multiple thought steps and generates multiple thoughts per step, creating a tree structure. The search process can be BFS (breadth-first search) or DFS (depth-first search) with each state evaluated by a classifier (via a prompt) or majority vote.\\nTask decomposition can be done (1) by LLM with simple prompting like \"Steps for XYZ.\\\\n1.\", \"What are the subgoals for achieving XYZ?\", (2) by using task-specific instructions; e.g. \"Write a story outline.\" for writing a novel, or (3) with human inputs.')]"
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "step[\"context\"]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b437da5d-ca09-4d15-9be2-c35e5a1ace77",
   "metadata": {},
   "source": [
    ":::tip\n",
    "\n",
    "Check out the [LangSmith trace](https://smith.langchain.com/public/cc25515d-2e46-44fa-8bb2-b9cb0f451504/r).\n",
    "\n",
    ":::"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}