langchain/docs/docs/how_to/output_parser_retry.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "4d6c0c86",
   "metadata": {},
   "source": [
    "# How to retry when a parsing error occurs\n",
    "\n",
    "While in some cases it is possible to fix any parsing mistakes by only looking at the output, in other cases it isn't. An example of this is when the output is not just in the incorrect format, but is partially complete. Consider the below example."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "f28526bd",
   "metadata": {},
   "outputs": [],
   "source": [
    "from langchain.output_parsers import OutputFixingParser\n",
    "from langchain_core.exceptions import OutputParserException\n",
    "from langchain_core.output_parsers import PydanticOutputParser\n",
    "from langchain_core.prompts import PromptTemplate\n",
    "from langchain_openai import ChatOpenAI, OpenAI\n",
    "from pydantic import BaseModel, Field"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "67c5e1ac",
   "metadata": {},
   "outputs": [],
   "source": [
    "template = \"\"\"Based on the user question, provide an Action and Action Input for what step should be taken.\n",
    "{format_instructions}\n",
    "Question: {query}\n",
    "Response:\"\"\"\n",
    "\n",
    "\n",
    "class Action(BaseModel):\n",
    "    action: str = Field(description=\"action to take\")\n",
    "    action_input: str = Field(description=\"input to the action\")\n",
    "\n",
    "\n",
    "parser = PydanticOutputParser(pydantic_object=Action)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "007aa87f",
   "metadata": {},
   "outputs": [],
   "source": [
    "prompt = PromptTemplate(\n",
    "    template=\"Answer the user query.\\n{format_instructions}\\n{query}\\n\",\n",
    "    input_variables=[\"query\"],\n",
    "    partial_variables={\"format_instructions\": parser.get_format_instructions()},\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "10d207ff",
   "metadata": {},
   "outputs": [],
   "source": [
    "prompt_value = prompt.format_prompt(query=\"who is leo di caprios gf?\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "68622837",
   "metadata": {},
   "outputs": [],
   "source": [
    "bad_response = '{\"action\": \"search\"}'"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "25631465",
   "metadata": {},
   "source": [
    "If we try to parse this response as is, we will get an error:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "5c760353-bba1-432c-b1f1-eb33f183be74",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Failed to parse Action from completion {\"action\": \"search\"}. Got: 1 validation error for Action\n",
      "action_input\n",
      "  Field required [type=missing, input_value={'action': 'search'}, input_type=dict]\n",
      "    For further information visit https://errors.pydantic.dev/2.9/v/missing\n",
      "For troubleshooting, visit: https://python.langchain.com/docs/troubleshooting/errors/OUTPUT_PARSING_FAILURE\n"
     ]
    }
   ],
   "source": [
    "try:\n",
    "    parser.parse(bad_response)\n",
    "except OutputParserException as e:\n",
    "    print(e)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f6b64696",
   "metadata": {},
   "source": [
    "If we try to use the `OutputFixingParser` to fix this error, it will be confused - namely, it doesn't know what to actually put for action input."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "78b2b40d",
   "metadata": {},
   "outputs": [],
   "source": [
    "fix_parser = OutputFixingParser.from_llm(parser=parser, llm=ChatOpenAI())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "4fe1301d",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Action(action='search', action_input='input')"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "fix_parser.parse(bad_response)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9bd9ea7d",
   "metadata": {},
   "source": [
    "Instead, we can use the RetryOutputParser, which passes in the prompt (as well as the original output) to try again to get a better response."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "7e8a8a28",
   "metadata": {},
   "outputs": [],
   "source": [
    "from langchain.output_parsers import RetryOutputParser"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "5c86e141",
   "metadata": {},
   "outputs": [],
   "source": [
    "retry_parser = RetryOutputParser.from_llm(parser=parser, llm=OpenAI(temperature=0))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "9c04f731",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Action(action='search', action_input='leo di caprio girlfriend')"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "retry_parser.parse_with_prompt(bad_response, prompt_value)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "16827256-5801-4388-b6fa-608991e29961",
   "metadata": {},
   "source": [
    "We can also add the RetryOutputParser easily with a custom chain which transform the raw LLM/ChatModel output into a more workable format."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "7eaff2fb-56d3-481c-99a1-a968a49d0654",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Action(action='search', action_input='leo di caprio girlfriend')\n"
     ]
    }
   ],
   "source": [
    "from langchain_core.runnables import RunnableLambda, RunnableParallel\n",
    "\n",
    "completion_chain = prompt | OpenAI(temperature=0)\n",
    "\n",
    "main_chain = RunnableParallel(\n",
    "    completion=completion_chain, prompt_value=prompt\n",
    ") | RunnableLambda(lambda x: retry_parser.parse_with_prompt(**x))\n",
    "\n",
    "\n",
    "main_chain.invoke({\"query\": \"who is leo di caprios gf?\"})"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e3a2513a",
   "metadata": {},
   "source": [
    "Find out api documentation for [RetryOutputParser](https://python.langchain.com/api_reference/langchain/output_parsers/langchain.output_parsers.retry.RetryOutputParser.html#langchain.output_parsers.retry.RetryOutputParser)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "a2f94fd8",
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}