map rerank chain (#516)

add a chain that applies a prompt to all inputs and then returns not
only an answer but scores it

add examples for question answering and question answering with sources

docs/modules/chains/combine_docs.md

@@ -6,7 +6,7 @@ For more information on specific use cases as well as different methods for **fe
 
 This documentation now picks up from after you've fetched your documents - now what?
 How do you pass them to the language model in a format it can understand?
-There are a few different methods, or chains, for doing so. LangChain supports three of the more common ones - and
+There are a few different methods, or chains, for doing so. LangChain supports four of the more common ones - and
 we are actively looking to include more, so if you have any ideas please reach out! Note that there is not
 one best method - the decision of which one to use is often very context specific. In order from simplest to
 most complex:
@@ -39,3 +39,13 @@ asking the LLM to refine the output based on the new document.
 **Pros:** Can pull in more relevant context, and may be less lossy than `MapReduceDocumentsChain`.
 
 **Cons:** Requires many more calls to the LLM than `StuffDocumentsChain`. The calls are also NOT independent, meaning they cannot be paralleled like `MapReduceDocumentsChain`. There is also some potential dependencies on the ordering of the documents.
+
+
+## Map-Rerank
+This method involves running an initial prompt on each chunk of data, that not only tries to complete a
+task but also gives a score for how certain it is in its answer. The responses are then
+ranked according to this score, and the highest score is returned.
+
+**Pros:** Similar pros as `MapReduceDocumentsChain`. Compared to `MapReduceDocumentsChain`, it requires fewer calls.
+
+**Cons:** Cannot combine information between documents. This means it is most useful when you expect there to be a single simple answer in a single document.

docs/modules/chains/combine_docs_examples/qa_with_sources.ipynb

@@ -7,7 +7,7 @@
    "source": [
     "# Question Answering with Sources\n",
     "\n",
-    "This notebook walks through how to use LangChain for question answering with sources over a list of documents. It covers three different chain types: `stuff`, `map_reduce`, and `refine`. For a more in depth explanation of what these chain types are, see [here](../combine_docs.md)."
+    "This notebook walks through how to use LangChain for question answering with sources over a list of documents. It covers four different chain types: `stuff`, `map_reduce`, `refine`,`map-rerank`. For a more in depth explanation of what these chain types are, see [here](../combine_docs.md)."
    ]
   },
   {
@@ -259,7 +259,7 @@
    "source": [
     "**Intermediate Steps**\n",
     "\n",
-    "We can also return the intermediate steps for `refine` chains, should we want to inspect them. This is done with the `return_refine_steps` variable."
+    "We can also return the intermediate steps for `refine` chains, should we want to inspect them. This is done with the `return_intermediate_steps` variable."
    ]
   },
   {
@@ -297,10 +297,87 @@
     "chain({\"input_documents\": docs, \"question\": query}, return_only_outputs=True)"
    ]
   },
+  {
+   "cell_type": "markdown",
+   "id": "07ff756e",
+   "metadata": {},
+   "source": [
+    "## The `map-rerank` Chain\n",
+    "\n",
+    "This sections shows results of using the `map-rerank` Chain to do question answering with sources."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 10,
+   "id": "46b52ef9",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "chain = load_qa_with_sources_chain(OpenAI(temperature=0), chain_type=\"map_rerank\", metadata_keys=['source'], return_intermediate_steps=True)"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 11,
+   "id": "7ce2da04",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "query = \"What did the president say about Justice Breyer\"\n",
+    "result = chain({\"input_documents\": docs, \"question\": query}, return_only_outputs=True)"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 12,
+   "id": "cbdcd3c5",
+   "metadata": {},
+   "outputs": [
+    {
+     "data": {
+      "text/plain": [
+       "' The President thanked Justice Breyer for his service and honored him for dedicating his life to serve the country.'"
+      ]
+     },
+     "execution_count": 12,
+     "metadata": {},
+     "output_type": "execute_result"
+    }
+   ],
+   "source": [
+    "result[\"output_text\"]"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 14,
+   "id": "6f0b3d03",
+   "metadata": {},
+   "outputs": [
+    {
+     "data": {
+      "text/plain": [
+       "[{'answer': ' The President thanked Justice Breyer for his service and honored him for dedicating his life to serve the country.',\n",
+       "  'score': '100'},\n",
+       " {'answer': ' This document does not answer the question', 'score': '0'},\n",
+       " {'answer': ' This document does not answer the question', 'score': '0'},\n",
+       " {'answer': ' This document does not answer the question', 'score': '0'}]"
+      ]
+     },
+     "execution_count": 14,
+     "metadata": {},
+     "output_type": "execute_result"
+    }
+   ],
+   "source": [
+    "result[\"intermediate_steps\"]"
+   ]
+  },
   {
    "cell_type": "code",
    "execution_count": null,
-   "id": "aa2b8db9",
+   "id": "e66b8160",
    "metadata": {},
    "outputs": [],
    "source": []

docs/modules/chains/combine_docs_examples/question_answering.ipynb

@@ -7,7 +7,7 @@
    "source": [
     "# Question Answering\n",
     "\n",
-    "This notebook walks through how to use LangChain for question answering over a list of documents. It covers three different types of chaings: `stuff`, `map_reduce`, and `refine`. For a more in depth explanation of what these chain types are, see [here](../combine_docs.md)."
+    "This notebook walks through how to use LangChain for question answering over a list of documents. It covers four different types of chaings: `stuff`, `map_reduce`, `refine`, `map-rerank`. For a more in depth explanation of what these chain types are, see [here](../combine_docs.md)."
    ]
   },
   {
@@ -21,7 +21,7 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 1,
+   "execution_count": 11,
    "id": "17fcbc0f",
    "metadata": {},
    "outputs": [],
@@ -34,7 +34,7 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 2,
+   "execution_count": 12,
    "id": "291f0117",
    "metadata": {},
    "outputs": [],
@@ -49,7 +49,7 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 3,
+   "execution_count": 13,
    "id": "fd9666a9",
    "metadata": {},
    "outputs": [],
@@ -59,7 +59,7 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 4,
+   "execution_count": 14,
    "id": "d1eaf6e6",
    "metadata": {},
    "outputs": [],
@@ -70,7 +70,7 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 5,
+   "execution_count": 15,
    "id": "a16e3453",
    "metadata": {},
    "outputs": [],
@@ -294,6 +294,94 @@
    "source": [
     "chain({\"input_documents\": docs, \"question\": query}, return_only_outputs=True)"
    ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "521a77cb",
+   "metadata": {},
+   "source": [
+    "## The `map-rerank` Chain\n",
+    "\n",
+    "This sections shows results of using the `map-rerank` Chain to do question answering with sources."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 16,
+   "id": "e2bfe203",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "chain = load_qa_chain(OpenAI(temperature=0), chain_type=\"map_rerank\", return_intermediate_steps=True)"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 17,
+   "id": "5c28880c",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "query = \"What did the president say about Justice Breyer\"\n",
+    "results = chain({\"input_documents\": docs, \"question\": query}, return_only_outputs=True)"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 18,
+   "id": "80ac2db3",
+   "metadata": {},
+   "outputs": [
+    {
+     "data": {
+      "text/plain": [
+       "' The president thanked Justice Breyer for his service and honored him for dedicating his life to serving the country. '"
+      ]
+     },
+     "execution_count": 18,
+     "metadata": {},
+     "output_type": "execute_result"
+    }
+   ],
+   "source": [
+    "results[\"output_text\"]"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 19,
+   "id": "b428fcb9",
+   "metadata": {},
+   "outputs": [
+    {
+     "data": {
+      "text/plain": [
+       "[{'answer': ' The president thanked Justice Breyer for his service and honored him for dedicating his life to serving the country. ',\n",
+       "  'score': '100'},\n",
+       " {'answer': \" The president said that Justice Breyer is a former top litigator in private practice, a former federal public defender, and from a family of public school educators and police officers. He also said that since she's been nominated, she's received a broad range of support from the Fraternal Order of Police to former judges appointed by Democrats and Republicans, and that she is a consensus builder.\",\n",
+       "  'score': '100'},\n",
+       " {'answer': ' The president did not mention Justice Breyer in this context.',\n",
+       "  'score': '0'},\n",
+       " {'answer': ' The president did not mention Justice Breyer in the given context. ',\n",
+       "  'score': '0'}]"
+      ]
+     },
+     "execution_count": 19,
+     "metadata": {},
+     "output_type": "execute_result"
+    }
+   ],
+   "source": [
+    "results[\"intermediate_steps\"]"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "c4f86521",
+   "metadata": {},
+   "outputs": [],
+   "source": []
   }
  ],
  "metadata": {