The Fellowship of the Vectors: New Embeddings Filter using clustering. (#7015)

Continuing with Tolkien inspired series of langchain tools. I bring to
you:
**The Fellowship of the Vectors**, AKA EmbeddingsClusteringFilter.
This document filter uses embeddings to group vectors together into
clusters, then allows you to pick an arbitrary number of documents
vector based on proximity to the cluster centers. That's a
representative sample of the cluster.

The original idea is from [Greg Kamradt](https://github.com/gkamradt)
from this video (Level4):
https://www.youtube.com/watch?v=qaPMdcCqtWk&t=365s

I added few tricks to make it a bit more versatile, so you can
parametrize what to do with duplicate documents in case of cluster
overlap: replace the duplicates with the next closest document or remove
it. This allow you to use it as an special kind of redundant filter too.
Additionally you can choose 2 diff orders: grouped by cluster or
respecting the original retriever scores.
In my use case I was using the docs grouped by cluster to run refine
chains per cluster to generate summarization over a large corpus of
documents.
Let me know if you want to change anything!

@rlancemartin, @eyurtsev, @hwchase17,

---------

Co-authored-by: rlm <pexpresss31@gmail.com>

docs/extras/modules/data_connection/retrievers/integrations/merger_retriever.ipynb

@@ -1,6 +1,7 @@
 {
  "cells": [
   {
+   "attachments": {},
    "cell_type": "markdown",
    "id": "fc0db1bc",
    "metadata": {},
@@ -25,7 +26,7 @@
     "from langchain.vectorstores import Chroma\n",
     "from langchain.embeddings import HuggingFaceEmbeddings\n",
     "from langchain.embeddings import OpenAIEmbeddings\n",
-    "from langchain.document_transformers import EmbeddingsRedundantFilter\n",
+    "from langchain.document_transformers import EmbeddingsRedundantFilter,EmbeddingsClusteringFilter\n",
     "from langchain.retrievers.document_compressors import DocumentCompressorPipeline\n",
     "from langchain.retrievers import ContextualCompressionRetriever\n",
     "\n",
@@ -70,6 +71,7 @@
    ]
   },
   {
+   "attachments": {},
    "cell_type": "markdown",
    "id": "c152339d",
    "metadata": {},
@@ -92,6 +94,46 @@
     "    base_compressor=pipeline, base_retriever=lotr\n",
     ")"
    ]
+  },
+  {
+   "attachments": {},
+   "cell_type": "markdown",
+   "id": "c10022fa",
+   "metadata": {},
+   "source": [
+    "## Pick a representative sample of documents from the merged retrievers."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "b3885482",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# This filter will divide the documents vectors into clusters or \"centers\" of meaning.\n",
+    "# Then it will pick the closest document to that center for the final results.\n",
+    "# By default the result document will be ordered/grouped by clusters.\n",
+    "filter_ordered_cluster = EmbeddingsClusteringFilter(\n",
+    "            embeddings=filter_embeddings,\n",
+    "            num_clusters=10,\n",
+    "            num_closest=1,\n",
+    "        )\n",
+    "\n",
+    "# If you want the final document to be ordered by the original retriever scores\n",
+    "# you need to add the \"sorted\" parameter.\n",
+    "filter_ordered_by_retriever = EmbeddingsClusteringFilter(\n",
+    "            embeddings=filter_embeddings,\n",
+    "            num_clusters=10,\n",
+    "            num_closest=1,\n",
+    "            sorted = True,\n",
+    "        )\n",
+    "\n",
+    "pipeline = DocumentCompressorPipeline(transformers=[filter_ordered_by_retriever])\n",
+    "compression_retriever = ContextualCompressionRetriever(\n",
+    "    base_compressor=pipeline, base_retriever=lotr\n",
+    ")\n"
+   ]
   }
  ],
  "metadata": {