{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "25bce5eb-8599-40fe-947e-4932cfae8184",
   "metadata": {},
   "source": [
    "# Vald\n",
    "\n",
    "> [Vald](https://github.com/vdaas/vald) is a highly scalable distributed fast approximate nearest neighbor (ANN) dense vector search engine.\n",
    "\n",
    "This notebook shows how to use functionality related to the `Vald` database.\n",
    "\n",
    "To run this notebook you need a running Vald cluster.\n",
    "Check [Get Started](https://github.com/vdaas/vald#get-started) for more information.\n",
    "\n",
    "See the [installation instructions](https://github.com/vdaas/vald-client-python#install)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f45f46f2-7229-4859-9797-30bbead1b8e0",
   "metadata": {},
   "outputs": [],
   "source": [
    "!pip install vald-client-python"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2f65caa9-8383-409a-bccb-6e91fc8d5e8f",
   "metadata": {},
   "source": [
    "## Basic Example"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "eab0b1e4-9793-4be7-a2ba-e4455c21ea22",
   "metadata": {},
   "outputs": [],
   "source": [
    "from langchain.document_loaders import TextLoader\n",
    "from langchain.embeddings import HuggingFaceEmbeddings\n",
    "from langchain.text_splitter import CharacterTextSplitter\n",
    "from langchain.vectorstores import Vald\n",
    "\n",
    "raw_documents = TextLoader('state_of_the_union.txt').load()\n",
    "text_splitter = CharacterTextSplitter(chunk_size=1000, chunk_overlap=0)\n",
    "documents = text_splitter.split_documents(raw_documents)\n",
    "embeddings = HuggingFaceEmbeddings()\n",
    "db = Vald.from_documents(documents, embeddings, host=\"localhost\", port=8080)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b0a6797c-2bb0-45db-a636-5d2437f7a4c0",
   "metadata": {},
   "outputs": [],
   "source": [
    "query = \"What did the president say about Ketanji Brown Jackson\"\n",
    "docs = db.similarity_search(query)\n",
    "docs[0].page_content"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c4c4e06d-6def-44ce-ac9a-4c01673c29a2",
   "metadata": {},
   "source": [
    "### Similarity search by vector"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "1eb72610-d451-4158-880c-9f0d45fa5909",
   "metadata": {},
   "outputs": [],
   "source": [
    "embedding_vector = embeddings.embed_query(query)\n",
    "docs = db.similarity_search_by_vector(embedding_vector)\n",
    "docs[0].page_content"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d33588d4-67c2-4bd3-b251-76ae783cbafb",
   "metadata": {},
   "source": [
    "### Similarity search with score"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "1a41e382-0336-4e6d-b2ef-44cc77db2696",
   "metadata": {},
   "outputs": [],
   "source": [
    "docs_and_scores = db.similarity_search_with_score(query)\n",
    "docs_and_scores[0]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "57f930f2-41a0-4795-ad9e-44a33c8f88ec",
   "metadata": {},
   "source": [
    "## Maximal Marginal Relevance Search (MMR)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4790e437-3207-45cb-b121-d857ab5aabd8",
   "metadata": {},
   "source": [
    "In addition to using similarity search in the retriever object, you can also use `mmr` as retriever."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "495754b1-5cdb-4af6-9733-f68700bb7232",
   "metadata": {},
   "outputs": [],
   "source": [
    "retriever = db.as_retriever(search_type=\"mmr\")\n",
    "retriever.get_relevant_documents(query)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e213d957-e439-4bd6-90f2-8909323f5f09",
   "metadata": {},
   "source": [
    "Or use `max_marginal_relevance_search` directly:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "99d928d0-3b79-4588-925e-32230e12af47",
   "metadata": {},
   "outputs": [],
   "source": [
    "db.max_marginal_relevance_search(query, k=2, fetch_k=10)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}