langchain/docs/docs/integrations/vectorstores/clickhouse.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "683953b3",
   "metadata": {},
   "source": [
    "# ClickHouse\n",
    "\n",
    "> [ClickHouse](https://clickhouse.com/) is the fastest and most resource efficient open-source database for real-time apps and analytics with full SQL support and a wide range of functions to assist users in writing analytical queries. Lately added data structures and distance search functions (like `L2Distance`) as well as [approximate nearest neighbor search indexes](https://clickhouse.com/docs/en/engines/table-engines/mergetree-family/annindexes) enable ClickHouse to be used as a high performance and scalable vector database to store and search vectors with SQL.\n",
    "\n",
    "This notebook shows how to use functionality related to the `ClickHouse` vector store.\n",
    "\n",
    "## Setup\n",
    "\n",
    "First set up a local clickhouse server with docker:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "8c4d2e16",
   "metadata": {},
   "outputs": [],
   "source": [
    "! docker run -d -p 8123:8123 -p9000:9000 --name langchain-clickhouse-server --ulimit nofile=262144:262144 clickhouse/clickhouse-server:24.7.6.8"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0acb2a8d",
   "metadata": {},
   "source": [
    "You'll need to install `langchain-community` and `clickhouse-connect` to use this integration"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d454fb7c",
   "metadata": {},
   "outputs": [],
   "source": [
    "pip install -qU langchain-community clickhouse-connect"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3df5501b",
   "metadata": {},
   "source": [
    "### Credentials\n",
    "\n",
    "There are no credentials for this notebook, just make sure you have installed the packages as shown above."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "54d5276f",
   "metadata": {},
   "source": [
    "If you want to get best in-class automated tracing of your model calls you can also set your [LangSmith](https://docs.smith.langchain.com/) API key by uncommenting below:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f6fd5b03",
   "metadata": {},
   "outputs": [],
   "source": [
    "# os.environ[\"LANGSMITH_API_KEY\"] = getpass.getpass(\"Enter your LangSmith API key: \")\n",
    "# os.environ[\"LANGSMITH_TRACING\"] = \"true\""
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2b87fe34",
   "metadata": {},
   "source": [
    "## Instantiation\n",
    "\n",
    "import EmbeddingTabs from \"@theme/EmbeddingTabs\";\n",
    "\n",
    "<EmbeddingTabs/>\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "60276097",
   "metadata": {},
   "outputs": [],
   "source": [
    "# | output: false\n",
    "# | echo: false\n",
    "from langchain_openai import OpenAIEmbeddings\n",
    "\n",
    "embeddings = OpenAIEmbeddings(model=\"text-embedding-3-large\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "aac9563e",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2023-06-03T08:33:31.554934Z",
     "start_time": "2023-06-03T08:33:31.549590Z"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "from langchain_community.vectorstores import Clickhouse, ClickhouseSettings\n",
    "\n",
    "settings = ClickhouseSettings(table=\"clickhouse_example\")\n",
    "vector_store = Clickhouse(embeddings, config=settings)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "32dd3f67",
   "metadata": {},
   "source": [
    "## Manage vector store\n",
    "\n",
    "Once you have created your vector store, we can interact with it by adding and deleting different items.\n",
    "\n",
    "### Add items to vector store\n",
    "\n",
    "We can add items to our vector store by using the `add_documents` function."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "944743ee",
   "metadata": {},
   "outputs": [],
   "source": [
    "from uuid import uuid4\n",
    "\n",
    "from langchain_core.documents import Document\n",
    "\n",
    "document_1 = Document(\n",
    "    page_content=\"I had chocolate chip pancakes and scrambled eggs for breakfast this morning.\",\n",
    "    metadata={\"source\": \"tweet\"},\n",
    ")\n",
    "\n",
    "document_2 = Document(\n",
    "    page_content=\"The weather forecast for tomorrow is cloudy and overcast, with a high of 62 degrees.\",\n",
    "    metadata={\"source\": \"news\"},\n",
    ")\n",
    "\n",
    "document_3 = Document(\n",
    "    page_content=\"Building an exciting new project with LangChain - come check it out!\",\n",
    "    metadata={\"source\": \"tweet\"},\n",
    ")\n",
    "\n",
    "document_4 = Document(\n",
    "    page_content=\"Robbers broke into the city bank and stole $1 million in cash.\",\n",
    "    metadata={\"source\": \"news\"},\n",
    ")\n",
    "\n",
    "document_5 = Document(\n",
    "    page_content=\"Wow! That was an amazing movie. I can't wait to see it again.\",\n",
    "    metadata={\"source\": \"tweet\"},\n",
    ")\n",
    "\n",
    "document_6 = Document(\n",
    "    page_content=\"Is the new iPhone worth the price? Read this review to find out.\",\n",
    "    metadata={\"source\": \"website\"},\n",
    ")\n",
    "\n",
    "document_7 = Document(\n",
    "    page_content=\"The top 10 soccer players in the world right now.\",\n",
    "    metadata={\"source\": \"website\"},\n",
    ")\n",
    "\n",
    "document_8 = Document(\n",
    "    page_content=\"LangGraph is the best framework for building stateful, agentic applications!\",\n",
    "    metadata={\"source\": \"tweet\"},\n",
    ")\n",
    "\n",
    "document_9 = Document(\n",
    "    page_content=\"The stock market is down 500 points today due to fears of a recession.\",\n",
    "    metadata={\"source\": \"news\"},\n",
    ")\n",
    "\n",
    "document_10 = Document(\n",
    "    page_content=\"I have a bad feeling I am going to get deleted :(\",\n",
    "    metadata={\"source\": \"tweet\"},\n",
    ")\n",
    "\n",
    "documents = [\n",
    "    document_1,\n",
    "    document_2,\n",
    "    document_3,\n",
    "    document_4,\n",
    "    document_5,\n",
    "    document_6,\n",
    "    document_7,\n",
    "    document_8,\n",
    "    document_9,\n",
    "    document_10,\n",
    "]\n",
    "uuids = [str(uuid4()) for _ in range(len(documents))]\n",
    "\n",
    "vector_store.add_documents(documents=documents, ids=uuids)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "18af81cc",
   "metadata": {},
   "source": [
    "### Delete items from vector store\n",
    "\n",
    "We can delete items from our vector store by ID by using the `delete` function."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "12b32762",
   "metadata": {},
   "outputs": [],
   "source": [
    "vector_store.delete(ids=uuids[-1])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ada27577",
   "metadata": {},
   "source": [
    "## Query vector store\n",
    "\n",
    "Once your vector store has been created and the relevant documents have been added you will most likely wish to query it during the running of your chain or agent. \n",
    "\n",
    "### Query directly\n",
    "\n",
    "#### Similarity search\n",
    "\n",
    "Performing a simple similarity search can be done as follows:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "015831a3",
   "metadata": {},
   "outputs": [],
   "source": [
    "results = vector_store.similarity_search(\n",
    "    \"LangChain provides abstractions to make working with LLMs easy\", k=2\n",
    ")\n",
    "for res in results:\n",
    "    print(f\"* {res.page_content} [{res.metadata}]\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "623d3b9d",
   "metadata": {},
   "source": [
    "#### Similarity search with score\n",
    "\n",
    "You can also search with score:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e7d43430",
   "metadata": {},
   "outputs": [],
   "source": [
    "results = vector_store.similarity_search_with_score(\"Will it be hot tomorrow?\", k=1)\n",
    "for res, score in results:\n",
    "    print(f\"* [SIM={score:3f}] {res.page_content} [{res.metadata}]\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f5a90c12",
   "metadata": {},
   "source": [
    "## Filtering\n",
    "\n",
    "You can have direct access to ClickHouse SQL where statement. You can write `WHERE` clause following standard SQL.\n",
    "\n",
    "**NOTE**: Please be aware of SQL injection, this interface must not be directly called by end-user.\n",
    "\n",
    "If you custimized your `column_map` under your setting, you search with filter like this:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "169d01d1",
   "metadata": {},
   "outputs": [],
   "source": [
    "meta = vector_store.metadata_column\n",
    "results = vector_store.similarity_search_with_relevance_scores(\n",
    "    \"What did I eat for breakfast?\",\n",
    "    k=4,\n",
    "    where_str=f\"{meta}.source = 'tweet'\",\n",
    ")\n",
    "for res in results:\n",
    "    print(f\"* {res.page_content} [{res.metadata}]\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d86fa4bf",
   "metadata": {},
   "source": [
    "#### Other search methods\n",
    "\n",
    "There are a variety of other search methods that are not covered in this notebook, such as MMR search or searching by vector. For a full list of the search abilities available for `Clickhouse` vector store check out the [API reference](https://python.langchain.com/api_reference/community/vectorstores/langchain_community.vectorstores.clickhouse.Clickhouse.html)."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "afacfd4e",
   "metadata": {},
   "source": [
    "### Query by turning into retriever\n",
    "\n",
    "You can also transform the vector store into a retriever for easier usage in your chains. \n",
    "\n",
    "Here is how to transform your vector store into a retriever and then invoke the retreiever with a simple query and filter."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "97187188",
   "metadata": {},
   "outputs": [],
   "source": [
    "retriever = vector_store.as_retriever(\n",
    "    search_type=\"similarity_score_threshold\",\n",
    "    search_kwargs={\"k\": 1, \"score_threshold\": 0.5},\n",
    ")\n",
    "retriever.invoke(\"Stealing from the bank is a crime\", filter={\"source\": \"news\"})"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "57fade30",
   "metadata": {},
   "source": [
    "## Usage for retrieval-augmented generation\n",
    "\n",
    "For guides on how to use this vector store for retrieval-augmented generation (RAG), see the following sections:\n",
    "\n",
    "- [Tutorials](/docs/tutorials/rag)\n",
    "- [How-to: Question and answer with RAG](https://python.langchain.com/docs/how_to/#qa-with-rag)\n",
    "- [Retrieval conceptual docs](https://python.langchain.com/docs/concepts/retrieval)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "db24787c",
   "metadata": {},
   "source": [
    "For more, check out a complete RAG template using Astra DB [here](https://github.com/langchain-ai/langchain/tree/master/templates/rag-astradb)."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "02452d34",
   "metadata": {},
   "source": [
    "## API reference\n",
    "\n",
    "For detailed documentation of all `Clickhouse` features and configurations head to the API reference:https://python.langchain.com/api_reference/community/vectorstores/langchain_community.vectorstores.clickhouse.Clickhouse.html"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.9"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}