langchain/docs/versioned_docs/version-0.2.x/integrations/text_embedding/openvino.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "ed47bb62",
   "metadata": {},
   "source": [
    "# OpenVINO\n",
    "[OpenVINO™](https://github.com/openvinotoolkit/openvino) is an open-source toolkit for optimizing and deploying AI inference. The OpenVINO™ Runtime supports various hardware [devices](https://github.com/openvinotoolkit/openvino?tab=readme-ov-file#supported-hardware-matrix) including x86 and ARM CPUs, and Intel GPUs. It can help to boost deep learning performance in Computer Vision, Automatic Speech Recognition, Natural Language Processing and other common tasks.\n",
    "\n",
    "Hugging Face embedding model can be supported by OpenVINO through ``OpenVINOEmbeddings`` class. If you have an Intel GPU, you can specify `model_kwargs={\"device\": \"GPU\"}` to run inference on it."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "16b20335-da1d-46ba-aa23-fbf3e2c6fe60",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Note: you may need to restart the kernel to use updated packages.\n"
     ]
    }
   ],
   "source": [
    "%pip install --upgrade-strategy eager \"optimum[openvino,nncf]\" --quiet"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "861521a9",
   "metadata": {},
   "outputs": [],
   "source": [
    "from langchain_community.embeddings import OpenVINOEmbeddings"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "ff9be586",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/home/ethan/intel/langchain_test/lib/python3.10/site-packages/tqdm/auto.py:21: TqdmWarning: IProgress not found. Please update jupyter and ipywidgets. See https://ipywidgets.readthedocs.io/en/stable/user_install.html\n",
      "  from .autonotebook import tqdm as notebook_tqdm\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO:nncf:NNCF initialized successfully. Supported frameworks detected: torch, onnx, openvino\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/home/ethan/intel/langchain_test/lib/python3.10/site-packages/transformers/utils/import_utils.py:519: FutureWarning: `is_torch_tpu_available` is deprecated and will be removed in 4.41.0. Please use the `is_torch_xla_available` instead.\n",
      "  warnings.warn(\n",
      "Framework not specified. Using pt to export the model.\n",
      "Using the export variant default. Available variants are:\n",
      "    - default: The default ONNX variant.\n",
      "Using framework PyTorch: 2.2.1+cu121\n",
      "/home/ethan/intel/langchain_test/lib/python3.10/site-packages/transformers/modeling_utils.py:4225: FutureWarning: `_is_quantized_training_enabled` is going to be deprecated in transformers 4.39.0. Please use `model.hf_quantizer.is_trainable` instead\n",
      "  warnings.warn(\n",
      "Compiling the model to CPU ...\n"
     ]
    }
   ],
   "source": [
    "model_name = \"sentence-transformers/all-mpnet-base-v2\"\n",
    "model_kwargs = {\"device\": \"CPU\"}\n",
    "encode_kwargs = {\"mean_pooling\": True, \"normalize_embeddings\": True}\n",
    "\n",
    "ov_embeddings = OpenVINOEmbeddings(\n",
    "    model_name_or_path=model_name,\n",
    "    model_kwargs=model_kwargs,\n",
    "    encode_kwargs=encode_kwargs,\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "d0a98ae9",
   "metadata": {},
   "outputs": [],
   "source": [
    "text = \"This is a test document.\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "5d6c682b",
   "metadata": {},
   "outputs": [],
   "source": [
    "query_result = ov_embeddings.embed_query(text)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "b57b8ce9-ef7d-4e63-979e-aa8763d1f9a8",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[-0.048951778560876846, -0.03986183926463127, -0.02156277745962143]"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "query_result[:3]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "bb5e74c0",
   "metadata": {},
   "outputs": [],
   "source": [
    "doc_result = ov_embeddings.embed_documents([text])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9a6da5ba",
   "metadata": {},
   "source": [
    "## Export IR model\n",
    "It is possible to export your embedding model to the OpenVINO IR format with ``OVModelForFeatureExtraction``, and load the model from local folder."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "a6544a65",
   "metadata": {},
   "outputs": [],
   "source": [
    "from pathlib import Path\n",
    "\n",
    "ov_model_dir = \"all-mpnet-base-v2-ov\"\n",
    "if not Path(ov_model_dir).exists():\n",
    "    from optimum.intel.openvino import OVModelForFeatureExtraction\n",
    "    from transformers import AutoTokenizer\n",
    "\n",
    "    ov_model = OVModelForFeatureExtraction.from_pretrained(\n",
    "        model_name, compile=False, export=True\n",
    "    )\n",
    "    tokenizer = AutoTokenizer.from_pretrained(model_name)\n",
    "    ov_model.half()\n",
    "    ov_model.save_pretrained(ov_model_dir)\n",
    "    tokenizer.save_pretrained(ov_model_dir)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "162004c4",
   "metadata": {},
   "outputs": [],
   "source": [
    "ov_embeddings = OpenVINOEmbeddings(\n",
    "    model_name_or_path=ov_model_dir,\n",
    "    model_kwargs=model_kwargs,\n",
    "    encode_kwargs=encode_kwargs,\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "92019ef1-5d30-4985-b4e6-c0d98bdfe265",
   "metadata": {},
   "source": [
    "## BGE with OpenVINO\n",
    "We can also access BGE embedding models via the ``OpenVINOBgeEmbeddings`` class with OpenVINO. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "66f5c6ba-1446-43e1-b012-800d17cef300",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/home/ethan/intel/langchain_test/lib/python3.10/site-packages/tqdm/auto.py:21: TqdmWarning: IProgress not found. Please update jupyter and ipywidgets. See https://ipywidgets.readthedocs.io/en/stable/user_install.html\n",
      "  from .autonotebook import tqdm as notebook_tqdm\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO:nncf:NNCF initialized successfully. Supported frameworks detected: torch, onnx, openvino\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/home/ethan/intel/langchain_test/lib/python3.10/site-packages/transformers/utils/import_utils.py:519: FutureWarning: `is_torch_tpu_available` is deprecated and will be removed in 4.41.0. Please use the `is_torch_xla_available` instead.\n",
      "  warnings.warn(\n",
      "Framework not specified. Using pt to export the model.\n",
      "Using the export variant default. Available variants are:\n",
      "    - default: The default ONNX variant.\n",
      "Using framework PyTorch: 2.2.1+cu121\n",
      "Overriding 1 configuration item(s)\n",
      "\t- use_cache -> False\n",
      "/home/ethan/intel/langchain_test/lib/python3.10/site-packages/transformers/modeling_utils.py:4225: FutureWarning: `_is_quantized_training_enabled` is going to be deprecated in transformers 4.39.0. Please use `model.hf_quantizer.is_trainable` instead\n",
      "  warnings.warn(\n",
      "Compiling the model to CPU ...\n"
     ]
    }
   ],
   "source": [
    "from langchain_community.embeddings import OpenVINOBgeEmbeddings\n",
    "\n",
    "model_name = \"BAAI/bge-small-en\"\n",
    "model_kwargs = {\"device\": \"CPU\"}\n",
    "encode_kwargs = {\"normalize_embeddings\": True}\n",
    "ov_embeddings = OpenVINOBgeEmbeddings(\n",
    "    model_name_or_path=model_name,\n",
    "    model_kwargs=model_kwargs,\n",
    "    encode_kwargs=encode_kwargs,\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "72001afb",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "384"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "embedding = ov_embeddings.embed_query(\"hi this is harrison\")\n",
    "len(embedding)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7e86c9ae-ec63-48e9-97ba-f23f7a042ed1",
   "metadata": {},
   "source": [
    "For more information refer to:\n",
    "\n",
    "* [OpenVINO LLM guide](https://docs.openvino.ai/2024/learn-openvino/llm_inference_guide.html).\n",
    "\n",
    "* [OpenVINO Documentation](https://docs.openvino.ai/2024/home.html).\n",
    "\n",
    "* [OpenVINO Get Started Guide](https://www.intel.com/content/www/us/en/content-details/819067/openvino-get-started-guide.html).\n",
    "\n",
    "* [RAG Notebook with LangChain](https://github.com/openvinotoolkit/openvino_notebooks/blob/latest/notebooks/llm-chatbot/rag-chatbot.ipynb)."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.12"
  },
  "vscode": {
   "interpreter": {
    "hash": "7377c2ccc78bc62c2683122d48c8cd1fb85a53850a1b1fc29736ed39852c9885"
   }
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}