langchain/docs/docs/integrations/llms/oci_model_deployment_endpoint.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# OCI Data Science Model Deployment Endpoint\n",
    "\n",
    "[OCI Data Science](https://docs.oracle.com/en-us/iaas/data-science/using/home.htm) is a fully managed and serverless platform for data science teams to build, train, and manage machine learning models in the Oracle Cloud Infrastructure.\n",
    "\n",
    "This notebooks goes over how to use an LLM hosted on a [OCI Data Science Model Deployment](https://docs.oracle.com/en-us/iaas/data-science/using/model-dep-about.htm).\n",
    "\n",
    "To authenticate, [oracle-ads](https://accelerated-data-science.readthedocs.io/en/latest/user_guide/cli/authentication.html) has been used to automatically load credentials for invoking endpoint."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!pip3 install oracle-ads"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Prerequisite\n",
    "\n",
    "### Deploy model\n",
    "Check [Oracle GitHub samples repository](https://github.com/oracle-samples/oci-data-science-ai-samples/tree/main/model-deployment/containers/llama2) on how to deploy your llm on OCI Data Science Model deployment.\n",
    "\n",
    "### Policies\n",
    "Make sure to have the required [policies](https://docs.oracle.com/en-us/iaas/data-science/using/model-dep-policies-auth.htm#model_dep_policies_auth__predict-endpoint) to access the OCI Data Science Model Deployment endpoint.\n",
    "\n",
    "## Set up\n",
    "\n",
    "### vLLM\n",
    "After having deployed model, you have to set up following required parameters of the `OCIModelDeploymentVLLM` call:\n",
    "\n",
    "- **`endpoint`**: The model HTTP endpoint from the deployed model, e.g. `https://<MD_OCID>/predict`. \n",
    "- **`model`**: The location of the model.\n",
    "\n",
    "### Text generation inference (TGI)\n",
    "You have to set up following required parameters of the `OCIModelDeploymentTGI` call:\n",
    "\n",
    "- **`endpoint`**: The model HTTP endpoint from the deployed model, e.g. `https://<MD_OCID>/predict`. \n",
    "\n",
    "### Authentication\n",
    "\n",
    "You can set authentication through either ads or environment variables. When you are working in OCI Data Science Notebook Session, you can leverage resource principal to access other OCI resources. Check out [here](https://accelerated-data-science.readthedocs.io/en/latest/user_guide/cli/authentication.html) to see more options. \n",
    "\n",
    "## Example"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import ads\n",
    "from langchain_community.llms import OCIModelDeploymentVLLM\n",
    "\n",
    "# Set authentication through ads\n",
    "# Use resource principal are operating within a\n",
    "# OCI service that has resource principal based\n",
    "# authentication configured\n",
    "ads.set_auth(\"resource_principal\")\n",
    "\n",
    "# Create an instance of OCI Model Deployment Endpoint\n",
    "# Replace the endpoint uri and model name with your own\n",
    "llm = OCIModelDeploymentVLLM(endpoint=\"https://<MD_OCID>/predict\", model=\"model_name\")\n",
    "\n",
    "# Run the LLM\n",
    "llm.invoke(\"Who is the first president of United States?\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import os\n",
    "\n",
    "from langchain_community.llms import OCIModelDeploymentTGI\n",
    "\n",
    "# Set authentication through environment variables\n",
    "# Use API Key setup when you are working from a local\n",
    "# workstation or on platform which does not support\n",
    "# resource principals.\n",
    "os.environ[\"OCI_IAM_TYPE\"] = \"api_key\"\n",
    "os.environ[\"OCI_CONFIG_PROFILE\"] = \"default\"\n",
    "os.environ[\"OCI_CONFIG_LOCATION\"] = \"~/.oci\"\n",
    "\n",
    "# Set endpoint through environment variables\n",
    "# Replace the endpoint uri with your own\n",
    "os.environ[\"OCI_LLM_ENDPOINT\"] = \"https://<MD_OCID>/predict\"\n",
    "\n",
    "# Create an instance of OCI Model Deployment Endpoint\n",
    "llm = OCIModelDeploymentTGI()\n",
    "\n",
    "# Run the LLM\n",
    "llm.invoke(\"Who is the first president of United States?\")"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "langchain",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.18"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}