Add OCI Generative AI new model support (#22880)

- [x] PR title: community: Add OCI Generative AI new model support - [x] PR message: - Description: adding support for new models offered by OCI Generative AI services. This is a moderate update of our initial integration PR 16548 and includes a new integration for our chat models under /langchain_community/chat_models/oci_generative_ai.py - Issue: NA - Dependencies: No new Dependencies, just latest version of our OCI sdk - Twitter handle: NA - [x] Add tests and docs: 1. we have updated our unit tests 2. we have updated our documentation including a new ipynb for our new chat integration - [x] Lint and test: `make format`, `make lint`, and `make test` run successfully --------- Co-authored-by: RHARPAZ <RHARPAZ@RHARPAZ-5750.us.oracle.com> Co-authored-by: Arthur Cheng <arthur.cheng@oracle.com>
2025-09-10 15:33:11 +00:00 · 2024-06-24 11:48:23 -07:00
parent 753edf9c80
commit f5ff7f178b
10 changed files with 885 additions and 146 deletions
--- a/docs/docs/integrations/chat/oci_generative_ai.ipynb
+++ b/docs/docs/integrations/chat/oci_generative_ai.ipynb
@@ -0,0 +1,190 @@
+{
+ "cells": [
+  {
+   "cell_type": "raw",
+   "id": "afaf8039",
+   "metadata": {},
+   "source": [
+    "---\n",
+    "sidebar_label: OCIGenAI\n",
+    "---"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "e49f1e0d",
+   "metadata": {},
+   "source": [
+    "# ChatOCIGenAI\n",
+    "\n",
+    "This notebook provides a quick overview for getting started with OCIGenAI [chat models](/docs/concepts/#chat-models). For detailed documentation of all ChatOCIGenAI features and configurations head to the [API reference](https://api.python.langchain.com/en/latest/chat_models/langchain_community.chat_models.oci_generative_ai.ChatOCIGenAI.html).\n",
+    "\n",
+    "Oracle Cloud Infrastructure (OCI) Generative AI is a fully managed service that provides a set of state-of-the-art, customizable large language models (LLMs) that cover a wide range of use cases, and which is available through a single API.\n",
+    "Using the OCI Generative AI service you can access ready-to-use pretrained models, or create and host your own fine-tuned custom models based on your own data on dedicated AI clusters. Detailed documentation of the service and API is available __[here](https://docs.oracle.com/en-us/iaas/Content/generative-ai/home.htm)__ and __[here](https://docs.oracle.com/en-us/iaas/api/#/en/generative-ai/20231130/)__.\n",
+    "\n",
+    "\n",
+    "## Overview\n",
+    "### Integration details\n",
+    "\n",
+    "| Class | Package | Local | Serializable | [JS support](https://js.langchain.com/v0.2/docs/integrations/chat/oci_generative_ai) | Package downloads | Package latest |\n",
+    "| :--- | :--- | :---: | :---: |  :---: | :---: | :---: |\n",
+    "| [ChatOCIGenAI](https://api.python.langchain.com/en/latest/chat_models/langchain_community.chat_models.oci_generative_ai.ChatOCIGenAI.html) | [langchain-community](https://api.python.langchain.com/en/latest/community_api_reference.html) | ❌ | ❌ | ❌ | ![PyPI - Downloads](https://img.shields.io/pypi/dm/langchain-oci-generative-ai?style=flat-square&label=%20) | ![PyPI - Version](https://img.shields.io/pypi/v/langchain-oci-generative-ai?style=flat-square&label=%20) |\n",
+    "\n",
+    "### Model features\n",
+    "| [Tool calling](/docs/how_to/tool_calling/) | [Structured output](/docs/how_to/structured_output/) | JSON mode | [Image input](/docs/how_to/multimodal_inputs/) | Audio input | Video input | [Token-level streaming](/docs/how_to/chat_streaming/) | Native async | [Token usage](/docs/how_to/chat_token_usage_tracking/) | [Logprobs](/docs/how_to/logprobs/) |\n",
+    "| :---: | :---: | :---: | :---: |  :---: | :---: | :---: | :---: | :---: | :---: |\n",
+    "| ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ | ❌ | ✅ | ❌ | \n",
+    "\n",
+    "## Setup\n",
+    "\n",
+    "To access OCIGenAI models you'll need to install the `oci` and `langchain-community` packages.\n",
+    "\n",
+    "### Credentials\n",
+    "\n",
+    "The credentials and authentication methods supported for this integration are equivalent to those used with other OCI services and follow the __[standard SDK authentication](https://docs.oracle.com/en-us/iaas/Content/API/Concepts/sdk_authentication_methods.htm)__ methods, specifically API Key, session token, instance principal, and resource principal.\n",
+    "\n",
+    "API key is the default authentication method used in the examples above. The following example demonstrates how to use a different authentication method (session token)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "0730d6a1-c893-4840-9817-5e5251676d5d",
+   "metadata": {},
+   "source": [
+    "### Installation\n",
+    "\n",
+    "The LangChain OCIGenAI integration lives in the `langchain-community` package and you will also need to install the `oci` package:"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "652d6238-1f87-422a-b135-f5abbb8652fc",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "%pip install -qU langchain-community oci"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "a38cde65-254d-4219-a441-068766c0d4b5",
+   "metadata": {},
+   "source": [
+    "## Instantiation\n",
+    "\n",
+    "Now we can instantiate our model object and generate chat completions:\n"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "cb09c344-1836-4e0c-acf8-11d13ac1dbae",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "from langchain_community.chat_models.oci_generative_ai import ChatOCIGenAI\n",
+    "from langchain_core.messages import AIMessage, HumanMessage, SystemMessage\n",
+    "\n",
+    "chat = ChatOCIGenAI(\n",
+    "    model_id=\"cohere.command-r-16k\",\n",
+    "    service_endpoint=\"https://inference.generativeai.us-chicago-1.oci.oraclecloud.com\",\n",
+    "    compartment_id=\"MY_OCID\",\n",
+    "    model_kwargs={\"temperature\": 0.7, \"max_tokens\": 500},\n",
+    ")"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "2b4f3e15",
+   "metadata": {},
+   "source": [
+    "## Invocation"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "62e0dbc3",
+   "metadata": {
+    "tags": []
+   },
+   "outputs": [],
+   "source": [
+    "messages = [\n",
+    "    SystemMessage(content=\"your are an AI assistant.\"),\n",
+    "    AIMessage(content=\"Hi there human!\"),\n",
+    "    HumanMessage(content=\"tell me a joke.\"),\n",
+    "]\n",
+    "response = chat.invoke(messages)"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "d86145b3-bfef-46e8-b227-4dda5c9c2705",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "print(response.content)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "18e2bfc0-7e78-4528-a73f-499ac150dca8",
+   "metadata": {},
+   "source": [
+    "## Chaining\n",
+    "\n",
+    "We can [chain](/docs/how_to/sequence/) our model with a prompt template like so:\n"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "e197d1d7-a070-4c96-9f8a-a0e86d046e0b",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "from langchain_core.prompts import ChatPromptTemplate\n",
+    "\n",
+    "prompt = ChatPromptTemplate.from_template(\"Tell me a joke about {topic}\")\n",
+    "chain = prompt | chat\n",
+    "\n",
+    "response = chain.invoke({\"topic\": \"dogs\"})\n",
+    "print(response.content)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "3a5bb5ca-c3ae-4a58-be67-2cd18574b9a3",
+   "metadata": {},
+   "source": [
+    "## API reference\n",
+    "\n",
+    "For detailed documentation of all ChatOCIGenAI features and configurations head to the API reference: https://api.python.langchain.com/en/latest/chat_models/langchain_community.chat_models.oci_generative_ai.ChatOCIGenAI.html"
+   ]
+  }
+ ],
+ "metadata": {
+  "kernelspec": {
+   "display_name": "Python 3 (ipykernel)",
+   "language": "python",
+   "name": "python3"
+  },
+  "language_info": {
+   "codemirror_mode": {
+    "name": "ipython",
+    "version": 3
+   },
+   "file_extension": ".py",
+   "mimetype": "text/x-python",
+   "name": "python",
+   "nbconvert_exporter": "python",
+   "pygments_lexer": "ipython3",
+   "version": "3.9.1"
+  }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 5
+}
--- a/docs/docs/integrations/llms/oci_generative_ai.ipynb
+++ b/docs/docs/integrations/llms/oci_generative_ai.ipynb
@@ -14,15 +14,15 @@
    "Oracle Cloud Infrastructure (OCI) Generative AI is a fully managed service that provides a set of state-of-the-art, customizable large language models (LLMs) that cover a wide range of use cases, and which is available through a single API.\n",
    "Using the OCI Generative AI service you can access ready-to-use pretrained models, or create and host your own fine-tuned custom models based on your own data on dedicated AI clusters. Detailed documentation of the service and API is available __[here](https://docs.oracle.com/en-us/iaas/Content/generative-ai/home.htm)__ and __[here](https://docs.oracle.com/en-us/iaas/api/#/en/generative-ai/20231130/)__.\n",
    "\n",
-    "This notebook explains how to use OCI's Genrative AI models with LangChain."
+    "This notebook explains how to use OCI's Generative AI complete models with LangChain."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
-    "### Prerequisite\n",
-    "We will need to install the oci sdk"
+    "## Setup\n",
+    "Ensure that the oci sdk and the langchain-community package are installed"
   ]
  },
  {
@@ -31,31 +31,7 @@
   "metadata": {},
   "outputs": [],
   "source": [
-    "!pip install -U oci"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "metadata": {},
-   "source": [
-    "### OCI Generative AI API endpoint \n",
-    "https://inference.generativeai.us-chicago-1.oci.oraclecloud.com"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "metadata": {},
-   "source": [
-    "## Authentication\n",
-    "The authentication methods supported for this langchain integration are:\n",
-    "\n",
-    "1. API Key\n",
-    "2. Session token\n",
-    "3. Instance principal\n",
-    "4. Resource principal \n",
-    "\n",
-    "These follows the standard SDK authentication methods detailed __[here](https://docs.oracle.com/en-us/iaas/Content/API/Concepts/sdk_authentication_methods.htm)__.\n",
-    " "
+    "!pip install -U oci langchain-community"
   ]
  },
  {
@@ -71,13 +47,13 @@
   "metadata": {},
   "outputs": [],
   "source": [
-    "from langchain_community.llms import OCIGenAI\n",
+    "from langchain_community.llms.oci_generative_ai import OCIGenAI\n",
    "\n",
-    "# use default authN method API-key\n",
    "llm = OCIGenAI(\n",
-    "    model_id=\"MY_MODEL\",\n",
+    "    model_id=\"cohere.command\",\n",
    "    service_endpoint=\"https://inference.generativeai.us-chicago-1.oci.oraclecloud.com\",\n",
    "    compartment_id=\"MY_OCID\",\n",
+    "    model_kwargs={\"temperature\": 0, \"max_tokens\": 500},\n",
    ")\n",
    "\n",
    "response = llm.invoke(\"Tell me one fact about earth\", temperature=0.7)\n",
@@ -85,30 +61,10 @@
   ]
  },
  {
-   "cell_type": "code",
-   "execution_count": null,
+   "cell_type": "markdown",
   "metadata": {},
-   "outputs": [],
   "source": [
-    "from langchain.chains import LLMChain\n",
-    "from langchain_core.prompts import PromptTemplate\n",
-    "\n",
-    "# Use Session Token to authN\n",
-    "llm = OCIGenAI(\n",
-    "    model_id=\"MY_MODEL\",\n",
-    "    service_endpoint=\"https://inference.generativeai.us-chicago-1.oci.oraclecloud.com\",\n",
-    "    compartment_id=\"MY_OCID\",\n",
-    "    auth_type=\"SECURITY_TOKEN\",\n",
-    "    auth_profile=\"MY_PROFILE\",  # replace with your profile name\n",
-    "    model_kwargs={\"temperature\": 0.7, \"top_p\": 0.75, \"max_tokens\": 200},\n",
-    ")\n",
-    "\n",
-    "prompt = PromptTemplate(input_variables=[\"query\"], template=\"{query}\")\n",
-    "\n",
-    "llm_chain = LLMChain(llm=llm, prompt=prompt)\n",
-    "\n",
-    "response = llm_chain.invoke(\"what is the capital of france?\")\n",
-    "print(response)"
+    "#### Chaining with prompt templates"
   ]
  },
  {
@@ -117,49 +73,95 @@
   "metadata": {},
   "outputs": [],
   "source": [
-    "from langchain_community.embeddings import OCIGenAIEmbeddings\n",
-    "from langchain_community.vectorstores import FAISS\n",
-    "from langchain_core.output_parsers import StrOutputParser\n",
-    "from langchain_core.runnables import RunnablePassthrough\n",
-    "\n",
-    "embeddings = OCIGenAIEmbeddings(\n",
-    "    model_id=\"MY_EMBEDDING_MODEL\",\n",
-    "    service_endpoint=\"https://inference.generativeai.us-chicago-1.oci.oraclecloud.com\",\n",
-    "    compartment_id=\"MY_OCID\",\n",
-    ")\n",
-    "\n",
-    "vectorstore = FAISS.from_texts(\n",
-    "    [\n",
-    "        \"Larry Ellison co-founded Oracle Corporation in 1977 with Bob Miner and Ed Oates.\",\n",
-    "        \"Oracle Corporation is an American multinational computer technology company headquartered in Austin, Texas, United States.\",\n",
-    "    ],\n",
-    "    embedding=embeddings,\n",
-    ")\n",
-    "\n",
-    "retriever = vectorstore.as_retriever()\n",
-    "\n",
-    "template = \"\"\"Answer the question based only on the following context:\n",
-    "{context}\n",
-    " \n",
-    "Question: {question}\n",
-    "\"\"\"\n",
-    "prompt = PromptTemplate.from_template(template)\n",
+    "from langchain_core.prompts import PromptTemplate\n",
    "\n",
    "llm = OCIGenAI(\n",
-    "    model_id=\"MY_MODEL\",\n",
+    "    model_id=\"cohere.command\",\n",
    "    service_endpoint=\"https://inference.generativeai.us-chicago-1.oci.oraclecloud.com\",\n",
    "    compartment_id=\"MY_OCID\",\n",
+    "    model_kwargs={\"temperature\": 0, \"max_tokens\": 500},\n",
    ")\n",
    "\n",
-    "chain = (\n",
-    "    {\"context\": retriever, \"question\": RunnablePassthrough()}\n",
-    "    | prompt\n",
-    "    | llm\n",
-    "    | StrOutputParser()\n",
+    "prompt = PromptTemplate(input_variables=[\"query\"], template=\"{query}\")\n",
+    "llm_chain = prompt | llm\n",
+    "\n",
+    "response = llm_chain.invoke(\"what is the capital of france?\")\n",
+    "print(response)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "#### Streaming"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "llm = OCIGenAI(\n",
+    "    model_id=\"cohere.command\",\n",
+    "    service_endpoint=\"https://inference.generativeai.us-chicago-1.oci.oraclecloud.com\",\n",
+    "    compartment_id=\"MY_OCID\",\n",
+    "    model_kwargs={\"temperature\": 0, \"max_tokens\": 500},\n",
    ")\n",
    "\n",
-    "print(chain.invoke(\"when was oracle founded?\"))\n",
-    "print(chain.invoke(\"where is oracle headquartered?\"))"
+    "for chunk in llm.stream(\"Write me a song about sparkling water.\"):\n",
+    "    print(chunk, end=\"\", flush=True)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Authentication\n",
+    "The authentication methods supported for LlamaIndex are equivalent to those used with other OCI services and follow the __[standard SDK authentication](https://docs.oracle.com/en-us/iaas/Content/API/Concepts/sdk_authentication_methods.htm)__ methods, specifically API Key, session token, instance principal, and resource principal.\n",
+    "\n",
+    "API key is the default authentication method used in the examples above. The following example demonstrates how to use a different authentication method (session token)"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "llm = OCIGenAI(\n",
+    "    model_id=\"cohere.command\",\n",
+    "    service_endpoint=\"https://inference.generativeai.us-chicago-1.oci.oraclecloud.com\",\n",
+    "    compartment_id=\"MY_OCID\",\n",
+    "    auth_type=\"SECURITY_TOKEN\",\n",
+    "    auth_profile=\"MY_PROFILE\",  # replace with your profile name\n",
+    ")"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Dedicated AI Cluster\n",
+    "To access models hosted in a dedicated AI cluster __[create an endpoint](https://docs.oracle.com/en-us/iaas/api/#/en/generative-ai-inference/20231130/)__ whose assigned OCID (currently prefixed by ‘ocid1.generativeaiendpoint.oc1.us-chicago-1’) is used as your model ID.\n",
+    "\n",
+    "When accessing models hosted in a dedicated AI cluster you will need to initialize the OCIGenAI interface with two extra required params (\"provider\" and \"context_size\")."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "llm = OCIGenAI(\n",
+    "    model_id=\"ocid1.generativeaiendpoint.oc1.us-chicago-1....\",\n",
+    "    service_endpoint=\"https://inference.generativeai.us-chicago-1.oci.oraclecloud.com\",\n",
+    "    compartment_id=\"DEDICATED_COMPARTMENT_OCID\",\n",
+    "    auth_profile=\"MY_PROFILE\",  # replace with your profile name,\n",
+    "    provider=\"MODEL_PROVIDER\",  # e.g., \"cohere\" or \"meta\"\n",
+    "    context_size=\"MODEL_CONTEXT_SIZE\",  # e.g., 128000\n",
+    ")"
   ]
  }
 ],
--- a/docs/docs/integrations/providers/oci.mdx
+++ b/docs/docs/integrations/providers/oci.mdx
@@ -2,27 +2,29 @@

 The `LangChain` integrations related to [Oracle Cloud Infrastructure](https://www.oracle.com/artificial-intelligence/).

-## LLMs
-
-### OCI Generative AI
+## OCI Generative AI
 > Oracle Cloud Infrastructure (OCI) [Generative AI](https://docs.oracle.com/en-us/iaas/Content/generative-ai/home.htm) is a fully managed service that provides a set of state-of-the-art,
 > customizable large language models (LLMs) that cover a wide range of use cases, and which are available through a single API.
 > Using the OCI Generative AI service you can access ready-to-use pretrained models, or create and host your own fine-tuned
 > custom models based on your own data on dedicated AI clusters. 

-To use, you should have the latest `oci` python SDK installed.
+To use, you should have the latest `oci` python SDK and the langchain_community package installed.

 ```bash
-pip install -U oci
+pip install -U oci langchain-community
 ```

-See [usage examples](/docs/integrations/llms/oci_generative_ai).
+See [chat](/docs/integrations/llms/oci_generative_ai), [complete](/docs/integrations/chat/oci_generative_ai), and [embedding](/docs/integrations/text_embedding/oci_generative_ai) usage examples.

 ```python
+from langchain_community.chat_models import ChatOCIGenAI
+
 from langchain_community.llms import OCIGenAI
+
+from langchain_community.embeddings import OCIGenAIEmbeddings
 ```

-### OCI Data Science Model Deployment Endpoint
+## OCI Data Science Model Deployment Endpoint

 > [OCI Data Science](https://docs.oracle.com/en-us/iaas/data-science/using/home.htm) is a
 > fully managed and serverless platform for data science teams. Using the OCI Data Science
@@ -47,12 +49,3 @@ from langchain_community.llms import OCIModelDeploymentVLLM
 from langchain_community.llms import OCIModelDeploymentTGI
 ```

-## Text Embedding Models
-
-### OCI Generative AI
-
-See [usage examples](/docs/integrations/text_embedding/oci_generative_ai).
-
-```python
-from langchain_community.embeddings import OCIGenAIEmbeddings
-```