mirror of
https://github.com/hwchase17/langchain.git
synced 2025-04-29 12:25:37 +00:00
461 lines
16 KiB
Plaintext
461 lines
16 KiB
Plaintext
{
|
|
"cells": [
|
|
{
|
|
"cell_type": "raw",
|
|
"metadata": {
|
|
"vscode": {
|
|
"languageId": "raw"
|
|
}
|
|
},
|
|
"source": [
|
|
"---\n",
|
|
"sidebar_label: ChatOCIModelDeployment\n",
|
|
"---"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"# ChatOCIModelDeployment\n",
|
|
"\n",
|
|
"This will help you getting started with OCIModelDeployment [chat models](/docs/concepts/chat_models). For detailed documentation of all ChatOCIModelDeployment features and configurations head to the [API reference](https://python.langchain.com/api_reference/community/chat_models/langchain_community.chat_models.oci_data_science.ChatOCIModelDeployment.html).\n",
|
|
"\n",
|
|
"[OCI Data Science](https://docs.oracle.com/en-us/iaas/data-science/using/home.htm) is a fully managed and serverless platform for data science teams to build, train, and manage machine learning models in the Oracle Cloud Infrastructure. You can use [AI Quick Actions](https://blogs.oracle.com/ai-and-datascience/post/ai-quick-actions-in-oci-data-science) to easily deploy LLMs on [OCI Data Science Model Deployment Service](https://docs.oracle.com/en-us/iaas/data-science/using/model-dep-about.htm). You may choose to deploy the model with popular inference frameworks such as vLLM or TGI. By default, the model deployment endpoint mimics the OpenAI API protocol.\n",
|
|
"\n",
|
|
"> For the latest updates, examples and experimental features, please see [ADS LangChain Integration](https://accelerated-data-science.readthedocs.io/en/latest/user_guide/large_language_model/langchain_models.html).\n",
|
|
"\n",
|
|
"## Overview\n",
|
|
"### Integration details\n",
|
|
"\n",
|
|
"| Class | Package | Local | Serializable | JS support | Package downloads | Package latest |\n",
|
|
"| :--- | :--- | :---: | :---: | :---: | :---: | :---: |\n",
|
|
"| [ChatOCIModelDeployment](https://python.langchain.com/api_reference/community/chat_models/langchain_community.chat_models.oci_data_science.ChatOCIModelDeployment.html) | [langchain-community](https://python.langchain.com/api_reference/community/index.html) | ❌ | beta | ❌ |  |  |\n",
|
|
"\n",
|
|
"### Model features\n",
|
|
"\n",
|
|
"| [Tool calling](/docs/how_to/tool_calling) | [Structured output](/docs/how_to/structured_output/) | JSON mode | [Image input](/docs/how_to/multimodal_inputs/) | Audio input | Video input | [Token-level streaming](/docs/how_to/chat_streaming/) | Native async | [Token usage](/docs/how_to/chat_token_usage_tracking/) | [Logprobs](/docs/how_to/logprobs/) |\n",
|
|
"| :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: |\n",
|
|
"| depends | depends | depends | depends | depends | depends | ✅ | ✅ | ✅ | ✅ | \n",
|
|
"\n",
|
|
"Some model features, including tool calling, structured output, JSON mode and multi-modal inputs, are depending on deployed model.\n",
|
|
"\n",
|
|
"\n",
|
|
"## Setup\n",
|
|
"\n",
|
|
"To use ChatOCIModelDeployment you'll need to deploy a chat model with chat completion endpoint and install the `langchain-community`, `langchain-openai` and `oracle-ads` integration packages.\n",
|
|
"\n",
|
|
"You can easily deploy foundation models using the [AI Quick Actions](https://github.com/oracle-samples/oci-data-science-ai-samples/blob/main/ai-quick-actions/model-deployment-tips.md) on OCI Data Science Model deployment. For additional deployment examples, please visit the [Oracle GitHub samples repository](https://github.com/oracle-samples/oci-data-science-ai-samples/tree/main/ai-quick-actions).\n",
|
|
"\n",
|
|
"### Policies\n",
|
|
"Make sure to have the required [policies](https://docs.oracle.com/en-us/iaas/data-science/using/model-dep-policies-auth.htm#model_dep_policies_auth__predict-endpoint) to access the OCI Data Science Model Deployment endpoint.\n"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"### Credentials\n",
|
|
"\n",
|
|
"You can set authentication through Oracle ADS. When you are working in OCI Data Science Notebook Session, you can leverage resource principal to access other OCI resources."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"import ads\n",
|
|
"\n",
|
|
"# Set authentication through ads\n",
|
|
"# Use resource principal are operating within a\n",
|
|
"# OCI service that has resource principal based\n",
|
|
"# authentication configured\n",
|
|
"ads.set_auth(\"resource_principal\")"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"Alternatively, you can configure the credentials using the following environment variables. For example, to use API key with specific profile:"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"import os\n",
|
|
"\n",
|
|
"# Set authentication through environment variables\n",
|
|
"# Use API Key setup when you are working from a local\n",
|
|
"# workstation or on platform which does not support\n",
|
|
"# resource principals.\n",
|
|
"os.environ[\"OCI_IAM_TYPE\"] = \"api_key\"\n",
|
|
"os.environ[\"OCI_CONFIG_PROFILE\"] = \"default\"\n",
|
|
"os.environ[\"OCI_CONFIG_LOCATION\"] = \"~/.oci\""
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"Check out [Oracle ADS docs](https://accelerated-data-science.readthedocs.io/en/latest/user_guide/cli/authentication.html) to see more options."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"### Installation\n",
|
|
"\n",
|
|
"The LangChain OCIModelDeployment integration lives in the `langchain-community` package. The following command will install `langchain-community` and the required dependencies."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"%pip install -qU langchain-community langchain-openai oracle-ads"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"## Instantiation\n",
|
|
"\n",
|
|
"You may instantiate the model with the generic `ChatOCIModelDeployment` or framework specific class like `ChatOCIModelDeploymentVLLM`.\n",
|
|
"\n",
|
|
"* Using `ChatOCIModelDeployment` when you need a generic entry point for deploying models. You can pass model parameters through `model_kwargs` during the instantiation of this class. This allows for flexibility and ease of configuration without needing to rely on framework-specific details."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 1,
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"from langchain_community.chat_models import ChatOCIModelDeployment\n",
|
|
"\n",
|
|
"# Create an instance of OCI Model Deployment Endpoint\n",
|
|
"# Replace the endpoint uri with your own\n",
|
|
"# Using generic class as entry point, you will be able\n",
|
|
"# to pass model parameters through model_kwargs during\n",
|
|
"# instantiation.\n",
|
|
"chat = ChatOCIModelDeployment(\n",
|
|
" endpoint=\"https://modeldeployment.<region>.oci.customer-oci.com/<ocid>/predict\",\n",
|
|
" streaming=True,\n",
|
|
" max_retries=1,\n",
|
|
" model_kwargs={\n",
|
|
" \"temperature\": 0.2,\n",
|
|
" \"max_tokens\": 512,\n",
|
|
" }, # other model params...\n",
|
|
")"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"* Using framework specific class like `ChatOCIModelDeploymentVLLM`: This is suitable when you are working with a specific framework (e.g. `vLLM`) and need to pass model parameters directly through the constructor, streamlining the setup process."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"from langchain_community.chat_models import ChatOCIModelDeploymentVLLM\n",
|
|
"\n",
|
|
"# Create an instance of OCI Model Deployment Endpoint\n",
|
|
"# Replace the endpoint uri with your own\n",
|
|
"# Using framework specific class as entry point, you will\n",
|
|
"# be able to pass model parameters in constructor.\n",
|
|
"chat = ChatOCIModelDeploymentVLLM(\n",
|
|
" endpoint=\"https://modeldeployment.<region>.oci.customer-oci.com/<md_ocid>/predict\",\n",
|
|
")"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"## Invocation"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 2,
|
|
"metadata": {},
|
|
"outputs": [
|
|
{
|
|
"data": {
|
|
"text/plain": [
|
|
"AIMessage(content=\"J'adore programmer.\", response_metadata={'token_usage': {'prompt_tokens': 44, 'total_tokens': 52, 'completion_tokens': 8}, 'model_name': 'odsc-llm', 'system_fingerprint': '', 'finish_reason': 'stop'}, id='run-ca145168-efa9-414c-9dd1-21d10766fdd3-0')"
|
|
]
|
|
},
|
|
"execution_count": 2,
|
|
"metadata": {},
|
|
"output_type": "execute_result"
|
|
}
|
|
],
|
|
"source": [
|
|
"messages = [\n",
|
|
" (\n",
|
|
" \"system\",\n",
|
|
" \"You are a helpful assistant that translates English to French. Translate the user sentence.\",\n",
|
|
" ),\n",
|
|
" (\"human\", \"I love programming.\"),\n",
|
|
"]\n",
|
|
"\n",
|
|
"ai_msg = chat.invoke(messages)\n",
|
|
"ai_msg"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 3,
|
|
"metadata": {},
|
|
"outputs": [
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"\n",
|
|
"J'adore programmer.\n"
|
|
]
|
|
}
|
|
],
|
|
"source": [
|
|
"print(ai_msg.content)"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"## Chaining"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 4,
|
|
"metadata": {},
|
|
"outputs": [
|
|
{
|
|
"data": {
|
|
"text/plain": [
|
|
"AIMessage(content='Ich liebe Programmierung.', response_metadata={'token_usage': {'prompt_tokens': 38, 'total_tokens': 48, 'completion_tokens': 10}, 'model_name': 'odsc-llm', 'system_fingerprint': '', 'finish_reason': 'stop'}, id='run-5dd936b0-b97e-490e-9869-2ad3dd524234-0')"
|
|
]
|
|
},
|
|
"execution_count": 4,
|
|
"metadata": {},
|
|
"output_type": "execute_result"
|
|
}
|
|
],
|
|
"source": [
|
|
"from langchain_core.prompts import ChatPromptTemplate\n",
|
|
"\n",
|
|
"prompt = ChatPromptTemplate.from_messages(\n",
|
|
" [\n",
|
|
" (\n",
|
|
" \"system\",\n",
|
|
" \"You are a helpful assistant that translates {input_language} to {output_language}.\",\n",
|
|
" ),\n",
|
|
" (\"human\", \"{input}\"),\n",
|
|
" ]\n",
|
|
")\n",
|
|
"\n",
|
|
"chain = prompt | chat\n",
|
|
"chain.invoke(\n",
|
|
" {\n",
|
|
" \"input_language\": \"English\",\n",
|
|
" \"output_language\": \"German\",\n",
|
|
" \"input\": \"I love programming.\",\n",
|
|
" }\n",
|
|
")"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"## Asynchronous calls"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 6,
|
|
"metadata": {},
|
|
"outputs": [
|
|
{
|
|
"data": {
|
|
"text/plain": [
|
|
"AIMessage(content='我喜欢编程', response_metadata={'token_usage': {'prompt_tokens': 37, 'total_tokens': 50, 'completion_tokens': 13}, 'model_name': 'odsc-llm', 'system_fingerprint': '', 'finish_reason': 'stop'}, id='run-a2dc9393-f269-41a4-b908-b1d8a92cf827-0')"
|
|
]
|
|
},
|
|
"execution_count": 6,
|
|
"metadata": {},
|
|
"output_type": "execute_result"
|
|
}
|
|
],
|
|
"source": [
|
|
"from langchain_community.chat_models import ChatOCIModelDeployment\n",
|
|
"\n",
|
|
"system = \"You are a helpful translator that translates {input_language} to {output_language}.\"\n",
|
|
"human = \"{text}\"\n",
|
|
"prompt = ChatPromptTemplate.from_messages([(\"system\", system), (\"human\", human)])\n",
|
|
"\n",
|
|
"chat = ChatOCIModelDeployment(\n",
|
|
" endpoint=\"https://modeldeployment.us-ashburn-1.oci.customer-oci.com/<ocid>/predict\"\n",
|
|
")\n",
|
|
"chain = prompt | chat\n",
|
|
"\n",
|
|
"await chain.ainvoke(\n",
|
|
" {\n",
|
|
" \"input_language\": \"English\",\n",
|
|
" \"output_language\": \"Chinese\",\n",
|
|
" \"text\": \"I love programming\",\n",
|
|
" }\n",
|
|
")"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"## Streaming calls"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 7,
|
|
"metadata": {},
|
|
"outputs": [
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"\n",
|
|
"\n",
|
|
"1. California\n",
|
|
"2. Texas\n",
|
|
"3. Florida\n",
|
|
"4. New York\n",
|
|
"5. Illinois"
|
|
]
|
|
}
|
|
],
|
|
"source": [
|
|
"import os\n",
|
|
"import sys\n",
|
|
"\n",
|
|
"from langchain_community.chat_models import ChatOCIModelDeployment\n",
|
|
"from langchain_core.prompts import ChatPromptTemplate\n",
|
|
"\n",
|
|
"prompt = ChatPromptTemplate.from_messages(\n",
|
|
" [(\"human\", \"List out the 5 states in the United State.\")]\n",
|
|
")\n",
|
|
"\n",
|
|
"chat = ChatOCIModelDeployment(\n",
|
|
" endpoint=\"https://modeldeployment.us-ashburn-1.oci.customer-oci.com/<ocid>/predict\"\n",
|
|
")\n",
|
|
"\n",
|
|
"chain = prompt | chat\n",
|
|
"\n",
|
|
"for chunk in chain.stream({}):\n",
|
|
" sys.stdout.write(chunk.content)\n",
|
|
" sys.stdout.flush()"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"## Structured output"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 8,
|
|
"metadata": {},
|
|
"outputs": [
|
|
{
|
|
"data": {
|
|
"text/plain": [
|
|
"{'setup': 'Why did the cat get stuck in the tree?',\n",
|
|
" 'punchline': 'Because it was chasing its tail!'}"
|
|
]
|
|
},
|
|
"execution_count": null,
|
|
"metadata": {},
|
|
"output_type": "execute_result"
|
|
}
|
|
],
|
|
"source": [
|
|
"from langchain_community.chat_models import ChatOCIModelDeployment\n",
|
|
"from pydantic import BaseModel\n",
|
|
"\n",
|
|
"\n",
|
|
"class Joke(BaseModel):\n",
|
|
" \"\"\"A setup to a joke and the punchline.\"\"\"\n",
|
|
"\n",
|
|
" setup: str\n",
|
|
" punchline: str\n",
|
|
"\n",
|
|
"\n",
|
|
"chat = ChatOCIModelDeployment(\n",
|
|
" endpoint=\"https://modeldeployment.us-ashburn-1.oci.customer-oci.com/<ocid>/predict\",\n",
|
|
")\n",
|
|
"structured_llm = chat.with_structured_output(Joke, method=\"json_mode\")\n",
|
|
"output = structured_llm.invoke(\n",
|
|
" \"Tell me a joke about cats, respond in JSON with `setup` and `punchline` keys\"\n",
|
|
")\n",
|
|
"\n",
|
|
"output.dict()"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"## API reference\n",
|
|
"\n",
|
|
"For comprehensive details on all features and configurations, please refer to the API reference documentation for each class:\n",
|
|
"\n",
|
|
"* [ChatOCIModelDeployment](https://python.langchain.com/api_reference/community/chat_models/langchain_community.chat_models.oci_data_science.ChatOCIModelDeployment.html)\n",
|
|
"* [ChatOCIModelDeploymentVLLM](https://python.langchain.com/api_reference/community/chat_models/langchain_community.chat_models.oci_data_science.ChatOCIModelDeploymentVLLM.html)\n",
|
|
"* [ChatOCIModelDeploymentTGI](https://python.langchain.com/api_reference/community/chat_models/langchain_community.chat_models.oci_data_science.ChatOCIModelDeploymentTGI.html)"
|
|
]
|
|
}
|
|
],
|
|
"metadata": {
|
|
"kernelspec": {
|
|
"display_name": "langchain",
|
|
"language": "python",
|
|
"name": "python3"
|
|
},
|
|
"language_info": {
|
|
"codemirror_mode": {
|
|
"name": "ipython",
|
|
"version": 3
|
|
},
|
|
"file_extension": ".py",
|
|
"mimetype": "text/x-python",
|
|
"name": "python",
|
|
"nbconvert_exporter": "python",
|
|
"pygments_lexer": "ipython3",
|
|
"version": "3.9.18"
|
|
}
|
|
},
|
|
"nbformat": 4,
|
|
"nbformat_minor": 2
|
|
}
|