langchain/docs/docs/integrations/llms/sambanova.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# SambaNova\n",
    "\n",
    "**[SambaNova](https://sambanova.ai/)'s** [Sambaverse](https://sambaverse.sambanova.ai/) and [Sambastudio](https://sambanova.ai/technology/full-stack-ai-platform) are platforms for running your own open-source models\n",
    "\n",
    "This example goes over how to use LangChain to interact with SambaNova models"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Sambaverse"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Sambaverse** allows you to interact with multiple open-source models. You can view the list of available models and interact with them in the [playground](https://sambaverse.sambanova.ai/playground).\n",
    " **Please note that Sambaverse's free offering is performance-limited.** Companies that are ready to evaluate the production tokens-per-second performance, volume throughput, and 10x lower total cost of ownership (TCO) of SambaNova should [contact us](https://sambaverse.sambanova.ai/contact-us) for a non-limited evaluation instance."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "An API key is required to access Sambaverse models. To get a key, create an account at [sambaverse.sambanova.ai](https://sambaverse.sambanova.ai/)\n",
    "\n",
    "The [sseclient-py](https://pypi.org/project/sseclient-py/) package is required to run streaming predictions "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%pip install --quiet sseclient-py==1.8.0"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Register your API key as an environment variable:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "import os\n",
    "\n",
    "sambaverse_api_key = \"<Your sambaverse API key>\"\n",
    "\n",
    "# Set the environment variables\n",
    "os.environ[\"SAMBAVERSE_API_KEY\"] = sambaverse_api_key"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Call Sambaverse models directly from LangChain!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from langchain_community.llms.sambanova import Sambaverse\n",
    "\n",
    "llm = Sambaverse(\n",
    "    sambaverse_model_name=\"Meta/llama-2-7b-chat-hf\",\n",
    "    streaming=False,\n",
    "    model_kwargs={\n",
    "        \"do_sample\": True,\n",
    "        \"max_tokens_to_generate\": 1000,\n",
    "        \"temperature\": 0.01,\n",
    "        \"process_prompt\": True,\n",
    "        \"select_expert\": \"llama-2-7b-chat-hf\",\n",
    "        # \"stop_sequences\": '\\\"sequence1\\\",\\\"sequence2\\\"',\n",
    "        # \"repetition_penalty\":  1.0,\n",
    "        # \"top_k\": 50,\n",
    "        # \"top_p\": 1.0\n",
    "    },\n",
    ")\n",
    "\n",
    "print(llm.invoke(\"Why should I use open source models?\"))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## SambaStudio"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**SambaStudio** allows you to train, run batch inference jobs, and deploy online inference endpoints to run open source models that you fine tuned yourself."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "A SambaStudio environment is required to deploy a model. Get more information at [sambanova.ai/products/enterprise-ai-platform-sambanova-suite](https://sambanova.ai/products/enterprise-ai-platform-sambanova-suite)\n",
    "\n",
    "The [sseclient-py](https://pypi.org/project/sseclient-py/) package is required to run streaming predictions "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%pip install --quiet sseclient-py==1.8.0"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Register your environment variables:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import os\n",
    "\n",
    "sambastudio_base_url = \"<Your SambaStudio environment URL>\"\n",
    "sambastudio_project_id = \"<Your SambaStudio project id>\"\n",
    "sambastudio_endpoint_id = \"<Your SambaStudio endpoint id>\"\n",
    "sambastudio_api_key = \"<Your SambaStudio endpoint API key>\"\n",
    "\n",
    "# Set the environment variables\n",
    "os.environ[\"SAMBASTUDIO_BASE_URL\"] = sambastudio_base_url\n",
    "os.environ[\"SAMBASTUDIO_PROJECT_ID\"] = sambastudio_project_id\n",
    "os.environ[\"SAMBASTUDIO_ENDPOINT_ID\"] = sambastudio_endpoint_id\n",
    "os.environ[\"SAMBASTUDIO_API_KEY\"] = sambastudio_api_key"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Call SambaStudio models directly from LangChain!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from langchain_community.llms.sambanova import SambaStudio\n",
    "\n",
    "llm = SambaStudio(\n",
    "    streaming=False,\n",
    "    model_kwargs={\n",
    "        \"do_sample\": True,\n",
    "        \"max_tokens_to_generate\": 1000,\n",
    "        \"temperature\": 0.01,\n",
    "        # \"repetition_penalty\":  1.0,\n",
    "        # \"top_k\": 50,\n",
    "        # \"top_logprobs\": 0,\n",
    "        # \"top_p\": 1.0\n",
    "    },\n",
    ")\n",
    "\n",
    "print(llm.invoke(\"Why should I use open source models?\"))"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.1"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}