langchain/docs/versioned_docs/version-0.2.x/integrations/llms/modal.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Modal\n",
    "\n",
    "The [Modal cloud platform](https://modal.com/docs/guide) provides convenient, on-demand access to serverless cloud compute from Python scripts on your local computer. \n",
    "Use `modal` to run your own custom LLM models instead of depending on LLM APIs.\n",
    "\n",
    "This example goes over how to use LangChain to interact with a `modal` HTTPS [web endpoint](https://modal.com/docs/guide/webhooks).\n",
    "\n",
    "[_Question-answering with LangChain_](https://modal.com/docs/guide/ex/potus_speech_qanda) is another example of how to use LangChain alonside `Modal`. In that example, Modal runs the LangChain application end-to-end and uses OpenAI as its LLM API."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "%pip install --upgrade --quiet  modal"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Launching login page in your browser window...\n",
      "If this is not showing up, please copy this URL into your web browser manually:\n",
      "https://modal.com/token-flow/tf-Dzm3Y01234mqmm1234Vcu3\n"
     ]
    }
   ],
   "source": [
    "# Register an account with Modal and get a new token.\n",
    "\n",
    "!modal token new"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The [`langchain.llms.modal.Modal`](https://github.com/langchain-ai/langchain/blame/master/langchain/llms/modal.py) integration class requires that you deploy a Modal application with a web endpoint that complies with the following JSON interface:\n",
    "\n",
    "1. The LLM prompt is accepted as a `str` value under the key `\"prompt\"`\n",
    "2. The LLM response returned as a `str` value under the key `\"prompt\"`\n",
    "\n",
    "**Example request JSON:**\n",
    "\n",
    "```json\n",
    "{\n",
    "    \"prompt\": \"Identify yourself, bot!\",\n",
    "    \"extra\": \"args are allowed\",\n",
    "}\n",
    "```\n",
    "\n",
    "**Example response JSON:**\n",
    "\n",
    "```json\n",
    "{\n",
    "    \"prompt\": \"This is the LLM speaking\",\n",
    "}\n",
    "```\n",
    "\n",
    "An example 'dummy' Modal web endpoint function fulfilling this interface would be\n",
    "\n",
    "```python\n",
    "...\n",
    "...\n",
    "\n",
    "class Request(BaseModel):\n",
    "    prompt: str\n",
    "\n",
    "@stub.function()\n",
    "@modal.web_endpoint(method=\"POST\")\n",
    "def web(request: Request):\n",
    "    _ = request  # ignore input\n",
    "    return {\"prompt\": \"hello world\"}\n",
    "```\n",
    "\n",
    "* See Modal's [web endpoints](https://modal.com/docs/guide/webhooks#passing-arguments-to-web-endpoints) guide for the basics of setting up an endpoint that fulfils this interface.\n",
    "* See Modal's ['Run Falcon-40B with AutoGPTQ'](https://modal.com/docs/guide/ex/falcon_gptq) open-source LLM example as a starting point for your custom LLM!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Once you have a deployed Modal web endpoint, you can pass its URL into the `langchain.llms.modal.Modal` LLM class. This class can then function as a building block in your chain."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from langchain.chains import LLMChain\n",
    "from langchain_community.llms import Modal\n",
    "from langchain_core.prompts import PromptTemplate"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "template = \"\"\"Question: {question}\n",
    "\n",
    "Answer: Let's think step by step.\"\"\"\n",
    "\n",
    "prompt = PromptTemplate.from_template(template)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "endpoint_url = \"https://ecorp--custom-llm-endpoint.modal.run\"  # REPLACE ME with your deployed Modal web endpoint's URL\n",
    "llm = Modal(endpoint_url=endpoint_url)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "llm_chain = LLMChain(prompt=prompt, llm=llm)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "question = \"What NFL team won the Super Bowl in the year Justin Beiber was born?\"\n",
    "\n",
    "llm_chain.run(question)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.6"
  },
  "vscode": {
   "interpreter": {
    "hash": "a0a0263b650d907a3bfe41c0f8d6a63a071b884df3cfdc1579f00cdc1aed6b03"
   }
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}