Update Modal.com integration docs (#8014)

Hey, I'm a Modal Labs engineer and I'm making this docs update after
getting a user question in [our beta Slack
space](https://join.slack.com/t/modalbetatesters/shared_invite/zt-1xl9gbob8-1QDgUY7_PRPg6dQ49hqEeQ)
about the Langchain integration docs.

🔗 [Modal beta-testers link to docs discussion
thread](https://modalbetatesters.slack.com/archives/C031Z7DBQFL/p1689777700594819?thread_ts=1689775859.855849&cid=C031Z7DBQFL)
This commit is contained in:
Jonathon Belotti
2023-07-20 18:53:06 -04:00
committed by GitHub
parent 62d0475c29
commit 021bb9be84
2 changed files with 119 additions and 49 deletions

View File

@@ -6,12 +6,12 @@
"source": [
"# Modal\n",
"\n",
"The [Modal Python Library](https://modal.com/docs/guide) provides convenient, on-demand access to serverless cloud compute from Python scripts on your local computer. \n",
"The `Modal` itself does not provide any LLMs but only the infrastructure.\n",
"The [Modal cloud platform](https://modal.com/docs/guide) provides convenient, on-demand access to serverless cloud compute from Python scripts on your local computer. \n",
"Use `modal` to run your own custom LLM models instead of depending on LLM APIs.\n",
"\n",
"This example goes over how to use LangChain to interact with `Modal`.\n",
"This example goes over how to use LangChain to interact with a `modal` HTTPS [web endpoint](https://modal.com/docs/guide/webhooks).\n",
"\n",
"[Here](https://modal.com/docs/guide/ex/potus_speech_qanda) is another example how to use LangChain to interact with `Modal`."
"[_Question-answering with LangChain_](https://modal.com/docs/guide/ex/potus_speech_qanda) is another example of how to use LangChain alonside `Modal`. In that example, Modal runs the LangChain application end-to-end and uses OpenAI as its LLM API."
]
},
{
@@ -22,7 +22,7 @@
},
"outputs": [],
"source": [
"!pip install modal-client"
"!pip install modal"
]
},
{
@@ -36,20 +36,14 @@
"name": "stdout",
"output_type": "stream",
"text": [
"\u001b[?25lLaunching login page in your browser window\u001b[33m...\u001b[0m\n",
"\u001b[2KIf this is not showing up, please copy this URL into your web browser manually:\n",
"\u001b[2Km⠙\u001b[0m Waiting for authentication in the web browser...\n",
"\u001b]8;id=417802;https://modal.com/token-flow/tf-ptEuGecm7T1T5YQe42kwM1\u001b\\\u001b[4;94mhttps://modal.com/token-flow/tf-ptEuGecm7T1T5YQe42kwM1\u001b[0m\u001b]8;;\u001b\\\n",
"\n",
"\u001b[2K\u001b[32m⠙\u001b[0m Waiting for authentication in the web browser...\n",
"\u001b[1A\u001b[2K^C\n",
"\n",
"\u001b[31mAborted.\u001b[0m\n"
"Launching login page in your browser window...\n",
"If this is not showing up, please copy this URL into your web browser manually:\n",
"https://modal.com/token-flow/tf-Dzm3Y01234mqmm1234Vcu3\n"
]
}
],
"source": [
"# register and get a new token\n",
"# Register an account with Modal and get a new token.\n",
"\n",
"!modal token new"
]
@@ -58,7 +52,53 @@
"cell_type": "markdown",
"metadata": {},
"source": [
"Follow [these instructions](https://modal.com/docs/guide/secrets) to deal with secrets."
"The [`langchain.llms.modal.Modal`](https://github.com/hwchase17/langchain/blame/master/langchain/llms/modal.py) integration class requires that you deploy a Modal application with a web endpoint that complies with the following JSON interface:\n",
"\n",
"1. The LLM prompt is accepted as a `str` value under the key `\"prompt\"`\n",
"2. The LLM response returned as a `str` value under the key `\"prompt\"`\n",
"\n",
"**Example request JSON:**\n",
"\n",
"```json\n",
"{\n",
" \"prompt\": \"Identify yourself, bot!\",\n",
" \"extra\": \"args are allowed\",\n",
"}\n",
"```\n",
"\n",
"**Example response JSON:**\n",
"\n",
"```json\n",
"{\n",
" \"prompt\": \"This is the LLM speaking\",\n",
"}\n",
"```\n",
"\n",
"An example 'dummy' Modal web endpoint function fulfilling this interface would be\n",
"\n",
"```python\n",
"...\n",
"...\n",
"\n",
"class Request(BaseModel):\n",
" prompt: str\n",
"\n",
"@stub.function()\n",
"@modal.web_endpoint(method=\"POST\")\n",
"def web(request: Request):\n",
" _ = request # ignore input\n",
" return {\"prompt\": \"hello world\"}\n",
"```\n",
"\n",
"* See Modal's [web endpoints](https://modal.com/docs/guide/webhooks#passing-arguments-to-web-endpoints) guide for the basics of setting up an endpoint that fulfils this interface.\n",
"* See Modal's ['Run Falcon-40B with AutoGPTQ'](https://modal.com/docs/guide/ex/falcon_gptq) open-source LLM example as a starting point for your custom LLM!"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Once you have a deployed Modal web endpoint, you can pass its URL into the `langchain.llms.modal.Modal` LLM class. This class can then function as a building block in your chain."
]
},
{
@@ -90,7 +130,8 @@
"metadata": {},
"outputs": [],
"source": [
"llm = Modal(endpoint_url=\"YOUR_ENDPOINT_URL\")"
"endpoint_url = \"https://ecorp--custom-llm-endpoint.modal.run\" # REPLACE ME with your deployed Modal web endpoint's URL\n",
"llm = Modal(endpoint_url=endpoint_url)"
]
},
{