databricks: Add partner package directory and ChatDatabricks implementation (#25430)

### Summary

Create `langchain-databricks` as a new partner packages. This PR does
not migrate all existing Databricks integration, but the package will
eventually contain:

* `ChatDatabricks` (implemented in this PR)
* `DatabricksVectorSearch`
* `DatabricksEmbeddings`
* ~`UCFunctionToolkit`~ (will be done after UC SDK work which
drastically simplify implementation)

Also, this PR does not add integration tests yet. This will be added
once the Databricks test workspace is ready.

Tagging @efriis as POC


### Tracker
[✍️] Create a package and imgrate ChatDatabricks
[ ] Migrate DatabricksVectorSearch, DatabricksEmbeddings, and their docs
~[ ] Migrate UCFunctionToolkit and its doc~
[ ] Add provider document and update README.md
[ ] Add integration tests and set up secrets (after moved to an external
package)
[ ] Add deprecation note to the community implementations.

---------

Signed-off-by: B-Step62 <yuki.watanabe@databricks.com>
Co-authored-by: Erick Friis <erick@langchain.dev>

docs/Makefile

@@ -18,7 +18,7 @@ for dir; do \
     if find "$$dir" -maxdepth 1 -type f \( -name "pyproject.toml" -o -name "setup.py" \) | grep -q .; then \
         echo "$$dir"; \
     fi \
-done' sh {} + | grep -vE "airbyte|ibm|couchbase" | tr '\n' ' ')
+done' sh {} + | grep -vE "airbyte|ibm|couchbase|databricks" | tr '\n' ' ')
 
 PORT ?= 3001
 

docs/docs/integrations/chat/databricks.ipynb

@@ -31,7 +31,7 @@
     "\n",
     "| Class | Package | Local | Serializable | Package downloads | Package latest |\n",
     "| :--- | :--- | :---: | :---: |  :---: | :---: |\n",
-    "| [ChatDatabricks](https://api.python.langchain.com/en/latest/chat_models/langchain_community.chat_models.databricks.ChatDatabricks.html) | [langchain-community](https://api.python.langchain.com/en/latest/community_api_reference.html) | ❌ | beta | ![PyPI - Downloads](https://img.shields.io/pypi/dm/langchain-community?style=flat-square&label=%20) | ![PyPI - Version](https://img.shields.io/pypi/v/langchain-community?style=flat-square&label=%20) |\n",
+    "| [ChatDatabricks](https://api.python.langchain.com/en/latest/chat_models/langchain_community.chat_models.databricks.ChatDatabricks.html) | [langchain-databricks](https://api.python.langchain.com/en/latest/databricks_api_reference.html) | ❌ | beta | ![PyPI - Downloads](https://img.shields.io/pypi/dm/langchain-databricks?style=flat-square&label=%20) | ![PyPI - Version](https://img.shields.io/pypi/v/langchain-databricks?style=flat-square&label=%20) |\n",
     "\n",
     "### Model features\n",
     "| [Tool calling](/docs/how_to/tool_calling/) | [Structured output](/docs/how_to/structured_output/) | JSON mode | [Image input](/docs/how_to/multimodal_inputs/) | Audio input | Video input | [Token-level streaming](/docs/how_to/chat_streaming/) | Native async | [Token usage](/docs/how_to/chat_token_usage_tracking/) | [Logprobs](/docs/how_to/logprobs/) |\n",
@@ -99,7 +99,7 @@
    "source": [
     "### Installation\n",
     "\n",
-    "The LangChain Databricks integration lives in the `langchain-community` package. Also, `mlflow >= 2.9 ` is required to run the code in this notebook."
+    "The LangChain Databricks integration lives in the `langchain-databricks` package."
    ]
   },
   {
@@ -108,7 +108,7 @@
    "metadata": {},
    "outputs": [],
    "source": [
-    "%pip install -qU langchain-community mlflow>=2.9.0"
+    "%pip install -qU langchain-databricks"
    ]
   },
   {
@@ -133,7 +133,7 @@
    "metadata": {},
    "outputs": [],
    "source": [
-    "from langchain_community.chat_models import ChatDatabricks\n",
+    "from langchain_databricks import ChatDatabricks\n",
     "\n",
     "chat_model = ChatDatabricks(\n",
     "    endpoint=\"databricks-dbrx-instruct\",\n",
@@ -245,9 +245,7 @@
    "cell_type": "markdown",
    "metadata": {},
    "source": [
-    "## Invocation (streaming)\n",
-    "\n",
-    "`ChatDatabricks` supports streaming response by `stream` method since `langchain-community>=0.2.1`."
+    "## Invocation (streaming)"
    ]
   },
   {
@@ -299,7 +297,7 @@
     "* An LLM was registered and deployed to [a Databricks serving endpoint](https://docs.databricks.com/machine-learning/model-serving/index.html) via MLflow. The endpoint must have OpenAI-compatible chat input/output format ([reference](https://mlflow.org/docs/latest/llms/deployments/index.html#chat))\n",
     "* You have [\"Can Query\" permission](https://docs.databricks.com/security/auth-authz/access-control/serving-endpoint-acl.html) to the endpoint.\n",
     "\n",
-    "Once the endpoint is ready, the usage pattern is completely same as Foundation Models."
+    "Once the endpoint is ready, the usage pattern is identical to that of Foundation Models."
    ]
   },
   {
@@ -332,7 +330,7 @@
     "\n",
     "First, create a new Databricks serving endpoint that proxies requests to the target external model. The endpoint creation should be fairy quick for proxying external models.\n",
     "\n",
-    "This requires registering OpenAI API Key in Databricks secret manager with the following comment:\n",
+    "This requires registering your OpenAI API Key within the Databricks secret manager as follows:\n",
     "```sh\n",
     "# Replace `<scope>` with your scope\n",
     "databricks secrets create-scope <scope>\n",
@@ -417,8 +415,6 @@
    "metadata": {},
    "outputs": [],
    "source": [
-    "from langchain_community.chat_models.databricks import ChatDatabricks\n",
-    "\n",
     "llm = ChatDatabricks(endpoint=\"databricks-meta-llama-3-70b-instruct\")\n",
     "tools = [\n",
     "    {\n",
@@ -461,7 +457,7 @@
    "source": [
     "## API reference\n",
     "\n",
-    "For detailed documentation of all ChatDatabricks features and configurations head to the API reference: https://api.python.langchain.com/en/latest/chat_models/langchain_community.chat_models.ChatDatabricks.html"
+    "For detailed documentation of all ChatDatabricks features and configurations head to the API reference: https://api.python.langchain.com/en/latest/chat_models/langchain_databricks.chat_models.ChatDatabricks.html"
    ]
   }
  ],