github/langchain
1
0
Fork 0
mirror of https://github.com/hwchase17/langchain.git synced 2025-08-01 00:49:25 +00:00
Code Issues Packages Projects Releases Wiki Activity

docs: how to contribute integrations (#28143)

Browse Source
This commit is contained in:
Erick Friis 2024-11-15 14:52:17 -08:00 committed by GitHub
parent 683644320b
commit 709c418022
No known key found for this signature in database
GPG Key ID: B5690EEEBB952194
9 changed files with 374 additions and 214 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

2
docs/docs/contributing/how_to/code/guidelines.mdx
View File

@ -29,7 +29,7 @@ or new agents/chains from outside contributors without an existing GitHub discus
- New features must come with docs, unit tests, and (if appropriate) integration tests.
- New integrations must come with docs, unit tests, and (if appropriate) integration tests.
- See [this page](../integrations.mdx) for more details on contributing new integrations.
- See [this page](../integrations/index.mdx) for more details on contributing new integrations.
- New functionality should not inherit from or use deprecated methods or classes.
- We will reject features that are likely to lead to security vulnerabilities or reports.
- Do not add any hard dependencies. Integrations may add optional dependencies.

2
docs/docs/contributing/how_to/index.mdx
View File

@ -2,4 +2,4 @@
- [**Documentation**](documentation/index.mdx): Help improve our docs, including this one!
- [**Code**](code/index.mdx): Help us write code, fix bugs, or improve our infrastructure.
- [**Integrations**](integrations.mdx): Help us integrate with your favorite vendors and tools.
- [**Integrations**](integrations/index.mdx): Help us integrate with your favorite vendors and tools.

203
docs/docs/contributing/how_to/integrations.mdx
View File

@ -1,203 +0,0 @@
---
sidebar_position: 5
---
# Contribute Integrations
To begin, make sure you have all the dependencies outlined in guide on [Contributing Code](code/index.mdx).
There are a few different places you can contribute integrations for LangChain:
- **Community**: For lighter-weight integrations that are primarily maintained by LangChain and the Open Source Community.
- **Partner Packages**: For independent packages that are co-maintained by LangChain and a partner.
For the most part, **new integrations should be added to the Community package**. Partner packages require more maintenance as separate packages, so please confirm with the LangChain team before creating a new partner package.
In the following sections, we'll walk through how to contribute to each of these packages from a fake company, `Parrot Link AI`.
## Community package
The `langchain-community` package is in `libs/community` and contains most integrations.
It can be installed with `pip install langchain-community`, and exported members can be imported with code like
```python
from langchain_community.chat_models import ChatParrotLink
from langchain_community.llms import ParrotLinkLLM
from langchain_community.vectorstores import ParrotLinkVectorStore
```
The `community` package relies on manually-installed dependent packages, so you will see errors
if you try to import a package that is not installed. In our fake example, if you tried to import `ParrotLinkLLM` without installing `parrot-link-sdk`, you will see an `ImportError` telling you to install it when trying to use it.
Let's say we wanted to implement a chat model for Parrot Link AI. We would create a new file in `libs/community/langchain_community/chat_models/parrot_link.py` with the following code:
```python
from langchain_core.language_models.chat_models import BaseChatModel
class ChatParrotLink(BaseChatModel):
"""ChatParrotLink chat model.
Example:
.. code-block:: python
from langchain_community.chat_models import ChatParrotLink
model = ChatParrotLink()
"""
...
```
And we would write tests in:
- Unit tests: `libs/community/tests/unit_tests/chat_models/test_parrot_link.py`
- Integration tests: `libs/community/tests/integration_tests/chat_models/test_parrot_link.py`
And add documentation to:
- `docs/docs/integrations/chat/parrot_link.ipynb`
## Partner package in LangChain repo
:::caution
Before starting a **partner** package, please confirm your intent with the LangChain team. Partner packages require more maintenance as separate packages, so we will close PRs that add new partner packages without prior discussion. See the above section for how to add a community integration.
:::
Partner packages can be hosted in the `LangChain` monorepo or in an external repo.
Partner package in the `LangChain` repo is placed in `libs/partners/{partner}`
and the package source code is in `libs/partners/{partner}/langchain_{partner}`.
A package is
installed by users with `pip install langchain-{partner}`, and the package members
can be imported with code like:
```python
from langchain_{partner} import X
```
### Set up a new package
To set up a new partner package, use the latest version of the LangChain CLI. You can install or update it with:
```bash
pip install -U langchain-cli
```
Let's say you want to create a new partner package working for a company called Parrot Link AI.
Then, run the following command to create a new partner package:
```bash
cd libs/partners
langchain-cli integration new
> Name: parrot-link
> Name of integration in PascalCase [ParrotLink]: ParrotLink
```
This will create a new package in `libs/partners/parrot-link` with the following structure:
```
libs/partners/parrot-link/
langchain_parrot_link/ # folder containing your package
...
tests/
...
docs/ # bootstrapped docs notebooks, must be moved to /docs in monorepo root
...
scripts/ # scripts for CI
...
LICENSE
README.md # fill out with information about your package
Makefile # default commands for CI
pyproject.toml # package metadata, mostly managed by Poetry
poetry.lock # package lockfile, managed by Poetry
.gitignore
```
### Implement your package
First, add any dependencies your package needs, such as your company's SDK:
```bash
poetry add parrot-link-sdk
```
If you need separate dependencies for type checking, you can add them to the `typing` group with:
```bash
poetry add --group typing types-parrot-link-sdk
```
Then, implement your package in `libs/partners/parrot-link/langchain_parrot_link`.
By default, this will include stubs for a Chat Model, an LLM, and/or a Vector Store. You should delete any of the files you won't use and remove them from `__init__.py`.
### Write Unit and Integration Tests
Some basic tests are presented in the `tests/` directory. You should add more tests to cover your package's functionality.
For information on running and implementing tests, see the [Testing guide](testing.mdx).
### Write documentation
Documentation is generated from Jupyter notebooks in the `docs/` directory. You should place the notebooks with examples
to the relevant `docs/docs/integrations` directory in the monorepo root.
### (If Necessary) Deprecate community integration
Note: this is only necessary if you're migrating an existing community integration into
a partner package. If the component you're integrating is net-new to LangChain (i.e.
not already in the `community` package), you can skip this step.
Let's pretend we migrated our `ChatParrotLink` chat model from the community package to
the partner package. We would need to deprecate the old model in the community package.
We would do that by adding a `@deprecated` decorator to the old model as follows, in
`libs/community/langchain_community/chat_models/parrot_link.py`.
Before our change, our chat model might look like this:
```python
class ChatParrotLink(BaseChatModel):
...
```
After our change, it would look like this:
```python
from langchain_core._api.deprecation import deprecated
@deprecated(
since="0.0.<next community version>",
removal="0.2.0",
alternative_import="langchain_parrot_link.ChatParrotLink"
)
class ChatParrotLink(BaseChatModel):
...
```
You should do this for *each* component that you're migrating to the partner package.
### Additional steps
Contributor steps:
- [ ] Add secret names to manual integrations workflow in `.github/workflows/_integration_test.yml`
- [ ] Add secrets to release workflow (for pre-release testing) in `.github/workflows/_release.yml`
Maintainer steps (Contributors should **not** do these):
- [ ] set up pypi and test pypi projects
- [ ] add credential secrets to Github Actions
- [ ] add package to conda-forge
## Partner package in external repo
Partner packages in external repos must be coordinated between the LangChain team and
the partner organization to ensure that they are maintained and updated.
If you're interested in creating a partner package in an external repo, please start
with one in the LangChain repo, and then reach out to the LangChain team to discuss
how to move it to an external repo.

50
docs/docs/contributing/how_to/integrations/community.mdx Normal file
View File

@ -0,0 +1,50 @@
## How to add a community integration (deprecated)
:::danger
We are no longer accepting new community integrations. Please see the
[main integration guide](./index.mdx) for more information on contributing new
integrations.
:::
The `langchain-community` package is in `libs/community`.
It can be installed with `pip install langchain-community`, and exported members can be imported with code like
```python
from langchain_community.chat_models import ChatParrotLink
from langchain_community.llms import ParrotLinkLLM
from langchain_community.vectorstores import ParrotLinkVectorStore
```
The `community` package relies on manually-installed dependent packages, so you will see errors
if you try to import a package that is not installed. In our fake example, if you tried to import `ParrotLinkLLM` without installing `parrot-link-sdk`, you will see an `ImportError` telling you to install it when trying to use it.
Let's say we wanted to implement a chat model for Parrot Link AI. We would create a new file in `libs/community/langchain_community/chat_models/parrot_link.py` with the following code:
```python
from langchain_core.language_models.chat_models import BaseChatModel
class ChatParrotLink(BaseChatModel):
"""ChatParrotLink chat model.
Example:
.. code-block:: python
from langchain_community.chat_models import ChatParrotLink
model = ChatParrotLink()
"""
...
```
And we would write tests in:
- Unit tests: `libs/community/tests/unit_tests/chat_models/test_parrot_link.py`
- Integration tests: `libs/community/tests/integration_tests/chat_models/test_parrot_link.py`
And add documentation to:
- `docs/docs/integrations/chat/parrot_link.ipynb`

132
docs/docs/contributing/how_to/integrations/from_template.mdx Normal file
View File

@ -0,0 +1,132 @@
# How to publish an integration package from a template
:::danger
This guide is a work-in-progress.
:::
First, duplicate this template repository: https://github.com/langchain-ai/integration-repo-template
In this guide, we will create a `libs/langchain-parrot-link` folder, simulating the creation
of a partner package for a fake company, "Parrot Link AI".
A package is
installed by users with `pip install langchain-{partner}`, and the package members
can be imported with code like:
```python
from langchain_{partner} import X
```
## Set up a new package
To set up a new partner package, use the latest version of the LangChain CLI. You can install or update it with:
```bash
pip install -U langchain-cli
```
Let's say you want to create a new partner package working for a company called Parrot Link AI.
Then, run the following command to create a new partner package:
```bash
mkdir libs
cd libs/
langchain-cli integration new
> Name: parrot-link
> Name of integration in PascalCase [ParrotLink]: ParrotLink
```
This will create a new package in `libs/parrot-link` with the following structure:
```
libs/parrot-link/
langchain_parrot_link/ # folder containing your package
...
tests/
...
docs/ # bootstrapped docs notebooks, must be moved to /docs in monorepo root
...
scripts/ # scripts for CI
...
LICENSE
README.md # fill out with information about your package
Makefile # default commands for CI
pyproject.toml # package metadata, mostly managed by Poetry
poetry.lock # package lockfile, managed by Poetry
.gitignore
```
## Implement your package
First, add any dependencies your package needs, such as your company's SDK:
```bash
poetry add parrot-link-sdk
```
If you need separate dependencies for type checking, you can add them to the `typing` group with:
```bash
poetry add --group typing types-parrot-link-sdk
```
Then, implement your package in `libs/partners/parrot-link/langchain_parrot_link`.
By default, this will include stubs for a Chat Model, an LLM, and/or a Vector Store. You should delete any of the files you won't use and remove them from `__init__.py`.
## Write Unit and Integration Tests
Some basic tests are presented in the `tests/` directory. You should add more tests to cover your package's functionality.
For information on running and implementing tests, see the [Testing guide](../testing.mdx).
## Write documentation
Documentation is generated from Jupyter notebooks in the `docs/` directory. You should place the notebooks with examples
to the relevant `docs/docs/integrations` directory in the monorepo root.
## (If Necessary) Deprecate community integration
Note: this is only necessary if you're migrating an existing community integration into
a partner package. If the component you're integrating is net-new to LangChain (i.e.
not already in the `community` package), you can skip this step.
Let's pretend we migrated our `ChatParrotLink` chat model from the community package to
the partner package. We would need to deprecate the old model in the community package.
We would do that by adding a `@deprecated` decorator to the old model as follows, in
`libs/community/langchain_community/chat_models/parrot_link.py`.
Before our change, our chat model might look like this:
```python
class ChatParrotLink(BaseChatModel):
...
```
After our change, it would look like this:
```python
from langchain_core._api.deprecation import deprecated
@deprecated(
since="0.0.<next community version>",
removal="0.2.0",
alternative_import="langchain_parrot_link.ChatParrotLink"
)
class ChatParrotLink(BaseChatModel):
...
```
You should do this for *each* component that you're migrating to the partner package.
## Additional steps
Contributor steps:
- [ ] Add secret names to manual integrations workflow in `.github/workflows/_integration_test.yml`
- [ ] Add secrets to release workflow (for pre-release testing) in `.github/workflows/_release.yml`
- [ ] set up pypi and test pypi projects
- [ ] add credential secrets to Github Actions
- [ ] add package to conda-forge

79
docs/docs/contributing/how_to/integrations/index.mdx Normal file
View File

@ -0,0 +1,79 @@
---
sidebar_position: 5
---
# Contribute Integrations
LangChain integrations are packages that provide access to language models, vector stores, and other components that can be used in LangChain.
This guide will walk you through how to contribute new integrations to LangChain, by
publishing an integration package to PyPi, and adding documentation for it
to the LangChain Monorepo.
These instructions will evolve over the next few months as we improve our integration
processes.
## Components to Integrate
:::info
See the [Conceptual Guide](../../../concepts/index.mdx) for an overview of all components
supported in LangChain
:::
While any component can be integrated into LangChain, at this time we are only accepting
new integrations in the docs of the following kinds:
<table>
<tr>
<th>Integrate these ✅</th>
<th>Not these ❌</th>
</tr>
<tr>
<td>
<ul>
<li>Chat Models</li>
<li>Tools/Toolkits</li>
<li>Retrievers</li>
<li>Document Loaders</li>
<li>Vector Stores</li>
<li>Embedding Models</li>
</ul>
</td>
<td>
<ul>
<li>LLMs (Text-Completion Models)</li>
<li>Key-Value Stores</li>
<li>Document Transformers</li>
<li>Model Caches</li>
<li>Graphs</li>
<li>Message Histories</li>
<li>Callbacks</li>
<li>Chat Loaders</li>
<li>Adapters</li>
<li>Other abstractions</li>
</ul>
</td>
</tr>
</table>
## How to contribute an integration
The only step necessary to "be" a LangChain integration is to add documentation
that will render on this site (https://python.langchain.com/).
As a prerequisite to adding your integration to our documentation, you must:
1. Confirm that your integration is in the list of components we are currently accepting.
2. Ensure that your integration is in a separate package that can be installed with `pip install <your-package>`.
3. Implement the standard tests for your integration and successfully run them.
3. Write documentation for your integration in the `docs/docs/integrations` directory of the LangChain monorepo.
4. Add a provider page for your integration in the `docs/docs/integrations/providers` directory of the LangChain monorepo.
Once you have completed these steps, you can submit a PR to the LangChain monorepo to add your integration to the documentation.
## Further Reading
If you're starting from scratch, you can follow the [Integration Template Guide](./from_template.mdx) to create and publish a new integration package
to the above spec.

116
docs/docs/how_to/tools_standard_tests.ipynb → docs/docs/contributing/how_to/integrations/standard_tests.ipynb
View File

@ -4,9 +4,11 @@
"cell_type": "markdown",
"metadata": {},
"source": [
"# How to add standard tests to a tool\n",
"# How to add standard tests to an integration\n",
"\n",
"When creating either a custom tool or a new tool to publish in a LangChain integration, it is important to add standard tests to ensure the tool works as expected. This guide will show you how to add standard tests to a tool.\n",
"Implementing standard tests \n",
"\n",
"When creating either a custom class for yourself or a new tool to publish in a LangChain integration, it is important to add standard tests to ensure it works as expected. This guide will show you how to add standard tests to a tool, and the templates for implementing each different kind of integration are linked [at the bottom](#standard-test-templates-per-component).\n",
"\n",
"## Setup\n",
"\n",
@ -116,14 +118,14 @@
},
"outputs": [],
"source": [
"# title=\"tests/unit_tests/test_custom_tool.py\"\n",
"# title=\"tests/unit_tests/test_tools.py\"\n",
"from typing import Type\n",
"\n",
"from langchain_parrot_link.tools import ParrotMultiplyTool\n",
"from langchain_standard_tests.unit_tests import ToolsUnitTests\n",
"\n",
"\n",
"class MultiplyToolUnitTests(ToolsUnitTests):\n",
"class ParrotMultiplyToolUnitTests(ToolsUnitTests):\n",
" @property\n",
" def tool_constructor(self) -> Type[ParrotMultiplyTool]:\n",
" return ParrotMultiplyTool\n",
@ -150,14 +152,14 @@
"metadata": {},
"outputs": [],
"source": [
"# title=\"tests/integration_tests/test_custom_tool.py\"\n",
"# title=\"tests/integration_tests/test_tools.py\"\n",
"from typing import Type\n",
"\n",
"from langchain_parrot_link.tools import ParrotMultiplyTool\n",
"from langchain_standard_tests.integration_tests import ToolsIntegrationTests\n",
"\n",
"\n",
"class MultiplyToolIntegrationTests(ToolsIntegrationTests):\n",
"class ParrotMultiplyToolIntegrationTests(ToolsIntegrationTests):\n",
" @property\n",
" def tool_constructor(self) -> Type[ParrotMultiplyTool]:\n",
" return ParrotMultiplyTool\n",
@ -196,7 +198,107 @@
{
"cell_type": "markdown",
"metadata": {},
"source": []
"source": [
"## Standard test templates per component:\n",
"\n",
"Above, we implement the **unit** and **integration** standard tests for a tool. Below are the templates for implementing the standard tests for each component:\n",
"\n",
"<details>\n",
" <summary>Chat Models</summary>"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"# title=\"tests/unit_tests/test_chat_models.py\"\n",
"from typing import Tuple, Type\n",
"\n",
"from langchain_parrot_link.chat_models import ChatParrotLink\n",
"from langchain_standard_tests.unit_tests import ChatModelUnitTests\n",
"\n",
"\n",
"class ChatParrotLinkUnitTests(ChatModelUnitTests):\n",
" @property\n",
" def chat_model_class(self) -> Type[ChatParrotLink]:\n",
" return ChatParrotLink"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"# title=\"tests/integration_tests/test_chat_models.py\"\n",
"from typing import Type\n",
"\n",
"from langchain_parrot_link.chat_models import ChatParrotLink\n",
"from langchain_standard_tests.integration_tests import ChatModelIntegrationTests\n",
"\n",
"\n",
"class TestMistralStandard(ChatModelIntegrationTests):\n",
" @property\n",
" def chat_model_class(self) -> Type[ChatParrotLink]:\n",
" return ChatParrotLink\n",
"\n",
" @property\n",
" def chat_model_params(self) -> dict:\n",
" return {\"model\": \"bird-brain-001\", \"temperature\": 0}"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"</details>\n",
"\n",
"<div style={{display:\"none\"}}>\n",
"Work in progress:\n",
"<details>\n",
" <summary>Tools/Toolkits</summary>\n",
" TODO"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"</details>\n",
"<details>\n",
" <summary>Retrievers</summary>\n",
" TODO"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"</details>\n",
"<details>\n",
" <summary>Vector Stores</summary>\n",
" TODO"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"</details>\n",
"<details>\n",
" <summary>Embedding Models</summary>\n",
" TODO"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"</details>\n",
"</div>"
]
}
],
"metadata": {

2
docs/docs/contributing/index.mdx
View File

@ -16,7 +16,7 @@ More coming soon! We are working on tutorials to help you make your first contri
- [**Documentation**](how_to/documentation/index.mdx): Help improve our docs, including this one!
- [**Code**](how_to/code/index.mdx): Help us write code, fix bugs, or improve our infrastructure.
- [**Integrations**](how_to/integrations.mdx): Help us integrate with your favorite vendors and tools.
- [**Integrations**](how_to/integrations/index.mdx): Help us integrate with your favorite vendors and tools.
## Reference

2
docs/docs/contributing/reference/repo_structure.mdx
View File

@ -61,5 +61,5 @@ The `/libs` directory contains the code for the LangChain packages.
To learn more about how to contribute code see the following guidelines:
- [Code](../how_to/code/index.mdx): Learn how to develop in the LangChain codebase.
- [Integrations](../how_to/integrations.mdx): Learn how to contribute to third-party integrations to `langchain-community` or to start a new partner package.
- [Integrations](../how_to/integrations/index.mdx): Learn how to contribute to third-party integrations to `langchain-community` or to start a new partner package.
- [Testing](../how_to/testing.mdx): Guidelines to learn how to write tests for the packages.