community[minor]: add DashScope Rerank (#22403)

**Description:** this PR adds DashScope Rerank capability to Langchain,
you can find DashScope Rerank API from
[here](https://help.aliyun.com/document_detail/2780058.html?spm=a2c4g.2780059.0.0.6d995024FlrJ12)
&
[here](https://help.aliyun.com/document_detail/2780059.html?spm=a2c4g.2780058.0.0.63f75024cr11N9).
[DashScope](https://dashscope.aliyun.com/) is the generative AI service
from Alibaba Cloud (Aliyun). You can create DashScope API key from
[here](https://bailian.console.aliyun.com/?apiKey=1#/api-key).

**Dependencies:** DashScopeRerank depends on `dashscope` python package.

**Twitter handle:** my twitter/x account is https://x.com/LastMonopoly
and I'd like a mention, thanks you!


**Tests and docs**
  1. integration test: `test_dashscope_rerank.py`
  2. example notebook: `dashscope_rerank.ipynb`

**Lint and test**: I have run `make format`, `make lint` and `make test`
from the root of the package I've modified.

---------

Co-authored-by: Bagatur <baskaryan@gmail.com>
Co-authored-by: Bagatur <22008038+baskaryan@users.noreply.github.com>
This commit is contained in:
X-HAN
2024-06-06 06:40:21 +08:00
committed by GitHub
parent 29064848f9
commit 62f13f95e4
5 changed files with 536 additions and 0 deletions

View File

@@ -0,0 +1,24 @@
from langchain_core.documents import Document
from langchain_community.document_compressors.dashscope_rerank import (
DashScopeRerank,
)
def test_rerank() -> None:
reranker = DashScopeRerank(api_key=None)
docs = [
Document(page_content="量子计算是计算科学的一个前沿领域"),
Document(page_content="预训练语言模型的发展给文本排序模型带来了新的进展"),
Document(
page_content="文本排序模型广泛用于搜索引擎和推荐系统中,它们根据文本相关性对候选文本进行排序"
),
Document(page_content="random text for nothing"),
]
compressed = reranker.compress_documents(
query="什么是文本排序模型",
documents=docs,
)
assert len(compressed) == 3, "default top_n is 3"
assert compressed[0].page_content == docs[2].page_content, "rerank works"

View File

@@ -6,6 +6,7 @@ EXPECTED_ALL = [
"JinaRerank",
"RankLLMRerank",
"FlashrankRerank",
"DashScopeRerank",
]