add save and load tfidf vectorizer and docs for TFIDFRetriever (#8112)

This is to add save_local and load_local to tfidf_vectorizer and docs in
tfidf_retriever to make the vectorizer reusable.

<!-- Thank you for contributing to LangChain!

Replace this comment with:
- Description: add save_local and load_local to tfidf_vectorizer and
docs in tfidf_retriever
  - Issue: None
  - Dependencies: None
  - Tag maintainer: @rlancemartin, @eyurtsev
  - Twitter handle: @MlopsJ

Please make sure you're PR is passing linting and testing before
submitting. Run `make format`, `make lint` and `make test` to check this
locally.

If you're adding a new integration, please include:
1. a test for the integration, preferably unit tests that do not rely on
network access,
  2. an example notebook showing its use.

Maintainer responsibilities:
  - General / Misc / if you don't know who to tag: @baskaryan
  - DataLoaders / VectorStores / Retrievers: @rlancemartin, @eyurtsev
  - Models / Prompts: @hwchase17, @baskaryan
  - Memory: @hwchase17
  - Agents / Tools / Toolkits: @hinthornw
  - Tracing / Callbacks: @agola11
  - Async: @agola11

If no one reviews your PR within a few days, feel free to @-mention the
same people again.

See contribution guidelines for more information on how to write/run
tests, lint, etc:
https://github.com/hwchase17/langchain/blob/master/.github/CONTRIBUTING.md
 -->

---------

Co-authored-by: Harrison Chase <hw.chase.17@gmail.com>
This commit is contained in:
shibuiwilliam
2023-08-04 15:06:27 +09:00
committed by GitHub
parent 0f68054401
commit 2759e2d857
3 changed files with 141 additions and 4 deletions

View File

@@ -16,7 +16,7 @@
},
{
"cell_type": "code",
"execution_count": 1,
"execution_count": 2,
"id": "a801b57c",
"metadata": {},
"outputs": [],
@@ -26,7 +26,7 @@
},
{
"cell_type": "code",
"execution_count": null,
"execution_count": 3,
"id": "393ac030",
"metadata": {
"tags": []
@@ -46,7 +46,7 @@
},
{
"cell_type": "code",
"execution_count": 3,
"execution_count": 4,
"id": "98b1c017",
"metadata": {
"tags": []
@@ -133,6 +133,68 @@
"source": [
"result"
]
},
{
"cell_type": "markdown",
"id": "363f3c04",
"metadata": {},
"source": [
"## Save and load\n",
"\n",
"You can easily save and load this retriever, making it handy for local development!"
]
},
{
"cell_type": "code",
"execution_count": 8,
"id": "10c90d03",
"metadata": {},
"outputs": [],
"source": [
"retriever.save_local(\"testing.pkl\")"
]
},
{
"cell_type": "code",
"execution_count": 9,
"id": "fb3b153c",
"metadata": {},
"outputs": [],
"source": [
"retriever_copy = TFIDFRetriever.load_local(\"testing.pkl\")"
]
},
{
"cell_type": "code",
"execution_count": 10,
"id": "c03ff3c7",
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"[Document(page_content='foo', metadata={}),\n",
" Document(page_content='foo bar', metadata={}),\n",
" Document(page_content='hello', metadata={}),\n",
" Document(page_content='world', metadata={})]"
]
},
"execution_count": 10,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"retriever_copy.get_relevant_documents(\"foo\")"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "2d7c5728",
"metadata": {},
"outputs": [],
"source": []
}
],
"metadata": {
@@ -151,7 +213,7 @@
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.11.3"
"version": "3.10.1"
}
},
"nbformat": 4,