Harrison/multithreading directory loader (#4650)

Co-authored-by: PawelFaron <42373772+PawelFaron@users.noreply.github.com>
Co-authored-by: Pawel Faron <ext-pawel.faron@vaisala.com>
This commit is contained in:
Harrison Chase
2023-05-13 21:46:02 -07:00
committed by GitHub
parent b0c733e327
commit 44ae673388
2 changed files with 63 additions and 16 deletions

View File

@@ -112,6 +112,34 @@
"docs = loader.load()"
]
},
{
"cell_type": "markdown",
"id": "c16ed46a",
"metadata": {},
"source": [
"## Use multithreading"
]
},
{
"attachments": {},
"cell_type": "markdown",
"id": "5752e23e",
"metadata": {},
"source": [
"By default the loading happens in one thread. In order to utilize several threads set the `use_multithreading` flag to true."
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "f8d84f52",
"metadata": {},
"outputs": [],
"source": [
"loader = DirectoryLoader('../', glob=\"**/*.md\", use_multithreading=True)\n",
"docs = loader.load()"
]
},
{
"cell_type": "markdown",
"id": "c5652850",