community[patch]: Adding try-except block for GCSDirectoryLoader (#19591)

- **Description:** Implemented try-except block for
`GCSDirectoryLoader`. Reason: Users processing large number of
unstructured files in a folder may experience many different errors. A
try-exception block is added to capture these errors. A new argument
`use_try_except=True` is added to enable *silent failure* so that error
caused by processing one file does not break the whole function.
- **Issue:** N/A
- **Dependencies:** no new dependencies
- **Twitter handle:** timothywong731

---------

Co-authored-by: Bagatur <baskaryan@gmail.com>
This commit is contained in:
Timothy
2024-03-27 00:12:24 +00:00
committed by GitHub
parent aea2be5bf3
commit ad77fa15ee
2 changed files with 55 additions and 4 deletions

View File

@@ -126,17 +126,40 @@
]
},
{
"cell_type": "code",
"execution_count": null,
"cell_type": "markdown",
"id": "f9c0734f",
"metadata": {},
"source": [
"## Continue on failure to load a single file\n",
"Files in a GCS bucket may cause errors during processing. Enable the `continue_on_failure=True` argument to allow silent failure. This means failure to process a single file will not break the function, it will log a warning instead. "
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "3d774795",
"metadata": {},
"outputs": [],
"source": []
"source": [
"loader = GCSDirectoryLoader(\n",
" project_name=\"aist\", bucket=\"testing-hwc\", continue_on_failure=True\n",
")"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "0d15f536",
"metadata": {},
"outputs": [],
"source": [
"loader.load()"
]
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3 (ipykernel)",
"display_name": "Python 3.10.6 64-bit",
"language": "python",
"name": "python3"
},
@@ -151,6 +174,11 @@
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.10.6"
},
"vscode": {
"interpreter": {
"hash": "5f90d085fc70553c85f15dd96b84c64a94d58988a621c9dbc38cac6a7e6079b3"
}
}
},
"nbformat": 4,