Make recursive loader yield while crawling (#7568)

Support actual lazy_load since it can take a while to crawl larger
directories.

docs/extras/modules/data_connection/document_loaders/integrations/recursive_url_loader.ipynb

@@ -1,7 +1,6 @@
 {
  "cells": [
   {
-   "attachments": {},
    "cell_type": "markdown",
    "id": "5a7cc773",
    "metadata": {},
@@ -25,7 +24,7 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 2,
+   "execution_count": 1,
    "id": "2e3532b2",
    "metadata": {},
    "outputs": [],
@@ -34,7 +33,6 @@
    ]
   },
   {
-   "attachments": {},
    "cell_type": "markdown",
    "id": "6384c057",
    "metadata": {},
@@ -44,7 +42,7 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 3,
+   "execution_count": 2,
    "id": "d69e5620",
    "metadata": {},
    "outputs": [],
@@ -56,7 +54,7 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 4,
+   "execution_count": 3,
    "id": "084fb2ce",
    "metadata": {},
    "outputs": [
@@ -66,7 +64,7 @@
        "12"
       ]
      },
-     "execution_count": 4,
+     "execution_count": 3,
      "metadata": {},
      "output_type": "execute_result"
     }
@@ -77,17 +75,17 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 5,
+   "execution_count": 4,
    "id": "89355b7c",
    "metadata": {},
    "outputs": [
     {
      "data": {
       "text/plain": [
-       "'\\n\\n\\n\\n\\nDynamoDB-Backed Chat Memory | \\uf8ffü¶úÔ∏è\\uf8ffüîó Lan'"
+       "'\\n\\n\\n\\n\\nBuffer Window Memory | 🦜️🔗 Langchain\\n\\n\\n\\n\\n\\nSki'"
       ]
      },
-     "execution_count": 5,
+     "execution_count": 4,
      "metadata": {},
      "output_type": "execute_result"
     }
@@ -98,20 +96,20 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 6,
+   "execution_count": 5,
    "id": "13bd7e16",
    "metadata": {},
    "outputs": [
     {
      "data": {
       "text/plain": [
-       "{'source': 'https://js.langchain.com/docs/modules/memory/examples/dynamodb',\n",
-       " 'title': 'DynamoDB-Backed Chat Memory | \\uf8ffü¶úÔ∏è\\uf8ffüîó Langchain',\n",
-       " 'description': 'For longer-term persistence across chat sessions, you can swap out the default in-memory chatHistory that backs chat memory classes like BufferMemory for a DynamoDB instance.',\n",
+       "{'source': 'https://js.langchain.com/docs/modules/memory/examples/buffer_window_memory',\n",
+       " 'title': 'Buffer Window Memory | 🦜️🔗 Langchain',\n",
+       " 'description': 'BufferWindowMemory keeps track of the back-and-forths in conversation, and then uses a window of size k to surface the last k back-and-forths to use as memory.',\n",
        " 'language': 'en'}"
       ]
      },
-     "execution_count": 6,
+     "execution_count": 5,
      "metadata": {},
      "output_type": "execute_result"
     }
@@ -121,14 +119,29 @@
    ]
   },
   {
-   "attachments": {},
    "cell_type": "markdown",
    "id": "40fc13ef",
    "metadata": {},
    "source": [
     "Now, let's try a more extensive example, the `docs` root dir.\n",
     "\n",
-    "We will skip everything under `api`."
+    "We will skip everything under `api`.\n",
+    "\n",
+    "For this, we can `lazy_load`  each page as we crawl the tree, using `WebBaseLoader` to load each as we go."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "5c938b9f",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "url = \"https://js.langchain.com/docs/\"\n",
+    "exclude_dirs = [\"https://js.langchain.com/docs/api/\"]\n",
+    "loader = RecursiveUrlLoader(url=url, exclude_dirs=exclude_dirs)\n",
+    "# Lazy load each\n",
+    "docs = [print(doc) or doc for doc in loader.lazy_load()]"
    ]
   },
   {
@@ -138,9 +151,7 @@
    "metadata": {},
    "outputs": [],
    "source": [
-    "url = \"https://js.langchain.com/docs/\"\n",
-    "exclude_dirs = [\"https://js.langchain.com/docs/api/\"]\n",
-    "loader = RecursiveUrlLoader(url=url, exclude_dirs=exclude_dirs)\n",
+    "# Load all pages\n",
     "docs = loader.load()"
    ]
   },
@@ -148,12 +159,14 @@
    "cell_type": "code",
    "execution_count": 8,
    "id": "457e30f3",
-   "metadata": {},
+   "metadata": {
+    "scrolled": true
+   },
    "outputs": [
     {
      "data": {
       "text/plain": [
-       "176"
+       "188"
       ]
      },
      "execution_count": 8,
@@ -174,7 +187,7 @@
     {
      "data": {
       "text/plain": [
-       "'\\n\\n\\n\\n\\nHacker News | \\uf8ffü¶úÔ∏è\\uf8ffüîó Langchain\\n\\n\\n\\n\\n\\nSkip'"
+       "'\\n\\n\\n\\n\\nAgent Simulations | 🦜️🔗 Langchain\\n\\n\\n\\n\\n\\nSkip t'"
       ]
      },
      "execution_count": 9,
@@ -195,9 +208,9 @@
     {
      "data": {
       "text/plain": [
-       "{'source': 'https://js.langchain.com/docs/modules/indexes/document_loaders/examples/web_loaders/hn',\n",
-       " 'title': 'Hacker News | \\uf8ffü¶úÔ∏è\\uf8ffüîó Langchain',\n",
-       " 'description': 'This example goes over how to load data from the hacker news website, using Cheerio. One document will be created for each page.',\n",
+       "{'source': 'https://js.langchain.com/docs/use_cases/agent_simulations/',\n",
+       " 'title': 'Agent Simulations | 🦜️🔗 Langchain',\n",
+       " 'description': 'Agent simulations involve taking multiple agents and having them interact with each other.',\n",
        " 'language': 'en'}"
       ]
      },