mirror of
https://github.com/hwchase17/langchain.git
synced 2025-06-23 15:19:33 +00:00
Add langchain-yt-dlp Document Loader Documentation (#28775)
## Overview This PR adds documentation for the `langchain-yt-dlp` package, a YouTube document loader that uses `yt-dlp` for Youtube videos metadata extraaction. ## Changes - Added documentation notebook for YoutubeLoader - Updated packages.yml to include langchain-yt-dlp ## Motivation The existing LangChain YoutubeLoader was unable to fetch YouTube metadata due to changes in YouTube's structure. This package resolves those issues by leveraging the `yt-dlp` library. ## Features - Reliable YouTube metadata extraction ## Related - Package Repository: https://github.com/aqib0770/langchain-yt-dlp - PyPI Package: https://pypi.org/project/langchain-yt-dlp/ --------- Co-authored-by: Chester Curme <chester.curme@gmail.com>
This commit is contained in:
ANSARI MD AAQIB AHMED
GitHub
parent
33b1fb95b8
commit
91d28ef453
149
docs/docs/integrations/document_loaders/yt_dlp.ipynb
Normal file
149
docs/docs/integrations/document_loaders/yt_dlp.ipynb
Normal file
@ -0,0 +1,149 @@
|
||||
{
|
||||
"cells": [
|
||||
{
|
||||
"cell_type": "raw",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"---\n",
|
||||
"sidebar_label: YoutubeLoaderDL\n",
|
||||
"---"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"# YoutubeLoaderDL\n",
|
||||
"\n",
|
||||
"Loader for Youtube leveraging the `yt-dlp` library.\n",
|
||||
"\n",
|
||||
"This package implements a [document loader](/docs/concepts/document_loaders/) for Youtube. In contrast to the [YoutubeLoader](https://python.langchain.com/api_reference/community/document_loaders/langchain_community.document_loaders.youtube.YoutubeLoader.html) of `langchain-community`, which relies on `pytube`, `YoutubeLoaderDL` is able to fetch YouTube metadata. `langchain-yt-dlp` leverages the robust `yt-dlp` library, providing a more reliable and feature-rich YouTube document loader.\n",
|
||||
"\n",
|
||||
"## Overview\n",
|
||||
"### Integration details\n",
|
||||
"\n",
|
||||
"| Class | Package | Local | Serializable | JS Support |\n",
|
||||
"| :--- | :--- | :---: | :---: | :---: |\n",
|
||||
"| YoutubeLoader | langchain-yt-dlp | ✅ | ✅ | ❌ |\n",
|
||||
"\n",
|
||||
"## Setup\n",
|
||||
"\n",
|
||||
"### Installation\n",
|
||||
"\n",
|
||||
"```bash\n",
|
||||
"pip install langchain-yt-dlp\n",
|
||||
"```"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"### Initialization"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 10,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"from langchain_yt_dlp.youtube_loader import YoutubeLoaderDL\n",
|
||||
"\n",
|
||||
"# Basic transcript loading\n",
|
||||
"loader = YoutubeLoaderDL.from_youtube_url(\n",
|
||||
" \"https://www.youtube.com/watch?v=dQw4w9WgXcQ\", add_video_info=True\n",
|
||||
")"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"### Load"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 11,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"documents = loader.load()"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 12,
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"data": {
|
||||
"text/plain": [
|
||||
"{'source': 'dQw4w9WgXcQ',\n",
|
||||
" 'title': 'Rick Astley - Never Gonna Give You Up (Official Music Video)',\n",
|
||||
" 'description': 'The official video for “Never Gonna Give You Up” by Rick Astley. \\n\\nNever: The Autobiography 📚 OUT NOW! \\nFollow this link to get your copy and listen to Rick’s ‘Never’ playlist ❤️ #RickAstleyNever\\nhttps://linktr.ee/rickastleynever\\n\\n“Never Gonna Give You Up” was a global smash on its release in July 1987, topping the charts in 25 countries including Rick’s native UK and the US Billboard Hot 100. It also won the Brit Award for Best single in 1988. Stock Aitken and Waterman wrote and produced the track which was the lead-off single and lead track from Rick’s debut LP “Whenever You Need Somebody”. The album was itself a UK number one and would go on to sell over 15 million copies worldwide.\\n\\nThe legendary video was directed by Simon West – who later went on to make Hollywood blockbusters such as Con Air, Lara Croft – Tomb Raider and The Expendables 2. The video passed the 1bn YouTube views milestone on 28 July 2021.\\n\\nSubscribe to the official Rick Astley YouTube channel: https://RickAstley.lnk.to/YTSubID\\n\\nFollow Rick Astley:\\nFacebook: https://RickAstley.lnk.to/FBFollowID \\nTwitter: https://RickAstley.lnk.to/TwitterID \\nInstagram: https://RickAstley.lnk.to/InstagramID \\nWebsite: https://RickAstley.lnk.to/storeID \\nTikTok: https://RickAstley.lnk.to/TikTokID\\n\\nListen to Rick Astley:\\nSpotify: https://RickAstley.lnk.to/SpotifyID \\nApple Music: https://RickAstley.lnk.to/AppleMusicID \\nAmazon Music: https://RickAstley.lnk.to/AmazonMusicID \\nDeezer: https://RickAstley.lnk.to/DeezerID \\n\\nLyrics:\\nWe’re no strangers to love\\nYou know the rules and so do I\\nA full commitment’s what I’m thinking of\\nYou wouldn’t get this from any other guy\\n\\nI just wanna tell you how I’m feeling\\nGotta make you understand\\n\\nNever gonna give you up\\nNever gonna let you down\\nNever gonna run around and desert you\\nNever gonna make you cry\\nNever gonna say goodbye\\nNever gonna tell a lie and hurt you\\n\\nWe’ve known each other for so long\\nYour heart’s been aching but you’re too shy to say it\\nInside we both know what’s been going on\\nWe know the game and we’re gonna play it\\n\\nAnd if you ask me how I’m feeling\\nDon’t tell me you’re too blind to see\\n\\nNever gonna give you up\\nNever gonna let you down\\nNever gonna run around and desert you\\nNever gonna make you cry\\nNever gonna say goodbye\\nNever gonna tell a lie and hurt you\\n\\n#RickAstley #NeverGonnaGiveYouUp #WheneverYouNeedSomebody #OfficialMusicVideo',\n",
|
||||
" 'view_count': 1603360806,\n",
|
||||
" 'publish_date': datetime.datetime(2009, 10, 25, 0, 0),\n",
|
||||
" 'length': 212,\n",
|
||||
" 'author': 'Rick Astley',\n",
|
||||
" 'channel_id': 'UCuAXFkgsw1L7xaCfnd5JJOw',\n",
|
||||
" 'webpage_url': 'https://www.youtube.com/watch?v=dQw4w9WgXcQ'}"
|
||||
]
|
||||
},
|
||||
"execution_count": 12,
|
||||
"metadata": {},
|
||||
"output_type": "execute_result"
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"documents[0].metadata"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## Lazy Load"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"- No lazy loading is implemented"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## API reference:\n",
|
||||
"\n",
|
||||
"- [Github](https://github.com/aqib0770/langchain-yt-dlp)\n",
|
||||
"- [PyPi](https://pypi.org/project/langchain-yt-dlp/)"
|
||||
]
|
||||
}
|
||||
],
|
||||
"metadata": {
|
||||
"kernelspec": {
|
||||
"display_name": "Python 3 (ipykernel)",
|
||||
"language": "python",
|
||||
"name": "python3"
|
||||
},
|
||||
"language_info": {
|
||||
"codemirror_mode": {
|
||||
"name": "ipython",
|
||||
"version": 3
|
||||
},
|
||||
"file_extension": ".py",
|
||||
"mimetype": "text/x-python",
|
||||
"name": "python",
|
||||
"nbconvert_exporter": "python",
|
||||
"pygments_lexer": "ipython3",
|
||||
"version": "3.10.4"
|
||||
}
|
||||
},
|
||||
"nbformat": 4,
|
||||
"nbformat_minor": 4
|
||||
}
|
||||
@ -158,6 +158,9 @@ packages:
|
||||
- name: langchain-linkup
|
||||
repo: LinkupPlatform/langchain-linkup
|
||||
path: .
|
||||
- name: langchain-yt-dlp
|
||||
repo: aqib0770/langchain-yt-dlp
|
||||
path: .
|
||||
- name: langchain-oceanbase
|
||||
repo: oceanbase/langchain-oceanbase
|
||||
path: .
|
||||
path: .
|
||||
Loading…
Reference in New Issue
Block a user