Multi-Threaded downloading for playlists #255

Author

Owner

@phil2508 commented on GitHub (Jun 10, 2012):

This technique is used by desktop apps and is very useful in increasing download speed. I agree that 50k/s is way too slow.

@phil2508 commented on GitHub (Jun 10, 2012): This technique is used by desktop apps and is very useful in increasing download speed. I agree that 50k/s is way too slow.

deekerman commented

Author

Owner

@reicab commented on GitHub (Jul 9, 2012):

I agree that this is essential - it is the only issue I have with the youtube-dl library but its a big issue. multi-threaded dowloading and the resulting increase in speed is a dire necessity.

@reicab commented on GitHub (Jul 9, 2012): I agree that this is essential - it is the only issue I have with the youtube-dl library but its a big issue. multi-threaded dowloading and the resulting increase in speed is a dire necessity.

deekerman commented

Author

Owner

@joshua-hull commented on GitHub (Jul 9, 2012):

Well there might be other reasons you're only getting 50k/s. It could be an issue with your ISP or the site. All that being said I too would love to see concurrent or multi threaded downloads.

Sent via Hubroid

@joshua-hull commented on GitHub (Jul 9, 2012): Well there might be other reasons you're only getting 50k/s. It could be an issue with your ISP or the site. All that being said I too would love to see concurrent or multi threaded downloads. _Sent via Hubroid_

deekerman commented

Author

Owner

@phil2508 commented on GitHub (Jul 9, 2012):

It's actually Youtube slowing it down after the first 30% have been cached. I'm running this on a box with dedicated 200MB/s internet connection.

@phil2508 commented on GitHub (Jul 9, 2012): It's actually Youtube slowing it down after the first 30% have been cached. I'm running this on a box with dedicated 200MB/s internet connection.

deekerman commented

Author

Owner

@konisto commented on GitHub (Jul 9, 2012):

Yes, it is youtube and yes, PLEASE add multithreaded downloading. we promote a desktop app downloading youtube apps and multithreading is crucial for use experience.

we need to be able to offer the same speed for online conversions... thanks

@konisto commented on GitHub (Jul 9, 2012): Yes, it is youtube and yes, PLEASE add multithreaded downloading. we promote a desktop app downloading youtube apps and multithreading is crucial for use experience. we need to be able to offer the same speed for online conversions... thanks

deekerman commented

Author

Owner

@phil2508 commented on GitHub (Aug 13, 2012):

any chance this will be going somewhere? Are we the only ones annoyed by youtube throttling our downloads? thanks!

@phil2508 commented on GitHub (Aug 13, 2012): any chance this will be going somewhere? Are we the only ones annoyed by youtube throttling our downloads? thanks!

deekerman commented

Author

Owner

@dev0k commented on GitHub (Nov 12, 2012):

Hi!

use external downloaders (like wget, axel) and get full speed. :)

@dev0k commented on GitHub (Nov 12, 2012): Hi! use external downloaders (like wget, axel) and get full speed. :)

deekerman commented

Author

Owner

@phil2508 commented on GitHub (Nov 21, 2012):

Wonderful idea. I've just implemented Axel and get 15MB/s now instead of 200k/s. Very nice indeed. Thanks so much for pointing me this way.

The only thing I don't like is that did I not think of this myself??!! Seems so easy.

@phil2508 commented on GitHub (Nov 21, 2012): Wonderful idea. I've just implemented Axel and get 15MB/s now instead of 200k/s. Very nice indeed. Thanks so much for pointing me this way. The only thing I don't like is that did I not think of this myself??!! Seems so easy.

deekerman commented

Author

Owner

@phil2508 commented on GitHub (Nov 21, 2012):

By the way, wget does not make sense as it is single threaded, too and gets the same throttling as youtube-dl.py

@phil2508 commented on GitHub (Nov 21, 2012): By the way, wget does not make sense as it is single threaded, too and gets the same throttling as youtube-dl.py

deekerman commented

Author

Owner

@singhpratyush commented on GitHub (Sep 6, 2016):

I tried many options and here is the best that I came up with - use aria2.

The `--external-downloader` option

$ youtube-dl --external-downloader aria2c --external-downloader-args '-x <number of threads>' https://www.youtube.com/playlist?list=PLsPUh22kYmNBkabv9M4fXo6HMLKnc7iR6

But this will download only one file at a time. To download many videos simultaneously, you will have to prepopulate download links.

Prepopulate download links

$ youtube-dl -g https://www.youtube.com/playlist?list=PLsPUh22kYmNBkabv9M4fXo6HMLKnc7iR6 > dlinks.txt
$ aria2c -i dlinks.txt -j <number of simultaneous downloads> -x <number of threads for each download>

You will have to use >> instead of > in the first command to append more links.

The two commands may also be combined -

$ aria2c -j <number of simultaneous downloads> -x <number of threads for each download> `youtube-dl -g https://www.youtube.com/playlist?list=PLsPUh22kYmNBkabv9M4fXo6HMLKnc7iR6`

But this will result in no backup of your links in case download interrupts.

Note: In case of prepopulating links, you may end up in a state when your links get expired because your previous downloads would have taken a while.

@singhpratyush commented on GitHub (Sep 6, 2016): I tried many options and here is the best that I came up with - use [aria2](https://github.com/aria2/aria2). ### The `--external-downloader` option ``` bash $ youtube-dl --external-downloader aria2c --external-downloader-args '-x <number of threads>' https://www.youtube.com/playlist?list=PLsPUh22kYmNBkabv9M4fXo6HMLKnc7iR6 ``` But this will download only one file at a time. To download many videos simultaneously, you will have to prepopulate download links. ### Prepopulate download links ``` bash $ youtube-dl -g https://www.youtube.com/playlist?list=PLsPUh22kYmNBkabv9M4fXo6HMLKnc7iR6 > dlinks.txt $ aria2c -i dlinks.txt -j <number of simultaneous downloads> -x <number of threads for each download> ``` You will have to use `>>` instead of `>` in the first command to append more links. The two commands may also be combined - ``` bash $ aria2c -j <number of simultaneous downloads> -x <number of threads for each download> `youtube-dl -g https://www.youtube.com/playlist?list=PLsPUh22kYmNBkabv9M4fXo6HMLKnc7iR6` ``` But this will result in no backup of your links in case download interrupts. **Note**: In case of prepopulating links, you may end up in a state when your links get expired because your previous downloads would have taken a while.

deekerman commented

Author

Owner

@yan12125 commented on GitHub (Oct 27, 2016):

Let me write some summary. Here are several levels to implement threaded downloading in youtube-dl:

User-input URLs, including command line options and batch files. (#109)
Playlists (this issue)
Segments in segmented streams (HLS, HDS, DASH) (#11044)
Chunk level in a single URL (#1498)

Only the last case works with external downloaders like aria2c or axel. For the first three kinds, more codes are necessary.

@yan12125 commented on GitHub (Oct 27, 2016): Let me write some summary. Here are several levels to implement threaded downloading in youtube-dl: 1. User-input URLs, including command line options and batch files. (#109) 2. Playlists (this issue) 3. Segments in segmented streams (HLS, HDS, DASH) (#11044) 4. Chunk level in a single URL (#1498) Only the last case works with external downloaders like aria2c or axel. For the first three kinds, more codes are necessary.

deekerman commented

Author

Owner

@d3fault commented on GitHub (Feb 21, 2017):

Multi-Threaded postproc for playlists

Was tempted to open another issue for it, but this is too similar (edit: as suggested, I put this in a separate issue #12215). I'm fine with single threaded downloads (multi threaded downloads introduce the possibility of getting your IP banned), but I'd like to see the postproc done in separate thread(s) (up to N threads, N being the number of CPU cores). As soon as a download is finished, the postproc job should be dispatched and the next download should start immediately. But if all N cores are doing postproc, wait for one to finish before downloading another playlist item. If the machine has only 1 core then the the behavior is exactly the same as it already is.

On my setup postproc is the bottleneck, not bandwidth. Postproc benefits much more from mult-threading, whereas downloading only benefits from multi-threading if the server is throttling you (which implies they don't want you to use multiple connections simultaneously, and might ban you if you do).

@d3fault commented on GitHub (Feb 21, 2017): Multi-Threaded postproc for playlists Was tempted to open another issue for it, but this is too similar (edit: as suggested, I put this in a separate issue #12215). I'm fine with single threaded downloads (multi threaded downloads introduce the possibility of getting your IP banned), but I'd like to see the postproc done in separate thread(s) (up to N threads, N being the number of CPU cores). As soon as a download is finished, the postproc job should be dispatched and the next download should start immediately. But if all N cores are doing postproc, wait for one to finish before downloading another playlist item. If the machine has only 1 core then the the behavior is exactly the same as it already is. On my setup postproc is the bottleneck, not bandwidth. Postproc benefits much more from mult-threading, whereas downloading only benefits from multi-threading if the server is throttling you (which implies they don't want you to use multiple connections simultaneously, and might ban you if you do).

deekerman commented

Author

Owner

@yan12125 commented on GitHub (Feb 22, 2017):

@d3fault: Thanks for writing this. Technically, postprocessing and downloading are implemented in different modules, it's better to have an separate issue.

@yan12125 commented on GitHub (Feb 22, 2017): @d3fault: Thanks for writing this. Technically, postprocessing and downloading are implemented in different modules, it's better to have an separate issue.

deekerman commented

Author

Owner

@TyraelElDruin commented on GitHub (Mar 19, 2017):

I wrote a multithreaded youtube-dl GUI. It basically just wraps the youtube-dl process and calls it for as many links as you set. The playlist portion does not run multithreaded however - before I wrapped youtube-dl I wrote the code to pull the links automatically, but since youtube-dl already handled it in a single thread, I left it alone. You could use it to download multiple playlists at a time though. One thing I've done for personal use is inject some javascript in the youtube playlist browser that just pulls the links into a text box at the bottom. You end up with a list of links and you can just paste them in and watch the multithreaded magic go. You can see it here https://www.youtube.com/watch?v=ihEH4DuTcTg

@TyraelElDruin commented on GitHub (Mar 19, 2017): I wrote a multithreaded youtube-dl GUI. It basically just wraps the youtube-dl process and calls it for as many links as you set. The playlist portion does not run multithreaded however - before I wrapped youtube-dl I wrote the code to pull the links automatically, but since youtube-dl already handled it in a single thread, I left it alone. You could use it to download multiple playlists at a time though. One thing I've done for personal use is inject some javascript in the youtube playlist browser that just pulls the links into a text box at the bottom. You end up with a list of links and you can just paste them in and watch the multithreaded magic go. You can see it here https://www.youtube.com/watch?v=ihEH4DuTcTg

deekerman commented

Author

Owner

@ncovercash commented on GitHub (Jun 10, 2017):

Will this ever be put in the main program?

@ncovercash commented on GitHub (Jun 10, 2017): Will this ever be put in the main program?

deekerman commented

Author

Owner

@Gaploid commented on GitHub (May 10, 2018):

Hey Guys, any plans to add multithreaded HLS downloading feature?

@Gaploid commented on GitHub (May 10, 2018): Hey Guys, any plans to add multithreaded HLS downloading feature?

deekerman commented

Author

Owner

@crypdick commented on GitHub (Jul 6, 2018):

@singhpratyush Is it possible to modify your command to extract audio? In particular, I want to extract audio-- I usually run youtube-dl --extract-audio --format bestaudio --external-downloader aria2c --external-downloader-args '--max-connection-per-server=15' --yes-playlist "<plist_url>"

@crypdick commented on GitHub (Jul 6, 2018): @singhpratyush Is it possible to modify your command to extract audio? In particular, I want to extract audio-- I usually run `youtube-dl --extract-audio --format bestaudio --external-downloader aria2c --external-downloader-args '--max-connection-per-server=15' --yes-playlist "<plist_url>"`

deekerman commented

Author

Owner

@singhpratyush commented on GitHub (Jul 7, 2018):

@crypdick: Yes, it would work fine. You can also do postprocessing on the audio if you want, e.g. --audio-format mp3.

@singhpratyush commented on GitHub (Jul 7, 2018): @crypdick: Yes, it would work fine. You can also do [postprocessing](https://github.com/rg3/youtube-dl#post-processing-options) on the audio if you want, e.g. `--audio-format mp3`.

deekerman commented

Author

Owner

@oltodosel commented on GitHub (Sep 4, 2018):

my sh snippet

ytall() {
	special_o='%(upload_date)s__%(title)s-%(id)s.%(ext)s'
	default_threads=3
	
	if [[ -z $1 ]]; then
		echo "ytall [channel_url/playlist] [threads] [special -o ${special_o}]"
		return
	fi
	
	if [[ -n $3 ]]; then
		youtube-dl -j --flat-playlist $1 | jq -r '.url' | parallel -j $2 --progress youtube-dl -i -o \'$special_o\' -- 
	elif [[ -n $2 ]]; then
		youtube-dl -j --flat-playlist $1 | jq -r '.url' | parallel -j $2 --progress youtube-dl -i --
	else
		youtube-dl -j --flat-playlist $1 | jq -r '.url' | parallel -j $default_threads --progress youtube-dl -i --
	fi
}

@oltodosel commented on GitHub (Sep 4, 2018): my sh snippet ```zsh ytall() { special_o='%(upload_date)s__%(title)s-%(id)s.%(ext)s' default_threads=3 if [[ -z $1 ]]; then echo "ytall [channel_url/playlist] [threads] [special -o ${special_o}]" return fi if [[ -n $3 ]]; then youtube-dl -j --flat-playlist $1 | jq -r '.url' | parallel -j $2 --progress youtube-dl -i -o \'$special_o\' -- elif [[ -n $2 ]]; then youtube-dl -j --flat-playlist $1 | jq -r '.url' | parallel -j $2 --progress youtube-dl -i -- else youtube-dl -j --flat-playlist $1 | jq -r '.url' | parallel -j $default_threads --progress youtube-dl -i -- fi } ```

deekerman commented

Author

Owner

@rautamiekka commented on GitHub (Sep 4, 2018):

@oltodosel that's a good snippet, but there are 2 things I notice right away:

You're limiting the max number of channels/playlists to 1 through the order of the parameters.
You're referencing parameter 1 without quoting it.

@rautamiekka commented on GitHub (Sep 4, 2018): @oltodosel that's a good snippet, but there are 2 things I notice right away: 1) You're limiting the max number of channels/playlists to 1 through the order of the parameters. 2) You're referencing parameter 1 without quoting it.

deekerman commented

Author

Owner

@oltodosel commented on GitHub (Sep 4, 2018):

@rautamiekka

I don't need it.
You're right, guess I got too reliant on oh-my-zsh's automatic url quotation.

@oltodosel commented on GitHub (Sep 4, 2018): @rautamiekka 1. I don't need it. 2. You're right, guess I got too reliant on oh-my-zsh's automatic url quotation.

deekerman commented

Author

Owner

@MPeti1 commented on GitHub (Dec 5, 2019):

I tried many options and here is the best that I came up with - use aria2.

The --external-downloader option
$ youtube-dl --external-downloader aria2c --external-downloader-args '-x <number of threads>' https://www.youtube.com/playlist?list=PLsPUh22kYmNBkabv9M4fXo6HMLKnc7iR6
But this will download only one file at a time. To download many videos simultaneously, you will have to prepopulate download links.

Prepopulate download links
$ youtube-dl -g https://www.youtube.com/playlist?list=PLsPUh22kYmNBkabv9M4fXo6HMLKnc7iR6 > dlinks.txt
$ aria2c -i dlinks.txt -j <number of simultaneous downloads> -x <number of threads for each download>
You will have to use >> instead of > in the first command to append more links.

The two commands may also be combined -
$ aria2c -j <number of simultaneous downloads> -x <number of threads for each download> `youtube-dl -g https://www.youtube.com/playlist?list=PLsPUh22kYmNBkabv9M4fXo6HMLKnc7iR6`
But this will result in no backup of your links in case download interrupts.

Note: In case of prepopulating links, you may end up in a state when your links get expired because your previous downloads would have taken a while.

There are playlists for which the -g option returns links that start with manifest.googlevideo.com, and if such a link is opened in a browser it will download a few hundred KB file. Is there a solution for cases like that?
Note: your playlist still returns the old links which aria2c can download

@MPeti1 commented on GitHub (Dec 5, 2019): > > > I tried many options and here is the best that I came up with - use [aria2](https://github.com/aria2/aria2). > ### The `--external-downloader` option > > ```shell > $ youtube-dl --external-downloader aria2c --external-downloader-args '-x <number of threads>' https://www.youtube.com/playlist?list=PLsPUh22kYmNBkabv9M4fXo6HMLKnc7iR6 > ``` > > But this will download only one file at a time. To download many videos simultaneously, you will have to prepopulate download links. > ### Prepopulate download links > > ```shell > $ youtube-dl -g https://www.youtube.com/playlist?list=PLsPUh22kYmNBkabv9M4fXo6HMLKnc7iR6 > dlinks.txt > $ aria2c -i dlinks.txt -j <number of simultaneous downloads> -x <number of threads for each download> > ``` > > You will have to use `>>` instead of `>` in the first command to append more links. > > The two commands may also be combined - > > ```shell > $ aria2c -j <number of simultaneous downloads> -x <number of threads for each download> `youtube-dl -g https://www.youtube.com/playlist?list=PLsPUh22kYmNBkabv9M4fXo6HMLKnc7iR6` > ``` > > But this will result in no backup of your links in case download interrupts. > > **Note**: In case of prepopulating links, you may end up in a state when your links get expired because your previous downloads would have taken a while. There are playlists for which the -g option returns links that start with manifest.googlevideo.com, and if such a link is opened in a browser it will download a few hundred KB file. Is there a solution for cases like that? Note: your playlist still returns the old links which aria2c can download

deekerman commented

Author

Owner

@Pramodh-G commented on GitHub (Aug 8, 2021):

Hi, just wanted to say that you can also use GNU parallel to download videos in parallel

Thanks!

@Pramodh-G commented on GitHub (Aug 8, 2021): Hi, just wanted to say that you can also use GNU parallel to download videos in parallel Thanks!

deekerman commented

Author

Owner

@aamkye commented on GitHub (Dec 30, 2021):

My bash function for parallel: https://github.com/amadeuszkryze/yt-functions

@aamkye commented on GitHub (Dec 30, 2021): My bash function for parallel: https://github.com/amadeuszkryze/yt-functions

deekerman commented

Author

Owner

@Vagelis-Prokopiou commented on GitHub (May 16, 2023):

I created a multithreaded Rust command line wrapper: https://github.com/Vagelis-Prokopiou/youtube-dl-rs

Currently it serves my own needs, but if it helps anyone else, be my guest and use it.

@Vagelis-Prokopiou commented on GitHub (May 16, 2023): I created a multithreaded Rust command line wrapper: https://github.com/Vagelis-Prokopiou/youtube-dl-rs Currently it serves my own needs, but if it helps anyone else, be my guest and use it.

deekerman commented

Author

Owner

@dirkf commented on GitHub (May 16, 2023):

The comments from 2012 regarding download speed that began this issue are out of date. yt-dl's YouTube download speeds have been fixed, broken and fixed again many times since then.

@dirkf commented on GitHub (May 16, 2023): The comments from 2012 regarding download speed that began this issue are out of date. yt-dl's YouTube download speeds have been fixed, broken and fixed again many times since then.

deekerman commented