Twitch VOD download is not as fast as it could be #3920

Author

Owner

@dstftw commented on GitHub (Jan 25, 2015):

@jaimeMF, VODs are delivered differently rather than simple videos, namely it's HLS with m3u8 containing lots of small chunks.

@dstftw commented on GitHub (Jan 25, 2015): @jaimeMF, VODs are delivered differently rather than simple videos, namely it's HLS with m3u8 containing lots of small chunks.

deekerman commented

Author

Owner

@jaimeMF commented on GitHub (Jan 25, 2015):

Sorry, I thought it was about about past broadcastasts. The method I posted doesn't apply then.

@jaimeMF commented on GitHub (Jan 25, 2015): Sorry, I thought it was about about past broadcastasts. The method I posted doesn't apply then.

deekerman commented

Author

Owner

@phihag commented on GitHub (Jan 25, 2015):

We should reuse the same TCP connection; that would probably give a great speedup.

@phihag commented on GitHub (Jan 25, 2015): We should reuse the same TCP connection; that would probably give a great speedup.

deekerman commented

Author

Owner

@Diazonium commented on GitHub (Jan 28, 2015):

Verbose output

youtube-dl --verbose http://www.twitch.tv/ara_gaming/v/3711528
[debug] System config: []
[debug] User config: []
[debug] Command-line args: ['--verbose', 'http://www.twitch.tv/ara_gaming/v/3711528']
[debug] Encodings: locale cp1250, fs mbcs, out cp852, pref cp1250
[debug] youtube-dl version 2015.01.25
[debug] Python version 2.7.8 - Windows-7-6.1.7601-SP1
[debug] exe versions: ffmpeg N-65738-g3c197442
[debug] Proxy map: {}
[twitch:vod] 3711528: Downloading vod info JSON
[twitch:vod] 3711528: Downloading vod access token
[twitch:vod] 3711528: Downloading m3u8 information
[debug] Invoking downloader on u'http://vod.ak.hls.ttvnw.net/v1/AUTH_system/vods_58ae/ara_
gaming_12760920816_193300649/chunked/index-dvr.m3u8'
[download] Destination: Hair of the dog is for crazy people. Tea please.-v3711528.mp4
ffmpeg version N-65738-g3c197442 Copyright (c) 2000-2014 the FFmpeg developers
built on Aug 18 2014 22:06:41 with gcc 4.8.3 (GCC)
configuration: --disable-static --enable-shared --enable-gpl --enable-version3 --disable
-w32threads --enable-avisynth --enable-bzlib --enable-fontconfig --enable-frei0r --enable-
gnutls --enable-iconv --enable-libass --enable-libbluray --enable-libbs2b --enable-libcaca
--enable-libfreetype --enable-libgme --enable-libgsm --enable-libilbc --enable-libmodplug
--enable-libmp3lame --enable-libopencore-amrnb --enable-libopencore-amrwb --enable-libope
njpeg --enable-libopus --enable-librtmp --enable-libschroedinger --enable-libsoxr --enable
-libspeex --enable-libtheora --enable-libtwolame --enable-libvidstab --enable-libvo-aacenc
--enable-libvo-amrwbenc --enable-libvorbis --enable-libvpx --enable-libwavpack --enable-l
ibwebp --enable-libx264 --enable-libx265 --enable-libxavs --enable-libxvid --enable-deckli
nk --enable-zlib
libavutil 54. 5.100 / 54. 5.100
libavcodec 56. 0.101 / 56. 0.101
libavformat 56. 1.100 / 56. 1.100
libavdevice 56. 0.100 / 56. 0.100
libavfilter 5. 0.100 / 5. 0.100
libswscale 3. 0.100 / 3. 0.100
libswresample 1. 1.100 / 1. 1.100
libpostproc 53. 0.100 / 53. 0.100
Input #0, hls,applehttp, from 'http://vod.ak.hls.ttvnw.net/v1/AUTH_system/vods_58ae/ara_ga
ming_12760920816_193300649/chunked/index-dvr.m3u8':
Duration: 02:58:33.71, start: 64.118000, bitrate: 0 kb/s
Program 0
Metadata:
variant_bitrate : 0
Stream #0:0: Audio: aac ([15][0][0][0] / 0x000F), 48000 Hz, stereo, fltp, 107 kb/s
Stream #0:1: Video: h264 (Main) ([27][0][0][0] / 0x001B), yuv420p(tv, bt709), 1920x108
0, 30 fps, 30 tbr, 90k tbn, 60 tbc
Stream #0:2: Data: timed_id3 (ID3 / 0x20334449)
Output #0, mp4, to 'Hair of the dog is for crazy people. Tea please.-v3711528.mp4.part':
Metadata:
encoder : Lavf56.1.100
Stream #0:0: Video: h264 ([33][0][0][0] / 0x0021), yuv420p, 1920x1080, q=2-31, 30 fps,
90k tbn, 90k tbc
Stream #0:1: Audio: aac ([64][0][0][0] / 0x0040), 48000 Hz, stereo, 107 kb/s
Stream mapping:
Stream #0:1 -> #0:0 (copy)
Stream #0:0 -> #0:1 (copy)
Press [q] to stop, [?] for help
frame= 3308 fps= 81 q=-1.0 Lsize= 42271kB time=00:01:50.74 bitrate=3126.9kbits/s
video:40394kB audio:1732kB subtitle:0kB other streams:0kB global headers:0kB muxing overhe
ad: 0.342595%
[ffmpeg] 43284996 bytes

@Diazonium commented on GitHub (Jan 28, 2015): Verbose output youtube-dl --verbose http://www.twitch.tv/ara_gaming/v/3711528 [debug] System config: [] [debug] User config: [] [debug] Command-line args: ['--verbose', 'http://www.twitch.tv/ara_gaming/v/3711528'] [debug] Encodings: locale cp1250, fs mbcs, out cp852, pref cp1250 [debug] youtube-dl version 2015.01.25 [debug] Python version 2.7.8 - Windows-7-6.1.7601-SP1 [debug] exe versions: ffmpeg N-65738-g3c197442 [debug] Proxy map: {} [twitch:vod] 3711528: Downloading vod info JSON [twitch:vod] 3711528: Downloading vod access token [twitch:vod] 3711528: Downloading m3u8 information [debug] Invoking downloader on u'http://vod.ak.hls.ttvnw.net/v1/AUTH_system/vods_58ae/ara_ gaming_12760920816_193300649/chunked/index-dvr.m3u8' [download] Destination: Hair of the dog is for crazy people. Tea please.-v3711528.mp4 ffmpeg version N-65738-g3c197442 Copyright (c) 2000-2014 the FFmpeg developers built on Aug 18 2014 22:06:41 with gcc 4.8.3 (GCC) configuration: --disable-static --enable-shared --enable-gpl --enable-version3 --disable -w32threads --enable-avisynth --enable-bzlib --enable-fontconfig --enable-frei0r --enable- gnutls --enable-iconv --enable-libass --enable-libbluray --enable-libbs2b --enable-libcaca --enable-libfreetype --enable-libgme --enable-libgsm --enable-libilbc --enable-libmodplug --enable-libmp3lame --enable-libopencore-amrnb --enable-libopencore-amrwb --enable-libope njpeg --enable-libopus --enable-librtmp --enable-libschroedinger --enable-libsoxr --enable -libspeex --enable-libtheora --enable-libtwolame --enable-libvidstab --enable-libvo-aacenc --enable-libvo-amrwbenc --enable-libvorbis --enable-libvpx --enable-libwavpack --enable-l ibwebp --enable-libx264 --enable-libx265 --enable-libxavs --enable-libxvid --enable-deckli nk --enable-zlib libavutil 54. 5.100 / 54. 5.100 libavcodec 56. 0.101 / 56. 0.101 libavformat 56. 1.100 / 56. 1.100 libavdevice 56. 0.100 / 56. 0.100 libavfilter 5. 0.100 / 5. 0.100 libswscale 3. 0.100 / 3. 0.100 libswresample 1. 1.100 / 1. 1.100 libpostproc 53. 0.100 / 53. 0.100 Input #0, hls,applehttp, from 'http://vod.ak.hls.ttvnw.net/v1/AUTH_system/vods_58ae/ara_ga ming_12760920816_193300649/chunked/index-dvr.m3u8': Duration: 02:58:33.71, start: 64.118000, bitrate: 0 kb/s Program 0 Metadata: variant_bitrate : 0 Stream #0:0: Audio: aac ([15][0][0][0] / 0x000F), 48000 Hz, stereo, fltp, 107 kb/s Stream #0:1: Video: h264 (Main) ([27][0][0][0] / 0x001B), yuv420p(tv, bt709), 1920x108 0, 30 fps, 30 tbr, 90k tbn, 60 tbc Stream #0:2: Data: timed_id3 (ID3 / 0x20334449) Output #0, mp4, to 'Hair of the dog is for crazy people. Tea please.-v3711528.mp4.part': Metadata: encoder : Lavf56.1.100 Stream #0:0: Video: h264 ([33][0][0][0] / 0x0021), yuv420p, 1920x1080, q=2-31, 30 fps, 90k tbn, 90k tbc Stream #0:1: Audio: aac ([64][0][0][0] / 0x0040), 48000 Hz, stereo, 107 kb/s Stream mapping: Stream #0:1 -> #0:0 (copy) Stream #0:0 -> #0:1 (copy) Press [q] to stop, [?] for help frame= 3308 fps= 81 q=-1.0 Lsize= 42271kB time=00:01:50.74 bitrate=3126.9kbits/s video:40394kB audio:1732kB subtitle:0kB other streams:0kB global headers:0kB muxing overhe ad: 0.342595% [ffmpeg] 43284996 bytes

deekerman commented

Author

Owner

@csenger commented on GitHub (Feb 19, 2015):

This might be worth a separate issue additional to a general problem with slow http connections. But regarding @Diazonium's comment, there was a change in behavior on the twitch side.

Some Links to videos have the format http://www.twitch.tv/<username>/b/<videoid>, e.g. http://www.twitch.tv/riotgames/b/579679336 . youtube-dl downloads a bunch of flv files pretty fast.

csenger@sympathikus:~/Videos$ youtube-dl http://www.twitch.tv/riotgames/b/579679336
[twitch:video] 579679336: Downloading video info JSON
[twitch:video] 579679336: Downloading video playlist JSON
[download] Downloading playlist: 2014 Worlds Final - Samsung White vs. Star Horn Royal Club - 10/18/14
[twitch:video] playlist 2014 Worlds Final - Samsung White vs. Star Horn Royal Club - 10/18/14: Collected 13 video ids (downloading 13 of them)
[download] Downloading video 1 of 13
...

For other twitch users the URL format is different: http://www.twitch.tv/<username>/v/<videoid>, e.g. http://www.twitch.tv/planetside2/v/3810046 .
In this case youtube-dl tries to pipe the download through avconv. Maybe to combine the best audio with the best video, but I'm not sure. This produces the verbose output in the last comment and is pretty slow. Also aborted downloads can not be resumed.

@csenger commented on GitHub (Feb 19, 2015): This might be worth a separate issue additional to a general problem with slow http connections. But regarding @Diazonium's comment, there was a change in behavior on the twitch side. Some Links to videos have the format `http://www.twitch.tv/<username>/b/<videoid>`, e.g. http://www.twitch.tv/riotgames/b/579679336 . youtube-dl downloads a bunch of flv files pretty fast. ``` csenger@sympathikus:~/Videos$ youtube-dl http://www.twitch.tv/riotgames/b/579679336 [twitch:video] 579679336: Downloading video info JSON [twitch:video] 579679336: Downloading video playlist JSON [download] Downloading playlist: 2014 Worlds Final - Samsung White vs. Star Horn Royal Club - 10/18/14 [twitch:video] playlist 2014 Worlds Final - Samsung White vs. Star Horn Royal Club - 10/18/14: Collected 13 video ids (downloading 13 of them) [download] Downloading video 1 of 13 ... ``` For other twitch users the URL format is different: `http://www.twitch.tv/<username>/v/<videoid>`, e.g. http://www.twitch.tv/planetside2/v/3810046 . In this case youtube-dl tries to pipe the download through avconv. Maybe to combine the best audio with the best video, but I'm not sure. This produces the verbose output in the last comment and is pretty slow. Also aborted downloads can not be resumed.

deekerman commented

Author

Owner

@weedy commented on GitHub (Jul 18, 2016):

[download] Downloading video 3 of 116
[generic] 67998156: Requesting header
[redirect] Following redirect to https://www.twitch.tv/moonmoon_ow/v/67998156
[twitch:vod] 67998156: Downloading vod info JSON
[twitch:vod] 67998156: Downloading vod access token
[twitch:vod] 67998156: Downloading m3u8 information
[download] Sleeping 3.0 seconds...
[hlsnative] Downloading m3u8 manifest
[hlsnative] Total fragments: 4515
[download] Destination: 20160522-v67998156-video_games.mp4
[download]  10.7% of ~3.70GiB at  523.27KiB/s ETA 01:21:38

If we know we are going to get a million chunks can we call --external-downloader with a list/file of all the chunks and use --external-downloader-args to parallel from there?

@weedy commented on GitHub (Jul 18, 2016): ``` [download] Downloading video 3 of 116 [generic] 67998156: Requesting header [redirect] Following redirect to https://www.twitch.tv/moonmoon_ow/v/67998156 [twitch:vod] 67998156: Downloading vod info JSON [twitch:vod] 67998156: Downloading vod access token [twitch:vod] 67998156: Downloading m3u8 information [download] Sleeping 3.0 seconds... [hlsnative] Downloading m3u8 manifest [hlsnative] Total fragments: 4515 [download] Destination: 20160522-v67998156-video_games.mp4 [download] 10.7% of ~3.70GiB at 523.27KiB/s ETA 01:21:38 ``` If we know we are going to get a million chunks can we call --external-downloader with a list/file of all the chunks and use --external-downloader-args to parallel from there?

deekerman commented

Author

Owner

@yan12125 commented on GitHub (Jul 18, 2016):

If we know we are going to get a million chunks can we call --external-downloader with a list/file of all the chunks and use --external-downloader-args to parallel from there?

Such a feature is not implemented yet. See #6069

@yan12125 commented on GitHub (Jul 18, 2016): > If we know we are going to get a million chunks can we call --external-downloader with a list/file of all the chunks and use --external-downloader-args to parallel from there? Such a feature is not implemented yet. See #6069

deekerman commented

Author

Owner

@lilydjwg commented on GitHub (Oct 17, 2017):

I don't need to parallel those files. I prefer connection reuse which can normally speed up massive web page downloading by over 2x. For video chunks I haven't tested, but I can only run at about 200-500KiB/s, while downloading one large file gives me over 5MiB/s, from YouTube.

@lilydjwg commented on GitHub (Oct 17, 2017): I don't need to parallel those files. I prefer connection reuse which can normally speed up massive web page downloading by over 2x. For video chunks I haven't tested, but I can only run at about 200-500KiB/s, while downloading one large file gives me over 5MiB/s, from YouTube.

deekerman commented

Author

Owner

@yan12125 commented on GitHub (Oct 17, 2017):

@lilydjwg: Could you open a new issue for connection reuse? That's a different problem than the original one.

@yan12125 commented on GitHub (Oct 17, 2017): @lilydjwg: Could you open a new issue for connection reuse? That's a different problem than the original one.

deekerman commented

Author

Owner

@lilydjwg commented on GitHub (Oct 17, 2017):

@yan12125 done at #14523.

@lilydjwg commented on GitHub (Oct 17, 2017): @yan12125 done at #14523.

deekerman commented

Author

Owner

@rosshadden commented on GitHub (Oct 20, 2017):

I know based on the above comments this isn't exactly what the OP was asking about, but I just wanted to say thank you to @jaimeMF for suggesting aria2c. It made my past broadcasts download at roughly 10-15x the speed as without.

@rosshadden commented on GitHub (Oct 20, 2017): I know based on the above comments this isn't exactly what the OP was asking about, but I just wanted to say thank you to @jaimeMF for suggesting `aria2c`. It made my past broadcasts download at roughly 10-15x the speed as without.

deekerman commented

Author

Owner

@Gaploid commented on GitHub (Oct 21, 2017):

@rosshadden How are you using aria2c? As I understand it does not support as an external downloader for HLS videos.

@Gaploid commented on GitHub (Oct 21, 2017): @rosshadden How are you using aria2c? As I understand it does not support as an external downloader for HLS videos.

deekerman commented

Author

Owner

@rosshadden commented on GitHub (Oct 21, 2017):

@Gaploid I passed --external-downloader aria2c. It worked great, and I have already watched and edited the resulting videos so I know it worked.

@rosshadden commented on GitHub (Oct 21, 2017): @Gaploid I passed `--external-downloader aria2c`. It worked great, and I have already watched and edited the resulting videos so I know it worked.

deekerman commented

Author

Owner

@Gaploid commented on GitHub (Oct 21, 2017):

@rosshadden Ive tested a lot and it does not work for me. Is it [hlsnative] in logs when you are downloading?

@Gaploid commented on GitHub (Oct 21, 2017): @rosshadden Ive tested a lot and it does not work for me. Is it [hlsnative] in logs when you are downloading?

deekerman commented

Author

Owner

@rosshadden commented on GitHub (Oct 21, 2017):

I was talking about past broadcasts, not explicitly saved VODs. I believe the comments here went in the direction of VODs, which is what I meant when I said "this isn't exactly what the OP was asking about". Sorry for getting off topic; just wanted to say thanks. If we are actually on the same page and past broadcasts shouldn't be working or benefiting from aria2c, let me know and I'll do it again and look at logs.

@rosshadden commented on GitHub (Oct 21, 2017): I was talking about past broadcasts, not explicitly saved VODs. I believe the comments here went in the direction of VODs, which is what I meant when I said "this isn't exactly what the OP was asking about". Sorry for getting off topic; just wanted to say thanks. If we are actually on the same page and past broadcasts shouldn't be working or benefiting from aria2c, let me know and I'll do it again and look at logs.

deekerman commented

Author

Owner

@Gaploid commented on GitHub (Oct 23, 2017):

Yep, I`m talking about past broadcasts. Maybe the low quality is saved in one file but 720p, 1080p is stored in hls format and youtube-dl does not support external downloaders.

@Gaploid commented on GitHub (Oct 23, 2017): Yep, I`m talking about past broadcasts. Maybe the low quality is saved in one file but 720p, 1080p is stored in hls format and youtube-dl does not support external downloaders.

deekerman commented