starred/youtube-dl

Fork 0

mirror of https://github.com/ytdl-org/youtube-dl.git synced 2026-03-02 23:07:04 -05:00

Memory leak program on big Youtube playlist #11401

New issue

Closed

opened 2026-02-21 04:39:12 -05:00 by deekerman · 4 comments

deekerman commented

2026-02-21 04:39:12 -05:00

Owner

Originally created by @damamasss on GitHub (Aug 2, 2017).

When I download big playlist (more than 500 videos) from Youtube with --datebefore (may be and without datebefore), on Debian 8 machine, youtube-dl have leak memory - full 4G RAM is occupied by the youtube-dl. And PC is frozen afret ~450 skipped videos

/usr/local/bin/youtube-dl -v -ci -x --audio-format mp3 --audio-quality 128K --datebefore 20170329 -o '%(upload_date)s-%(uploader)s-%(title)s-%(id)s.%(ext)s' https://www.youtube.com/channel/UCTwUWzrI3iAxQwzbj5yJsvw/videos
[debug] System config: []
[debug] User config: []
[debug] Custom config: []
[debug] Command-line args: [u'-v', u'-ci', u'-x', u'--audio-format', u'mp3', u'--audio-quality', u'128K', u'--datebefore', u'20170329', u'-o', u'%(upload_date)s-%(uploader)s-%(title)s-%(id)s.%(ext)s', u'https://www.youtube.com/channel/UCTwUWzrI3iAxQwzbj5yJsvw/videos']
[debug] Encodings: locale UTF-8, fs UTF-8, out UTF-8, pref UTF-8
[debug] youtube-dl version 2017.07.30.1
[debug] Python version 2.7.9 - Linux-3.16.0-4-amd64-x86_64-with-debian-8.9
[debug] exe versions: ffmpeg 2.6.9, ffprobe 2.6.9
[debug] Proxy map: {}
[youtube:channel] UCTwUWzrI3iAxQwzbj5yJsvw: Downloading channel page
[youtube:playlist] UUTwUWzrI3iAxQwzbj5yJsvw: Downloading webpage
[download] Downloading playlist: Uploads from БМ Знания
[youtube:playlist] UUTwUWzrI3iAxQwzbj5yJsvw: Downloading page #1
[youtube:playlist] UUTwUWzrI3iAxQwzbj5yJsvw: Downloading page #2
[youtube:playlist] UUTwUWzrI3iAxQwzbj5yJsvw: Downloading page #3
[youtube:playlist] UUTwUWzrI3iAxQwzbj5yJsvw: Downloading page #4
[youtube:playlist] UUTwUWzrI3iAxQwzbj5yJsvw: Downloading page #5
[youtube:playlist] playlist Uploads from БМ Знания: Downloading 545 videos
[download] Downloading video 1 of 545
[youtube] F0Ldm10xbn4: Downloading webpage
[youtube] F0Ldm10xbn4: Downloading video info webpage
[youtube] F0Ldm10xbn4: Extracting video information
[youtube] F0Ldm10xbn4: Downloading MPD manifest
[youtube] F0Ldm10xbn4: Downloading MPD manifest
[download] 2017-08-01 upload date is not in range 0001-01-01 - 2017-03-29
[download] Downloading video 2 of 545
[youtube] x1ryKOZU0DY: Downloading webpage
[youtube] x1ryKOZU0DY: Downloading video info webpage
[youtube] x1ryKOZU0DY: Extracting video information
[youtube] x1ryKOZU0DY: Downloading MPD manifest
[youtube] x1ryKOZU0DY: Downloading MPD manifest
[download] 2017-08-01 upload date is not in range 0001-01-01 - 2017-03-29
[download] Downloading video 3 of 545
[youtube] AWXrhlPMvFM: Downloading webpage
^C
ERROR: Interrupted by user

Originally created by @damamasss on GitHub (Aug 2, 2017). When I download big playlist (more than 500 videos) from Youtube with --datebefore (may be and without datebefore), on Debian 8 machine, youtube-dl have leak memory - full 4G RAM is occupied by the youtube-dl. And PC is frozen afret ~450 skipped videos /usr/local/bin/youtube-dl -v -ci -x --audio-format mp3 --audio-quality 128K --datebefore 20170329 -o '%(upload_date)s-%(uploader)s-%(title)s-%(id)s.%(ext)s' https://www.youtube.com/channel/UCTwUWzrI3iAxQwzbj5yJsvw/videos [debug] System config: [] [debug] User config: [] [debug] Custom config: [] [debug] Command-line args: [u'-v', u'-ci', u'-x', u'--audio-format', u'mp3', u'--audio-quality', u'128K', u'--datebefore', u'20170329', u'-o', u'%(upload_date)s-%(uploader)s-%(title)s-%(id)s.%(ext)s', u'https://www.youtube.com/channel/UCTwUWzrI3iAxQwzbj5yJsvw/videos'] [debug] Encodings: locale UTF-8, fs UTF-8, out UTF-8, pref UTF-8 [debug] youtube-dl version 2017.07.30.1 [debug] Python version 2.7.9 - Linux-3.16.0-4-amd64-x86_64-with-debian-8.9 [debug] exe versions: ffmpeg 2.6.9, ffprobe 2.6.9 [debug] Proxy map: {} [youtube:channel] UCTwUWzrI3iAxQwzbj5yJsvw: Downloading channel page [youtube:playlist] UUTwUWzrI3iAxQwzbj5yJsvw: Downloading webpage [download] Downloading playlist: Uploads from БМ Знания [youtube:playlist] UUTwUWzrI3iAxQwzbj5yJsvw: Downloading page #1 [youtube:playlist] UUTwUWzrI3iAxQwzbj5yJsvw: Downloading page #2 [youtube:playlist] UUTwUWzrI3iAxQwzbj5yJsvw: Downloading page #3 [youtube:playlist] UUTwUWzrI3iAxQwzbj5yJsvw: Downloading page #4 [youtube:playlist] UUTwUWzrI3iAxQwzbj5yJsvw: Downloading page #5 [youtube:playlist] playlist Uploads from БМ Знания: Downloading 545 videos [download] Downloading video 1 of 545 [youtube] F0Ldm10xbn4: Downloading webpage [youtube] F0Ldm10xbn4: Downloading video info webpage [youtube] F0Ldm10xbn4: Extracting video information [youtube] F0Ldm10xbn4: Downloading MPD manifest [youtube] F0Ldm10xbn4: Downloading MPD manifest [download] 2017-08-01 upload date is not in range 0001-01-01 - 2017-03-29 [download] Downloading video 2 of 545 [youtube] x1ryKOZU0DY: Downloading webpage [youtube] x1ryKOZU0DY: Downloading video info webpage [youtube] x1ryKOZU0DY: Extracting video information [youtube] x1ryKOZU0DY: Downloading MPD manifest [youtube] x1ryKOZU0DY: Downloading MPD manifest [download] 2017-08-01 upload date is not in range 0001-01-01 - 2017-03-29 [download] Downloading video 3 of 545 [youtube] AWXrhlPMvFM: Downloading webpage ^C ERROR: Interrupted by user

deekerman

2026-02-21 04:39:12 -05:00

closed this issue
added the
not-a-bug
label

deekerman commented

2026-02-21 04:39:16 -05:00

Author

Owner

@damamasss commented on GitHub (Aug 3, 2017):

With download we also see same

...
[download] Downloading video 369 of 550
[youtube] fl5C8LIR1Zk: Downloading webpage
[youtube] fl5C8LIR1Zk: Downloading video info webpage
[youtube] fl5C8LIR1Zk: Extracting video information
[youtube] fl5C8LIR1Zk: Downloading MPD manifest
[youtube] fl5C8LIR1Zk: Downloading MPD manifest
[dashsegments] Total fragments: 393
[download] Destination: 20170424-БМ Знания-Строим Самолёт. Учеба в Бизнес Молодости. Работа в команде. «Любой может стать любым» [БМ]-fl5C8LIR1Zk.m4a
[download] 100% of 29.67MiB in 00:37
WARNING: fl5C8LIR1Zk: writing DASH m4a. Only some players support this container. Install ffmpeg or avconv to fix this automatically.
ERROR: WARNING: unable to obtain file audio codec with ffprobe
...

@damamasss commented on GitHub (Aug 3, 2017): With download we also see same ... [download] Downloading video 369 of 550 [youtube] fl5C8LIR1Zk: Downloading webpage [youtube] fl5C8LIR1Zk: Downloading video info webpage [youtube] fl5C8LIR1Zk: Extracting video information [youtube] fl5C8LIR1Zk: Downloading MPD manifest [youtube] fl5C8LIR1Zk: Downloading MPD manifest [dashsegments] Total fragments: 393 [download] Destination: 20170424-БМ Знания-Строим Самолёт. Учеба в Бизнес Молодости. Работа в команде. «Любой может стать любым» [БМ]-fl5C8LIR1Zk.m4a [download] 100% of 29.67MiB in 00:37 WARNING: fl5C8LIR1Zk: writing DASH m4a. Only some players support this container. Install ffmpeg or avconv to fix this automatically. ERROR: WARNING: unable to obtain file audio codec with ffprobe ...

deekerman commented

2026-02-21 04:39:17 -05:00

Author

Owner

@damamasss commented on GitHub (Aug 3, 2017):

But my ffmpeg can convert m4a

@damamasss commented on GitHub (Aug 3, 2017): But my ffmpeg can convert m4a

deekerman commented

2026-02-21 04:39:17 -05:00

Author

Owner

@dstftw commented on GitHub (Aug 4, 2017):

There is no memory leak. High memory consumption is due to lots of DASH segments formats that carry fragments metafield. fragments metafield may be quite heavy (ranging from 0.5 to 15 MiB per video, see --write-info-json) since it contains a list of all fragments of a DASH video. Due to a high number of such videos collected metadata grows.
You can use --playlist-start and --playlist-end to process playlist in parts.

@dstftw commented on GitHub (Aug 4, 2017): There is no memory leak. High memory consumption is due to lots of DASH segments formats that carry [`fragments` metafield](https://github.com/rg3/youtube-dl/blob/master/youtube_dl/extractor/common.py#L134-L144). `fragments` metafield may be quite heavy (ranging from 0.5 to 15 MiB per video, see `--write-info-json`) since it contains a list of all fragments of a DASH video. Due to a high number of such videos collected metadata grows. You can use `--playlist-start` and `--playlist-end` to process playlist in parts.

deekerman commented

2026-02-21 04:39:17 -05:00

Author

Owner

@ivan commented on GitHub (Jun 19, 2019):

github.com/ytdl-org/youtube-dl@7c24a58bdb/youtube_dl/YoutubeDL.py (L1007) is responsible for sometimes retaining > 2.5GB of objects when downloading very long playlists

@ivan commented on GitHub (Jun 19, 2019): https://github.com/ytdl-org/youtube-dl/blob/7c24a58bdb60af80137beac85c8804c70194a455/youtube_dl/YoutubeDL.py#L1007 is responsible for sometimes retaining > 2.5GB of objects when downloading very long playlists

No milestone

No project

No assignees

1 participant

Notifications

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference

starred/youtube-dl#11401

No description provided.

Rows
Columns