starred/youtube-dl-ytdl-org

Fork 0

mirror of https://github.com/ytdl-org/youtube-dl.git synced 2026-03-02 19:17:00 -05:00

Multi-Threaded postproc for playlists #10014

New issue

Open

opened 2026-02-21 05:20:16 -05:00 by deekerman · 5 comments

deekerman commented

2026-02-21 05:20:16 -05:00

Owner

Originally created by @d3fault on GitHub (Feb 22, 2017).

Would be nice if postproc was done in separate thread(s) (up to N threads, N being the number of CPU cores). As soon as a download is finished, the postproc job should be dispatched and the next download should start immediately. But if all N cores are doing postproc, wait for one to finish before downloading another playlist item. If the machine has only 1 core then the the behavior is exactly the same as it currently is.

Originally created by @d3fault on GitHub (Feb 22, 2017). Would be nice if postproc was done in separate thread(s) (up to N threads, N being the number of CPU cores). As soon as a download is finished, the postproc job should be dispatched and the next download should start immediately. But if all N cores are doing postproc, wait for one to finish before downloading another playlist item. If the machine has only 1 core then the the behavior is exactly the same as it currently is.

deekerman added the

request

postprocessors

labels

2026-02-21 05:20:16 -05:00

deekerman commented

2026-02-21 05:20:20 -05:00

Author

Owner

@yan12125 commented on GitHub (Feb 22, 2017):

FYI: #12077 is useful in some cases.

@yan12125 commented on GitHub (Feb 22, 2017): FYI: #12077 is useful in some cases.

deekerman commented

2026-02-21 05:20:20 -05:00

Author

Owner

@d3fault commented on GitHub (Feb 22, 2017):

Indeed, thanks. N ffmpeg processes working on their own files almost always parallelizes better than 1x ffmpeg process passed -threads N, because not all routines in ffmpeg can-be/are parallelized: it depends on the underlying codec to support multi-threading. but you already knew this ;-P

@d3fault commented on GitHub (Feb 22, 2017): Indeed, thanks. N ffmpeg processes working on their own files almost always parallelizes better than 1x ffmpeg process passed -threads N, because not all routines in ffmpeg can-be/are parallelized: it depends on the underlying codec to support multi-threading. but you already knew this ;-P

deekerman commented

2026-02-21 05:20:22 -05:00

Author

Owner

@cometjc commented on GitHub (May 24, 2017):

Using parallel to utilize CPU cores, run N postproc at same time.
parallel's default is 100% which will run one job per CPU core on each machine.

e.g.

youtube-dl --get-id PLAYLIST_ID -i | tee ids
parallel --linebuffer -- youtube-dl -x -k --embed-thumbnail --add-metadata --audio-format mp3 <ids

@cometjc commented on GitHub (May 24, 2017): Using parallel to utilize CPU cores, run N postproc at same time. parallel's default is 100% which will run one job per CPU core on each machine. e.g. ``` youtube-dl --get-id PLAYLIST_ID -i | tee ids parallel --linebuffer -- youtube-dl -x -k --embed-thumbnail --add-metadata --audio-format mp3 <ids ```

deekerman commented

2026-02-21 05:20:22 -05:00

Author

Owner

@d3fault commented on GitHub (May 24, 2017):

Nice thanks Jethro that looks like itll do exactly what I want (except
that the downloads are parallel too, slightly increasing my chance of
getting banned lol).

Maybe there could be a flag "--external-jobrunner parallel" or some
such that simply wraps that command (and gracefully defaults back to
internal/regular/single-threaded if the external jobrunner dne).
sounds ez enough, but Im terrible at python :(

On 5/23/17, Jethro Yu notifications@github.com wrote:

Using parallel to utilize CPU cores, run N postproc in same time.
parallel's default is 100% which will run one job per CPU core on each
machine.

e.g.
youtube-dl --get-id PLAYLIST_ID -i | tee ids
parallel --linebuffer -- youtube-dl -x -k --embed-thumbnail --add-metadata
--audio-format mp3 <ids
--
You are receiving this because you authored the thread.
Reply to this email directly or view it on GitHub:
https://github.com/rg3/youtube-dl/issues/12215#issuecomment-303622004

@d3fault commented on GitHub (May 24, 2017): Nice thanks Jethro that looks like itll do exactly what I want (except that the downloads are parallel too, slightly increasing my chance of getting banned lol). Maybe there could be a flag "--external-jobrunner parallel" or some such that simply wraps that command (and gracefully defaults back to internal/regular/single-threaded if the external jobrunner dne). sounds ez enough, but Im terrible at python :( On 5/23/17, Jethro Yu <notifications@github.com> wrote: > Using parallel to utilize CPU cores, run N postproc in same time. > parallel's default is 100% which will run one job per CPU core on each > machine. > > e.g. > ``` > youtube-dl --get-id PLAYLIST_ID -i | tee ids > parallel --linebuffer -- youtube-dl -x -k --embed-thumbnail --add-metadata > --audio-format mp3 <ids > ``` > > -- > You are receiving this because you authored the thread. > Reply to this email directly or view it on GitHub: > https://github.com/rg3/youtube-dl/issues/12215#issuecomment-303622004

deekerman commented

2026-02-21 05:20:22 -05:00

Author

Owner

@yan12125 commented on GitHub (Nov 8, 2017):

As a side note: #3746 is related

@yan12125 commented on GitHub (Nov 8, 2017): As a side note: #3746 is related

deekerman referenced this issue

2026-02-21 06:42:21 -05:00

generic extractor cognito idp - import library #12591

deekerman referenced this issue

2026-02-21 11:33:34 -05:00