Auto Batch Mode?

deekerman commented

2026-02-20 23:03:11 -05:00

Owner

Originally created by @w524e on GitHub (Nov 27, 2012).

Remove the link from a text file assigned by --batch-file (or move to a backup file) when a file completed, and automatically resume the list for incomplete download when the last line of the text is done (instead of quitting, process the batch file again--sorry for my poor english), until the list is empty or no more .part file in the current directory. It would be useful for interrupted batch download due to wifi problem or other reasons. Now i'm using sed with some Windows batch code to do this, but it's a bit troublesome. it would be nice if youtube-dl can do this. thanks

Originally created by @w524e on GitHub (Nov 27, 2012). Remove the link from a text file assigned by `--batch-file` (or move to a backup file) when a file completed, and automatically resume the list for incomplete download when the last line of the text is done (instead of quitting, process the batch file again--sorry for my poor english), until the list is empty or no more .part file in the current directory. It would be useful for interrupted batch download due to wifi problem or other reasons. Now i'm using sed with some Windows batch code to do this, but it's a bit troublesome. it would be nice if youtube-dl can do this. thanks

deekerman

2026-02-20 23:03:11 -05:00

closed this issue
added the
request

outdated-version
labels

deekerman commented

2026-02-20 23:03:15 -05:00

Author

Owner

@phihag commented on GitHub (Nov 27, 2012):

Sounds like a great idea.

@phihag commented on GitHub (Nov 27, 2012): Sounds like a great idea.

deekerman commented

2026-02-20 23:03:15 -05:00

Author

Owner

@FiloSottile commented on GitHub (Nov 27, 2012):

Meanwhile, if you restart your batch, youtube-dl will simply skip the already downloaded videos at the beginning of the list.

@FiloSottile commented on GitHub (Nov 27, 2012): Meanwhile, if you restart your batch, youtube-dl will simply skip the already downloaded videos at the beginning of the list.

deekerman commented

2026-02-20 23:03:16 -05:00

Author

Owner

@w524e commented on GitHub (Nov 27, 2012):

just tried with build 2012.11.27, it will still connect to youtube, download webpage, and extract information...and get file name base on parameters, if .part with then same file name exist, resume; if file name + ext exist then skip...... correct me if my guess are wrong... but don't you thinks it's unnecessary? and if you happen to change your rename rules, it will download everything again!....

@w524e commented on GitHub (Nov 27, 2012): just tried with build 2012.11.27, it will still connect to youtube, download webpage, and extract information...and get file name base on parameters, if .part with then same file name exist, resume; if file name + ext exist then skip...... correct me if my guess are wrong... but don't you thinks it's unnecessary? and if you happen to change your rename rules, it will download everything again!....

deekerman commented

2026-02-20 23:03:16 -05:00

Author

Owner

@FiloSottile commented on GitHub (Nov 27, 2012):

Yeah, it's suboptimal, I pointed you at it only as a workaround.
We can implement your feature, but we can't make the skipping method any better (easily, I'd rather prefer avoiding complex heuristics). Or do you have an idea?

@FiloSottile commented on GitHub (Nov 27, 2012): Yeah, it's suboptimal, I pointed you at it only as a workaround. We can implement your feature, but we can't make the skipping method any better (easily, I'd rather prefer avoiding complex heuristics). Or do you have an idea?

deekerman commented

2026-02-20 23:03:16 -05:00

Author

Owner

@w524e commented on GitHub (Nov 27, 2012):

the way youtube-dl doing is fine and very "python" :).
i was thinking of annotating the complete link in the batch file before i used the above method. Just put a # in front and youtube-dl will skip the line, just like the todo list--done that and cross it out.

@w524e commented on GitHub (Nov 27, 2012): the way youtube-dl doing is fine and very "python" :). i was thinking of annotating the complete link in the batch file before i used the above method. Just put a # in front and youtube-dl will skip the line, just like the todo list--done that and cross it out.

deekerman commented

2026-02-20 23:03:17 -05:00

Author

Owner

@Plaque-fcc commented on GitHub (Nov 29, 2012):

… and if you happen to change
your rename rules, it will download everything again!....

And if they publish their videos with a larger available format which
implies files of larger size (or other names originated), there may
come troubles you definetely did not wish to rush into. The only way to
identify the files/links you've already downloaded is making youtube-dl
«remember» what it did times before, which implies having a database
global at least to whole user account.

Do you actually NEED this, @w524e ?

@Plaque-fcc commented on GitHub (Nov 29, 2012): > … and if you happen to change > your rename rules, it will download everything again!.... And if they publish their videos with a larger available format which implies files of larger size (or other names originated), there may come troubles you definetely did not wish to rush into. The only way to identify the files/links you've already downloaded is making youtube-dl «remember» what it did times before, which implies having a database global at least to whole user account. Do you actually NEED this, @w524e ?

deekerman commented

2026-02-20 23:03:17 -05:00

Author

Owner

@FiloSottile commented on GitHub (Nov 29, 2012):

I think he's asking to keep state not globally, but of the current batch.

@FiloSottile commented on GitHub (Nov 29, 2012): I think he's asking to keep state not globally, but of the current batch.

deekerman commented

2026-02-20 23:03:17 -05:00

Author

Owner

@Plaque-fcc commented on GitHub (Nov 29, 2012):

Then it implies that there each batch task will finish succeeding which
no-one guaranties; this means, youtube-dl have to store some info
between its runtimes. This, IMHO, is a quite implementable thing
generally storing (at least, where the playlist file lies) another
file where youtube-dl can store what it got before (video ID,
filename, format ID and optionally download completeness). This way
the tool can read (if told so) and update the info on downloads,
neither harming format/size nor making dupes.

Hit me where I am wrong.

@Plaque-fcc commented on GitHub (Nov 29, 2012): Then it implies that there each batch task will finish succeeding which no-one guaranties; this means, youtube-dl have to store some info between its runtimes. This, IMHO, is a quite implementable thing generally storing (at least, where the playlist file lies) another file where youtube-dl can store what it got before (video ID, filename, format ID and optionally download completeness). This way the tool can read (if told so) and update the info on downloads, neither harming format/size nor making dupes. Hit me where I am wrong.

deekerman commented

2026-02-20 23:03:17 -05:00

Author

Owner

@w524e commented on GitHub (Nov 29, 2012):

NO....I need youtube-dl "forget" what it did.
No need datebase. this is what i did:
let's say batchlist.txt has:
http//wxw.youtube.com/watch?v=00000000000
http//wxw.youtube.com/watch?v=11111111111
...
http//wxw.youtube.com/watch?v=zzzzzzzzzzz

first, "youtube-dl.exe -x -y -z --batch-file "batchlist.txt"
maybe some "unable to download..." error happen...
then check the result:
If exist *00000000000.mp4 sed.exe -i "/00000000000/D" batchlist.txt
If exist *11111111111.mp4 sed.exe -i "/11111111111/D" batchlist.txt
...

Again,
IF EXIST .*.part or size of batchlist.txt is not zero goto loop
"youtube-dl.exe -x -y -z --batch-file "batchlist.txt"
until, if %%~za==0 GOTO EOF

if DOS can do that, Python can do it better, right? :D

A better way it's like this:
first, youtube-dl read a line from batchlist.txt, say:
"http://www.youtube.com/watch?V=11111111111"
when download is completed 100%, just append prefix (let's say,"# ") to this line so it becomes
"# http://www.youtube.com/watch?V=11111111111"
and then write it back to batchlist.txt by replacing the original line

When youtube-dl reads it, only read the lines that don't start with "# "

Just a different way to handle the "--batch-file" file...

@w524e commented on GitHub (Nov 29, 2012): NO....I need youtube-dl "forget" what it did. No need datebase. this is what i did: let's say batchlist.txt has: http//wxw.youtube.com/watch?v=00000000000 http//wxw.youtube.com/watch?v=11111111111 ... http//wxw.youtube.com/watch?v=zzzzzzzzzzz first, "youtube-dl.exe -x -y -z --batch-file "batchlist.txt" maybe some "unable to download..." error happen... then check the result: If exist *00000000000.mp4 sed.exe -i "/00000000000/D" batchlist.txt If exist *11111111111.mp4 sed.exe -i "/11111111111/D" batchlist.txt ... Again, IF EXIST .*.part or size of batchlist.txt is not zero goto loop "youtube-dl.exe -x -y -z --batch-file "batchlist.txt" until, if %%~za==0 GOTO EOF if DOS can do that, Python can do it better, right? :D A better way it's like this: first, youtube-dl read a line from batchlist.txt, say: "http://www.youtube.com/watch?V=11111111111" when download is completed 100%, just append prefix (let's say,"# ") to this line so it becomes "# http://www.youtube.com/watch?V=11111111111" and then write it back to batchlist.txt by replacing the original line When youtube-dl reads it, only read the lines that don't start with "# " Just a different way to handle the "--batch-file" file...

deekerman commented

2026-02-20 23:03:17 -05:00

Author

Owner

@Plaque-fcc commented on GitHub (Nov 30, 2012):

I never asked youtube-dl team to add wget flavours; instead, I prefer
using bash shell scripts (analogue to DOS batch files with kind of
VBS/JS control; I am not interested in DOS/NT). This way lets me
separate functional related to web connectivity and my own requisites
in appropriate locations. I guess you may find (using MS WinNT 6.1+)
some Epic PowerShell useful much more than MS-DOS batch files; at least
when I touched Microsoft PowerShell at Windows 7, I found it quite
mimic to POSIX subsystems.

And yes, you say «cmd can do that, so Python too, indeed». Well,
indeed. Ain't Python a scripting language? It is. Ain't DOS
batch/powershell/bash/sh/etc files some sort of scripts too? They all
are. I can imagine the way I use youtube-dl and other stuff for
gathering info about which video I do possess already, load percentage,
codec IDs, so can let youtube-dl know which and how should it download
further, if any. Yes, this may appear harder to script it up than just
having a «guess-it-yourself-you-blackbox» option, but I've never even
read any other sane proposition on how should youtube-dl act to
implement the manner you request.

And yes, you wrote there how your DOS batch file works. And you
called your '--batch-file' file 'not a playlist' while it actually IS
your local custom playlist, heh. Maybe I confused you with this little
terms mixing I did not produce? Then pardon meh, plz. But tell me, if
your batch file handles your trivial workaround, why asking youtube-dl
team implementing this trivial workaround, already handled pretty fine?

@Plaque-fcc commented on GitHub (Nov 30, 2012): I never asked youtube-dl team to add wget flavours; instead, I prefer using bash shell scripts (analogue to DOS batch files with kind of VBS/JS control; I am not interested in DOS/NT). This way lets me separate functional related to web connectivity and my own requisites in appropriate locations. I guess you may find (using MS WinNT 6.1+) some Epic PowerShell useful much more than MS-DOS batch files; at least when I touched Microsoft PowerShell at Windows 7, I found it quite mimic to POSIX subsystems. And yes, you say «cmd can do that, so Python too, indeed». Well, indeed. Ain't Python a scripting language? It is. Ain't DOS batch/powershell/bash/sh/etc files some sort of scripts too? They all are. I can imagine the way I use youtube-dl and other stuff for gathering info about which video I do possess already, load percentage, codec IDs, so can let youtube-dl know which and how should it download further, if any. Yes, this may appear harder to script it up than just having a «guess-it-yourself-you-blackbox» option, but I've never even read any other sane proposition on how should youtube-dl act to implement the manner you request. And yes, you wrote there how your DOS batch file works. And you called your '--batch-file' file 'not a playlist' while it actually IS your local custom playlist, heh. Maybe I confused you with this little terms mixing I did not produce? Then pardon meh, plz. But tell me, if your batch file handles your trivial workaround, why asking youtube-dl team implementing this trivial workaround, already handled pretty fine?

deekerman commented

2026-02-20 23:03:17 -05:00

Author

Owner

@w524e commented on GitHub (Nov 30, 2012):

That how youtube-dl calls it, (see --help) :D
<< -a, --batch-file FILE file containing URLs to download ('-' for stdin)>>

the way i did can do the job, but it has limitations :( .

Anyway, re-downloading every ~200k Html file (for youtube) might not be the best way to check skipping for those who has a lot (say 1000+ :D) of to "batch". (*edit by @phihag: snip the zen *)

@w524e commented on GitHub (Nov 30, 2012): That how youtube-dl calls it, (see --help) :D << -a, --batch-file FILE file containing URLs to download ('-' for stdin)>> the way i did can do the job, but it has limitations :( . Anyway, re-downloading every ~200k Html file (for youtube) might not be the best way to check skipping for those who has a lot (say 1000+ :D) of to "batch". (*edit by @phihag: snip the [zen](http://www.python.org/dev/peps/pep-0020/) *)

deekerman commented

2026-02-20 23:03:17 -05:00

Author

Owner

@phihag commented on GitHub (Nov 30, 2012):

This discussion is getting way out of hand. The feature like a good idea, the data format and proposed algorithms are sufficiently simple to be worthy of being implemented. Feel free to supply a patch request.

@phihag commented on GitHub (Nov 30, 2012): This discussion is getting way out of hand. The feature like a good idea, the data format and proposed algorithms are sufficiently simple to be worthy of being implemented. Feel free to supply a patch request.

deekerman commented

2026-02-20 23:03:17 -05:00

Author

Owner

@dirkf commented on GitHub (Feb 5, 2024):

Nowadays, --download-archive ....

@dirkf commented on GitHub (Feb 5, 2024): Nowadays, `--download-archive ...`.

deekerman referenced this issue

2026-02-21 01:29:51 -05:00

Vine error #3643

Rows
Columns

Auto Batch Mode? #406