Sometimes I get 403 errors #410

Closed
opened 2026-02-20 21:12:09 -05:00 by deekerman · 20 comments
Owner

Originally created by @dev0k on GitHub (Nov 12, 2012).

Hello guys,

i use youtube-dl for download my music. But sometimes I get a 403 error.

url=youtube-dl --user-agent 'Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.11 (KHTML, like Gecko) Chrome/23.0.1271.64 Safari/537.11' -g --skip-download --write-info-json --cookies cookie.y2m -o source $1 | tail -n 1 -

cookie=$( cat cookie.y2m | awk '{printf("%s=%s;",$6,$7)}' )

axel -U 'Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.11 (KHTML, like Gecko) Chrome/23.0.1271.64 Safari/537.11 ' -H "Cookie: $cookie" $url

(axel is a downloader (like wget))

Very funny! Sometimes it works, sometimes not! But why..?

Thanks :)

Originally created by @dev0k on GitHub (Nov 12, 2012). Hello guys, i use youtube-dl for download my music. But sometimes I get a 403 error. url=`youtube-dl --user-agent 'Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.11 (KHTML, like Gecko) Chrome/23.0.1271.64 Safari/537.11' -g --skip-download --write-info-json --cookies cookie.y2m -o source $1 | tail -n 1 -` cookie=$( cat cookie.y2m | awk '{printf("%s=%s;",$6,$7)}' ) axel -U 'Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.11 (KHTML, like Gecko) Chrome/23.0.1271.64 Safari/537.11 ' -H "Cookie: $cookie" $url (axel is a downloader (like wget)) Very funny! Sometimes it works, sometimes not! But why..? Thanks :)
Author
Owner

@dev0k commented on GitHub (Nov 12, 2012):

What the fuck ...

root@mybase: curl 'http://o-o---preferred---sn-25g7rn7z---v3---lscache7.c.youtube.com/.....' -L > video2
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
19 12.1M 19 2429k 0 0 507k 0 0:00:24 0:00:04 0:00:20 581k^C

curl works, without cookies! (only with the parameter ('-L'))

and now axel... super fast but..

root@mybase:# axel 'http://o-o---preferred---sn-25g7rn7z---v3---lscache7.c.youtube.com/.....' > video3
Start Request
HTTP/1.1 403 Forbidden

but why?!

@dev0k commented on GitHub (Nov 12, 2012): What the fuck ... root@mybase: curl 'http://o-o---preferred---sn-25g7rn7z---v3---lscache7.c.youtube.com/.....' -L > video2 % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 19 12.1M 19 2429k 0 0 507k 0 0:00:24 0:00:04 0:00:20 581k^C curl works, without cookies! (only with the parameter ('-L')) and now axel... super fast but.. root@mybase:# axel 'http://o-o---preferred---sn-25g7rn7z---v3---lscache7.c.youtube.com/.....' > video3 Start Request HTTP/1.1 403 Forbidden but why?!
Author
Owner

@Tailszefox commented on GitHub (Nov 12, 2012):

Hm, I'm not getting any issue, neither with curl nor axel, the video just gets downloaded correctly. Do you have any particular URLs for which it doesn't always work, or does it seem random?

As a side note, if you used the same URLs twice, that may be the reason your second download got refused. If you try to download a video and then try it again a bit later, it may not work since the link may have expired, or be deleted since it's already been used. Do you get a 403 error if you try with axel first?

@Tailszefox commented on GitHub (Nov 12, 2012): Hm, I'm not getting any issue, neither with curl nor axel, the video just gets downloaded correctly. Do you have any particular URLs for which it doesn't always work, or does it seem random? As a side note, if you used the same URLs twice, that may be the reason your second download got refused. If you try to download a video and then try it again a bit later, it may not work since the link may have expired, or be deleted since it's already been used. Do you get a 403 error if you try with axel first?
Author
Owner

@dev0k commented on GitHub (Nov 12, 2012):

Thanks. I've found the problem. In my script the url has not escaped.

axel $url

Sometimes error. why?!

axel "$url"

No Exception.

But why has he given only SOMETIMES an error message? Thats funny..

@dev0k commented on GitHub (Nov 12, 2012): Thanks. I've found the problem. In my script the url has not escaped. axel $url Sometimes error. why?! axel "$url" No Exception. But why has he given only SOMETIMES an error message? Thats funny..
Author
Owner

@Tailszefox commented on GitHub (Nov 12, 2012):

Ah, right, I thought you already did that. Not sure, might be because of the way YouTube reacts to malformed URLs. Regardless, as long as it works now, it's all good!

@Tailszefox commented on GitHub (Nov 12, 2012): Ah, right, I thought you already did that. Not sure, might be because of the way YouTube reacts to malformed URLs. Regardless, as long as it works now, it's all good!
Author
Owner

@FiloSottile commented on GitHub (Nov 12, 2012):

Probably breaks only if shell meaningful chars get into the url, and this might be random. Nice to hear the issue is solved, closing.

@FiloSottile commented on GitHub (Nov 12, 2012): Probably breaks only if shell meaningful chars get into the url, and this might be random. Nice to hear the issue is solved, closing.
Author
Owner

@dev0k commented on GitHub (Nov 13, 2012):

root@y:/home/product# axel 'YOUTUBE-URL' -o test
Start Request: YOUTUBE-URL
HTTP/1.1 403 Forbidden

root@y:/home/product# curl -L 'YOUTUBE-URL' > test
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
4 12.1M 4 579k 0 0 109k 0 0:01:54 0:00:05 0:01:49 114k^C

curl works.. axel dont work :-/ yesterday it still worked. Now comes the 403 error. What now? :-/

@dev0k commented on GitHub (Nov 13, 2012): root@y:/home/product# axel 'YOUTUBE-URL' -o test Start Request: YOUTUBE-URL HTTP/1.1 403 Forbidden root@y:/home/product# curl -L 'YOUTUBE-URL' > test % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 4 12.1M 4 579k 0 0 109k 0 0:01:54 0:00:05 0:01:49 114k^C curl works.. axel dont work :-/ yesterday it still worked. Now comes the 403 error. What now? :-/
Author
Owner

@FiloSottile commented on GitHub (Nov 13, 2012):

Umh... the User-Agent? You should supply the youtube-dl one to axel.

@FiloSottile commented on GitHub (Nov 13, 2012): Umh... the User-Agent? You should supply the youtube-dl one to axel.
Author
Owner

@dev0k commented on GitHub (Nov 13, 2012):

But why it works that way with wget, curl, etc? Sometimes it is also with axel, but only sometimes.

@dev0k commented on GitHub (Nov 13, 2012): But why it works that way with wget, curl, etc? Sometimes it is also with axel, but only sometimes.
Author
Owner

@FiloSottile commented on GitHub (Nov 13, 2012):

Well, it might be the different User-Agent, but my best bet would be that the axel download gets killed as it opens too much concurrent connections.

@FiloSottile commented on GitHub (Nov 13, 2012): Well, it might be the different User-Agent, but my best bet would be that the axel download gets killed as it opens too much concurrent connections.
Author
Owner

@dev0k commented on GitHub (Nov 13, 2012):

root@y:~# axel -n 1 -U 'Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.11 (KHTML, like Gecko) Chrome/23.0.1271.64 Safari/537.11' '$url'

Only 1 Connection and same useragent.

and.. 403 error :-/

Can you maybe try with Axel? The YT Url is: http://www.youtube.com/watch?v=4wOoLLDXbDY.

@dev0k commented on GitHub (Nov 13, 2012): root@y:~# axel -n 1 -U 'Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.11 (KHTML, like Gecko) Chrome/23.0.1271.64 Safari/537.11' '$url' Only 1 Connection and same useragent. and.. 403 error :-/ Can you maybe try with Axel? The YT Url is: http://www.youtube.com/watch?v=4wOoLLDXbDY.
Author
Owner

@Tailszefox commented on GitHub (Nov 13, 2012):

% axel `youtube-dl -g "http://www.youtube.com/watch?v=4wOoLLDXbDY"`
Initializing download: http://o-o---preferred---sn-5hn7sned---v14---lscache6.c.youtube.com/videoplayback?upn=2w9wpU4czQ8&sparams=cp%2Cgcr%2Cid%2Cip%2Cipbits%2Citag%2Cratebypass%2Csource%2Cupn%2Cexpire&fexp=919318%2C904461%2C916617%2C922401%2C920704%2C912806%2C927201%2C922403%2C925003%2C913546%2C913556%2C916805%2C920201%2C901451&ms=au&expire=1352863289&itag=22&ipbits=48&gcr=fr&sver=3&ratebypass=yes&mt=1352839511&ip=2a02%3A8420%3A5722%3A6100%3Ad0b9%3Afc58%3A6278%3A66f2&mv=m&source=youtube&key=yt1&cp=U0hUR1hRUV9NU0NONF9QS1VDOnN0eTRXZ1NpSU0x&id=e303a82cb0d76c36&signature=9A3B839DFC5C00F81F65A77D2800CD002E111AE4.C2957062DF9B83E4D70A13629A260E9576A6F9C7
File size: 48520653 bytes
Opening output file videoplayback
Starting download

[  0%]  .......... .......... .......... .......... ..........  [  14,6KB/s]
[  0%]  .......... .......... .......... .......... ..........  [  21,6KB/s]
[  0%]  .......... .......... .......... .......... ..........  [  21,6KB/s]
[  0%]  .......... .......... .......... .......... ..........  [  25,7KB/s]
[...]

Download then continues and finishes normally. I tried it at least five times, each time getting a different URL from YouTube since I'm making youtube-dl make a new request each time, and it seems to work with or without a "proper" user agent, at least for me.

@Tailszefox commented on GitHub (Nov 13, 2012): ``` % axel `youtube-dl -g "http://www.youtube.com/watch?v=4wOoLLDXbDY"` Initializing download: http://o-o---preferred---sn-5hn7sned---v14---lscache6.c.youtube.com/videoplayback?upn=2w9wpU4czQ8&sparams=cp%2Cgcr%2Cid%2Cip%2Cipbits%2Citag%2Cratebypass%2Csource%2Cupn%2Cexpire&fexp=919318%2C904461%2C916617%2C922401%2C920704%2C912806%2C927201%2C922403%2C925003%2C913546%2C913556%2C916805%2C920201%2C901451&ms=au&expire=1352863289&itag=22&ipbits=48&gcr=fr&sver=3&ratebypass=yes&mt=1352839511&ip=2a02%3A8420%3A5722%3A6100%3Ad0b9%3Afc58%3A6278%3A66f2&mv=m&source=youtube&key=yt1&cp=U0hUR1hRUV9NU0NONF9QS1VDOnN0eTRXZ1NpSU0x&id=e303a82cb0d76c36&signature=9A3B839DFC5C00F81F65A77D2800CD002E111AE4.C2957062DF9B83E4D70A13629A260E9576A6F9C7 File size: 48520653 bytes Opening output file videoplayback Starting download [ 0%] .......... .......... .......... .......... .......... [ 14,6KB/s] [ 0%] .......... .......... .......... .......... .......... [ 21,6KB/s] [ 0%] .......... .......... .......... .......... .......... [ 21,6KB/s] [ 0%] .......... .......... .......... .......... .......... [ 25,7KB/s] [...] ``` Download then continues and finishes normally. I tried it at least five times, each time getting a different URL from YouTube since I'm making youtube-dl make a new request each time, and it seems to work with or without a "proper" user agent, at least for me.
Author
Owner
@dev0k commented on GitHub (Nov 13, 2012): root@y:# axel `./youtube-dl -g "http://www.youtube.com/watch?v=4wOoLLDXbDY"` (469, 591) (498, 591) (498, 1182) (498, 1773) (498, 2364) (498, 2955) (498, 3546) (498, 4137) (498, 4728) (498, 5319) (498, 5910) (498, 6501) Starte Abruf: http://o-o---preferred---sn-25g7rn7k---v14---lscache6.c.youtube.com/videoplayback?upn=ioWaURvK7Mc&sparams=cp%2Cid%2Cip%2Cipbits%2Citag%2Cratebypass%2Csource%2Cupn%2Cexpire&fexp=900080%2C922401%2C920704%2C912806%2C927201%2C922403%2C925003%2C913546%2C913556%2C916805%2C920201%2C901451&key=yt1&expire=1352863289&itag=22&ipbits=8&sver=3&ratebypass=yes&mt=1352840172&ip=94.23.233.25&mv=m&source=youtube&ms=au&cp=U0hUR1hRUV9NU0NONF9QS1VDOi1PaVMzeXVSYjdG&id=e303a82cb0d76c36&signature=2134E4C08FE1EF2EE94E0619DEC5FBED3CADC53D.1D8AEFC904E0F87358D337CDC6CD167B82D4957B HTTP/1.1 403 Forbidden Can you please paste this: axel --version youtube-dl --version
Author
Owner

@Tailszefox commented on GitHub (Nov 13, 2012):

Seems you're using an old version of youtube-dl, given that debug output you get. You should try compiling the latest version, I don't know if it's related to your issue, I doubt it, but it's worth trying. Anyway:

% axel --version
Axel version 2.4 (Linux)

Copyright 2001-2002 Wilmer van der Gaast.

% youtube-dl --version
2012.10.09

The youtube-dl version I use is the latest as I regularly update my local repository to match the current version.

@Tailszefox commented on GitHub (Nov 13, 2012): Seems you're using an old version of youtube-dl, given that debug output you get. You should try compiling the latest version, I don't know if it's related to your issue, I doubt it, but it's worth trying. Anyway: ``` % axel --version Axel version 2.4 (Linux) Copyright 2001-2002 Wilmer van der Gaast. % youtube-dl --version 2012.10.09 ``` The youtube-dl version I use is the latest as I regularly update my local repository to match the current version.
Author
Owner

@dev0k commented on GitHub (Nov 13, 2012):

root@y:# axel youtube-dl -g "http://www.youtube.com/watch?v=4wOoLLDXbDY "
(469, 591)
(498, 591)
(498, 1182)
(498, 1773)
(498, 2364)
(498, 2955)
(498, 3546)
(498, 4137)
(498, 4728)
(498, 5319)
(498, 5910)
(498, 6501)
Starte Abruf: http://o-o---preferred---sn-25g7rn7k---v14---lscache6.c.youtube.co m/videoplayback?upn=Frb8DUNFPYg&sparams=cp%2Cid%2Cip%2Cipbits%2Citag%2Cratebypas s%2Csource%2Cupn%2Cexpire&fexp=906437%2C910013%2C922401%2C920704%2C912806%2C9272 01%2C922403%2C925003%2C913546%2C913556%2C916805%2C920201%2C901451&key=yt1&expire =1352866889&itag=22&ipbits=8&sver=3&ratebypass=yes&mt=1352841791&ip=94.23.233.25 &mv=m&source=youtube&ms=au&cp=U0hUR1hRVF9NU0NONF9QS1hJOlRSSjVRdTAyRERS&id=e303a8 2cb0d76c36&signature=C6D0D10A4FD37715B0801F7858CEC20C3D85F8E1.CCEC1976655171BDB1 F6324EB98C76BE0B8C0E90
HTTP/1.1 403 Forbidden
root@y:
# youtube-dl --version
2012.10.09

What do the numbers at the beginning?

I've even tested on a different server, there it worked.

@dev0k commented on GitHub (Nov 13, 2012): root@y:~# axel `youtube-dl -g "http://www.youtube.com/watch?v=4wOoLLDXbDY "` (469, 591) (498, 591) (498, 1182) (498, 1773) (498, 2364) (498, 2955) (498, 3546) (498, 4137) (498, 4728) (498, 5319) (498, 5910) (498, 6501) Starte Abruf: http://o-o---preferred---sn-25g7rn7k---v14---lscache6.c.youtube.co m/videoplayback?upn=Frb8DUNFPYg&sparams=cp%2Cid%2Cip%2Cipbits%2Citag%2Cratebypas s%2Csource%2Cupn%2Cexpire&fexp=906437%2C910013%2C922401%2C920704%2C912806%2C9272 01%2C922403%2C925003%2C913546%2C913556%2C916805%2C920201%2C901451&key=yt1&expire =1352866889&itag=22&ipbits=8&sver=3&ratebypass=yes&mt=1352841791&ip=94.23.233.25 &mv=m&source=youtube&ms=au&cp=U0hUR1hRVF9NU0NONF9QS1hJOlRSSjVRdTAyRERS&id=e303a8 2cb0d76c36&signature=C6D0D10A4FD37715B0801F7858CEC20C3D85F8E1.CCEC1976655171BDB1 F6324EB98C76BE0B8C0E90 HTTP/1.1 403 Forbidden root@y:~# youtube-dl --version 2012.10.09 What do the numbers at the beginning? I've even tested on a different server, there it worked.
Author
Owner

@Tailszefox commented on GitHub (Nov 13, 2012):

This is some debug output left there by mistake. It's been fixed in commit 5d40a470a2, which means you're using an outdated version. As I said, I don't think the issues are related, but it's always good to have the latest version.

The only thing I can think of that would cause this problem is that axel downloads the video with a different IP that youtube-dl uses when requesting the URL of the video. Maybe one is using a proxy while the other isn't. If it works with curl or wget, maybe you could try something like this:

% axel -o ip.txt "http://ifconfig.me/ip"
% wget -O ip2.txt "http://ifconfig.me/ip"

Then, check the files ip.txt and ip2.txt. If they don't have the same IP, that's the issue. If they do, then...I'm afraid I'm out of ideas, sorry.

@Tailszefox commented on GitHub (Nov 13, 2012): This is some debug output left there by mistake. It's been fixed in commit 5d40a470a2c62f33de54f14a04dfab997e6503fe, which means you're using an outdated version. As I said, I don't think the issues are related, but it's always good to have the latest version. The only thing I can think of that would cause this problem is that axel downloads the video with a different IP that youtube-dl uses when requesting the URL of the video. Maybe one is using a proxy while the other isn't. If it works with curl or wget, maybe you could try something like this: ``` % axel -o ip.txt "http://ifconfig.me/ip" % wget -O ip2.txt "http://ifconfig.me/ip" ``` Then, check the files ip.txt and ip2.txt. If they don't have the same IP, that's the issue. If they do, then...I'm afraid I'm out of ideas, sorry.
Author
Owner

@dev0k commented on GitHub (Nov 13, 2012):

Thanks for your help. The IP addresses are the same.
One thing I ask myself, I have youtube-dl updated to the latest version, but I still get the numbers at the beginning. (10/09/2012).
Then I have the same installed on another server. Then I tried it, and there it works. What could be wrong?

@dev0k commented on GitHub (Nov 13, 2012): Thanks for your help. The IP addresses are the same. One thing I ask myself, I have youtube-dl updated to the latest version, but I still get the numbers at the beginning. (10/09/2012). Then I have the same installed on another server. Then I tried it, and there it works. What could be wrong?
Author
Owner

@Tailszefox commented on GitHub (Nov 13, 2012):

Do you mean the debug output, or the version number from youtube-dl? The version number isn't updated automatically, and the youtube-dl binary provided here isn't always up to date either. The only way to ensure you actually do have the latest version is to clone the repository and use make youtube-dl to compile it.

As for why it works on a server and not on another, I can only guess it's a connectivity or proxy issue that only happens on one of the server, and not the other. That sounds inconvenient but I'm afraid there's nothing youtube-dl can do about that, as it seems to be more related to axel than anything.

@Tailszefox commented on GitHub (Nov 13, 2012): Do you mean the debug output, or the version number from youtube-dl? The version number isn't updated automatically, and the youtube-dl binary provided here isn't always up to date either. The only way to ensure you actually do have the latest version is to clone the repository and use `make youtube-dl` to compile it. As for why it works on a server and not on another, I can only guess it's a connectivity or proxy issue that only happens on one of the server, and not the other. That sounds inconvenient but I'm afraid there's nothing youtube-dl can do about that, as it seems to be more related to axel than anything.
Author
Owner

@FiloSottile commented on GitHub (Nov 13, 2012):

As a last chance, I would try (updating and) passing also the cookies to axel. We have an option for that.

@FiloSottile commented on GitHub (Nov 13, 2012): As a last chance, I would try (updating and) passing also the cookies to axel. We have an option for that.
Author
Owner

@dev0k commented on GitHub (Nov 14, 2012):

Okay.

root@y:/home# youtube-dl "http://www.youtube.com/watch?v=RQaz8VFTzlQ&feature=related"
[youtube] Setting language
[youtube] RQaz8VFTzlQ: Downloading video webpage
[youtube] RQaz8VFTzlQ: Downloading video info webpage
[youtube] RQaz8VFTzlQ: Extracting video information
(456, 591)
(485, 591)
(485, 1182)
(485, 1773)
(485, 2364)
(485, 2955)
(485, 3546)
(485, 4137)
(485, 4728)
(485, 5319)
(485, 5910)
(485, 6501)
[download] Destination: RQaz8VFTzlQ.flv
root@y:/home# youtube-dl --version
2012.10.09

These numbers are still there, although "# print >> sys.stderr, self.getpos ()" is commented out. Why?

@dev0k commented on GitHub (Nov 14, 2012): Okay. root@y:/home# youtube-dl "http://www.youtube.com/watch?v=RQaz8VFTzlQ&feature=related" [youtube] Setting language [youtube] RQaz8VFTzlQ: Downloading video webpage [youtube] RQaz8VFTzlQ: Downloading video info webpage [youtube] RQaz8VFTzlQ: Extracting video information (456, 591) (485, 591) (485, 1182) (485, 1773) (485, 2364) (485, 2955) (485, 3546) (485, 4137) (485, 4728) (485, 5319) (485, 5910) (485, 6501) [download] Destination: RQaz8VFTzlQ.flv root@y:/home# youtube-dl --version 2012.10.09 These numbers are still there, although "# print >> sys.stderr, self.getpos ()" is commented out. Why?
Author
Owner

@dev0k commented on GitHub (Nov 14, 2012):

So, I have the solution!

root @ y :/ home # youtube-dl-g 'http://www.youtube.com/watch?v=4wOoLLDXbDY' | xargs curl-I
HTTP/1.1 302 Moved Temporarily
Last-Modified: Wed, 02 May 2007 10:26:10 GMT
Date: Wed, 14 November 2012 09:23:52 GMT
Expires: Wed, 14 November 2012 09:23:52 GMT
Cache-Control: private, max-age = 900
Location: NEW YOUTUBE URL
Connection: close
X-Content-Type-Options: nosniff
Content-Type: text / html
Server: gvs 1.0

And then: axel "$ new url" and WORKS!

Thanks to you :)

@dev0k commented on GitHub (Nov 14, 2012): So, I have the solution! root @ y :/ home # youtube-dl-g 'http://www.youtube.com/watch?v=4wOoLLDXbDY' | xargs curl-I HTTP/1.1 302 Moved Temporarily Last-Modified: Wed, 02 May 2007 10:26:10 GMT Date: Wed, 14 November 2012 09:23:52 GMT Expires: Wed, 14 November 2012 09:23:52 GMT Cache-Control: private, max-age = 900 Location: NEW YOUTUBE URL Connection: close X-Content-Type-Options: nosniff Content-Type: text / html Server: gvs 1.0 And then: axel "$ new url" and WORKS! Thanks to you :)
Sign in to join this conversation.
No milestone
No project
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
starred/youtube-dl#410
No description provided.