[core] Networking enhancements #27054

New issue

Open

opened 2026-02-21 14:37:15 -05:00 by deekerman · 0 comments

deekerman commented

2026-02-21 14:37:15 -05:00

Owner

Originally created by @dirkf on GitHub (Feb 7, 2025).

In #33045, we found an AWS server that rejected HTTP/1.1.

Originally posted by @dirkf in https://github.com/ytdl-org/youtube-dl/issues/33045#issuecomment-2642506283

I confirmed that adding --http1.1 to the working curl command causes the same 464 error.

Other sites are deploying web-breaking technology via Cloudflare, Fastly, etc, that are primarily intended to protect against large-scale DDoS attacks but have a possibly not unwelcome effect on tools like yt-dl. The magic words include "Enable JavaScript to continue", "Verifying your browser" (I'm at your site, so it's a browser, what is there to verify?), and so on. Some of the main video sites (Vimeo, eg) are fingerprinting requests to deny access.

yt-dlp has re-implemented networking so as to take advantage of the more modern standard and external library modules available to it. This allows it to support, eg, curl-impersonate to make network requests, and so satisfy the door policy of some of the more choosy sites. While that might be impractical for some of yt-dl's target platforms, it would be nice to have the option.

In the linked comment I continued:

One way of looking at this is that Python's urllib2, which doesn't support HTTP/2, is ancient: yt-dl should get with the programme and join the third decade of the 21st century. OTOH, it's the best we can do with the standard library of supported Python versions.
...
Should we really need to support HTTP/2, we can wrap the curl command-line. There is a potentially better technical solution in which libcurl is called directly from Python, but only the 5-year-old version 7.43.0.5 of the pycurl PyPi package that would avoid having to re-implement that is compatible with our Python platforms. Wrapping the curl command-line would also allow for versions with additional functionality, such as curl-impersonate.

This issue invites suggestions of other potential approaches and especially of features that should be supported.

These issues were linked as resolved by yt-dlp's network rework:

socks5h proxy support.

Originally created by @dirkf on GitHub (Feb 7, 2025). In #33045, we found an AWS server that rejected HTTP/1.1. _Originally posted by @dirkf in https://github.com/ytdl-org/youtube-dl/issues/33045#issuecomment-2642506283_ >I confirmed that adding `--http1.1` to the working _curl_ command causes the same 464 error. Other sites are deploying web-breaking technology via Cloudflare, Fastly, etc, that are primarily intended to protect against large-scale DDoS attacks but have a possibly not unwelcome effect on tools like _yt-dl_. The magic words include "Enable JavaScript to continue", "Verifying your browser" (I'm at your site, so it's a browser, what is there to verify?), and so on. Some of the main video sites (Vimeo, eg) are fingerprinting requests to deny access. _yt-dlp_ has re-implemented networking so as to take advantage of the more modern standard and external library modules available to it. This allows it to support, eg, _curl-impersonate_ to make network requests, and so satisfy the [door policy](https://tvtropes.org/pmwiki/pmwiki.php/Main/NotOnTheList) of some of the more choosy sites. While that might be impractical for some of _yt-dl_'s target platforms, it would be nice to have the option. In the linked comment I continued: >One way of looking at this is that Python's `urllib2`, which doesn't support HTTP/2, is ancient: _yt-dl_ should get with the programme and join the third decade of the 21st century. OTOH, it's the best we can do with the standard library of supported Python versions. >... >Should we really need to support HTTP/2, we can wrap the _curl_ command-line. There is a potentially better technical solution in which _libcurl_ is called directly from Python, but only the 5-year-old version 7.43.0.5 of the [_pycurl_ PyPi package](https://pypi.org/project/pycurl) that would avoid having to re-implement that is compatible with our Python platforms. Wrapping the _curl_ command-line would also allow for versions with additional functionality, such as [_curl-impersonate_](https://github.com/lwthiker/curl-impersonate). This issue invites suggestions of other potential approaches and especially of features that should be supported. These issues were linked as resolved by [_yt-dlp_'s network rework](https://github.com/yt-dlp/yt-dlp/pull/3668): * #13734 * #14523 * #32549 Related: * [socks5h proxy support](https://github.com/yt-dlp/yt-dlp/issues/6325#issuecomment-1441293009).

deekerman added the

request

question

labels

2026-02-21 14:37:15 -05:00

No milestone

No project

No assignees

1 participant

Notifications

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference

starred/youtube-dl-ytdl-org#27054

No description provided.

Rows
Columns