New Release? #25785

Author

Owner

@Grub4K commented on GitHub (Feb 19, 2023):

What are those issues that need sorting out? Is there an issue/PR tracking this? Creating a PR could bring in people that are willing to help with the process (like myself)

@Grub4K commented on GitHub (Feb 19, 2023): What are those issues that need sorting out? Is there an issue/PR tracking this? Creating a PR could bring in people that are willing to help with the process (like myself)

deekerman commented

Author

Owner

@dirkf commented on GitHub (Feb 19, 2023):

#30644, but it should probably be refreshed now.

This is very far down my stack today, but as I recall there were various possible blockers:

change to pytest for Py3.10+ (mostly achieved in my test branch with a pytest build that should run on the supported platforms)
especially Jython was tricky, if anyone cares (but we have found a Py2.6 user today)
various workflow upgrades because of GH dependencies
the usual maze of twisty build system dependencies, all alike.

Obviously I need to dive back into this especially in view of https://github.com/yt-dlp/yt-dlp/pull/6220, which I would also like to acquire, if only for more regular versioned releases.

@dirkf commented on GitHub (Feb 19, 2023): #30644, but it should probably be refreshed now. This is very far down my stack today, but as I recall there were various possible blockers: * change to _pytest_ for Py3.10+ (mostly achieved in my [test branch](https://github.com/dirkf/youtube-dl) with a _pytest_ build that should run on the supported platforms) * especially Jython was tricky, if anyone cares (but we have found a Py2.6 user today) * various workflow upgrades because of GH dependencies * the usual maze of twisty build system dependencies, all alike. Obviously I need to dive back into this especially in view of https://github.com/yt-dlp/yt-dlp/pull/6220, which I would also like to acquire, if only for more regular versioned releases.

deekerman commented

Author

Owner

@dirkf commented on GitHub (Feb 20, 2023):

But isn't the auto-update installer already there, depending on the how the yt-dl instance was installed? Once a build has been deployed to the release page and to PyPi, it can be pulled by almost all installation types.

The problem being addressed is really to get the small number of builds in the release tested on all the different platforms.

@dirkf commented on GitHub (Feb 20, 2023): But isn't the auto-update installer already there, depending on the how the yt-dl instance was installed? Once a build has been deployed to the release page and to PyPi, it can be pulled by almost all installation types. The problem being addressed is really to get the small number of builds in the release tested on all the different platforms.

deekerman commented

Author

Owner

@ghost commented on GitHub (Feb 20, 2023):

My understanding is that PyPi and other options would only update on release, and not necessarily auto update at that (i.e. without running pip install again separately).

My suggestion would auto update on either release or master branch updates before each run of the program. That way you don't need to rush to cut release, at the same time users could stay up to date if they wish.

One thing I didn't really point out is that youtube_dl itself would also be initiated through this script, so it would run the checks to update and then pass the argument to youtube_dl and run.

Basically it's a simple hack to make updating against the master branch painless and easy to the user.

@ghost commented on GitHub (Feb 20, 2023): My understanding is that PyPi and other options would only update on release, and not necessarily auto update at that (i.e. without running pip install again separately). My suggestion would auto update on either release or master branch updates before each run of the program. That way you don't need to rush to cut release, at the same time users could stay up to date if they wish. One thing I didn't really point out is that youtube_dl itself would also be initiated through this script, so it would run the checks to update and then pass the argument to youtube_dl and run. Basically it's a simple hack to make updating against the master branch painless and easy to the user.

deekerman commented

Author

Owner

@Grub4K commented on GitHub (Feb 20, 2023):

Will #30644 keep tracking this and should I post my remarks on there?

I think maybe instead of perfecting the release process it might make sense to release something so as to get updates to the end user. If releases were not automated, whats problematic about building like that, then passing those executables to the user to be tried on various platforms? The related release can be marked as pre-release if need be, but that way there at least are executables which will work for most people while we can get reports for the edge cases and squash those specifically.

I would also advise to look into the release.yml/release-nightly.yml workflow in https://github.com/yt-dlp/yt-dlp/pull/6220. It is crafted so that it creates a release in the yt-dlp style regardless of build output, adding all of the build artifacts from build.yml, it's checksums and the _update_spec. A build.yml satisfying simplistic conditions can be created at first, which would build some more general artifacts, gathering info about possible incompatibilities.

These are the repository configuration options of the release and nighly action:

vars.PUSH_VERSION_COMMIT: Update master branch with a push
vars.BUILD_NIGHTLY: Build a nightly release on every commit

# Both need to be set to push build to archive repo as release
vars.ARCHIVE_REPO: repo like `yt-dlp/yt-dlp-builds`
secrets.ARCHIVE_REPO_TOKEN: token with contents:write

# Push to PyPI and brew if each is set
secrets.PYPI_TOKEN: Token for PyPI
secrets.BREW_TOKEN: Token for brew

update-version.py would have to be backported since the format changed and it's output is now a bit different. Maybe the naming could also be changed to update_version.py?

make_changelog.py, while configurable, is most likely not configurable enough to be able to support the ydl changelog. The author assumption would have to be corrected (Authored by: ) or the changelog would have to be crafted and read manually for now (would be simple to do a shim that just reads from ChangeLog for now).

change to pytest for Py3.10+ (mostly achieved in my test branch with a pytest build that should run on the supported platforms)

Are the test adjustments needed for the release for now? Is that preparation to be testing if the builds would work as expected? If so, I'd think that can be moved back for now?

especially Jython was tricky, if anyone cares (but we have found a Py2.6 user today)

Do we need to supply youtube-dl builds that embed JPython? Otherwise I can't really see the direct relation of that and a release.

various workflow upgrades because of GH dependencies

This should be handled by 6220. I can help with the remaining blockers there.

the usual maze of twisty build system dependencies, all alike.

Yes, understandable. If we encounter anything in particular we might need to push that back until somebody with experience on that system or with those dependencies can chime in. I think those issues will become apparent once we create/try creating the actual build.yml.

The steps that I would propose for now are:

Create the release.yml and adjust to the format we would want
Fix/port update_version.py and make_changelog.py
Create a simplified build.yml handling the most common targets.
Publish a pre-release with the created executables (people will be _{^{sort of}} happy here)
Fix problems, improve and extend build
Employ extended testing and the sort

Feel free to contact me via E-Mail if you would like to move to a different communication channel for this. Alternatively let me know of anything else I can do.

@Grub4K commented on GitHub (Feb 20, 2023): Will #30644 keep tracking this and should I post my remarks on there? I think maybe instead of perfecting the release process it might make sense to release *something* so as to get updates to the end user. If releases were not automated, whats problematic about building like that, then passing those executables to the user to be tried on various platforms? The related release can be marked as `pre-release` if need be, but that way there at least are executables which will work for *most* people while we can get reports for the edge cases and squash those specifically. I would also advise to look into the `release.yml`/`release-nightly.yml` workflow in https://github.com/yt-dlp/yt-dlp/pull/6220. It is crafted so that it creates a release in the `yt-dlp` style regardless of build output, adding all of the build artifacts from `build.yml`, it's checksums and the `_update_spec`. A `build.yml` satisfying simplistic conditions can be created at first, which would build some more general artifacts, gathering info about possible incompatibilities. These are the repository configuration options of the release and nighly action: ```yml vars.PUSH_VERSION_COMMIT: Update master branch with a push vars.BUILD_NIGHTLY: Build a nightly release on every commit # Both need to be set to push build to archive repo as release vars.ARCHIVE_REPO: repo like `yt-dlp/yt-dlp-builds` secrets.ARCHIVE_REPO_TOKEN: token with contents:write # Push to PyPI and brew if each is set secrets.PYPI_TOKEN: Token for PyPI secrets.BREW_TOKEN: Token for brew ``` `update-version.py` would have to be backported since the format changed and it's output is now a bit different. Maybe the naming could also be changed to `update_version.py`? `make_changelog.py`, while configurable, is most likely not configurable enough to be able to support the ydl changelog. The author assumption would have to be corrected (`Authored by: `) or the changelog would have to be crafted and read manually for now (would be simple to do a shim that just reads from `ChangeLog` for now). > - change to pytest for Py3.10+ (mostly achieved in my [test branch](https://github.com/dirkf/youtube-dl) with a pytest build that should run on the supported platforms) Are the test adjustments needed for the release for now? Is that preparation to be testing if the builds would work as expected? If so, I'd think that can be moved back for now? > - especially Jython was tricky, if anyone cares (but we have found a Py2.6 user today) Do we need to supply `youtube-dl` builds that embed JPython? Otherwise I can't really see the direct relation of that and a release. > - various workflow upgrades because of GH dependencies This should be handled by 6220. I can help with the remaining blockers there. > - the usual maze of twisty build system dependencies, all alike. Yes, understandable. If we encounter anything in particular we might need to push that back until somebody with experience on that system or with those dependencies can chime in. I think those issues will become apparent once we create/try creating the actual `build.yml`. The steps that I would propose for now are: - [ ] Create the `release.yml` and adjust to the format we would want - [ ] Fix/port `update_version.py` and `make_changelog.py` - [ ] Create a simplified `build.yml` handling the most common targets. - [ ] Publish a pre-release with the created executables (people will be <sub><sup>sort of</sup></sub> happy here) - [ ] Fix problems, improve and extend build - [ ] Employ extended testing and the sort Feel free to contact me via [E-Mail](mailto:contact@grub4k.xyz) if you would like to move to a different communication channel for this. Alternatively let me know of anything else I can do.

deekerman commented

Author

Owner

@iBobik commented on GitHub (Mar 13, 2023):

This fork works and has updated releases:
https://github.com/yt-dlp/yt-dlp

@iBobik commented on GitHub (Mar 13, 2023): This fork works and has updated releases: https://github.com/yt-dlp/yt-dlp

deekerman commented

From https://github.com/ytdl-org/youtube-dl/issues/31530#issuecomment-1472127323

Author

Owner

@dirkf commented on GitHub (Mar 16, 2023):

Just to show that a release might appear quite soon, and not in the far future.

... I was wonder why so big deal to create new release. I came across to #30568 and #31535 and I hope I understand. @dirkf decided to support old unsupported versions of python and probably this is reason why is not possible to use continuous integration to build and release new version.

No, although it's true that the GH workflow ecosystem forces you into an upgrade race as various dependencies are moved to new versions, much as in the rest of the s/w world. Actually, for yt-dl itself, Py2.7 is fine and dandy for the most part: the #31530 fix did reveal some 2.6 hold-outs but also surprisingly many verbose logs showing 2.7 still in use.

The previous build and release system, not well documented, depended on some local setup that had been passed down through the former maintainers. Lured by the gleaming promise of online builds, I completely failed to reproduce the necessary local setup. That brings us to last month.

Now, I find that GH is about to junk ("deprecated") versions of Linux that support the Pythons needed for yt-dl's CI testing. So we'll have to rework that completely using our own runners, etc. This is too hard for now and the existing tests will have to do: boo to Python >= 3.10 (though I don't expect significant issues there). Therefore the build and release workflow doesn't depend on the test workflow.

I took the yt-dlp scheme as a pattern, as suggested, rather than building directly on the excellent work in #30644.

Thus build.yml is a simplified script merging the one from #30644 with the yt-dlp-derived version, removing the bits that are handled elsewhere in the yt-dlp-derived workflows. For now, only the existing targets will be built, ie: POSIX (Linux/macOS/BSD/etc) self-extracting executable, matching .tar.gz, Windows self-extracting executable with built-in Python 3.4.4. As the unofficial nightly builds seem unproblematic I don't expect to see a lot of build issues.

Users of new macOSes that have no Python will still have to install a Python and arrange for it to be found by the shebang of the POSIX build.

Currently a tricky part is make_changelog.py. yt-dl wants a plain-text changelog, but the Changelog class in the script can be specialised and the punch-line becomes:

print((Changelog if args.markdown else PlainChangelog)(commits.groups(), args.repo))

Less simple, the 200 commits since the last release aren't as disciplined as the yt-dlp script expected. I think the only practical design for grouping these commits automatically is to use git commands to identify what each commit changed and then map back from there. For instance:

git log --format= --dirstat=files commit gives a count of changed files by directory that can be sorted to pick the significant directory, eg if the top directory in the commit is youtube_dl/extractors, the group should be CommitGroup.EXTRACTOR
git log --format= --numstat commit gives a count of lines changed in each file that can be sorted to give the most changed file (should be relative to the line count of the file) and the unsuffixed basename of that file is a useful input to classification.

Given this, the solution is to take some output of an as-good-as-can-be-expected hack on the existing script and modify the CI process to accept some manually tweaked changelog delta that will be prepended with its version ID to the ChangeLog file and uploaded in the release notes.

Obviously any other suggestions would be happily considered.

@dirkf commented on GitHub (Mar 16, 2023): Just to show that a release might appear quite soon, and not in the far future. From https://github.com/ytdl-org/youtube-dl/issues/31530#issuecomment-1472127323 >... I was wonder why so big deal to create new release. I came across to #30568 and #31535 and I hope I understand. @dirkf decided to support old unsupported versions of python and probably this is reason why is not possible to use continuous integration to build and release new version. No, although it's true that the GH workflow ecosystem forces you into an upgrade race as various dependencies are moved to new versions, much as in the rest of the s/w world. Actually, for yt-dl itself, Py2.7 is fine and dandy for the most part: the #31530 fix did reveal some 2.6 hold-outs but also surprisingly many verbose logs showing 2.7 still in use. The previous build and release system, not well documented, depended on some local setup that had been passed down through the former maintainers. Lured by the gleaming promise of online builds, I completely failed to reproduce the necessary local setup. That brings us to last month. Now, I find that GH is about to junk ("deprecated") versions of Linux that support the Pythons needed for yt-dl's CI testing. So we'll have to rework that completely using our own runners, etc. This is too hard for now and the existing tests will have to do: boo to Python >= 3.10 (though I don't expect significant issues there). Therefore the build and release workflow doesn't depend on the test workflow. I took the yt-dlp scheme as a pattern, as [suggested](https://github.com/ytdl-org/youtube-dl/issues/31585#issuecomment-1437817585), rather than building directly on the excellent work in #30644. Thus `build.yml` is a simplified script merging the one from #30644 with the yt-dlp-derived version, removing the bits that are handled elsewhere in the yt-dlp-derived workflows. For now, only the existing targets will be built, ie: POSIX (Linux/macOS/BSD/etc) self-extracting executable, matching .tar.gz, Windows self-extracting executable with built-in Python 3.4.4. As the unofficial nightly builds seem unproblematic I don't expect to see a lot of build issues. Users of new macOSes that have no Python will still have to install a Python and arrange for it to be found by the shebang of the POSIX build. Currently a tricky part is `make_changelog.py`. yt-dl wants a plain-text changelog, but the `Changelog` class in the script can be specialised and the punch-line becomes: ```py print((Changelog if args.markdown else PlainChangelog)(commits.groups(), args.repo)) ``` Less simple, the 200 commits since the last release aren't as disciplined as the yt-dlp script expected. I think the only practical design for grouping these commits automatically is to use _git_ commands to identify what each commit changed and then map back from there. For instance: * `git log --format= --dirstat=files commit` gives a count of changed files by directory that can be sorted to pick the significant directory, eg if the top directory in the commit is `youtube_dl/extractors`, the group should be `CommitGroup.EXTRACTOR` * `git log --format= --numstat commit` gives a count of lines changed in each file that can be sorted to give the most changed file (should be relative to the line count of the file) and the unsuffixed basename of that file is a useful input to classification. Given this, the solution is to take some output of an as-good-as-can-be-expected hack on the existing script and modify the CI process to accept some manually tweaked changelog delta that will be prepended with its version ID to the `ChangeLog` file and uploaded in the release notes. Obviously any other suggestions would be happily considered.

deekerman commented

Author

Owner

@ReenigneArcher commented on GitHub (Mar 16, 2023):

Good to hear there is a path forward!

Now, I find that GH is about to junk ("deprecated") versions of Linux

Yes, I believe the 18.04 runner is being removed in April if I remember correctly. I assume you're aware of this action to setup Python? https://github.com/actions/setup-python ... Example: https://github.com/LizardByte/Themerr-plex/blob/master/.github/workflows/CI.yml#L36-L49

Ubuntu 20.04 and 22.04 also still have python 2.7 that can be manually installed. Here's a build in an Ubuntu 22.04 dockerfile that uses Python 2.7 for reference: https://github.com/LizardByte/PlexyGlass/blob/nightly/Dockerfile#L4-L24 ... Debian/Ubuntu has too many packages that still rely on Python 2.x to completely remove it.

I didn't see a mention of publishing to PyPi in your comment. Are you still going to publish there?

@ReenigneArcher commented on GitHub (Mar 16, 2023): Good to hear there is a path forward! > Now, I find that GH is about to junk ("deprecated") versions of Linux Yes, I believe the 18.04 runner is being removed in April if I remember correctly. I assume you're aware of this action to setup Python? https://github.com/actions/setup-python ... Example: https://github.com/LizardByte/Themerr-plex/blob/master/.github/workflows/CI.yml#L36-L49 Ubuntu 20.04 and 22.04 also still have python 2.7 that can be manually installed. Here's a build in an Ubuntu 22.04 dockerfile that uses Python 2.7 for reference: https://github.com/LizardByte/PlexyGlass/blob/nightly/Dockerfile#L4-L24 ... Debian/Ubuntu has too many packages that still rely on Python 2.x to completely remove it. I didn't see a mention of publishing to PyPi in your comment. Are you still going to publish there?

deekerman commented

Author

Owner

@Grub4K commented on GitHub (Mar 16, 2023):

Is there a new PR, some branch or similar that I can take a look at and implement missing factors to help out?

I think the only practical design for grouping these commits automatically is to use git commands to identify what each commit changed and then map back from there.

This is why I suggested shimming it for now, as manual mapping will be required anyways. Regarding future commits, it might make sense to adjust the commit message to be closer to dlp format OR (I thought about it before but scrapped it since its easier to just rely on message only) search for the files using file or text search if the format should be kept the same ([YouTube] => extractor/youtube.py => EXTRACTOR). Let me know of any changes to that script that I could do.

I didn't see a mention of publishing to PyPi in your comment. Are you still going to publish there?

Assuming that this is building on the release workflow of dlp the PyPI publish is integrated in those steps

It might make sense to diff with dlp build workflow as some new changes were coming. A relevant branch can be found here (though many changes were directly related to nightly and --update-to and are therefore irrelevant)

@Grub4K commented on GitHub (Mar 16, 2023): Is there a new PR, some branch or similar that I can take a look at and implement missing factors to help out? > I think the only practical design for grouping these commits automatically is to use git commands to identify what each commit changed and then map back from there. This is why I suggested shimming it for now, as manual mapping will be required anyways. Regarding future commits, it might make sense to adjust the commit message to be closer to dlp format OR (I thought about it before but scrapped it since its easier to just rely on message only) search for the files using file or text search if the format should be kept the same (`[YouTube]` => `extractor/youtube.py` => `EXTRACTOR`). Let me know of any changes to that script that I could do. > I didn't see a mention of publishing to PyPi in your comment. Are you still going to publish there? Assuming that this is building on the release workflow of dlp the PyPI publish is integrated in those steps It might make sense to diff with dlp build workflow as some new changes were coming. A relevant branch can be found [here](https://github.com/Grub4K/yt-dlp/tree/cp/releases-stuff) (though many changes were directly related to nightly and `--update-to` and are therefore irrelevant)

deekerman commented

Author

Owner

@MikeMcQuaid commented on GitHub (Mar 18, 2023):

Homebrew project leader here 👋🏻. I think even getting a tagged prerelease here with failing CI will allow Homebrew to update to a version that will work for downloading from YouTube and stop you getting so many issues filed by our users.

@MikeMcQuaid commented on GitHub (Mar 18, 2023): Homebrew project leader here 👋🏻. I think even getting a tagged prerelease here with failing CI will allow Homebrew to update to a version that will work for downloading from YouTube and stop you getting so many issues filed by our users.

deekerman commented

Author

Owner

@dirkf commented on GitHub (Mar 18, 2023):

Thanks for your interest!

Almost any commit to master is meant to be reliable and its CI test will have succeeded, having been exercised on another branch first.

If you would like a special tag applied to some commit, that's easy. If you're able to take an unofficial nightly build (from a downstream), which has the advantage of reporting a different version and showing a disclaimer, that's even easier.

@dirkf commented on GitHub (Mar 18, 2023): Thanks for your interest! Almost any commit to master is meant to be reliable and its CI test will have succeeded, having been exercised on another branch first. If you would like a special tag applied to some commit, that's easy. If you're able to take an unofficial nightly build (from a downstream), which has the advantage of reporting a different version and showing a disclaimer, that's even easier.

deekerman commented

Author

Owner

@dirkf commented on GitHub (Mar 18, 2023):

Is there a new PR, some branch or similar that I can take a look at and implement missing factors to help out?

Upcoming.

This is why I suggested shimming it for now, as manual mapping will be required anyways. ...

I suppose that a manually adjusted changelog delta could be offered to the workflow as a var, even if it's ~15kB?

I didn't see a mention of publishing to PyPi in your comment. Are you still going to publish there?

Assuming that this is building on the release workflow of dlp the PyPI publish is integrated in those steps

For now I commented out the package management interfaces. The existing PyPi update process would have been manual AFAIK with any account details held by former maintainers; presumably we can regenerate them if they can't be recalled.

It might make sense to diff with dlp build workflow as some new changes were coming. A relevant branch can be found here (though many changes were directly related to nightly and --update-to and are therefore irrelevant)

The workflow files

	new file:   .github/workflows/build.yml
	new file:   .github/workflows/publish.yml
	new file:   .github/workflows/release.yml
	new file:   devscripts/make_changelog.py
	modified:   devscripts/make_issue_template.py
	new file:   devscripts/update_formulae.py
	new file:   devscripts/update_version.py
	new file:   devscripts/utils.py

all started out from the yt-dlp equivalents in a meld session. So yes.

@dirkf commented on GitHub (Mar 18, 2023): > Is there a new PR, some branch or similar that I can take a look at and implement missing factors to help out? Upcoming. > This is why I suggested shimming it for now, as manual mapping will be required anyways. ... I suppose that a manually adjusted changelog delta could be offered to the workflow as a var, even if it's ~15kB? > > I didn't see a mention of publishing to PyPi in your comment. Are you still going to publish there? > > Assuming that this is building on the release workflow of dlp the PyPI publish is integrated in those steps For now I commented out the package management interfaces. The existing PyPi update process would have been manual AFAIK with any account details held by former maintainers; presumably we can regenerate them if they can't be recalled. > It might make sense to diff with dlp build workflow as some new changes were coming. A relevant branch can be found [here](https://github.com/Grub4K/yt-dlp/tree/cp/releases-stuff) (though many changes were directly related to nightly and `--update-to` and are therefore irrelevant) The workflow files ``` new file: .github/workflows/build.yml new file: .github/workflows/publish.yml new file: .github/workflows/release.yml new file: devscripts/make_changelog.py modified: devscripts/make_issue_template.py new file: devscripts/update_formulae.py new file: devscripts/update_version.py new file: devscripts/utils.py ``` all started out from the yt-dlp equivalents in a [_meld_](https://tooling.bennuttall.com/meld/) session. So yes.

deekerman commented

Author

Owner

@MikeMcQuaid commented on GitHub (Mar 19, 2023):

Thanks for your interest!

Thanks for your great software that I personally use!

If you would like a special tag applied to some commit, that's easy.

Yes, our hope would be any sort of tag applied to a release that would show up in https://github.com/ytdl-org/youtube-dl/releases that we could update to 🙇🏻

@MikeMcQuaid commented on GitHub (Mar 19, 2023): > Thanks for your interest! Thanks for your great software that I personally use! > If you would like a special tag applied to some commit, that's easy. Yes, our hope would be any sort of tag applied to a release that would show up in https://github.com/ytdl-org/youtube-dl/releases that we could update to 🙇🏻

deekerman commented

Author

Owner

@dirkf commented on GitHub (Mar 19, 2023):

Which release asset is used to create the Homebrew package?

@dirkf commented on GitHub (Mar 19, 2023): Which release asset is used to create the Homebrew package?

deekerman commented

Author

Owner

@MikeMcQuaid commented on GitHub (Mar 19, 2023):

The pypi URL: github.com/Homebrew/homebrew-core@a44ce7e234/Formula/youtube-dl.rb (L6)

@MikeMcQuaid commented on GitHub (Mar 19, 2023): The pypi URL: https://github.com/Homebrew/homebrew-core/blob/a44ce7e234fddf054ba589cc682d73d9d8499771/Formula/youtube-dl.rb#L6

deekerman commented

Author

Owner

@Grub4K commented on GitHub (Mar 19, 2023):

Which release asset is used to create the Homebrew package?

This is also reflected in the publish_pypi_homebrew job from the yt-dlp release workflow. It publishes to PyPI, then updates the respective Ruby script in the homebrew-taps repo using its wheels.

A PyPI publish only requires the PYPI_TOKEN secret on the repo to be updated and then should function automatically (this can be tested on test.pypi.org, like we did using yt-dlp-test). Certain changes might be required for adjusting readme and similar meta files.

@Grub4K commented on GitHub (Mar 19, 2023): > Which release asset is used to create the Homebrew package? This is also reflected in the `publish_pypi_homebrew` job from the yt-dlp release workflow. It publishes to PyPI, then updates the respective Ruby script in the `homebrew-taps` repo using its wheels. A PyPI publish only requires the `PYPI_TOKEN` secret on the repo to be updated and then should function automatically (this can be tested on [test.pypi.org](https://test.pypi.org), like we did using [yt-dlp-test](https://test.pypi.org/project/yt-dlp-test/)). Certain changes might be required for adjusting readme and similar meta files.

deekerman commented

Author

Owner

@dirkf commented on GitHub (Mar 19, 2023):

In that case the answer is either linking something like this

url "https://github.com/ytdl-patched/youtube-dl/releases/download/2023.03.19.43044/youtube-dl-2023.03.19.43044.tar.gz"
sha256 "56888be194b5c1f2f54b527a31d93fb34719d3f0f8265f7bc7d2d98c870d1858"

in the formula, or using --head.

@dirkf commented on GitHub (Mar 19, 2023): In that case the answer is either linking something like this ``` url "https://github.com/ytdl-patched/youtube-dl/releases/download/2023.03.19.43044/youtube-dl-2023.03.19.43044.tar.gz" sha256 "56888be194b5c1f2f54b527a31d93fb34719d3f0f8265f7bc7d2d98c870d1858" ``` in the formula, or using `--head`.

deekerman commented

Author

Owner

@MikeMcQuaid commented on GitHub (Mar 20, 2023):

@dirkf We're unlikely to change our upstream to a repo titled "Unofficial daily builds for youtube-dl". Our policy (not for youtube-dl but for all projects) is to require tagged (or otherwise versioned) official upstream releases. --head is not a good solution for our users; it's slower, untested by us and more error-prone so we don't support any failures that result from using it.

@MikeMcQuaid commented on GitHub (Mar 20, 2023): @dirkf We're unlikely to change our upstream to a repo titled "Unofficial daily builds for youtube-dl". Our policy (not for youtube-dl but for all projects) is to require tagged (or otherwise versioned) official upstream releases. `--head` is not a good solution for our users; it's slower, untested by us and more error-prone so we don't support any failures that result from using it.

deekerman commented

Author

Owner

@DmytroUsenko commented on GitHub (Mar 25, 2023):

@dirkf hey, do you have approx possible terms for the new release?

@DmytroUsenko commented on GitHub (Mar 25, 2023): @dirkf hey, do you have approx possible terms for the new release?

deekerman commented

Author

Owner

@daald commented on GitHub (Mar 27, 2023):

I didn't read everything here, but how about having just a patch release 2021.12.17.2b or so from a ad-hoc branch with minimal changes? For me it's kind of a blocker that the tool regularly fails with youtube links (#31530).
Patch releases are quite standard procedure for other projects. And note that even distribution packages like ubuntu's depend on these official releases

@daald commented on GitHub (Mar 27, 2023): I didn't read everything here, but how about having just a patch release 2021.12.17.2b or so from a ad-hoc branch with minimal changes? For me it's kind of a blocker that the tool regularly fails with youtube links (#31530). Patch releases are quite standard procedure for other projects. And note that even distribution packages like ubuntu's depend on these official releases

deekerman commented

Author

Owner

@ReenigneArcher commented on GitHub (Apr 16, 2023):

Just to show that a release might appear quite soon, and not in the far future.

Any update on getting a release published. It's been one month since this comment, and 2 months since the latest release is unusable (https://github.com/ytdl-org/youtube-dl/issues/31530).

Like many others it's not feasible for me to install from git when using this library as a dependency in other projects.

Almost any commit to master is meant to be reliable

If this is the case, perhaps you should automate GitHub and PyPi releases on push events.

... Anything you need help with specifically to get automated releases working? Or this is just not a priority for this project?

@ReenigneArcher commented on GitHub (Apr 16, 2023): > Just to show that a release might appear quite soon, and not in the far future. Any update on getting a release published. It's been one month since this comment, and 2 months since the latest release is unusable (https://github.com/ytdl-org/youtube-dl/issues/31530). Like many others it's not feasible for me to install from git when using this library as a dependency in other projects. > Almost any commit to master is meant to be reliable If this is the case, perhaps you should automate GitHub and PyPi releases on push events. ... Anything you need help with specifically to get automated releases working? Or this is just not a priority for this project?

deekerman commented

Author

Owner

@djjudas21 commented on GitHub (Apr 17, 2023):

Just to weigh in on the release cycle debate... I'm a devops engineer and so I do a lot of stuff with pipelines and releases. At work, it's important to pin my dependencies (usually libraries). But with an app like youtube-dl I'm just a user, I'm not in work mode. I install it via brew and I want it to "just work" when updates are available via package manager. I don't want to be backporting fixes or building my own version. So a patch release would be great, just to fix the current problem with YouTube. Thanks for your efforts

@djjudas21 commented on GitHub (Apr 17, 2023): Just to weigh in on the release cycle debate... I'm a devops engineer and so I do a lot of stuff with pipelines and releases. At work, it's important to pin my dependencies (usually libraries). But with an app like `youtube-dl` I'm just a user, I'm not in work mode. I install it via `brew` and I want it to "just work" when updates are available via package manager. I don't want to be backporting fixes or building my own version. So a patch release would be great, just to fix the current problem with YouTube. Thanks for your efforts

deekerman commented

Author

Owner

@TinaRussell commented on GitHub (May 16, 2023):

Just a heads-up: Google plans to start deleting inactive accounts later this year, and that includes YouTube content. https://www.blog.google/technology/safety-security/updating-our-inactive-account-policies/ So, saving historic videos from YouTube is now more important than ever. If we can get a new release pushed soon that works with YouTube, that would be great. Thanks for all your work, everyone!

@TinaRussell commented on GitHub (May 16, 2023): Just a heads-up: Google plans to start deleting inactive accounts later this year, and that includes YouTube content. https://www.blog.google/technology/safety-security/updating-our-inactive-account-policies/ So, saving historic videos from YouTube is now more important than ever. If we can get a new release pushed soon that works with YouTube, that would be great. Thanks for all your work, everyone!

deekerman commented

Author

Owner

@adeeb1 commented on GitHub (May 19, 2023):

Just a heads-up: Google plans to start deleting inactive accounts later this year, and that includes YouTube content. https://www.blog.google/technology/safety-security/updating-our-inactive-account-policies/ So, saving historic videos from YouTube is now more important than ever. If we can get a new release pushed soon that works with YouTube, that would be great. Thanks for all your work, everyone!

I agree with archiving historic videos. Just wanted to point out that this information is incorrect, though. In the announcement linked, it includes this sentence:

Additionally, we do not have plans to delete accounts with YouTube videos at this time.

So YouTube content is still safe.

@adeeb1 commented on GitHub (May 19, 2023): > Just a heads-up: Google plans to start deleting inactive accounts later this year, and that includes YouTube content. https://www.blog.google/technology/safety-security/updating-our-inactive-account-policies/ So, saving historic videos from YouTube is now more important than ever. If we can get a new release pushed soon that works with YouTube, that would be great. Thanks for all your work, everyone! I agree with archiving historic videos. Just wanted to point out that this information is incorrect, though. In the announcement linked, it includes this sentence: >Additionally, we do not have plans to delete accounts with YouTube videos at this time. So YouTube content is still safe.

deekerman commented

Author

Owner

@DianaNites commented on GitHub (May 19, 2023):

It wasn't incorrect at the time, that was a later addition, as noted at the end

We've updated this post to clarify how this announcement impacts creator accounts and accounts with YouTube videos.

and as visible archived

In fact they explictly said they would delete youtube

To reduce this risk, we are updating our inactivity policy for Google Accounts to 2 years across our products. Starting later this year, if a Google Account has not been used or signed into for at least 2 years, we may delete the account and its contents – including content within Google Workspace (Gmail, Docs, Drive, Meet, Calendar), YouTube and Google Photos.

@DianaNites commented on GitHub (May 19, 2023): It wasn't incorrect at the time, that was a later addition, as noted at the end > We've updated this post to clarify how this announcement impacts creator accounts and accounts with YouTube videos. and as visible [archived](https://web.archive.org/web/20230516150516/https://www.blog.google/technology/safety-security/updating-our-inactive-account-policies/) In fact they *explictly* said they would delete youtube > To reduce this risk, we are updating our inactivity policy for Google Accounts to 2 years across our products. Starting later this year, if a Google Account has not been used or signed into for at least 2 years, we may delete the account and its contents – including content within Google Workspace (Gmail, Docs, Drive, Meet, Calendar), **YouTube** and Google Photos.

deekerman commented

Author

Owner

@steckerhalter commented on GitHub (Jun 10, 2023):

I have been using youtube-dl for a long time and now it's just broken. Yeah, I can use it via Git but it's complicated and I'd like to just be able to use it. I suggest people switch to https://github.com/unrud/video-downloader
Why not just release a new version so it's at least not broken for everybody?
Thanks

@steckerhalter commented on GitHub (Jun 10, 2023): I have been using youtube-dl for a long time and now it's just broken. Yeah, I can use it via Git but it's complicated and I'd like to just be able to use it. I suggest people switch to https://github.com/unrud/video-downloader Why not just release a new version so it's at least not broken for everybody? Thanks

deekerman commented

Author

Owner

@dirkf commented on GitHub (Jul 2, 2023):

While the build and release workflow for the main repo is under construction, we have this, thanks @Lesmiscore: https://github.com/ytdl-org/ytdl-nightly/releases.

@dirkf commented on GitHub (Jul 2, 2023): While the build and release workflow for the main repo is under construction, we have this, thanks @Lesmiscore: https://github.com/ytdl-org/ytdl-nightly/releases.

deekerman commented

Author

Owner

@daald commented on GitHub (Jul 3, 2023):

btw here is the "patch release" bug report for Ubuntu: https://bugs.launchpad.net/ubuntu/+source/youtube-dl/+bug/2008009

@daald commented on GitHub (Jul 3, 2023): btw here is the "patch release" bug report for Ubuntu: https://bugs.launchpad.net/ubuntu/+source/youtube-dl/+bug/2008009

deekerman commented

https://github.com/ytdl-org/ytdl-nightly/releases/download/2023.07.03.19419/youtube-dl.exe

Author

Owner

@Vangelis66 commented on GitHub (Jul 4, 2023):

we have this, thanks @Lesmiscore:

https://github.com/ytdl-org/ytdl-nightly/releases.

As an old-time an avid yt-dl plain user, it's not a hyperbole to say I'm elated over this new development 🎉 ...

As a yt-dl tester though, and without any trace of entitlement/ungratefulness 😉 , I have to point out the results of my nit-picking:

Like the ytdl-patched/youtube-dl "daily" releases, this one is also built without native crypto support 😞 ; I have detailed this "problem" in the relevant PR #30644, specifically in comments 1, 2, 3; since the building environment is py3.4-based, it may simply be a case of

python -m pip install -U setuptools
python -m pip install -U pycrypto

however, in that case, VS2010 (32-bit?) also has to be pre-installed, so that GHA can build the binary parts of pycrypto (.pyd files) under Windows; or simply deploy:

python -m pip install "https://web.archive.org/web/20200627032153/http://www.voidspace.org.uk/python/pycrypto-2.6.1/pycrypto-2.6.1-cp34-none-win32.whl"

Currently, it's not easily evident the master branch code snapshot the ytdl-nightly binary is built from (unlike the nightly "downstream" builds, e.g.
[debug] yt-dlp version nightly@2023.07.03.104728 [4dc4d8473] (win_x86_exe)); for the latest nightly version, 2023.07.04.114514, I had to visually compare
https://github.com/ytdl-org/youtube-dl/commits/master
to
https://github.com/ytdl-org/ytdl-nightly/commits/master
to establish that nightly v2023.07.04.114514 reflects master branch code snapshot fa7f0effbe ...
Despite the rebrand to "nightly", --verbose mode still speaks of "unofficial daily" builds:

youtube-dl -v

[debug] System config: []
[debug] User config: []
[debug] Custom config: []
[debug] Command-line args: ['-v']
[debug] Encodings: locale cp1253, fs mbcs, out cp737, pref cp1253
[debug] youtube-dl version 2023.07.04.810 (single file build)
** This build is unofficial daily builds, provided for ease of use.
** Please do not ask for any support.
[debug] Python 3.4.4 (CPython x86 32bit) - Windows-Vista-6.0.6003-SP2 - OpenSSL 1.0.2d 9 Jul 2015
[debug] exe versions: none
[debug] Proxy map: {}
Usage: youtube-dl [OPTIONS] URL [URL...]

youtube-dl: error: You must provide at least one URL.
Type youtube-dl --help to see a list of all options.

The ytdl-nightly builds are still on the same update channel as the fork they "came" from ; STR:
a. Download previous ytdl-nightly binary release, i.e. v2023.07.03.19419:

b. Issue youtube-dl -vU:

debug] System config: []
[debug] User config: []
[debug] Custom config: []
[debug] Command-line args: ['-vU']
[debug] Encodings: locale cp1253, fs mbcs, out cp737, pref cp1253
[debug] youtube-dl version 2023.07.03.19419 (single file build)
** This build is unofficial daily builds, provided for ease of use.
** Please do not ask for any support.
[debug] Python 3.4.4 (CPython x86 32bit) - Windows-Vista-6.0.6003-SP2 - OpenSSL 1.0.2d 9 Jul 2015
[debug] exe versions: none
[debug] Proxy map: {}
Latest version: 2023.07.04.810, Current version: 2023.07.03.19419
Current Build Hash f6dfe7cff1211ada6e63b982c0dc429e5d3fc22b54cb7744087879ea80743fbe
Updating to version 2023.07.04.810 ...
Updated youtube-dl to version 2023.07.04.810

However, v2023.07.04.810 is the latest release on "ytdl-patched/youtube-dl":

https://github.com/ytdl-patched/youtube-dl/releases/tag/2023.07.04.810

NOT the latest "ytdl-org/ytdl-nightly" one, which is v2023.07.04.114514:

https://github.com/ytdl-org/ytdl-nightly/releases/tag/2023.07.04.114514

Incidentally, the "updated" build v2023.07.04.810 contains stale yt-dl code, thus the "update" is actually a "downgrade" 😉 ...

I propose that the ytdl-nightly update mechanism follow what is currently being implemented in the yt-dlp-nightly update channel: Build workflow runs should be triggered only when the actual master code gets updated; in the ytdl-patched/youtube-dl configuration, we had consecutive daily builds irrespective of yt-dl's master branch "activity": if no commit was pushed for, say, a period of a month, ytdl-patched/youtube-dl would have produced 30 (!) "daily" builds, all built on identical source code... Perhaps pukkandan 😄 could be kind enough to lend his GHA expertise to that goal..

In conclusion, please take my suggestions above as incentive to improve the ytdl-nightly "service" itself, not as unreasonable demands raised by an entitled non-coder 😜 ; far from that...

Kindest regards 😄 !

@Vangelis66 commented on GitHub (Jul 4, 2023): > we have this, thanks @Lesmiscore: > > https://github.com/ytdl-org/ytdl-nightly/releases. As an old-time an avid `yt-dl` plain user, it's not a hyperbole to say I'm elated over this new development 🎉 ... As a `yt-dl` **tester** though, and without any trace of entitlement/ungratefulness 😉 , I have to point out the results of my nit-picking: 1. Like the `ytdl-patched/youtube-dl` "daily" releases, this one is also built **without native crypto** support 😞 ; I have detailed this "problem" in the relevant PR #30644, specifically in comments [1](https://github.com/ytdl-org/youtube-dl/pull/30644#issuecomment-1436180682), [2](https://github.com/ytdl-org/youtube-dl/pull/30644#issuecomment-1436264475), [3](https://github.com/ytdl-org/youtube-dl/pull/30644#issuecomment-1449168914); since the building environment is **py3.4**-based, it may simply be a case of ```shell python -m pip install -U setuptools python -m pip install -U pycrypto ``` however, in that case, **VS2010** (32-bit?) also has to be pre-installed, so that GHA can build the binary parts of pycrypto (`.pyd` files) under Windows; or simply deploy: ```shell python -m pip install "https://web.archive.org/web/20200627032153/http://www.voidspace.org.uk/python/pycrypto-2.6.1/pycrypto-2.6.1-cp34-none-win32.whl" ``` 2. Currently, it's not easily evident the `master` branch code snapshot the `ytdl-nightly` binary is built from (unlike the nightly "downstream" builds, e.g. `[debug] yt-dlp version nightly@2023.07.03.104728 [4dc4d8473] (win_x86_exe)`); for the latest nightly version, `2023.07.04.114514`, I had to visually compare https://github.com/ytdl-org/youtube-dl/commits/master to https://github.com/ytdl-org/ytdl-nightly/commits/master to establish that nightly `v2023.07.04.114514` reflects `master` branch code snapshot fa7f0effbe4e14fcf70e1dc4496371c9862b64b9 ... 3. Despite the rebrand to "nightly", `--verbose` mode still speaks of "unofficial daily" builds: ```shell youtube-dl -v [debug] System config: [] [debug] User config: [] [debug] Custom config: [] [debug] Command-line args: ['-v'] [debug] Encodings: locale cp1253, fs mbcs, out cp737, pref cp1253 [debug] youtube-dl version 2023.07.04.810 (single file build) ** This build is unofficial daily builds, provided for ease of use. ** Please do not ask for any support. [debug] Python 3.4.4 (CPython x86 32bit) - Windows-Vista-6.0.6003-SP2 - OpenSSL 1.0.2d 9 Jul 2015 [debug] exe versions: none [debug] Proxy map: {} Usage: youtube-dl [OPTIONS] URL [URL...] youtube-dl: error: You must provide at least one URL. Type youtube-dl --help to see a list of all options. ``` 4. The `ytdl-nightly` builds are still **on the same update channel as the fork they "came" from** ; STR: a. Download previous `ytdl-nightly` binary release, i.e. `v2023.07.03.19419`: https://github.com/ytdl-org/ytdl-nightly/releases/download/2023.07.03.19419/youtube-dl.exe b. Issue `youtube-dl -vU`: ```shell debug] System config: [] [debug] User config: [] [debug] Custom config: [] [debug] Command-line args: ['-vU'] [debug] Encodings: locale cp1253, fs mbcs, out cp737, pref cp1253 [debug] youtube-dl version 2023.07.03.19419 (single file build) ** This build is unofficial daily builds, provided for ease of use. ** Please do not ask for any support. [debug] Python 3.4.4 (CPython x86 32bit) - Windows-Vista-6.0.6003-SP2 - OpenSSL 1.0.2d 9 Jul 2015 [debug] exe versions: none [debug] Proxy map: {} Latest version: 2023.07.04.810, Current version: 2023.07.03.19419 Current Build Hash f6dfe7cff1211ada6e63b982c0dc429e5d3fc22b54cb7744087879ea80743fbe Updating to version 2023.07.04.810 ... Updated youtube-dl to version 2023.07.04.810 ``` However, `v2023.07.04.810` is the latest release on "ytdl-patched/youtube-dl": https://github.com/ytdl-patched/youtube-dl/releases/tag/2023.07.04.810 NOT the latest "ytdl-org/ytdl-nightly" one, which is `v2023.07.04.114514`: https://github.com/ytdl-org/ytdl-nightly/releases/tag/2023.07.04.114514 Incidentally, the "updated" build `v2023.07.04.810` contains stale `yt-dl` code, thus the "update" is actually a "downgrade" 😉 ... 5. I propose that the `ytdl-nightly` update mechanism follow what is currently being implemented in the `yt-dlp-nightly` update channel: `Build` workflow runs should be triggered only when the actual `master` code gets updated; in the `ytdl-patched/youtube-dl` configuration, we had consecutive **daily** builds irrespective of `yt-dl`'s `master` branch "activity": if no commit was pushed for, say, a period of a month, `ytdl-patched/youtube-dl` would have produced 30 (!) "daily" builds, all built on **identical source code**... Perhaps **pukkandan** 😄 could be kind enough to lend his GHA expertise to that goal.. In conclusion, please take my suggestions above as incentive to **improve** the `ytdl-nightly` "service" itself, not as unreasonable demands raised by an entitled non-coder 😜 ; far from that... Kindest regards 😄 !

deekerman commented

Author

Owner

@dirkf commented on GitHub (Jul 4, 2023):

These are useful points. I was already considering item 5. It's fair to say that the "not completely trivial" work is not yet fully finished.

@dirkf commented on GitHub (Jul 4, 2023): These are useful points. I was already considering item 5. It's fair to say that the "not completely trivial" work is not yet fully finished.

deekerman commented

Author

Owner

@daald commented on GitHub (Jul 4, 2023):

These are useful points. I was already considering item 5. It's fair to say that the "not completely trivial" work is not yet fully finished.

clearly, for a daily release, there's more work to do.
for a simple patch of 2021.12.17, only two lines have to be changed.

@daald commented on GitHub (Jul 4, 2023): > These are useful points. I was already considering item 5. It's fair to say that the "not completely trivial" work is not yet fully finished. clearly, for a daily release, there's more work to do. for a simple patch of 2021.12.17, only two lines have to be changed.

deekerman commented

Author

Owner

@dirkf commented on GitHub (Jul 4, 2023):

No, that's now pointless. The master code fixes several more recent issues that prevent or obstruct downloading from YouTube. You don't get to see those if your yt-dl crashes at the uploader id line. The published source is a good basis for packaging, which will remove any internal updating functionality.

@dirkf commented on GitHub (Jul 4, 2023): No, that's now pointless. The master code fixes several more recent issues that prevent or obstruct downloading from YouTube. You don't get to see those if your yt-dl crashes at the `uploader id` line. The published source is a good basis for packaging, which will remove any internal updating functionality.

deekerman commented

Author

Owner

@daald commented on GitHub (Jul 5, 2023):

@dirkf I'm not lying. It's just a different approach. It works for me since I fixed it that way. Try it yourself if you don't believe. I just skipped all that refactoring.


--- /usr/lib/python3/dist-packages/youtube_dl/extractor/youtube.py_orig	2021-12-16 19:47:24.000000000 +0100
+++ /usr/lib/python3/dist-packages/youtube_dl/extractor/youtube.py	2023-03-28 00:06:43.307464258 +0200
@@ -1791,7 +1791,7 @@
                 microformat.get('uploadDate')
                 or search_meta('uploadDate')),
             'uploader': video_details['author'],
-            'uploader_id': self._search_regex(r'/(?:channel|user)/([^/?&#]+)', owner_profile_url, 'uploader id') if owner_profile_url else None,
+            'uploader_id': self._search_regex(r'/(?:channel/|user/|@)([^/?&#]+)', owner_profile_url, 'uploader id') if owner_profile_url else None,
             'uploader_url': owner_profile_url,
             'channel_id': channel_id,
             'channel_url': 'https://www.youtube.com/channel/' + channel_id if channel_id else None,

--- /usr/lib/python3/dist-packages/youtube_dl/jsinterp.py_orig	2021-12-16 19:47:24.000000000 +0100
+++ /usr/lib/python3/dist-packages/youtube_dl/jsinterp.py	2023-06-19 00:47:39.399800340 +0200
@@ -215,7 +215,7 @@
         obj = {}
         obj_m = re.search(
             r'''(?x)
-                (?<!this\.)%s\s*=\s*{\s*
+                (?<!\.)%s\s*=\s*{\s*
                     (?P<fields>(%s\s*:\s*function\s*\(.*?\)\s*{.*?}(?:,\s*)?)*)
                 }\s*;
             ''' % (re.escape(objname), _FUNC_NAME_RE),

@daald commented on GitHub (Jul 5, 2023): @dirkf I'm not lying. It's just a different approach. It works for me since I fixed it that way. Try it yourself if you don't believe. I just skipped all that refactoring. ``` --- /usr/lib/python3/dist-packages/youtube_dl/extractor/youtube.py_orig 2021-12-16 19:47:24.000000000 +0100 +++ /usr/lib/python3/dist-packages/youtube_dl/extractor/youtube.py 2023-03-28 00:06:43.307464258 +0200 @@ -1791,7 +1791,7 @@ microformat.get('uploadDate') or search_meta('uploadDate')), 'uploader': video_details['author'], - 'uploader_id': self._search_regex(r'/(?:channel|user)/([^/?&#]+)', owner_profile_url, 'uploader id') if owner_profile_url else None, + 'uploader_id': self._search_regex(r'/(?:channel/|user/|@)([^/?&#]+)', owner_profile_url, 'uploader id') if owner_profile_url else None, 'uploader_url': owner_profile_url, 'channel_id': channel_id, 'channel_url': 'https://www.youtube.com/channel/' + channel_id if channel_id else None, --- /usr/lib/python3/dist-packages/youtube_dl/jsinterp.py_orig 2021-12-16 19:47:24.000000000 +0100 +++ /usr/lib/python3/dist-packages/youtube_dl/jsinterp.py 2023-06-19 00:47:39.399800340 +0200 @@ -215,7 +215,7 @@ obj = {} obj_m = re.search( r'''(?x) - (?<!this\.)%s\s*=\s*{\s* + (?<!\.)%s\s*=\s*{\s* (?P<fields>(%s\s*:\s*function\s*$.*?$\s*{.*?}(?:,\s*)?)*) }\s*; ''' % (re.escape(objname), _FUNC_NAME_RE), ```

deekerman commented

Author

Owner

@dirkf commented on GitHub (Jul 5, 2023):

You're not fixing the nsig throttling with those two patches. Also you're not extracting the uploader-related metadata properly. Also you're omitting 18 months of fixes and improvements for YouTube and other extractors. There's no reason not to use the master code from the daily/nightly release.

@dirkf commented on GitHub (Jul 5, 2023): You're not fixing the nsig throttling with those two patches. Also you're not extracting the uploader-related metadata properly. Also you're omitting 18 months of fixes and improvements for YouTube and other extractors. There's no reason not to use the master code from the daily/nightly release.

deekerman commented

Author

Owner

@daald commented on GitHub (Jul 5, 2023):

everything is ok if it reaches the distributions (Ubuntu in my case). I downloaded multiple hundred files with this patch without problems, the only few errors I had during this time were regional restrictions which are out of scope here.

I was talking about a patch. You are talking about a release. Both have their good reasons to exist. Patches are quick and lean, but address only blocking issues. Releases are complete and add new features, but potentially also new bugs - and they need more time.

95% of your users would be satisfied with the patch.

@daald commented on GitHub (Jul 5, 2023): everything is ok if it reaches the distributions (Ubuntu in my case). I downloaded multiple hundred files with this patch without problems, the only few errors I had during this time were regional restrictions which are out of scope here. I was talking about a patch. You are talking about a release. Both have their good reasons to exist. Patches are quick and lean, but address only blocking issues. Releases are complete and add new features, but potentially also new bugs - and they need more time. 95% of your users would be satisfied with the patch.

deekerman commented

Author

Owner

@pukkandan commented on GitHub (Jul 6, 2023):

https://github.com/ytdl-org/youtube-dl/issues/31585#issuecomment-1620615706

2,3,4 should be trivial to fix

Build workflow runs should be triggered only when the actual master code gets updated

The easiest solution is to trigger the workflow when a push is made to this repo. But that requires the workflow (but not necessarily the releases) to be moved to this repo. This is how yt-dlp nightlies work, and can be done with minimal changes to #30644

But assuming that is not an option, another solution would be to combine the rebase and release workflows as follows:

Save git rev-parse HEAD in a var, say OLD
rebase
Save git rev-parse HEAD in a var, say NEW
If NEW != OLD, create release

A third option is to use GH API to check the commitish of the last nightly release, but that's unnecessarily complex imo

@pukkandan commented on GitHub (Jul 6, 2023): > https://github.com/ytdl-org/youtube-dl/issues/31585#issuecomment-1620615706 2,3,4 should be trivial to fix > `Build` workflow runs should be triggered only when the actual `master` code gets updated The easiest solution is to trigger the workflow when a push is made to this repo. But that requires the workflow (but not necessarily the releases) to be moved to this repo. This is how yt-dlp nightlies work, and can be done with minimal changes to #30644 But assuming that is not an option, another solution would be to combine the rebase and release workflows as follows: 1. Save `git rev-parse HEAD` in a var, say OLD 2. rebase 3. Save `git rev-parse HEAD` in a var, say NEW 4. If NEW != OLD, create release A third option is to use GH API to check the commitish of the last nightly release, but that's unnecessarily complex imo

deekerman commented

Author

Owner

@pukkandan commented on GitHub (Jul 6, 2023):

Btw, you may want to link the nightly directly from https://github.com/ytdl-org/youtube-dl/issues/30839

@pukkandan commented on GitHub (Jul 6, 2023): Btw, you may want to link the nightly directly from https://github.com/ytdl-org/youtube-dl/issues/30839

deekerman commented

Author

Owner

@dirkf commented on GitHub (Jul 6, 2023):

Isn't that making it too easy ... ?

Thanks, the git rev-parse HEAD thing should be fine, once a couple of other pressing issues have been sorted.

@dirkf commented on GitHub (Jul 6, 2023): Isn't that making it too easy ... ? Thanks, the `git rev-parse HEAD` thing should be fine, once a couple of other pressing issues have been sorted.

deekerman commented

Author

Owner

@dirkf commented on GitHub (Jul 14, 2023):

Issues with nightly:

Windows native crypto support
Identify commit for build
Given item 5 (or even without), users can see the commit on the left of the Release panel, but that isn't always the commit from the main repo: so now the main repo HEAD commit SHA is being passed into the build workflow and added to youtube_dl/version.py as RELEASE_GIT_HEAD, and this is reported in the debug info, similar to yt-dlp.
--verbose mode still speaks of "unofficial daily" builds
Wrong update channel
Build workflow runs should be triggered only when the actual master code gets updated
The scheduled rebase workflow now calls the build workflow if the rebase changes the HEAD, and the build workflow is no longer separately scheduled.

Pls report any issues with 2, 4, 5.

@dirkf commented on GitHub (Jul 14, 2023): Issues with nightly: 1. [x] Windows native crypto support 2. [x] Identify commit for build Given item 5 (or even without), users can see the commit on the left of the Release panel, but that isn't always the commit from the main repo: so now the main repo HEAD commit SHA is being passed into the build workflow and added to `youtube_dl/version.py` as `RELEASE_GIT_HEAD`, and this is reported in the debug info, similar to yt-dlp. 3. [x] `--verbose` mode still speaks of "unofficial daily" builds 4. [x] Wrong update channel 5. [x] `Build` workflow runs should be triggered only when the actual master code gets updated The scheduled rebase workflow now calls the build workflow if the rebase changes the HEAD, and the build workflow is no longer separately scheduled. Pls report any issues with 2, 4, 5.

deekerman commented

Author

Owner

@dirkf commented on GitHub (Jul 15, 2023):

At https://github.com/ytdl-org/ytdl-nightly/releases/download/2023.07.15/youtube-dl-pycrypto.exe there is a Windows build that includes the archived PyCrypto wheel linked in #30644, but is otherwise identical. Testing welcome, even encouraged.

@dirkf commented on GitHub (Jul 15, 2023): At https://github.com/ytdl-org/ytdl-nightly/releases/download/2023.07.15/youtube-dl-pycrypto.exe there is a Windows build that includes the archived PyCrypto wheel linked in #30644, but is otherwise identical. Testing welcome, even encouraged.

deekerman commented

Author

Owner

@Vangelis66 commented on GitHub (Jul 15, 2023):

Pls report any issues with 2, 4, 5.

OT: Hi dirkf 😄 ; currently under extreme heatwave conditions here (daytime maximum temps of 39-41C, nighttime minimum temps of 31(!) C), so not optimal for software testing 😉 ; be that as it may...

Identify commit for build
Given item 5 (or even without), users can see the commit on the left of the Release panel, but that isn't always the commit from the main repo: so now the main repo HEAD commit SHA is being passed into the build workflow and added to youtube_dl/version.py as RELEASE_GIT_HEAD, and this is reported in the debug info, similar to yt-dlp.

You've done a splendid job there 👍 , here's my verbose log with latest Windows build:

youtube-dl -v => 

[debug] System config: []
[debug] User config: []
[debug] Custom config: []
[debug] Command-line args: ['-v']
[debug] Encodings: locale cp1253, fs mbcs, out cp1253, pref cp1253
[debug] youtube-dl version 2023.07.15 [f24bc9272] (single file build)
[debug] ** This version was built from the latest master code at https://github.com/ytdl-org/youtube-dl.
[debug] ** For support, visit the main site.
[debug] Python 3.4.4 (CPython x86 32bit) - Windows-Vista-6.0.6003-SP2 - OpenSSL 1.0.2d 9 Jul 2015
[debug] exe versions: none
[debug] Proxy map: {}
Usage: youtube-dl [OPTIONS] URL [URL...]

youtube-dl: error: You must provide at least one URL.
Type youtube-dl --help to see a list of all options.

Wrong update channel

... Most sadly, while this has been fixed "per se" (update channel is the right one now 😉 ), the "internal update mechanism" is currently BROKEN 😢 ...

STR:

Download previous Windows build 2023.07.14
Launch a Windows Command Prompt in the same working directory as the EXE resides
Issue youtube-dl -vU
Result:

[debug] System config: []
[debug] User config: []
[debug] Custom config: []
[debug] Command-line args: ['-vU']
[debug] Encodings: locale cp1253, fs mbcs, out cp737, pref cp1253
[debug] youtube-dl version 2023.07.14 [f24bc9272] (single file build)
[debug] ** This version was built from the latest master code at https://github.com/ytdl-org/youtube-dl.
[debug] ** For support, visit the main site.
[debug] Python 3.4.4 (CPython x86 32bit) - Windows-Vista-6.0.6003-SP2 - OpenSSL 1.0.2d 9 Jul 2015
[debug] exe versions: none
[debug] Proxy map: {}
Latest version: 2023.07.15, Current version: 2023.07.14
Current Build Hash 1dead7ccce0def131e11babebd5aa2a69a50a7af719348fa26654da691cba4ca
Updating to version 2023.07.15 ...
Traceback (most recent call last):
  File "__main__.py", line 19, in <module>
  File "D:\a\ytdl-nightly\ytdl-nightly\youtube_dl\__init__.py", line 473, in main
  File "D:\a\ytdl-nightly\ytdl-nightly\youtube_dl\__init__.py", line 443, in _real_main
  File "D:\a\ytdl-nightly\ytdl-nightly\youtube_dl\update.py", line 260, in update_self
  File "D:\a\ytdl-nightly\ytdl-nightly\youtube_dl\update.py", line 169, in run_update
  File "D:\a\ytdl-nightly\ytdl-nightly\youtube_dl\update.py", line 135, in get_sha256sum
ValueError: dictionary update sequence element #2 has length 1; 2 is required

FWIW, a youtube-dl.exe.new binary file has been downloaded to completion, adjacent to youtube-dl.exe, but this is the exact point where the updating process stalls 😞 ; so, this will need to get fixed 😉 ...

Windows native crypto support
...
there is a Windows build that includes the archived PyCrypto wheel linked in #30644, but is otherwise identical. Testing welcome, even encouraged.

I've given that build a sample test download containing an AES-128-encrypted HLSe stream, and it performed as expected 👍 :

youtube-dl-pycrypto -vF "https://cdn.bitmovin.com/content/assets/art-of-motion_drm/m3u8s/11331.m3u8" => 

[debug] System config: []
[debug] User config: []
[debug] Custom config: []
[debug] Command-line args: ['-vF', 'https://cdn.bitmovin.com/content/assets/art-of-motion_drm/m3u8s/11331.m3u8']
[debug] Encodings: locale cp1253, fs mbcs, out cp737, pref cp1253
[debug] youtube-dl version 2023.07.15 [f24bc9272] (single file build)
[debug] ** This version was built from the latest master code at https://github.com/ytdl-org/youtube-dl.
[debug] ** For support, visit the main site.
[debug] Python 3.4.4 (CPython x86 32bit) - Windows-Vista-6.0.6003-SP2 - OpenSSL 1.0.2d 9 Jul 2015
[debug] exe versions: none
[debug] Proxy map: {}
[generic] 11331: Requesting header
[generic] 11331: Downloading m3u8 information
[info] Available formats for 11331:
format code                extension  resolution note
audio_high-english_stereo  mp4        audio only [en]
692                        mp4        320x180     692k , avc1.42c00d, video only
992                        mp4        480x270     992k , avc1.42c00d, video only
1792                       mp4        640x360    1792k , avc1.42c00d, video only
2592                       mp4        960x540    2592k , avc1.42c00d, video only
4992                       mp4        1280x720   4992k , avc1.42c00d, video only
9792                       mp4        1920x1080  9792k , avc1.42c00d, video only (best)

and then:

youtube-dl-pycrypto -v --console-title --hls-prefer-native --hls-use-mpegts -c --no-part --fixup never -f 692 "https://cdn.bitmovin.com/content/assets/art-of-motion_drm/m3u8s/11331.m3u8" => 

debug] System config: []
[debug] User config: []
[debug] Custom config: []
[debug] Command-line args: ['-v', '--console-title', '--hls-prefer-native', '--hls-use-mpegts', '-c', '--no-part', '--fixup', 'never', '-f', '692', 'https://cdn.bitmovin.com/content/assets/art-of-motion_drm/m3u8s/11331.m3u8']
[debug] Encodings: locale cp1253, fs mbcs, out cp737, pref cp1253
[debug] youtube-dl version 2023.07.15 [f24bc9272] (single file build)
[debug] ** This version was built from the latest master code at https://github.com/ytdl-org/youtube-dl.
[debug] ** For support, visit the main site.
[debug] Python 3.4.4 (CPython x86 32bit) - Windows-Vista-6.0.6003-SP2 - OpenSSL 1.0.2d 9 Jul 2015
[debug] exe versions: none
[debug] Proxy map: {}
[generic] 11331: Requesting header
[generic] 11331: Downloading m3u8 information
[debug] Invoking downloader on 'https://cdn.bitmovin.com/content/assets/art-of-motion_drm/m3u8s/11331_video_180_250000.m3u8'
[hlsnative] Downloading m3u8 manifest
[hlsnative] Total fragments: 53
[download] Destination: 11331-11331.mp4
[download] 100% of 7.29MiB in 00:49

where chosen variant:

https://cdn.bitmovin.com/content/assets/art-of-motion_drm/m3u8s/11331_video_180_250000.m3u8 =>

#EXTM3U

#EXT-X-VERSION:3
#EXT-X-MEDIA-SEQUENCE:0
#EXT-X-TARGETDURATION:4
#EXT-X-KEY:METHOD=AES-128,URI="../video/180_250000/enc_hls/encryption.key",IV=0x613E8B8CE9CE208C4EAD4A0E03636371

#EXTINF:4.0
../video/180_250000/enc_hls/segment_0.ts
...

NB: cdn.bitmovin.com will BLOCK your IP address if you perform several test downloads over a short period of time 😠 ...

I'd say it's ready for production, after all it's the same pycrypto-2.6.1 lib that was bundled inside the Windows binaries released by the previous team of devs...
However, are we now "bound" to the web archive service holding that wheel?

And while I'm at it, would it be unrealistic to kindly ask for an "embellishment" as the one to be found in yt-dlp ?

[debug] Optional libraries: Cryptodome-3.18.0, brotli-1.0.9, certifi-2023.05.07, mutagen-1.46.0, sqlite3-2.6.0, websockets-11.0.3

Could we get something similar for yt-dl when compiled with PyCrypto, e.g. :

[debug] Optional libraries: [Py]Crypto-2.6.1

In closing, gigantic thanks for your coding efforts 🥇 !

@Vangelis66 commented on GitHub (Jul 15, 2023): > Pls report any issues with 2, 4, 5. OT: Hi **dirkf** 😄 ; currently under **extreme heatwave conditions** here (daytime maximum temps of 39-41C, nighttime minimum temps of 31(!) C), so not optimal for software testing 😉 ; be that as it may... >- [x] **Identify commit for build** Given item 5 (or even without), users can see the commit on the left of the Release panel, but that isn't always the commit from the main repo: so now the main repo HEAD commit SHA is being passed into the build workflow and added to youtube_dl/version.py as RELEASE_GIT_HEAD, and this is reported in the debug info, similar to yt-dlp. You've done a splendid job there 👍 , here's my verbose log with latest Windows build: ```console youtube-dl -v => [debug] System config: [] [debug] User config: [] [debug] Custom config: [] [debug] Command-line args: ['-v'] [debug] Encodings: locale cp1253, fs mbcs, out cp1253, pref cp1253 [debug] youtube-dl version 2023.07.15 [f24bc9272] (single file build) [debug] ** This version was built from the latest master code at https://github.com/ytdl-org/youtube-dl. [debug] ** For support, visit the main site. [debug] Python 3.4.4 (CPython x86 32bit) - Windows-Vista-6.0.6003-SP2 - OpenSSL 1.0.2d 9 Jul 2015 [debug] exe versions: none [debug] Proxy map: {} Usage: youtube-dl [OPTIONS] URL [URL...] youtube-dl: error: You must provide at least one URL. Type youtube-dl --help to see a list of all options. ``` >- [x] Wrong update channel ... Most sadly, while this has been fixed "per se" (update channel _is_ **the right one** now 😉 ), the "**internal update mechanism**" is currently BROKEN 😢 ... **STR**: 1. Download **previous** Windows build [2023.07.14](https://github.com/ytdl-org/ytdl-nightly/releases/download/2023.07.14/youtube-dl.exe) 2. Launch a **Windows Command Prompt** in the same working directory as the EXE resides 3. Issue `youtube-dl -vU` 4. Result: ```console [debug] System config: [] [debug] User config: [] [debug] Custom config: [] [debug] Command-line args: ['-vU'] [debug] Encodings: locale cp1253, fs mbcs, out cp737, pref cp1253 [debug] youtube-dl version 2023.07.14 [f24bc9272] (single file build) [debug] ** This version was built from the latest master code at https://github.com/ytdl-org/youtube-dl. [debug] ** For support, visit the main site. [debug] Python 3.4.4 (CPython x86 32bit) - Windows-Vista-6.0.6003-SP2 - OpenSSL 1.0.2d 9 Jul 2015 [debug] exe versions: none [debug] Proxy map: {} Latest version: 2023.07.15, Current version: 2023.07.14 Current Build Hash 1dead7ccce0def131e11babebd5aa2a69a50a7af719348fa26654da691cba4ca Updating to version 2023.07.15 ... Traceback (most recent call last): File "__main__.py", line 19, in <module> File "D:\a\ytdl-nightly\ytdl-nightly\youtube_dl\__init__.py", line 473, in main File "D:\a\ytdl-nightly\ytdl-nightly\youtube_dl\__init__.py", line 443, in _real_main File "D:\a\ytdl-nightly\ytdl-nightly\youtube_dl\update.py", line 260, in update_self File "D:\a\ytdl-nightly\ytdl-nightly\youtube_dl\update.py", line 169, in run_update File "D:\a\ytdl-nightly\ytdl-nightly\youtube_dl\update.py", line 135, in get_sha256sum ValueError: dictionary update sequence element #2 has length 1; 2 is required ``` FWIW, a `youtube-dl.exe.new` binary file has been downloaded to completion, adjacent to `youtube-dl.exe`, but this is the exact point where the updating process stalls 😞 ; so, this will need to get fixed 😉 ... >- [ ] Windows native crypto support > ... > there is a **Windows build** that includes the archived **PyCrypto wheel** linked in #30644, but is otherwise identical. **Testing welcome, even encouraged.** I've given that build a sample test download containing an `AES-128`-encrypted HLSe stream, and it performed as expected 👍 : ```console youtube-dl-pycrypto -vF "https://cdn.bitmovin.com/content/assets/art-of-motion_drm/m3u8s/11331.m3u8" => [debug] System config: [] [debug] User config: [] [debug] Custom config: [] [debug] Command-line args: ['-vF', 'https://cdn.bitmovin.com/content/assets/art-of-motion_drm/m3u8s/11331.m3u8'] [debug] Encodings: locale cp1253, fs mbcs, out cp737, pref cp1253 [debug] youtube-dl version 2023.07.15 [f24bc9272] (single file build) [debug] ** This version was built from the latest master code at https://github.com/ytdl-org/youtube-dl. [debug] ** For support, visit the main site. [debug] Python 3.4.4 (CPython x86 32bit) - Windows-Vista-6.0.6003-SP2 - OpenSSL 1.0.2d 9 Jul 2015 [debug] exe versions: none [debug] Proxy map: {} [generic] 11331: Requesting header [generic] 11331: Downloading m3u8 information [info] Available formats for 11331: format code extension resolution note audio_high-english_stereo mp4 audio only [en] 692 mp4 320x180 692k , avc1.42c00d, video only 992 mp4 480x270 992k , avc1.42c00d, video only 1792 mp4 640x360 1792k , avc1.42c00d, video only 2592 mp4 960x540 2592k , avc1.42c00d, video only 4992 mp4 1280x720 4992k , avc1.42c00d, video only 9792 mp4 1920x1080 9792k , avc1.42c00d, video only (best) ``` and then: ```console youtube-dl-pycrypto -v --console-title --hls-prefer-native --hls-use-mpegts -c --no-part --fixup never -f 692 "https://cdn.bitmovin.com/content/assets/art-of-motion_drm/m3u8s/11331.m3u8" => debug] System config: [] [debug] User config: [] [debug] Custom config: [] [debug] Command-line args: ['-v', '--console-title', '--hls-prefer-native', '--hls-use-mpegts', '-c', '--no-part', '--fixup', 'never', '-f', '692', 'https://cdn.bitmovin.com/content/assets/art-of-motion_drm/m3u8s/11331.m3u8'] [debug] Encodings: locale cp1253, fs mbcs, out cp737, pref cp1253 [debug] youtube-dl version 2023.07.15 [f24bc9272] (single file build) [debug] ** This version was built from the latest master code at https://github.com/ytdl-org/youtube-dl. [debug] ** For support, visit the main site. [debug] Python 3.4.4 (CPython x86 32bit) - Windows-Vista-6.0.6003-SP2 - OpenSSL 1.0.2d 9 Jul 2015 [debug] exe versions: none [debug] Proxy map: {} [generic] 11331: Requesting header [generic] 11331: Downloading m3u8 information [debug] Invoking downloader on 'https://cdn.bitmovin.com/content/assets/art-of-motion_drm/m3u8s/11331_video_180_250000.m3u8' [hlsnative] Downloading m3u8 manifest [hlsnative] Total fragments: 53 [download] Destination: 11331-11331.mp4 [download] 100% of 7.29MiB in 00:49 ``` where chosen variant: `https://cdn.bitmovin.com/content/assets/art-of-motion_drm/m3u8s/11331_video_180_250000.m3u8` => ```xml #EXTM3U #EXT-X-VERSION:3 #EXT-X-MEDIA-SEQUENCE:0 #EXT-X-TARGETDURATION:4 #EXT-X-KEY:METHOD=AES-128,URI="../video/180_250000/enc_hls/encryption.key",IV=0x613E8B8CE9CE208C4EAD4A0E03636371 #EXTINF:4.0 ../video/180_250000/enc_hls/segment_0.ts ... ``` NB: `cdn.bitmovin.com` will **BLOCK your IP address** if you perform **several** test downloads over a **short period** of time :angry: ... I'd say it's ready for production, after all it's **the same** `pycrypto-2.6.1` lib that was bundled inside the Windows binaries released by the **previous team of devs**... However, are we now "bound" to the `web archive` service holding that wheel? And while I'm at it, would it be unrealistic to kindly ask for an "embellishment" as the one to be found in `yt-dlp` ? ```console [debug] Optional libraries: Cryptodome-3.18.0, brotli-1.0.9, certifi-2023.05.07, mutagen-1.46.0, sqlite3-2.6.0, websockets-11.0.3 ``` Could we get something similar for `yt-dl` when compiled with `PyCrypto`, e.g. : ```console [debug] Optional libraries: [Py]Crypto-2.6.1 ``` In closing, gigantic thanks for your coding efforts 🥇 !

deekerman commented

Author

Owner

@dirkf commented on GitHub (Jul 16, 2023):

... ValueError: dictionary update sequence element #2 has length 1; 2 is required

The workflow had to be changed because GH changed the API, but it should use $env:GITHUB_OUTPUT, not $GITHUB_OUTPUT, to access the environment variable GITHUB_OUTPUT because the default job shell for Windows is PowerShell, apparently. Not setting the sha256_win output causes the SHA256 file to look like this:

e70b9228bf32f5ec84d4b3187d5906faeab36019e1687526850966cdf906c2d2  youtube-dl
cf2e64efa2cd1e6b166c24dc449d1ae5ca061655b5ad5c88588e8d3356ec325d  youtube-dl-2023.07.15.tar.gz
  youtube-dl.exe

Lo, only one item where two should be. I expect that fixing the workflow will unbreak this. The POSIX-y update should work. The -pycrypto build is not supported as it's just a test.

... However, are we now "bound" to the web archive service holding that wheel?

Today, yes, but if it becomes the default we can either host it directly (requires a commit or maybe stashing in a gist) or use the workflow cache (depends on no-one flushing it if the source vanishes).

Could we get [dependency listing] for yt-dl when compiled with PyCrypto, e.g. :

yt-dlp has a module to handle this. One option is full plagiarism; another is to do what that does manually (ie, go through a list of deps, in this case of the one module crypto, try to import it, list it if that worked); and then there is the zero option, which has obvious attractions.

@dirkf commented on GitHub (Jul 16, 2023): >... ValueError: dictionary update sequence element #2 has length 1; 2 is required The workflow had to be changed because GH changed the API, but it should use `$env:GITHUB_OUTPUT`, not `$GITHUB_OUTPUT`, to access the environment variable `GITHUB_OUTPUT` because the default job shell for Windows is PowerShell, apparently. Not setting the `sha256_win` output causes the SHA256 file to look like this: ``` e70b9228bf32f5ec84d4b3187d5906faeab36019e1687526850966cdf906c2d2 youtube-dl cf2e64efa2cd1e6b166c24dc449d1ae5ca061655b5ad5c88588e8d3356ec325d youtube-dl-2023.07.15.tar.gz youtube-dl.exe ``` Lo, only one item where two should be. I expect that fixing the workflow will unbreak this. The POSIX-y update should work. The -pycrypto build is not supported as it's just a test. >... However, are we now "bound" to the web archive service holding that wheel? Today, yes, but if it becomes the default we can either host it directly (requires a commit or maybe stashing in a gist) or use the workflow cache (depends on no-one flushing it if the source vanishes). > Could we get [dependency listing] for yt-dl when compiled with PyCrypto, e.g. : yt-dlp has a module to handle this. One option is full plagiarism; another is to do what that does manually (ie, go through a list of deps, in this case of the one module `crypto`, try to import it, list it if that worked); and then there is the zero option, which has obvious attractions.

deekerman commented

Author

Owner

@pukkandan commented on GitHub (Jul 16, 2023):

yt-dlp has a module to handle this.

That is a rather recent change. We had it in compat before that - github.com/yt-dlp/yt-dlp@edf65256aa (diff-241045c3f3)

If you want to be able to distinguish pycrypto and pycryptodome, see github.com/yt-dlp/yt-dlp@1d485a1a79 (diff-27f5169677)

@pukkandan commented on GitHub (Jul 16, 2023): > yt-dlp has a module to handle this. That is a rather recent change. We had it in compat before that - https://github.com/yt-dlp/yt-dlp/commit/edf65256aa630a5ce011138e8957c95c9bef0584#diff-241045c3f394fb621a68db52d734b10ef32417760b036319df388519a2ef10fe If you want to be able to distinguish pycrypto and pycryptodome, see https://github.com/yt-dlp/yt-dlp/commit/1d485a1a799bbeeb2faea0595676ca7d4c0f3716#diff-27f516967719582a7725de284e1042b89a8c4d8f787301bff6a8ba6ec14cbb0f

deekerman commented

Author

Owner

@Vangelis66 commented on GitHub (Jul 16, 2023):

I expect that fixing the workflow will unbreak this.

... And, we have lift-off 😜 :

youtube-dl -vU =>

[debug] System config: []
[debug] User config: []
[debug] Custom config: []
[debug] Command-line args: ['-vU']
[debug] Encodings: locale cp1253, fs mbcs, out cp737, pref cp1253
[debug] youtube-dl version 2023.07.15 [f24bc9272] (single file build)
[debug] ** This version was built from the latest master code at https://github.com/ytdl-org/youtube-dl.
[debug] ** For support, visit the main site.
[debug] Python 3.4.4 (CPython x86 32bit) - Windows-Vista-6.0.6003-SP2 - OpenSSL 1.0.2d 9 Jul 2015
[debug] exe versions: none
[debug] Proxy map: {}
Latest version: 2023.07.17, Current version: 2023.07.15
Current Build Hash b9d3842fd62ab41b5d0da6202025ed8c6549bcb4053793e048e2ec618c7af7bb
Updating to version 2023.07.17 ...
Updated youtube-dl to version 2023.07.17

So, many congrats 👍 !

The -pycrypto build is not supported, as it's just a test.

Just for laughs 😉 , I did try:

youtube-dl-pycrypto -vU => 

[debug] System config: []
[debug] User config: []
[debug] Custom config: []
[debug] Command-line args: ['-vU']
[debug] Encodings: locale cp1253, fs mbcs, out cp737, pref cp1253
[debug] youtube-dl version 2023.07.15 [f24bc9272] (single file build)
[debug] ** This version was built from the latest master code at https://github.com/ytdl-org/youtube-dl.
[debug] ** For support, visit the main site.
[debug] Python 3.4.4 (CPython x86 32bit) - Windows-Vista-6.0.6003-SP2 - OpenSSL 1.0.2d 9 Jul 2015
[debug] exe versions: none
[debug] Proxy map: {}
Latest version: 2023.07.17, Current version: 2023.07.15
Current Build Hash 769996f4ddf23faf803095ed15e1a013fa8c2ec0451436217c5450ae1e5b759f
Updating to version 2023.07.17 ...
Updated youtube-dl to version 2023.07.17

(NB the different Current Build Hash values in this case), which did give an initial impression of a successful update (a youtube-dl-pycrypto.exe binary with file version 2023.07.17.0 is the end product of the update), but, of course/as said, this impression is false: the "updated" executable is just a renamed youtube-dl.exe one (of file version 2023.07.17.0), without bundling pycrypto; just as a FTR, nothing more 😄 ...

@Vangelis66 commented on GitHub (Jul 16, 2023): > I expect that fixing the workflow will **unbreak this.** ... And, we have lift-off 😜 : ```console youtube-dl -vU => [debug] System config: [] [debug] User config: [] [debug] Custom config: [] [debug] Command-line args: ['-vU'] [debug] Encodings: locale cp1253, fs mbcs, out cp737, pref cp1253 [debug] youtube-dl version 2023.07.15 [f24bc9272] (single file build) [debug] ** This version was built from the latest master code at https://github.com/ytdl-org/youtube-dl. [debug] ** For support, visit the main site. [debug] Python 3.4.4 (CPython x86 32bit) - Windows-Vista-6.0.6003-SP2 - OpenSSL 1.0.2d 9 Jul 2015 [debug] exe versions: none [debug] Proxy map: {} Latest version: 2023.07.17, Current version: 2023.07.15 Current Build Hash b9d3842fd62ab41b5d0da6202025ed8c6549bcb4053793e048e2ec618c7af7bb Updating to version 2023.07.17 ... Updated youtube-dl to version 2023.07.17 ``` So, many congrats 👍 ! > The `-pycrypto` build is **not supported**, as it's just a test. Just for laughs 😉 , I did try: ```console youtube-dl-pycrypto -vU => [debug] System config: [] [debug] User config: [] [debug] Custom config: [] [debug] Command-line args: ['-vU'] [debug] Encodings: locale cp1253, fs mbcs, out cp737, pref cp1253 [debug] youtube-dl version 2023.07.15 [f24bc9272] (single file build) [debug] ** This version was built from the latest master code at https://github.com/ytdl-org/youtube-dl. [debug] ** For support, visit the main site. [debug] Python 3.4.4 (CPython x86 32bit) - Windows-Vista-6.0.6003-SP2 - OpenSSL 1.0.2d 9 Jul 2015 [debug] exe versions: none [debug] Proxy map: {} Latest version: 2023.07.17, Current version: 2023.07.15 Current Build Hash 769996f4ddf23faf803095ed15e1a013fa8c2ec0451436217c5450ae1e5b759f Updating to version 2023.07.17 ... Updated youtube-dl to version 2023.07.17 ``` (NB the different `Current Build Hash` values in this case), which did give an _initial_ impression of a successful update (a `youtube-dl-pycrypto.exe` binary with file version `2023.07.17.0` is the end product of the update), but, of course/as said, this impression is **false**: the "updated" executable is just a **renamed** `youtube-dl.exe` one (of file version `2023.07.17.0`), without bundling `pycrypto`; just as a FTR, nothing more 😄 ...

deekerman commented

Author

Owner

@dirkf commented on GitHub (Jul 18, 2023):

@MikeMcQuaid

Yes, our hope would be any sort of tag applied to a release that would show up in https://github.com/ytdl-org/youtube-dl/releases that we could update to 🙇🏻

Maybe this: https://github.com/ytdl-org/ytdl-nightly/releases/download/2023.07.18/youtube-dl-2023.07.18.tar.gz ?

Somewhat similar to brew install --HEAD ... but tagged and identifies itself. I believe that a pip-like installation will say this if a user tries to update with -U:

It looks like you installed youtube-dl with a package manager, pip or setup.py; Use that to update

@dirkf commented on GitHub (Jul 18, 2023): @MikeMcQuaid > Yes, our hope would be any sort of tag applied to a release that would show up in https://github.com/ytdl-org/youtube-dl/releases that we could update to 🙇🏻 Maybe this: https://github.com/ytdl-org/ytdl-nightly/releases/download/2023.07.18/youtube-dl-2023.07.18.tar.gz ? Somewhat similar to `brew install --HEAD ...` but tagged and identifies itself. I believe that a pip-like installation will say this if a user tries to update with `-U`: ``` It looks like you installed youtube-dl with a package manager, pip or setup.py; Use that to update ```

deekerman commented

Author

Owner

@MikeMcQuaid commented on GitHub (Jul 18, 2023):

@dirkf Yeh, that's basically what --HEAD provides. Homebrew needs more than that, I'm afraid: an actual tagged release on this repository.

@MikeMcQuaid commented on GitHub (Jul 18, 2023): @dirkf Yeh, that's basically what `--HEAD` provides. Homebrew needs more than that, I'm afraid: an actual tagged release on this repository.

deekerman commented

Author

Owner

@Vangelis66 commented on GitHub (Jul 18, 2023):

Something has gone amiss 😞 ...

From time to time, I compile locally the master branch yt-dl code with CPython 2.7.18 (32-bit); building environment shown below:

python -m pip --no-python-version-warning list =>

Package    Version
---------- -------
pip        20.3.4
py2exe-py2 0.6.9
pycrypto   2.6.1
setuptools 44.1.1
wheel      0.37.1

My previous binary had been compiled from code snapshot master-git-20230705-gf24bc92 (f24bc92); with it, when I issue youtube-dl -vU, I get:

[debug] System config: []
[debug] User config: []
[debug] Custom config: []
[debug] Command-line args: [u'-vU']
[debug] Encodings: locale cp1253, fs mbcs, out cp737, pref cp1253
[debug] youtube-dl version 2023.07.05.2258 (single file build)
[debug] Lazy loading extractors enabled
[debug] Python 2.7.18 (CPython x86 32bit) - Windows-Vista-6.0.6003-SP2 - OpenSSL 1.0.2t  10 Sep 2019
[debug] exe versions: none
[debug] Proxy map: {}
youtube-dl is up to date (2023.07.05.2258)

Since this build is on the release update channel (with latest release being 2021.12.17), the outcome is to be expected 😉 ...

Half an hour ago, I finished compiling latest (public) master branch snapshot master-git-20230718-g47214e4 (47214e4); the compilation succeeded, but now, when I issue youtube-dl -vU on the newly produced binary, an ERROR is generated:

[debug] System config: []
[debug] User config: []
[debug] Custom config: []
[debug] Command-line args: [u'-vU']
[debug] Encodings: locale cp1253, fs mbcs, out cp737, pref cp1253
[debug] youtube-dl version 2023.07.18.0950
[debug] Lazy loading extractors enabled
[debug] Single file build
[debug] Python 2.7.18 (CPython x86 32bit) - Windows-Vista-6.0.6003-SP2 - OpenSSL 1.0.2t  10 Sep 2019
[debug] exe versions: none
[debug] Proxy map: {}
Traceback (most recent call last):
  File "youtube_dl\update.pyo", line 46, in update_self
  File "urllib2.pyo", line 435, in open
  File "urllib2.pyo", line 548, in http_response
  File "urllib2.pyo", line 467, in error
  File "urllib2.pyo", line 407, in _call_chain
  File "urllib2.pyo", line 654, in http_error_302
  File "urllib2.pyo", line 435, in open
  File "urllib2.pyo", line 548, in http_response
  File "urllib2.pyo", line 473, in error
  File "urllib2.pyo", line 407, in _call_chain
  File "urllib2.pyo", line 556, in http_error_default
HTTPError: HTTP Error 404: Not Found

ERROR: can't find the current version. Please try again later.

Please advise...

@Vangelis66 commented on GitHub (Jul 18, 2023): Something has gone **amiss** 😞 ... From time to time, I compile locally the `master` branch `yt-dl` code with CPython **2.7.18** (32-bit); building environment shown below: ``` python -m pip --no-python-version-warning list => Package Version ---------- ------- pip 20.3.4 py2exe-py2 0.6.9 pycrypto 2.6.1 setuptools 44.1.1 wheel 0.37.1 ``` My previous binary had been compiled from code snapshot `master-git-20230705-gf24bc92` (f24bc92); with it, when I issue `youtube-dl -vU`, I get: ```console [debug] System config: [] [debug] User config: [] [debug] Custom config: [] [debug] Command-line args: [u'-vU'] [debug] Encodings: locale cp1253, fs mbcs, out cp737, pref cp1253 [debug] youtube-dl version 2023.07.05.2258 (single file build) [debug] Lazy loading extractors enabled [debug] Python 2.7.18 (CPython x86 32bit) - Windows-Vista-6.0.6003-SP2 - OpenSSL 1.0.2t 10 Sep 2019 [debug] exe versions: none [debug] Proxy map: {} youtube-dl is up to date (2023.07.05.2258) ``` Since this build is on the `release` update channel (with _latest_ release being `2021.12.17`), the outcome is to be expected 😉 ... Half an hour ago, I finished compiling latest (public) `master` branch snapshot `master-git-20230718-g47214e4` (47214e4); the **compilation succeeded**, but now, when I issue `youtube-dl -vU` on the newly produced binary, an `ERROR` is generated: ```console [debug] System config: [] [debug] User config: [] [debug] Custom config: [] [debug] Command-line args: [u'-vU'] [debug] Encodings: locale cp1253, fs mbcs, out cp737, pref cp1253 [debug] youtube-dl version 2023.07.18.0950 [debug] Lazy loading extractors enabled [debug] Single file build [debug] Python 2.7.18 (CPython x86 32bit) - Windows-Vista-6.0.6003-SP2 - OpenSSL 1.0.2t 10 Sep 2019 [debug] exe versions: none [debug] Proxy map: {} Traceback (most recent call last): File "youtube_dl\update.pyo", line 46, in update_self File "urllib2.pyo", line 435, in open File "urllib2.pyo", line 548, in http_response File "urllib2.pyo", line 467, in error File "urllib2.pyo", line 407, in _call_chain File "urllib2.pyo", line 654, in http_error_302 File "urllib2.pyo", line 435, in open File "urllib2.pyo", line 548, in http_response File "urllib2.pyo", line 473, in error File "urllib2.pyo", line 407, in _call_chain File "urllib2.pyo", line 556, in http_error_default HTTPError: HTTP Error 404: Not Found ERROR: can't find the current version. Please try again later. ``` Please advise...

deekerman commented

Author

Owner

@nicolaasjan commented on GitHub (Jul 19, 2023):

Hmm...
Something similar here with my Python 3.4.4 build:

[debug] System config: []
[debug] User config: ['--no-mtime', '-o', 'C:\\Users\\nico\\Desktop\\%(title)s.%(ext)s', '-f', 'bestvideo[height<=1080][ext=mp4]+bestaudio[ext=m4a]/best[ext=mp4]/best', '--embed-thumbnail']
[debug] Custom config: []
[debug] Command-line args: ['-vU']
[debug] Encodings: locale cp1252, fs mbcs, out cp1252, pref cp1252
[debug] youtube-dl version 2023.07.18
[debug] Lazy loading extractors enabled
[debug] Single file build
[debug] Python 3.4.4 (CPython AMD64 32bit) - Windows-7-6.1.7601-SP1 - OpenSSL 1.0.2d 9 Jul 2015
[debug] exe versions: ffmpeg N-111280-gd51b0580e-2023-06-25-nonfree, ffprobe N-111280-gd51b0580e-2023-06-25-nonfree, phantomjs 2.1.1
[debug] Proxy map: {}
Traceback (most recent call last):
  File "C:\Users\nico\Desktop\youtube-dl_source\youtube_dl\update.py", line 46, in update_self
  File "c:\Program Files (x86)\Python3.4.4\lib\urllib\request.py", line 470, in open
    response = meth(req, response)
  File "c:\Program Files (x86)\Python3.4.4\lib\urllib\request.py", line 580, in http_response
    'http', request, response, code, msg, hdrs)
  File "c:\Program Files (x86)\Python3.4.4\lib\urllib\request.py", line 502, in error
    result = self._call_chain(*args)
  File "c:\Program Files (x86)\Python3.4.4\lib\urllib\request.py", line 442, in _call_chain
    result = func(*args)
  File "c:\Program Files (x86)\Python3.4.4\lib\urllib\request.py", line 685, in http_error_302
    return self.parent.open(new, timeout=req.timeout)
  File "c:\Program Files (x86)\Python3.4.4\lib\urllib\request.py", line 470, in open
    response = meth(req, response)
  File "c:\Program Files (x86)\Python3.4.4\lib\urllib\request.py", line 580, in http_response
    'http', request, response, code, msg, hdrs)
  File "c:\Program Files (x86)\Python3.4.4\lib\urllib\request.py", line 508, in error
    return self._call_chain(*args)
  File "c:\Program Files (x86)\Python3.4.4\lib\urllib\request.py", line 442, in _call_chain
    result = func(*args)
  File "c:\Program Files (x86)\Python3.4.4\lib\urllib\request.py", line 588, in http_error_default
    raise HTTPError(req.full_url, code, msg, hdrs, fp)
urllib.error.HTTPError: HTTP Error 404: Not Found

ERROR: can't find the current version. Please try again later.

@nicolaasjan commented on GitHub (Jul 19, 2023): Hmm... Something similar here with my Python 3.4.4 build: ```cmd [debug] System config: [] [debug] User config: ['--no-mtime', '-o', 'C:\\Users\\nico\\Desktop\\%(title)s.%(ext)s', '-f', 'bestvideo[height<=1080][ext=mp4]+bestaudio[ext=m4a]/best[ext=mp4]/best', '--embed-thumbnail'] [debug] Custom config: [] [debug] Command-line args: ['-vU'] [debug] Encodings: locale cp1252, fs mbcs, out cp1252, pref cp1252 [debug] youtube-dl version 2023.07.18 [debug] Lazy loading extractors enabled [debug] Single file build [debug] Python 3.4.4 (CPython AMD64 32bit) - Windows-7-6.1.7601-SP1 - OpenSSL 1.0.2d 9 Jul 2015 [debug] exe versions: ffmpeg N-111280-gd51b0580e-2023-06-25-nonfree, ffprobe N-111280-gd51b0580e-2023-06-25-nonfree, phantomjs 2.1.1 [debug] Proxy map: {} Traceback (most recent call last): File "C:\Users\nico\Desktop\youtube-dl_source\youtube_dl\update.py", line 46, in update_self File "c:\Program Files (x86)\Python3.4.4\lib\urllib\request.py", line 470, in open response = meth(req, response) File "c:\Program Files (x86)\Python3.4.4\lib\urllib\request.py", line 580, in http_response 'http', request, response, code, msg, hdrs) File "c:\Program Files (x86)\Python3.4.4\lib\urllib\request.py", line 502, in error result = self._call_chain(*args) File "c:\Program Files (x86)\Python3.4.4\lib\urllib\request.py", line 442, in _call_chain result = func(*args) File "c:\Program Files (x86)\Python3.4.4\lib\urllib\request.py", line 685, in http_error_302 return self.parent.open(new, timeout=req.timeout) File "c:\Program Files (x86)\Python3.4.4\lib\urllib\request.py", line 470, in open response = meth(req, response) File "c:\Program Files (x86)\Python3.4.4\lib\urllib\request.py", line 580, in http_response 'http', request, response, code, msg, hdrs) File "c:\Program Files (x86)\Python3.4.4\lib\urllib\request.py", line 508, in error return self._call_chain(*args) File "c:\Program Files (x86)\Python3.4.4\lib\urllib\request.py", line 442, in _call_chain result = func(*args) File "c:\Program Files (x86)\Python3.4.4\lib\urllib\request.py", line 588, in http_error_default raise HTTPError(req.full_url, code, msg, hdrs, fp) urllib.error.HTTPError: HTTP Error 404: Not Found ERROR: can't find the current version. Please try again later. ```

deekerman commented

Author

Owner

@dirkf commented on GitHub (Jul 19, 2023):

Please try again with a new download. I had to revamp the nightly repo while merging #32450. The POSIX bundled version from the 2023.07.18 page does this for me:

$ ./youtube-dl-20230718 -vU
[debug] System config: [u'--prefer-ffmpeg']
[debug] User config: []
[debug] Custom config: []
[debug] Command-line args: [u'-vU']
[debug] Encodings: locale UTF-8, fs UTF-8, out UTF-8, pref UTF-8
[debug] youtube-dl version 2023.07.18 [47214e46d] (single file build)
[debug] ** This version was built from the latest master code at https://github.com/ytdl-org/youtube-dl.
[debug] ** For support, visit the main site.
[debug] Python 2.7.18 (CPython i686 32bit) - Linux-4.4.0-210-generic-i686-with-Ubuntu-16.04-xenial - OpenSSL 1.1.1u  30 May 2023 - glibc 2.15
[debug] exe versions: avconv 4.3, avprobe 4.3, ffmpeg 4.3, ffprobe 4.3
[debug] Proxy map: {}
Latest version: 2023.07.18, Current version: 2023.07.18
youtube-dl is up to date (2023.07.18)
$

@dirkf commented on GitHub (Jul 19, 2023): Please try again with a new download. I had to revamp the nightly repo while merging #32450. The POSIX bundled version from the 2023.07.18 page does this for me: ```console $ ./youtube-dl-20230718 -vU [debug] System config: [u'--prefer-ffmpeg'] [debug] User config: [] [debug] Custom config: [] [debug] Command-line args: [u'-vU'] [debug] Encodings: locale UTF-8, fs UTF-8, out UTF-8, pref UTF-8 [debug] youtube-dl version 2023.07.18 [47214e46d] (single file build) [debug] ** This version was built from the latest master code at https://github.com/ytdl-org/youtube-dl. [debug] ** For support, visit the main site. [debug] Python 2.7.18 (CPython i686 32bit) - Linux-4.4.0-210-generic-i686-with-Ubuntu-16.04-xenial - OpenSSL 1.1.1u 30 May 2023 - glibc 2.15 [debug] exe versions: avconv 4.3, avprobe 4.3, ffmpeg 4.3, ffprobe 4.3 [debug] Proxy map: {} Latest version: 2023.07.18, Current version: 2023.07.18 youtube-dl is up to date (2023.07.18) $ ```

deekerman commented

Author

Owner

@nicolaasjan commented on GitHub (Jul 19, 2023):

That one is indeed OK:

ytd -vU
[debug] System config: []
[debug] User config: ['--rm-cache-dir', '-i', '-o', '/dev/shm/test-ytd/%(title)s.%(ext)s', '-f', 'bestvideo[height<=1080][ext=mp4]+bestaudio[ext=m4a]/best[ext=mp4]/best', '--no-mtime', '--embed-thumbnail', '--force-ipv4']
[debug] Custom config: []
[debug] Command-line args: ['-vU']
[debug] Encodings: locale UTF-8, fs utf-8, out utf-8, pref UTF-8
[debug] youtube-dl version 2023.07.18 [47214e46d] (single file build)
[debug] ** This version was built from the latest master code at https://github.com/ytdl-org/youtube-dl.
[debug] ** For support, visit the main site.
[debug] Python 3.8.10 (CPython x86_64 64bit) - Linux-5.4.0-153-generic-x86_64-with-glibc2.29 - OpenSSL 1.1.1f  31 Mar 2020 - glibc 2.31
[debug] exe versions: ffmpeg N-111355-g68e9d2835f-Nico-20230707, ffprobe N-111355-g68e9d2835f-Nico-20230707, phantomjs 2.1.1, rtmpdump 2.4
[debug] Proxy map: {}
Latest version: 2023.07.18, Current version: 2023.07.18
youtube-dl is up to date (2023.07.18)
Removing cache dir /home/nico/.cache/youtube-dl ..

@nicolaasjan commented on GitHub (Jul 19, 2023): That one is indeed OK: ```bash ytd -vU [debug] System config: [] [debug] User config: ['--rm-cache-dir', '-i', '-o', '/dev/shm/test-ytd/%(title)s.%(ext)s', '-f', 'bestvideo[height<=1080][ext=mp4]+bestaudio[ext=m4a]/best[ext=mp4]/best', '--no-mtime', '--embed-thumbnail', '--force-ipv4'] [debug] Custom config: [] [debug] Command-line args: ['-vU'] [debug] Encodings: locale UTF-8, fs utf-8, out utf-8, pref UTF-8 [debug] youtube-dl version 2023.07.18 [47214e46d] (single file build) [debug] ** This version was built from the latest master code at https://github.com/ytdl-org/youtube-dl. [debug] ** For support, visit the main site. [debug] Python 3.8.10 (CPython x86_64 64bit) - Linux-5.4.0-153-generic-x86_64-with-glibc2.29 - OpenSSL 1.1.1f 31 Mar 2020 - glibc 2.31 [debug] exe versions: ffmpeg N-111355-g68e9d2835f-Nico-20230707, ffprobe N-111355-g68e9d2835f-Nico-20230707, phantomjs 2.1.1, rtmpdump 2.4 [debug] Proxy map: {} Latest version: 2023.07.18, Current version: 2023.07.18 youtube-dl is up to date (2023.07.18) Removing cache dir /home/nico/.cache/youtube-dl .. ```

deekerman commented

Author

Owner

@nicolaasjan commented on GitHub (Jul 19, 2023):

Pardon my ignorance, but why does my own build from the same master (47214e46d) give that error?
(I always like to build my own with some fixes, i.e. #29318, #29581, #29593 and #30998)

@nicolaasjan commented on GitHub (Jul 19, 2023): Pardon my ignorance, but why does my _own_ build from the same `master` (47214e46d) give that error? (I always like to build my own with some fixes, i.e. #29318, #29581, #29593 and #30998)

deekerman commented

Author

Owner

@dirkf commented on GitHub (Jul 19, 2023):

In the nightly build:

version.py is updated with the actual version and the build SHA if available
there is a nightly version of update.py, similar to yt-dlp's, that knows about the nightly
the update URL is https://api.github.com/repos/ytdl-org/ytdl-nightly/releases/latest, which returns JSON details of the latest release.

In the main repo:

update.py looks at https://yt-dl.org/update/LATEST_VERSION, which currently returns 2021.12.17.

Make sure any patches leave the desired mechanism in place.

@dirkf commented on GitHub (Jul 19, 2023): In the nightly build: * `version.py` is updated with the actual version and the build SHA if available * there is a nightly version of `update.py`, similar to yt-dlp's, that knows about the nightly * the update URL is https://api.github.com/repos/ytdl-org/ytdl-nightly/releases/latest, which returns JSON details of the latest release. In the main repo: * `update.py` looks at https://yt-dl.org/update/LATEST_VERSION, which currently returns `2021.12.17`. Make sure any patches leave the desired mechanism in place.

deekerman commented

https://github.com/ytdl-org/youtube-dl/commits/master

Author

Owner

@Vangelis66 commented on GitHub (Jul 19, 2023):

Something similar here with my Python 3.4.4 build:

I can replicate; your previous build of version 2023.07.05 behaves as expected:

youtube-dl -vU

[debug] System config: []
[debug] User config: []
[debug] Custom config: []
[debug] Command-line args: ['-vU']
[debug] Encodings: locale cp1253, fs mbcs, out cp737, pref cp1253
[debug] youtube-dl version 2023.07.05 (single file build)
[debug] Lazy loading extractors enabled
[debug] Python 3.4.4 (CPython x86 32bit) - Windows-Vista-6.0.6003-SP2 - OpenSSL 1.0.2d 9 Jul 2015
[debug] exe versions: none
[debug] Proxy map: {}
youtube-dl is up to date (2023.07.05)

while your most recent one is BROKEN (in a similar fashion to mine):

[debug] System config: []
[debug] User config: []
[debug] Custom config: []
[debug] Command-line args: ['-vU']
[debug] Encodings: locale cp1253, fs mbcs, out cp737, pref cp1253
[debug] youtube-dl version 2023.07.18
[debug] Lazy loading extractors enabled
[debug] Single file build
[debug] Python 3.4.4 (CPython x86 32bit) - Windows-Vista-6.0.6003-SP2 - OpenSSL 1.0.2d 9 Jul 2015
[debug] exe versions: none
[debug] Proxy map: {}
Traceback (most recent call last):
  File "C:\Users\nico\Desktop\youtube-dl_source\youtube_dl\update.py", line 46, in update_self
  File "c:\Program Files (x86)\Python3.4.4\lib\urllib\request.py", line 470, in open
  File "c:\Program Files (x86)\Python3.4.4\lib\urllib\request.py", line 580, in http_response
  File "c:\Program Files (x86)\Python3.4.4\lib\urllib\request.py", line 502, in error
  File "c:\Program Files (x86)\Python3.4.4\lib\urllib\request.py", line 442, in_call_chain
  File "c:\Program Files (x86)\Python3.4.4\lib\urllib\request.py", line 685, in http_error_302
  File "c:\Program Files (x86)\Python3.4.4\lib\urllib\request.py", line 470, in open
  File "c:\Program Files (x86)\Python3.4.4\lib\urllib\request.py", line 580, in http_response
  File "c:\Program Files (x86)\Python3.4.4\lib\urllib\request.py", line 508, in error
  File "c:\Program Files (x86)\Python3.4.4\lib\urllib\request.py", line 442, in_call_chain
  File "c:\Program Files (x86)\Python3.4.4\lib\urllib\request.py", line 588, in http_error_default
urllib.error.HTTPError: HTTP Error 404: Not Found

ERROR: can't find the current version. Please try again later.

Please try again with a new download. I had to revamp the nightly repo

... Please, kindly acknowledge that "I" and @nicolaasjan are building from

and NOT from the ytdl-nightly repo:

https://github.com/ytdl-org/ytdl-nightly/commits/master

As I've written in my initial report (and you further clarified in your most recent reply), these two repos have different "internal update channels"; yes, the nightly channel currently works as planned 👍 , but the "release channel" doesn't...

In the main repo:

update.py looks at
https://yt-dl.org/update/LATEST_VERSION
, which currently returns 2021.12.17.

Make sure any patches leave the desired mechanism in place.

"I" am not applying any "patches" to the downloaded source prior to compilation, while Nico is, however his shouldn't touch at all files associated with the internal update mechanism...

My gut feeling is that the internal update mechanism in the main master repo has been broken due to the new code added to mitigate Security Advisory GHSA-9jqj-9wwh-r5mg, i.e. #32450; this new code handles HTTP[S] redirects differently, but a redirect is indeed necessary when the "release update channel" checks for the latest version:

github.com/ytdl-org/youtube-dl@b6dff40...47214e4

comprises 12 commits; later in the evening, when high temps somewhat subside, I might try compiling from each commit snapshot, to pinpoint the exact one where internal update broke...

Best wishes...

@Vangelis66 commented on GitHub (Jul 19, 2023): > Something similar here with my Python 3.4.4 build: I can replicate; your previous build of version `2023.07.05` behaves as expected: ```console youtube-dl -vU [debug] System config: [] [debug] User config: [] [debug] Custom config: [] [debug] Command-line args: ['-vU'] [debug] Encodings: locale cp1253, fs mbcs, out cp737, pref cp1253 [debug] youtube-dl version 2023.07.05 (single file build) [debug] Lazy loading extractors enabled [debug] Python 3.4.4 (CPython x86 32bit) - Windows-Vista-6.0.6003-SP2 - OpenSSL 1.0.2d 9 Jul 2015 [debug] exe versions: none [debug] Proxy map: {} youtube-dl is up to date (2023.07.05) ``` while your most recent one is BROKEN (in a similar fashion to mine): ```console [debug] System config: [] [debug] User config: [] [debug] Custom config: [] [debug] Command-line args: ['-vU'] [debug] Encodings: locale cp1253, fs mbcs, out cp737, pref cp1253 [debug] youtube-dl version 2023.07.18 [debug] Lazy loading extractors enabled [debug] Single file build [debug] Python 3.4.4 (CPython x86 32bit) - Windows-Vista-6.0.6003-SP2 - OpenSSL 1.0.2d 9 Jul 2015 [debug] exe versions: none [debug] Proxy map: {} Traceback (most recent call last): File "C:\Users\nico\Desktop\youtube-dl_source\youtube_dl\update.py", line 46, in update_self File "c:\Program Files (x86)\Python3.4.4\lib\urllib\request.py", line 470, in open File "c:\Program Files (x86)\Python3.4.4\lib\urllib\request.py", line 580, in http_response File "c:\Program Files (x86)\Python3.4.4\lib\urllib\request.py", line 502, in error File "c:\Program Files (x86)\Python3.4.4\lib\urllib\request.py", line 442, in_call_chain File "c:\Program Files (x86)\Python3.4.4\lib\urllib\request.py", line 685, in http_error_302 File "c:\Program Files (x86)\Python3.4.4\lib\urllib\request.py", line 470, in open File "c:\Program Files (x86)\Python3.4.4\lib\urllib\request.py", line 580, in http_response File "c:\Program Files (x86)\Python3.4.4\lib\urllib\request.py", line 508, in error File "c:\Program Files (x86)\Python3.4.4\lib\urllib\request.py", line 442, in_call_chain File "c:\Program Files (x86)\Python3.4.4\lib\urllib\request.py", line 588, in http_error_default urllib.error.HTTPError: HTTP Error 404: Not Found ERROR: can't find the current version. Please try again later. ``` > Please try again **with a new download**. I had to revamp **the nightly repo** ... Please, kindly acknowledge that "I" and @nicolaasjan are building from https://github.com/ytdl-org/youtube-dl/commits/master and NOT from the `ytdl-nightly` repo: https://github.com/ytdl-org/ytdl-nightly/commits/master As I've written in my [initial report](https://github.com/ytdl-org/youtube-dl/issues/31585#issuecomment-1641276548) (and you further clarified in your most recent reply), these two repos have different "internal update channels"; yes, _the nightly channel currently works as planned_ 👍 , but **the "release channel" doesn't**... > In the main repo: > >`update.py` looks at >https://yt-dl.org/update/LATEST_VERSION >, which currently returns `2021.12.17`. > >Make sure **any patches leave the desired mechanism in place**. "I" am not applying any "patches" to the downloaded source prior to compilation, while **Nico** is, however _his_ shouldn't touch at all files associated with the internal update mechanism... My gut feeling is that the internal update mechanism in the main `master` repo has been **broken** due to the **new code added** to mitigate [Security Advisory GHSA-9jqj-9wwh-r5mg](https://github.com/dirkf/youtube-dl/security/advisories/GHSA-9jqj-9wwh-r5mg), i.e. #32450; this new code handles **HTTP[S] redirects differently**, but **a redirect is indeed necessary** when the "release update channel" checks for the latest version: ![ytdl-redir](https://github.com/ytdl-org/youtube-dl/assets/9669492/fb3d0cf2-f79c-492e-9d21-4006cc191e6f) https://github.com/ytdl-org/youtube-dl/compare/b6dff40...47214e4 comprises **12** commits; later in the evening, when high temps somewhat subside, I might try compiling from each commit snapshot, to pinpoint the **exact one** where internal update broke... Best wishes...

deekerman commented

Author

Owner

@dirkf commented on GitHub (Jul 20, 2023):

Please, kindly acknowledge that "I" and @nicolaasjan are building from https://github.com/ytdl-org/youtube-dl/commits/master ...

Ah. What's happening is that the Host header from the original request is still being sent in the redirected request. This is because we use the header_items() method that is the documented interface to a urllib2.Request object's headers:

Request.header_items()

Return a list of tuples (header_name, header_value) of the Request headers.

Apparently this automagically inserts a Host header into the headers stored in the (undocumented) headers attribute of the Request. While this could be useful internally when preparing the Request to be sent, it's not helpful when we want to transfer the headers to a new Request for the redirect URL and not clearly signalled in the spec.

So this solves the problem:

--- old/youtube-dl/youtube_dl/utils.py
+++ new/youtube-dl/youtube_dl/utils.py
@@ -3014,6 +3014,8 @@
             new_data = None
             remove_headers.extend(['Content-Length', 'Content-Type'])
 
+        # remove unwanted and undocumented Host header for old URL
+        remove_headers.append('Host')
         # NB: don't use dict comprehension for python 2.6 compatibility
         new_headers = dict((k, v) for k, v in req.header_items()
                            if k.title() not in remove_headers)

@dirkf commented on GitHub (Jul 20, 2023): > Please, kindly acknowledge that "I" and @nicolaasjan are building from https://github.com/ytdl-org/youtube-dl/commits/master ... Ah. What's happening is that the `Host` header from the original request is still being sent in the redirected request. This is because we use the `header_items()` method that is the documented interface to a `urllib2.Request` object's headers: >Request.header_items() > > Return a list of tuples (header_name, header_value) of the Request headers. Apparently this automagically inserts a `Host` header into the headers stored in the (undocumented) `headers` attribute of the `Request`. While this could be useful internally when preparing the `Request` to be sent, it's not helpful when we want to transfer the headers to a new `Request` for the redirect URL and not clearly signalled in the spec. So this solves the problem: ```diff --- old/youtube-dl/youtube_dl/utils.py +++ new/youtube-dl/youtube_dl/utils.py @@ -3014,6 +3014,8 @@ new_data = None remove_headers.extend(['Content-Length', 'Content-Type']) + # remove unwanted and undocumented Host header for old URL + remove_headers.append('Host') # NB: don't use dict comprehension for python 2.6 compatibility new_headers = dict((k, v) for k, v in req.header_items() if k.title() not in remove_headers) ```

deekerman commented

Author

Owner

@nicolaasjan commented on GitHub (Jul 20, 2023):

Oops...
Just finished compiling a new version from b2ba24bb02. 😀️

Retrying from 1fa8b86f0b95f2e1488042ceeda8f356ea2a5448...:
OK, now no error:

ytd -vU
[debug] System config: []
[debug] User config: ['--rm-cache-dir', '-i', '-o', '/dev/shm/test-ytd/%(title)s.%(ext)s', '-f', 'bestvideo[height<=1080][ext=mp4]+bestaudio[ext=m4a]/best[ext=mp4]/best', '--no-mtime', '--embed-thumbnail', '--force-ipv4']
[debug] Custom config: []
[debug] Command-line args: ['-vU']
[debug] Encodings: locale UTF-8, fs utf-8, out utf-8, pref UTF-8
[debug] youtube-dl version 2023.07.20
[debug] Lazy loading extractors enabled
[debug] Single file build
[debug] Python 3.8.10 (CPython x86_64 64bit) - Linux-5.4.0-153-generic-x86_64-with-glibc2.29 - OpenSSL 1.1.1f  31 Mar 2020 - glibc 2.31
[debug] exe versions: ffmpeg N-111355-g68e9d2835f-Nico-20230707, ffprobe N-111355-g68e9d2835f-Nico-20230707, phantomjs 2.1.1, rtmpdump 2.4
[debug] Proxy map: {}
youtube-dl is up to date (2023.07.20)
Removing cache dir /home/nico/.cache/youtube-dl ..

@nicolaasjan commented on GitHub (Jul 20, 2023): Oops... Just finished compiling a new version from b2ba24bb026904f3503db71f65d2b1627f08edf1. 😀️ Retrying from 1fa8b86f0b95f2e1488042ceeda8f356ea2a5448...: OK, now no error: ```bash ytd -vU [debug] System config: [] [debug] User config: ['--rm-cache-dir', '-i', '-o', '/dev/shm/test-ytd/%(title)s.%(ext)s', '-f', 'bestvideo[height<=1080][ext=mp4]+bestaudio[ext=m4a]/best[ext=mp4]/best', '--no-mtime', '--embed-thumbnail', '--force-ipv4'] [debug] Custom config: [] [debug] Command-line args: ['-vU'] [debug] Encodings: locale UTF-8, fs utf-8, out utf-8, pref UTF-8 [debug] youtube-dl version 2023.07.20 [debug] Lazy loading extractors enabled [debug] Single file build [debug] Python 3.8.10 (CPython x86_64 64bit) - Linux-5.4.0-153-generic-x86_64-with-glibc2.29 - OpenSSL 1.1.1f 31 Mar 2020 - glibc 2.31 [debug] exe versions: ffmpeg N-111355-g68e9d2835f-Nico-20230707, ffprobe N-111355-g68e9d2835f-Nico-20230707, phantomjs 2.1.1, rtmpdump 2.4 [debug] Proxy map: {} youtube-dl is up to date (2023.07.20) Removing cache dir /home/nico/.cache/youtube-dl .. ```

deekerman commented

Author

Owner

@nicolaasjan commented on GitHub (Jul 20, 2023):

OT
B.t.w., what about those cherry-picked pull requests I mentioned in my previous post here.
Is it possible for you to merge these (or part of them)?

It would save me some time (and prevent future duplicate issues concerning RedTube and SoundCloud). 😀️

@nicolaasjan commented on GitHub (Jul 20, 2023): **OT** B.t.w., what about those cherry-picked pull requests I mentioned in my [previous post](https://github.com/ytdl-org/youtube-dl/issues/31585#issuecomment-1642020828) here. Is it possible for you to merge these (or part of them)? It would save me some time (and prevent future duplicate issues concerning RedTube and SoundCloud). 😀️

deekerman commented

Author

Owner

@Vangelis66 commented on GitHub (Jul 22, 2023):

Pls report any issues with 2, 4, 5.

... Hence 😉 ...

Identify commit for build
... so now the main repo HEAD commit SHA is being passed into the build workflow and added to youtube_dl/version.py as RELEASE_GIT_HEAD, and this is reported in the debug info, similar to yt-dlp.

Currently, Windows ytdl-nightly build 2023.07.18 is the last one that honours the above stipulation:

...
[debug] youtube-dl version 2023.07.18 [47214e46d] (single file build)
[debug] ** This version was built from the latest master code at https://github.com/ytdl-org/youtube-dl.
[debug] ** For support, visit the main site.
...

where 47214e46d is indeed:

https://github.com/ytdl-org/youtube-dl/commit/47214e46d
[compat] Fix old Pythons broken loading of valueless cookie attributes

of the main repo...

Next Windows ytdl-nightly build 2023.07.20 reports:

...
[debug] youtube-dl version 2023.07.20 [383dd024f] (single file build)
[debug] ** This version was built from the latest master code at https://github.com/ytdl-org/youtube-dl.
[debug] ** For support, visit the main site.
...

however 383dd024f is nowhere to be found inside the main repo 😞 :

https://github.com/ytdl-org/youtube-dl/commit/383dd024f =>

This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.

nor is it found inside the ytdl-nightly repo:

https://github.com/ytdl-org/ytdl-nightly/commit/383dd024f =>

This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.

And if we finally move to latest (at this time) Windows ytdl-nightly build 2023.07.21, it reports:

...
[debug] youtube-dl version 2023.07.21 [069471f38] (single file build)
[debug] ** This version was built from the latest master code at https://github.com/ytdl-org/youtube-dl.
[debug] ** For support, visit the main site.
...

where 069471f38, again, is nowhere to be found inside the main repo:

https://github.com/ytdl-org/youtube-dl/commit/069471f38 =>

This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.

but, surprise 😜 , it belongs instead to the ytdl-nightly repo:

https://github.com/ytdl-org/ytdl-nightly/commit/069471f38
[workflows/build.yml] Make PyCrypto the default for Windows

The plot thickens:

yes, the nightly channel currently works as planned 👍

At this time of writing, my assessment above isn't entirely true 😞 ...

Windows ytdl-nightly build 2023.07.17 is the LAST one that will successfully update (via the internal, nightly channel, update mechanism) to most current build 2023.07.21:

youtube-dl -vU => 

[debug] System config: []
[debug] User config: []
[debug] Custom config: []
[debug] Command-line args: ['-vU']
[debug] Encodings: locale cp1253, fs mbcs, out cp737, pref cp1253
[debug] youtube-dl version 2023.07.17 [f24bc9272] (single file build)
[debug] ** This version was built from the latest master code at https://github.com/ytdl-org/youtube-dl.
[debug] ** For support, visit the main site.
[debug] Python 3.4.4 (CPython x86 32bit) - Windows-Vista-6.0.6003-SP2 - OpenSSL 1.0.2d 9 Jul 2015
[debug] exe versions: none
[debug] Proxy map: {}
Latest version: 2023.07.21, Current version: 2023.07.17
Current Build Hash 25eb284c034b2df81e10328104e2da28606c2ef8608a90f0ecb36f46a4d07446
Updating to version 2023.07.21 ...
Updated youtube-dl to version 2023.07.21

Windows build 2023.07.18:

youtube-dl -vU => 

[debug] System config: []
[debug] User config: []
[debug] Custom config: []
[debug] Command-line args: ['-vU']
[debug] Encodings: locale cp1253, fs mbcs, out cp737, pref cp1253
[debug] youtube-dl version 2023.07.18 [47214e46d] (single file build)
[debug] ** This version was built from the latest master code at https://github.com/ytdl-org/youtube-dl.
[debug] ** For support, visit the main site.
[debug] Python 3.4.4 (CPython x86 32bit) - Windows-Vista-6.0.6003-SP2 - OpenSSL 1.0.2d 9 Jul 2015
[debug] exe versions: none
[debug] Proxy map: {}
Latest version: 2023.07.21, Current version: 2023.07.18
Current Build Hash 46739c3e35dd5d50de99ccd8f362127f69ee7707658e250abcaae85defc2c13b
Updating to version 2023.07.21 ...
ERROR: Unable to download latest version; Visit  https://github.com/ytdl-org/ytdl-nightly/releases/latest

and Windows build 2023.07.20:

youtube-dl -vU => 

[debug] System config: []
[debug] User config: []
[debug] Custom config: []
[debug] Command-line args: ['-vU']
[debug] Encodings: locale cp1253, fs mbcs, out cp737, pref cp1253
[debug] youtube-dl version 2023.07.20 [383dd024f] (single file build)
[debug] ** This version was built from the latest master code at https://github.com/ytdl-org/youtube-dl.
[debug] ** For support, visit the main site.
[debug] Python 3.4.4 (CPython x86 32bit) - Windows-Vista-6.0.6003-SP2 - OpenSSL 1.0.2d 9 Jul 2015
[debug] exe versions: none
[debug] Proxy map: {}
Latest version: 2023.07.21, Current version: 2023.07.20
Current Build Hash 710412c62f5907a5aca55e65f681fdd86cbced918a3548607a1c659c464f8d33
Updating to version 2023.07.21 ...
ERROR: Unable to download latest version; Visit  https://github.com/ytdl-org/ytdl-nightly/releases/latest

BOTH FAIL to do so 😭 ...

Question: What evil powers are jinxing this project and why? 😉 ...

@Vangelis66 commented on GitHub (Jul 22, 2023): > Pls report any issues with 2, 4, 5. ... Hence 😉 ... >- [x] **Identify commit for build** ... so **now the main repo HEAD commit SHA is being passed into the build workflow and added to youtube_dl/version.py as RELEASE_GIT_HEAD, and this is reported in the debug info**, similar to yt-dlp. Currently, Windows `ytdl-nightly` build [2023.07.18](https://github.com/ytdl-org/ytdl-nightly/releases/tag/2023.07.18) is the last one that honours the above stipulation: ```console ... [debug] youtube-dl version 2023.07.18 [47214e46d] (single file build) [debug] ** This version was built from the latest master code at https://github.com/ytdl-org/youtube-dl. [debug] ** For support, visit the main site. ... ``` where `47214e46d` is indeed: `https://github.com/ytdl-org/youtube-dl/commit/47214e46d` **[compat] Fix old Pythons broken loading of valueless cookie attributes** of _**the main repo**_... Next Windows `ytdl-nightly` build [2023.07.20](https://github.com/ytdl-org/ytdl-nightly/releases/tag/2023.07.20) reports: ```console ... [debug] youtube-dl version 2023.07.20 [383dd024f] (single file build) [debug] ** This version was built from the latest master code at https://github.com/ytdl-org/youtube-dl. [debug] ** For support, visit the main site. ... ``` however `383dd024f` is nowhere to be found inside **the main repo** 😞 : `https://github.com/ytdl-org/youtube-dl/commit/383dd024f` => > **This commit does not belong to any branch on this repository**, and may belong to a fork outside of the repository. nor is it found inside the `ytdl-nightly` repo: `https://github.com/ytdl-org/ytdl-nightly/commit/383dd024f` => > **This commit does not belong to any branch on this repository**, and may belong to a fork outside of the repository. And if we finally move to **latest** (at this time) Windows `ytdl-nightly` build [2023.07.21](https://github.com/ytdl-org/ytdl-nightly/releases/tag/2023.07.21), it reports: ```console ... [debug] youtube-dl version 2023.07.21 [069471f38] (single file build) [debug] ** This version was built from the latest master code at https://github.com/ytdl-org/youtube-dl. [debug] ** For support, visit the main site. ... ``` where `069471f38`, again, is nowhere to be found inside **the main repo**: `https://github.com/ytdl-org/youtube-dl/commit/069471f38` => > **This commit does not belong to any branch on this repository**, and may belong to a fork outside of the repository. but, surprise 😜 , it belongs instead to the `ytdl-nightly` repo: `https://github.com/ytdl-org/ytdl-nightly/commit/069471f38` **[workflows/build.yml] Make PyCrypto the default for Windows** The plot thickens: > yes, the **nightly channel currently works as planned** 👍 At this time of writing, my assessment above **isn't entirely true** 😞 ... Windows `ytdl-nightly` build [2023.07.17](https://github.com/ytdl-org/ytdl-nightly/releases/tag/2023.07.17) is the LAST one that will **successfully update** (via the internal, **nightly channel**, update mechanism) to most current build [2023.07.21](https://github.com/ytdl-org/ytdl-nightly/releases/tag/2023.07.21): ```console youtube-dl -vU => [debug] System config: [] [debug] User config: [] [debug] Custom config: [] [debug] Command-line args: ['-vU'] [debug] Encodings: locale cp1253, fs mbcs, out cp737, pref cp1253 [debug] youtube-dl version 2023.07.17 [f24bc9272] (single file build) [debug] ** This version was built from the latest master code at https://github.com/ytdl-org/youtube-dl. [debug] ** For support, visit the main site. [debug] Python 3.4.4 (CPython x86 32bit) - Windows-Vista-6.0.6003-SP2 - OpenSSL 1.0.2d 9 Jul 2015 [debug] exe versions: none [debug] Proxy map: {} Latest version: 2023.07.21, Current version: 2023.07.17 Current Build Hash 25eb284c034b2df81e10328104e2da28606c2ef8608a90f0ecb36f46a4d07446 Updating to version 2023.07.21 ... Updated youtube-dl to version 2023.07.21 ``` Windows build [2023.07.18](https://github.com/ytdl-org/ytdl-nightly/releases/tag/2023.07.18): ```console youtube-dl -vU => [debug] System config: [] [debug] User config: [] [debug] Custom config: [] [debug] Command-line args: ['-vU'] [debug] Encodings: locale cp1253, fs mbcs, out cp737, pref cp1253 [debug] youtube-dl version 2023.07.18 [47214e46d] (single file build) [debug] ** This version was built from the latest master code at https://github.com/ytdl-org/youtube-dl. [debug] ** For support, visit the main site. [debug] Python 3.4.4 (CPython x86 32bit) - Windows-Vista-6.0.6003-SP2 - OpenSSL 1.0.2d 9 Jul 2015 [debug] exe versions: none [debug] Proxy map: {} Latest version: 2023.07.21, Current version: 2023.07.18 Current Build Hash 46739c3e35dd5d50de99ccd8f362127f69ee7707658e250abcaae85defc2c13b Updating to version 2023.07.21 ... ERROR: Unable to download latest version; Visit https://github.com/ytdl-org/ytdl-nightly/releases/latest ``` and Windows build [2023.07.20](https://github.com/ytdl-org/ytdl-nightly/releases/tag/2023.07.20): ```console youtube-dl -vU => [debug] System config: [] [debug] User config: [] [debug] Custom config: [] [debug] Command-line args: ['-vU'] [debug] Encodings: locale cp1253, fs mbcs, out cp737, pref cp1253 [debug] youtube-dl version 2023.07.20 [383dd024f] (single file build) [debug] ** This version was built from the latest master code at https://github.com/ytdl-org/youtube-dl. [debug] ** For support, visit the main site. [debug] Python 3.4.4 (CPython x86 32bit) - Windows-Vista-6.0.6003-SP2 - OpenSSL 1.0.2d 9 Jul 2015 [debug] exe versions: none [debug] Proxy map: {} Latest version: 2023.07.21, Current version: 2023.07.20 Current Build Hash 710412c62f5907a5aca55e65f681fdd86cbced918a3548607a1c659c464f8d33 Updating to version 2023.07.21 ... ERROR: Unable to download latest version; Visit https://github.com/ytdl-org/ytdl-nightly/releases/latest ``` BOTH FAIL to do so 😭 ... **Question**: What evil powers are jinxing this project and why? 😉 ...

deekerman commented

Author

Owner

@dirkf commented on GitHub (Jul 23, 2023):

The redirect fix was only committed on 20 July so the problem with fetching the new version isn't fixed until 20230721 or later.

Also, instead of passing through the HEAD commit after the rebase, I think the latest commit from upstream/master should be passed through.

@dirkf commented on GitHub (Jul 23, 2023): The redirect fix was only committed on 20 July so the problem with fetching the new version isn't fixed until 20230721 or later. Also, instead of passing through the HEAD commit **after** the rebase, I think the latest commit from `upstream/master` should be passed through.

deekerman commented

Author

Owner

@Vangelis66 commented on GitHub (Jul 27, 2023):

Build workflow runs should be triggered only when the actual master code gets updated
The scheduled rebase workflow now calls the build workflow if the rebase changes the HEAD, and the build workflow is no longer separately scheduled.

Pardon me for asking 😉 , but will there be any newer ytdl-nightly release after the last one, 2023.07.21 ?
That one was built on master branch snapshot 1fa8b86f0b, while now (since July 25th, actually) the master HEAD is at 0861812d72; is the whole process still automated, or is human intervention required to trigger the Build workflow?

OT, but somehow related: @nicolaasjan: Will you be releasing anything newer than youtube-dl.exe v2023.07.22.1 ?

Thanks in advance to both! ❤️

@Vangelis66 commented on GitHub (Jul 27, 2023): > - [x] `Build` workflow runs should be triggered only when the actual master code gets updated The scheduled rebase workflow now calls the build workflow if the rebase changes the HEAD, and the build workflow is no longer separately scheduled. Pardon me for asking 😉 , but will there be any newer `ytdl-nightly` release after the last one, [2023.07.21](https://github.com/ytdl-org/ytdl-nightly/releases/tag/2023.07.21) ? That one was built on `master` branch snapshot 1fa8b86f0b95f2e1488042ceeda8f356ea2a5448, while now (since **July 25th**, actually) the `master` HEAD is at 0861812d7208310a03909502b1610f5e89d04401; is the whole process still automated, or is human intervention required to trigger the `Build` workflow? OT, but somehow related: @nicolaasjan: Will you be releasing anything newer than **youtube-dl.exe** v`2023.07.22.1` ? Thanks in advance to both! :heart:

deekerman commented

Author

Owner

@Vangelis66 commented on GitHub (Jul 27, 2023):

... So, either a) Google/Youtube 👿 are on to something new, b) they don't like recent yt-dl code or c) recent yt-dl code is in a bad state...

I have been encountering lately random errors on Youtube while using the latest ytdl-nightly build; not only age-gated IDs are affected, but also "normal" ones; this is a log from 30min ago:

yt-dl -vF "4jduuQh-Uho" => 

[debug] System config: []
[debug] User config: []
[debug] Custom config: []
[debug] Command-line args: ['--ffmpeg-location', '.\\FFmpeg', '--external-downloader-args', '-v 8 -stats', '-vF', '4jduuQh-Uho']
[debug] Encodings: locale cp1253, fs mbcs, out cp737, pref cp1253
[debug] youtube-dl version 2023.07.21 [069471f38] (single file build)
[debug] ** This version was built from the latest master code at https://github.com/ytdl-org/youtube-dl.
[debug] ** For support, visit the main site.
[debug] Python 3.4.4 (CPython x86 32bit) - Windows-Vista-6.0.6003-SP2 - OpenSSL 1.0.2d 9 Jul 2015
[debug] exe versions: ffmpeg n6.1-dev-1700-N-111584-ga4e6168, ffprobe n6.1-dev-1700-N-111584-ga4e6168, phantomjs 2.1.1, rtmpdump 2.4
[debug] Proxy map: {}
[youtube] 4jduuQh-Uho: Downloading webpage
WARNING: Unable to download webpage: HTTP Error 302: The HTTP server returned a redirect error that would lead to an infinite loop.
The last 30x error message was:
Found
[youtube] 4jduuQh-Uho: Downloading API JSON
WARNING: unable to extract player URL; please report this issue on https://yt-dl.org/bug . Make sure you are using the latest version; type  youtube-dl -U  to update. Be sure to call youtube-dl with the --verbose flag and include its complete output.
WARNING: [youtube] Unable to decode n-parameter: download likely to be throttled
 (expected string or buffer Traceback (most recent call last):
  File "D:\a\ytdl-nightly\ytdl-nightly\youtube_dl\extractor\youtube.py", line 1679, in _n_descramble
  File "D:\a\ytdl-nightly\ytdl-nightly\youtube_dl\extractor\youtube.py", line 1644, in _extract_n_function
  File "D:\a\ytdl-nightly\ytdl-nightly\youtube_dl\extractor\youtube.py", line 1476, in _extract_player_info
  File "C:\hostedtoolcache\windows\Python\3.4.4\x86\lib\re.py", line 170, in search
TypeError: expected string or buffer
)
[youtube] 4jduuQh-Uho: Downloading API JSON
[info] Available formats for 4jduuQh-Uho:
format code  extension  resolution note
249          webm       audio only tiny   47k , webm_dash container, opus  (48000Hz), 1.05MiB
250          webm       audio only tiny   59k , webm_dash container, opus  (48000Hz), 1.31MiB
251          webm       audio only tiny  105k , webm_dash container, opus  (48000Hz), 2.32MiB
140          m4a        audio only tiny  129k , m4a_dash container, mp4a.40.2 (44100Hz), 2.86MiB
160          mp4        256x144    144p   13k , mp4_dash container, avc1.4d400c, 30fps, video only, 310.72KiB
278          webm       256x144    144p   15k , webm_dash container, vp9, 30fps, video only, 350.62KiB
242          webm       426x240    240p   22k , webm_dash container, vp9, 30fps, video only, 509.02KiB
133          mp4        426x240    240p   23k , mp4_dash container, avc1.4d4015, 30fps, video only, 534.08KiB
243          webm       640x360    360p   38k , webm_dash container, vp9, 30fps, video only, 865.13KiB
134          mp4        640x360    360p   41k , mp4_dash container, avc1.4d401e, 30fps, video only, 938.46KiB
244          webm       854x480    480p   59k , webm_dash container, vp9, 30fps, video only, 1.31MiB
135          mp4        854x480    480p   64k , mp4_dash container, avc1.4d401f, 30fps, video only, 1.43MiB
247          webm       1280x720   720p  112k , webm_dash container, vp9, 30fps, video only, 2.48MiB
136          mp4        1280x720   720p  119k , mp4_dash container, avc1.4d401f, 30fps, video only, 2.62MiB
18           mp4        640x360    360p  208k , avc1.42001E, 30fps, mp4a.40.2 (44100Hz), 4.60MiB
22           mp4        1280x720   720p  247k , avc1.64001F, 30fps, mp4a.40.2 (44100Hz) (best)

... and a second one from ca. 20min ago:

yt-dl -vF "p7FCgw_GlWc"

[debug] System config: []
[debug] User config: []
[debug] Custom config: []
[debug] Command-line args: ['--ffmpeg-location', '.\\FFmpeg', '--external-downloader-args', '-v 8 -stats', '-vF', 'p7FCgw_GlWc']
[debug] Encodings: locale cp1253, fs mbcs, out cp737, pref cp1253
[debug] youtube-dl version 2023.07.21 [069471f38] (single file build)
[debug] ** This version was built from the latest master code at https://github.com/ytdl-org/youtube-dl.
[debug] ** For support, visit the main site.
[debug] Python 3.4.4 (CPython x86 32bit) - Windows-Vista-6.0.6003-SP2 - OpenSSL 1.0.2d 9 Jul 2015
[debug] exe versions: ffmpeg n6.1-dev-1700-N-111584-ga4e6168, ffprobe n6.1-dev-1700-N-111584-ga4e6168, phantomjs 2.1.1, rtmpdump 2.4
[debug] Proxy map: {}
[youtube] p7FCgw_GlWc: Downloading webpage
WARNING: Unable to download webpage: HTTP Error 302: The HTTP server returned a redirect error that would lead to an infinite loop.
The last 30x error message was:
Found
[youtube] p7FCgw_GlWc: Downloading API JSON
[youtube] Confirming age
WARNING: unable to extract player URL; please report this issue on https://yt-dl.org/bug . Make sure you are using the latest version; type  youtube-dl -U  to update. Be sure to call youtube-dl with the --verbose flag and include its complete output.
Traceback (most recent call last):
  File "__main__.py", line 19, in <module>
  File "D:\a\ytdl-nightly\ytdl-nightly\youtube_dl\__init__.py", line 473, in main
  File "D:\a\ytdl-nightly\ytdl-nightly\youtube_dl\__init__.py", line 463, in _real_main
  File "D:\a\ytdl-nightly\ytdl-nightly\youtube_dl\YoutubeDL.py", line 2226, in download
  File "D:\a\ytdl-nightly\ytdl-nightly\youtube_dl\YoutubeDL.py", line 859, in extract_info
  File "D:\a\ytdl-nightly\ytdl-nightly\youtube_dl\YoutubeDL.py", line 866, in wrapper
  File "D:\a\ytdl-nightly\ytdl-nightly\youtube_dl\YoutubeDL.py", line 962, in __extract_info
  File "D:\a\ytdl-nightly\ytdl-nightly\youtube_dl\extractor\common.py", line 564, in extract
  File "D:\a\ytdl-nightly\ytdl-nightly\youtube_dl\extractor\youtube.py", line 1891, in _real_extract
  File "D:\a\ytdl-nightly\ytdl-nightly\youtube_dl\extractor\youtube.py", line 319, in _extract_ytcfg
  File "D:\a\ytdl-nightly\ytdl-nightly\youtube_dl\extractor\common.py", line 1021, in _search_regex
  File "C:\hostedtoolcache\windows\Python\3.4.4\x86\lib\re.py", line 170, in search
TypeError: expected string or buffer

That second log isn't edited, code execution did not proceed to list any formats at all 😢 ...
Obviously, the info to pay attention to is:

WARNING: Unable to download webpage: HTTP Error 302: The HTTP server returned a redirect error that would lead to an infinite loop.

As I said beforehand, the errors encountered are completely random; next invocation of yt-dl with the exact command will work as expected...

Yesterday, I did compile locally latest master branch code snapshot (1fa8b86), so I did some testing there, too; while the exact wording on the errors is somewhat different, the net result is the same, e.g.:

yt-dl -vF "7t0SqerlBA0" => 

[debug] System config: []
[debug] User config: []
[debug] Custom config: []
[debug] Command-line args: ['--ffmpeg-location', '.\\FFmpeg', '--external-downloader-args', '-v 8 -stats', '-vF', '7t0SqerlBA0']
[debug] Encodings: locale cp1253, fs mbcs, out cp737, pref cp1253
[debug] youtube-dl version 2023.07.25.1411
[debug] Lazy loading extractors enabled
[debug] Single file build
[[debug] Python 3.4.4 (CPython x86 32bit) - Windows-Vista-6.0.6003-SP2 - OpenSSL 1.0.2d 9 Jul 2015
[debug] exe versions: ffmpeg n6.1-dev-1700-N-111584-ga4e6168, ffprobe n6.1-dev-1700-N-111584-ga4e6168, phantomjs 2.1.1, rtmpdump 2.4
[debug] Proxy map: {}
[youtube] 7t0SqerlBA0: Downloading webpage
[youtube] 7t0SqerlBA0: Downloading API JSON
[youtube] Confirming age
WARNING: unable to extract player URL; please report this issue on https://yt-dl.org/bug . Make sure you are using the latest version; type  youtube-dl -U  to update. Be sure to call youtube-dl with the --verbose flag and include its complete output.
WARNING: Cannot extract signature timestamp without player_url.
[youtube] 7t0SqerlBA0: Downloading API JSON
WARNING: unable to extract player URL; please report this issue on https://yt-dl.org/bug . Make sure you are using the latest version; type  youtube-dl -U  to update. Be sure to call youtube-dl with the --verbose flag and include its complete output.
WARNING: [youtube] Unable to decode n-parameter: download likely to be throttled
 (expected string or buffer Traceback (most recent call last):
  File "C:\Python3410-32\yt-dl_pycompile\youtube_dl\extractor\youtube.py", line 1679, in _n_descramble
    self._player_cache[player_id] = self._extract_n_function(video_id, player_url)
  File "C:\Python3410-32\yt-dl_pycompile\youtube_dl\extractor\youtube.py", line 1644, in _extract_n_function
    player_id = self._extract_player_info(player_url)
  File "C:\Python3410-32\yt-dl_pycompile\youtube_dl\extractor\youtube.py", line 1476, in _extract_player_info
    id_m = re.search(player_re, player_url)
  File "C:\Python3410-32\lib\re.py", line 170, in search
    return _compile(pattern, flags).search(string)
TypeError: expected string or buffer
)
[youtube] 7t0SqerlBA0: Downloading API JSON
[info] Available formats for 7t0SqerlBA0:
format code  extension  resolution note
249-0        webm       audio only tiny    3k , webm_dash container, opus  (48000Hz), 26.82KiB
250-0        webm       audio only tiny    3k , webm_dash container, opus  (48000Hz), 26.82KiB
251-0        webm       audio only tiny    3k , webm_dash container, opus  (48000Hz), 26.82KiB
249-1        webm       audio only tiny    3k , webm_dash container, opus  (48000Hz), 26.82KiB
250-1        webm       audio only tiny    3k , webm_dash container, opus  (48000Hz), 26.82KiB
251-1        webm       audio only tiny    3k , webm_dash container, opus  (48000Hz), 26.82KiB
140-0        m4a        audio only tiny  129k , m4a_dash container, mp4a.40.2 (44100Hz), 949.77KiB
140-1        m4a        audio only tiny  129k , m4a_dash container, mp4a.40.2 (44100Hz), 949.77KiB
278          webm       256x144    144p   34k , webm_dash container, vp9, 30fps, video only, 251.25KiB
160          mp4        256x144    144p   34k , mp4_dash container, avc1.4d400c, 30fps, video only, 253.46KiB
242          webm       426x240    240p   41k , webm_dash container, vp9, 30fps, video only, 301.54KiB
133          mp4        426x240    240p  113k , mp4_dash container, avc1.4d4015, 30fps, video only, 834.73KiB
243          webm       640x360    360p  108k , webm_dash container, vp9, 30fps, video only, 796.99KiB
134          mp4        640x360    360p  229k , mp4_dash container, avc1.4d401e, 30fps, video only, 1.64MiB
135          mp4        854x480    480p  451k , mp4_dash container, avc1.4d401f, 30fps, video only, 3.23MiB
244          webm       854x480    480p  473k , webm_dash container, vp9, 30fps, video only, 3.39MiB
136          mp4        1280x720   720p 1352k , mp4_dash container, avc1.4d401f, 30fps, video only, 9.67MiB
247          webm       1280x720   720p 2106k , webm_dash container, vp9, 30fps, video only, 15.07MiB
302          webm       1280x720   720p60 2631k , webm_dash container, vp9, 60fps, video only, 18.82MiB
298          mp4        1280x720   720p60 3467k , mp4_dash container, avc1.4d4020, 60fps, video only, 24.80MiB
303          webm       1920x1080  1080p60 4397k , webm_dash container, vp9, 60fps, video only, 31.45MiB
299          mp4        1920x1080  1080p60 5753k , mp4_dash container, avc1.64002a, 60fps, video only, 41.15MiB
308          webm       2560x1440  1440p60 13230k , webm_dash container, vp9, 60fps, video only, 94.63MiB
315          webm       3840x2160  2160p60 26523k , webm_dash container, vp9, 60fps, video only, 189.71MiB
18           mp4        640x360    360p  357k , avc1.42001E, 30fps, mp4a.40.2 (44100Hz)
22           mp4        1280x720   720p 1480k , avc1.64001F, 30fps, mp4a.40.2 (44100Hz) (best)

... And, 3min after, the same command succeeds:

yt-dl -vF "7t0SqerlBA0" =>

[debug] System config: []
[debug] User config: []
[debug] Custom config: []
[debug] Command-line args: ['--ffmpeg-location', '.\\FFmpeg', '--external-downloader-args', '-v 8 -stats', '-vF', '7t0SqerlBA0']
[debug] Encodings: locale cp1253, fs mbcs, out cp737, pref cp1253
[debug] youtube-dl version 2023.07.25.1411
[debug] Lazy loading extractors enabled
[debug] Single file build
[debug] Python 3.4.4 (CPython x86 32bit) - Windows-Vista-6.0.6003-SP2 - OpenSSL 1.0.2d 9 Jul 2015
[debug] exe versions: ffmpeg n6.1-dev-1700-N-111584-ga4e6168, ffprobe n6.1-dev-1700-N-111584-ga4e6168, phantomjs 2.1.1, rtmpdump 2.4
[debug] Proxy map: {}
[youtube] 7t0SqerlBA0: Downloading webpage
[youtube] Confirming age
[youtube] 7t0SqerlBA0: Downloading API JSON
[youtube] 7t0SqerlBA0: Downloading player 0e6aaa83
[debug] [youtube] Decrypted nsig proJhhsbIXZsTigaH_ => u9mwfbjD7Gvtyg
[debug] [youtube] Decrypted nsig 3nr2nEIoH10xrGVXqk => wP3tEGGA6hi0Zg
[info] Available formats for 7t0SqerlBA0:
format code  extension  resolution note
249-0        webm       audio only tiny    3k , webm_dash container, opus  (48000Hz), 26.82KiB
250-0        webm       audio only tiny    3k , webm_dash container, opus  (48000Hz), 26.82KiB
251-0        webm       audio only tiny    3k , webm_dash container, opus  (48000Hz), 26.82KiB
249-1        webm       audio only tiny    3k , webm_dash container, opus  (48000Hz), 26.82KiB
250-1        webm       audio only tiny    3k , webm_dash container, opus  (48000Hz), 26.82KiB
251-1        webm       audio only tiny    3k , webm_dash container, opus  (48000Hz), 26.82KiB
140-0        m4a        audio only tiny  129k , m4a_dash container, mp4a.40.2 (44100Hz), 949.77KiB
140-1        m4a        audio only tiny  129k , m4a_dash container, mp4a.40.2 (44100Hz), 949.77KiB
278          webm       256x144    144p   34k , webm_dash container, vp9, 30fps, video only, 251.25KiB
160          mp4        256x144    144p   34k , mp4_dash container, avc1.4d400c, 30fps, video only, 253.46KiB
242          webm       426x240    240p   41k , webm_dash container, vp9, 30fps, video only, 301.54KiB
133          mp4        426x240    240p  113k , mp4_dash container, avc1.4d4015, 30fps, video only, 834.73KiB
243          webm       640x360    360p  108k , webm_dash container, vp9, 30fps, video only, 796.99KiB
134          mp4        640x360    360p  229k , mp4_dash container, avc1.4d401e, 30fps, video only, 1.64MiB
135          mp4        854x480    480p  451k , mp4_dash container, avc1.4d401f, 30fps, video only, 3.23MiB
244          webm       854x480    480p  473k , webm_dash container, vp9, 30fps, video only, 3.39MiB
136          mp4        1280x720   720p 1352k , mp4_dash container, avc1.4d401f, 30fps, video only, 9.67MiB
247          webm       1280x720   720p 2106k , webm_dash container, vp9, 30fps, video only, 15.07MiB
302          webm       1280x720   720p60 2631k , webm_dash container, vp9, 60fps, video only, 18.82MiB
298          mp4        1280x720   720p60 3467k , mp4_dash container, avc1.4d4020, 60fps, video only, 24.80MiB
303          webm       1920x1080  1080p60 4397k , webm_dash container, vp9, 60fps, video only, 31.45MiB
299          mp4        1920x1080  1080p60 5753k , mp4_dash container, avc1.64002a, 60fps, video only, 41.15MiB
308          webm       2560x1440  1440p60 13230k , webm_dash container, vp9, 60fps, video only, 94.63MiB
315          webm       3840x2160  2160p60 26523k , webm_dash container, vp9, 60fps, video only, 189.71MiB
18           mp4        640x360    360p  357k , avc1.42001E, 30fps, mp4a.40.2 (44100Hz)
22           mp4        1280x720   720p 1480k , avc1.64001F, 30fps, mp4a.40.2 (44100Hz) (best)

In desperation, I even suspected possible interference from my AntiVirus/Security Suite, but whitelisting youtube-dl.exe or even temporarily switching it OFF had no effect on the Youtube errors (they still randomly happen) ...

Any ideas, anyone?

Since I have no reliable way of reproducing these errors, no New issue has been submitted...

@Vangelis66 commented on GitHub (Jul 27, 2023): ... So, either a) **Google/Youtube** 👿 are on to something new, b) they don't like **recent** `yt-dl` code or c) **recent** `yt-dl` code is in a bad state... I have been encountering lately **random** errors on `Youtube` while using the latest `ytdl-nightly` build; not only **age-gated** IDs are affected, but also "normal" ones; this is a log from 30min ago: ```console yt-dl -vF "4jduuQh-Uho" => [debug] System config: [] [debug] User config: [] [debug] Custom config: [] [debug] Command-line args: ['--ffmpeg-location', '.\\FFmpeg', '--external-downloader-args', '-v 8 -stats', '-vF', '4jduuQh-Uho'] [debug] Encodings: locale cp1253, fs mbcs, out cp737, pref cp1253 [debug] youtube-dl version 2023.07.21 [069471f38] (single file build) [debug] ** This version was built from the latest master code at https://github.com/ytdl-org/youtube-dl. [debug] ** For support, visit the main site. [debug] Python 3.4.4 (CPython x86 32bit) - Windows-Vista-6.0.6003-SP2 - OpenSSL 1.0.2d 9 Jul 2015 [debug] exe versions: ffmpeg n6.1-dev-1700-N-111584-ga4e6168, ffprobe n6.1-dev-1700-N-111584-ga4e6168, phantomjs 2.1.1, rtmpdump 2.4 [debug] Proxy map: {} [youtube] 4jduuQh-Uho: Downloading webpage WARNING: Unable to download webpage: HTTP Error 302: The HTTP server returned a redirect error that would lead to an infinite loop. The last 30x error message was: Found [youtube] 4jduuQh-Uho: Downloading API JSON WARNING: unable to extract player URL; please report this issue on https://yt-dl.org/bug . Make sure you are using the latest version; type youtube-dl -U to update. Be sure to call youtube-dl with the --verbose flag and include its complete output. WARNING: [youtube] Unable to decode n-parameter: download likely to be throttled (expected string or buffer Traceback (most recent call last): File "D:\a\ytdl-nightly\ytdl-nightly\youtube_dl\extractor\youtube.py", line 1679, in _n_descramble File "D:\a\ytdl-nightly\ytdl-nightly\youtube_dl\extractor\youtube.py", line 1644, in _extract_n_function File "D:\a\ytdl-nightly\ytdl-nightly\youtube_dl\extractor\youtube.py", line 1476, in _extract_player_info File "C:\hostedtoolcache\windows\Python\3.4.4\x86\lib\re.py", line 170, in search TypeError: expected string or buffer ) [youtube] 4jduuQh-Uho: Downloading API JSON [info] Available formats for 4jduuQh-Uho: format code extension resolution note 249 webm audio only tiny 47k , webm_dash container, opus (48000Hz), 1.05MiB 250 webm audio only tiny 59k , webm_dash container, opus (48000Hz), 1.31MiB 251 webm audio only tiny 105k , webm_dash container, opus (48000Hz), 2.32MiB 140 m4a audio only tiny 129k , m4a_dash container, mp4a.40.2 (44100Hz), 2.86MiB 160 mp4 256x144 144p 13k , mp4_dash container, avc1.4d400c, 30fps, video only, 310.72KiB 278 webm 256x144 144p 15k , webm_dash container, vp9, 30fps, video only, 350.62KiB 242 webm 426x240 240p 22k , webm_dash container, vp9, 30fps, video only, 509.02KiB 133 mp4 426x240 240p 23k , mp4_dash container, avc1.4d4015, 30fps, video only, 534.08KiB 243 webm 640x360 360p 38k , webm_dash container, vp9, 30fps, video only, 865.13KiB 134 mp4 640x360 360p 41k , mp4_dash container, avc1.4d401e, 30fps, video only, 938.46KiB 244 webm 854x480 480p 59k , webm_dash container, vp9, 30fps, video only, 1.31MiB 135 mp4 854x480 480p 64k , mp4_dash container, avc1.4d401f, 30fps, video only, 1.43MiB 247 webm 1280x720 720p 112k , webm_dash container, vp9, 30fps, video only, 2.48MiB 136 mp4 1280x720 720p 119k , mp4_dash container, avc1.4d401f, 30fps, video only, 2.62MiB 18 mp4 640x360 360p 208k , avc1.42001E, 30fps, mp4a.40.2 (44100Hz), 4.60MiB 22 mp4 1280x720 720p 247k , avc1.64001F, 30fps, mp4a.40.2 (44100Hz) (best) ``` ... and a second one from ca. 20min ago: ```console yt-dl -vF "p7FCgw_GlWc" [debug] System config: [] [debug] User config: [] [debug] Custom config: [] [debug] Command-line args: ['--ffmpeg-location', '.\\FFmpeg', '--external-downloader-args', '-v 8 -stats', '-vF', 'p7FCgw_GlWc'] [debug] Encodings: locale cp1253, fs mbcs, out cp737, pref cp1253 [debug] youtube-dl version 2023.07.21 [069471f38] (single file build) [debug] ** This version was built from the latest master code at https://github.com/ytdl-org/youtube-dl. [debug] ** For support, visit the main site. [debug] Python 3.4.4 (CPython x86 32bit) - Windows-Vista-6.0.6003-SP2 - OpenSSL 1.0.2d 9 Jul 2015 [debug] exe versions: ffmpeg n6.1-dev-1700-N-111584-ga4e6168, ffprobe n6.1-dev-1700-N-111584-ga4e6168, phantomjs 2.1.1, rtmpdump 2.4 [debug] Proxy map: {} [youtube] p7FCgw_GlWc: Downloading webpage WARNING: Unable to download webpage: HTTP Error 302: The HTTP server returned a redirect error that would lead to an infinite loop. The last 30x error message was: Found [youtube] p7FCgw_GlWc: Downloading API JSON [youtube] Confirming age WARNING: unable to extract player URL; please report this issue on https://yt-dl.org/bug . Make sure you are using the latest version; type youtube-dl -U to update. Be sure to call youtube-dl with the --verbose flag and include its complete output. Traceback (most recent call last): File "__main__.py", line 19, in <module> File "D:\a\ytdl-nightly\ytdl-nightly\youtube_dl\__init__.py", line 473, in main File "D:\a\ytdl-nightly\ytdl-nightly\youtube_dl\__init__.py", line 463, in _real_main File "D:\a\ytdl-nightly\ytdl-nightly\youtube_dl\YoutubeDL.py", line 2226, in download File "D:\a\ytdl-nightly\ytdl-nightly\youtube_dl\YoutubeDL.py", line 859, in extract_info File "D:\a\ytdl-nightly\ytdl-nightly\youtube_dl\YoutubeDL.py", line 866, in wrapper File "D:\a\ytdl-nightly\ytdl-nightly\youtube_dl\YoutubeDL.py", line 962, in __extract_info File "D:\a\ytdl-nightly\ytdl-nightly\youtube_dl\extractor\common.py", line 564, in extract File "D:\a\ytdl-nightly\ytdl-nightly\youtube_dl\extractor\youtube.py", line 1891, in _real_extract File "D:\a\ytdl-nightly\ytdl-nightly\youtube_dl\extractor\youtube.py", line 319, in _extract_ytcfg File "D:\a\ytdl-nightly\ytdl-nightly\youtube_dl\extractor\common.py", line 1021, in _search_regex File "C:\hostedtoolcache\windows\Python\3.4.4\x86\lib\re.py", line 170, in search TypeError: expected string or buffer ``` That second log isn't edited, **code execution did not proceed to list any formats at all** 😢 ... Obviously, the info to pay attention to is: ```console WARNING: Unable to download webpage: HTTP Error 302: The HTTP server returned a redirect error that would lead to an infinite loop. ``` As I said beforehand, the errors encountered are **completely random**; next invocation of `yt-dl` with the exact command will work as expected... Yesterday, I did compile locally latest `master` branch code snapshot (1fa8b86), so I did some testing there, too; while the exact wording on the errors is somewhat different, the net result is the same, e.g.: ```console yt-dl -vF "7t0SqerlBA0" => [debug] System config: [] [debug] User config: [] [debug] Custom config: [] [debug] Command-line args: ['--ffmpeg-location', '.\\FFmpeg', '--external-downloader-args', '-v 8 -stats', '-vF', '7t0SqerlBA0'] [debug] Encodings: locale cp1253, fs mbcs, out cp737, pref cp1253 [debug] youtube-dl version 2023.07.25.1411 [debug] Lazy loading extractors enabled [debug] Single file build [[debug] Python 3.4.4 (CPython x86 32bit) - Windows-Vista-6.0.6003-SP2 - OpenSSL 1.0.2d 9 Jul 2015 [debug] exe versions: ffmpeg n6.1-dev-1700-N-111584-ga4e6168, ffprobe n6.1-dev-1700-N-111584-ga4e6168, phantomjs 2.1.1, rtmpdump 2.4 [debug] Proxy map: {} [youtube] 7t0SqerlBA0: Downloading webpage [youtube] 7t0SqerlBA0: Downloading API JSON [youtube] Confirming age WARNING: unable to extract player URL; please report this issue on https://yt-dl.org/bug . Make sure you are using the latest version; type youtube-dl -U to update. Be sure to call youtube-dl with the --verbose flag and include its complete output. WARNING: Cannot extract signature timestamp without player_url. [youtube] 7t0SqerlBA0: Downloading API JSON WARNING: unable to extract player URL; please report this issue on https://yt-dl.org/bug . Make sure you are using the latest version; type youtube-dl -U to update. Be sure to call youtube-dl with the --verbose flag and include its complete output. WARNING: [youtube] Unable to decode n-parameter: download likely to be throttled (expected string or buffer Traceback (most recent call last): File "C:\Python3410-32\yt-dl_pycompile\youtube_dl\extractor\youtube.py", line 1679, in _n_descramble self._player_cache[player_id] = self._extract_n_function(video_id, player_url) File "C:\Python3410-32\yt-dl_pycompile\youtube_dl\extractor\youtube.py", line 1644, in _extract_n_function player_id = self._extract_player_info(player_url) File "C:\Python3410-32\yt-dl_pycompile\youtube_dl\extractor\youtube.py", line 1476, in _extract_player_info id_m = re.search(player_re, player_url) File "C:\Python3410-32\lib\re.py", line 170, in search return _compile(pattern, flags).search(string) TypeError: expected string or buffer ) [youtube] 7t0SqerlBA0: Downloading API JSON [info] Available formats for 7t0SqerlBA0: format code extension resolution note 249-0 webm audio only tiny 3k , webm_dash container, opus (48000Hz), 26.82KiB 250-0 webm audio only tiny 3k , webm_dash container, opus (48000Hz), 26.82KiB 251-0 webm audio only tiny 3k , webm_dash container, opus (48000Hz), 26.82KiB 249-1 webm audio only tiny 3k , webm_dash container, opus (48000Hz), 26.82KiB 250-1 webm audio only tiny 3k , webm_dash container, opus (48000Hz), 26.82KiB 251-1 webm audio only tiny 3k , webm_dash container, opus (48000Hz), 26.82KiB 140-0 m4a audio only tiny 129k , m4a_dash container, mp4a.40.2 (44100Hz), 949.77KiB 140-1 m4a audio only tiny 129k , m4a_dash container, mp4a.40.2 (44100Hz), 949.77KiB 278 webm 256x144 144p 34k , webm_dash container, vp9, 30fps, video only, 251.25KiB 160 mp4 256x144 144p 34k , mp4_dash container, avc1.4d400c, 30fps, video only, 253.46KiB 242 webm 426x240 240p 41k , webm_dash container, vp9, 30fps, video only, 301.54KiB 133 mp4 426x240 240p 113k , mp4_dash container, avc1.4d4015, 30fps, video only, 834.73KiB 243 webm 640x360 360p 108k , webm_dash container, vp9, 30fps, video only, 796.99KiB 134 mp4 640x360 360p 229k , mp4_dash container, avc1.4d401e, 30fps, video only, 1.64MiB 135 mp4 854x480 480p 451k , mp4_dash container, avc1.4d401f, 30fps, video only, 3.23MiB 244 webm 854x480 480p 473k , webm_dash container, vp9, 30fps, video only, 3.39MiB 136 mp4 1280x720 720p 1352k , mp4_dash container, avc1.4d401f, 30fps, video only, 9.67MiB 247 webm 1280x720 720p 2106k , webm_dash container, vp9, 30fps, video only, 15.07MiB 302 webm 1280x720 720p60 2631k , webm_dash container, vp9, 60fps, video only, 18.82MiB 298 mp4 1280x720 720p60 3467k , mp4_dash container, avc1.4d4020, 60fps, video only, 24.80MiB 303 webm 1920x1080 1080p60 4397k , webm_dash container, vp9, 60fps, video only, 31.45MiB 299 mp4 1920x1080 1080p60 5753k , mp4_dash container, avc1.64002a, 60fps, video only, 41.15MiB 308 webm 2560x1440 1440p60 13230k , webm_dash container, vp9, 60fps, video only, 94.63MiB 315 webm 3840x2160 2160p60 26523k , webm_dash container, vp9, 60fps, video only, 189.71MiB 18 mp4 640x360 360p 357k , avc1.42001E, 30fps, mp4a.40.2 (44100Hz) 22 mp4 1280x720 720p 1480k , avc1.64001F, 30fps, mp4a.40.2 (44100Hz) (best) ``` ... And, 3min after, the same command succeeds: ```console yt-dl -vF "7t0SqerlBA0" => [debug] System config: [] [debug] User config: [] [debug] Custom config: [] [debug] Command-line args: ['--ffmpeg-location', '.\\FFmpeg', '--external-downloader-args', '-v 8 -stats', '-vF', '7t0SqerlBA0'] [debug] Encodings: locale cp1253, fs mbcs, out cp737, pref cp1253 [debug] youtube-dl version 2023.07.25.1411 [debug] Lazy loading extractors enabled [debug] Single file build [debug] Python 3.4.4 (CPython x86 32bit) - Windows-Vista-6.0.6003-SP2 - OpenSSL 1.0.2d 9 Jul 2015 [debug] exe versions: ffmpeg n6.1-dev-1700-N-111584-ga4e6168, ffprobe n6.1-dev-1700-N-111584-ga4e6168, phantomjs 2.1.1, rtmpdump 2.4 [debug] Proxy map: {} [youtube] 7t0SqerlBA0: Downloading webpage [youtube] Confirming age [youtube] 7t0SqerlBA0: Downloading API JSON [youtube] 7t0SqerlBA0: Downloading player 0e6aaa83 [debug] [youtube] Decrypted nsig proJhhsbIXZsTigaH_ => u9mwfbjD7Gvtyg [debug] [youtube] Decrypted nsig 3nr2nEIoH10xrGVXqk => wP3tEGGA6hi0Zg [info] Available formats for 7t0SqerlBA0: format code extension resolution note 249-0 webm audio only tiny 3k , webm_dash container, opus (48000Hz), 26.82KiB 250-0 webm audio only tiny 3k , webm_dash container, opus (48000Hz), 26.82KiB 251-0 webm audio only tiny 3k , webm_dash container, opus (48000Hz), 26.82KiB 249-1 webm audio only tiny 3k , webm_dash container, opus (48000Hz), 26.82KiB 250-1 webm audio only tiny 3k , webm_dash container, opus (48000Hz), 26.82KiB 251-1 webm audio only tiny 3k , webm_dash container, opus (48000Hz), 26.82KiB 140-0 m4a audio only tiny 129k , m4a_dash container, mp4a.40.2 (44100Hz), 949.77KiB 140-1 m4a audio only tiny 129k , m4a_dash container, mp4a.40.2 (44100Hz), 949.77KiB 278 webm 256x144 144p 34k , webm_dash container, vp9, 30fps, video only, 251.25KiB 160 mp4 256x144 144p 34k , mp4_dash container, avc1.4d400c, 30fps, video only, 253.46KiB 242 webm 426x240 240p 41k , webm_dash container, vp9, 30fps, video only, 301.54KiB 133 mp4 426x240 240p 113k , mp4_dash container, avc1.4d4015, 30fps, video only, 834.73KiB 243 webm 640x360 360p 108k , webm_dash container, vp9, 30fps, video only, 796.99KiB 134 mp4 640x360 360p 229k , mp4_dash container, avc1.4d401e, 30fps, video only, 1.64MiB 135 mp4 854x480 480p 451k , mp4_dash container, avc1.4d401f, 30fps, video only, 3.23MiB 244 webm 854x480 480p 473k , webm_dash container, vp9, 30fps, video only, 3.39MiB 136 mp4 1280x720 720p 1352k , mp4_dash container, avc1.4d401f, 30fps, video only, 9.67MiB 247 webm 1280x720 720p 2106k , webm_dash container, vp9, 30fps, video only, 15.07MiB 302 webm 1280x720 720p60 2631k , webm_dash container, vp9, 60fps, video only, 18.82MiB 298 mp4 1280x720 720p60 3467k , mp4_dash container, avc1.4d4020, 60fps, video only, 24.80MiB 303 webm 1920x1080 1080p60 4397k , webm_dash container, vp9, 60fps, video only, 31.45MiB 299 mp4 1920x1080 1080p60 5753k , mp4_dash container, avc1.64002a, 60fps, video only, 41.15MiB 308 webm 2560x1440 1440p60 13230k , webm_dash container, vp9, 60fps, video only, 94.63MiB 315 webm 3840x2160 2160p60 26523k , webm_dash container, vp9, 60fps, video only, 189.71MiB 18 mp4 640x360 360p 357k , avc1.42001E, 30fps, mp4a.40.2 (44100Hz) 22 mp4 1280x720 720p 1480k , avc1.64001F, 30fps, mp4a.40.2 (44100Hz) (best) ``` In desperation, I even suspected possible interference from my AntiVirus/Security Suite, but whitelisting `youtube-dl.exe` or even temporarily switching it OFF had no effect on the Youtube errors (they still randomly happen) ... Any ideas, anyone? Since I have no **reliable way of reproducing** these errors, no `New issue` has been submitted...

deekerman commented

Author

Owner

@nicolaasjan commented on GitHub (Jul 28, 2023):

OT, but somehow related: @nicolaasjan: Will you be releasing anything newer than youtube-dl.exe v2023.07.22.1 ?

Somehow I seem to have forgotten...
But your wish is my command:
Here it is. 🌻

@nicolaasjan commented on GitHub (Jul 28, 2023): > OT, but somehow related: @nicolaasjan: Will you be releasing anything newer than **youtube-dl.exe** v`2023.07.22.1` ? Somehow I seem to have forgotten... But your wish is my command: [Here](https://dl.dropboxusercontent.com/scl/fi/y294fz1ufsvy2it4kt5iv/youtube-dl.zip?rlkey=8na2mvkcxabfdlm7ku0pw21o9) it is. 🌻

deekerman commented

Author

Owner

@nicolaasjan commented on GitHub (Jul 28, 2023):

Any ideas, anyone?

Since I have no reliable way of reproducing these errors, no New issue has been submitted...

No, but I experienced the same...

First try:

youtube-dl -v --ignore-config -vF "7t0SqerlBA0"
[debug] System config: []
[debug] User config: []
[debug] Custom config: []
[debug] Command-line args: ['-v', '--ignore-config', '-vF', '7t0SqerlBA0']
[debug] Encodings: locale UTF-8, fs utf-8, out utf-8, pref UTF-8
[debug] youtube-dl version 2023.07.25
[debug] Lazy loading extractors enabled
[debug] Single file build
[debug] Python 3.8.10 (CPython x86_64 64bit) - Linux-5.4.0-155-generic-x86_64-with-glibc2.29 - OpenSSL 1.1.1f  31 Mar 2020 - glibc 2.31
[debug] exe versions: ffmpeg N-111355-g68e9d2835f-Nico-20230707, ffprobe N-111355-g68e9d2835f-Nico-20230707, phantomjs 2.1.1, rtmpdump 2.4
[debug] Proxy map: {}
[youtube] 7t0SqerlBA0: Downloading webpage
[youtube] 7t0SqerlBA0: Downloading API JSON
[youtube] Confirming age
WARNING: unable to extract player URL; please report this issue on https://yt-dl.org/bug . Make sure you are using the latest version; type  youtube-dl -U  to update. Be sure to call youtube-dl with the --verbose flag and include its complete output.
WARNING: Cannot extract signature timestamp without player_url.
[youtube] 7t0SqerlBA0: Downloading API JSON
WARNING: unable to extract player URL; please report this issue on https://yt-dl.org/bug . Make sure you are using the latest version; type  youtube-dl -U  to update. Be sure to call youtube-dl with the --verbose flag and include its complete output.
WARNING: [youtube] Unable to decode n-parameter: download likely to be throttled (expected string or bytes-like object Traceback (most recent call last):
  File "/usr/local/bin/youtube-dl.orig/youtube_dl/extractor/youtube.py", line 1679, in _n_descramble
    self._player_cache[player_id] = self._extract_n_function(video_id, player_url)
  File "/usr/local/bin/youtube-dl.orig/youtube_dl/extractor/youtube.py", line 1644, in _extract_n_function
    player_id = self._extract_player_info(player_url)
  File "/usr/local/bin/youtube-dl.orig/youtube_dl/extractor/youtube.py", line 1476, in _extract_player_info
    id_m = re.search(player_re, player_url)
  File "/usr/lib/python3.8/re.py", line 201, in search
    return _compile(pattern, flags).search(string)
TypeError: expected string or bytes-like object
)
[youtube] 7t0SqerlBA0: Downloading API JSON
[info] Available formats for 7t0SqerlBA0:
format code  extension  resolution note
249-0        webm       audio only tiny    3k , webm_dash container, opus  (48000Hz), 26.82KiB
250-0        webm       audio only tiny    3k , webm_dash container, opus  (48000Hz), 26.82KiB
251-0        webm       audio only tiny    3k , webm_dash container, opus  (48000Hz), 26.82KiB
249-1        webm       audio only tiny    3k , webm_dash container, opus  (48000Hz), 26.82KiB
250-1        webm       audio only tiny    3k , webm_dash container, opus  (48000Hz), 26.82KiB
251-1        webm       audio only tiny    3k , webm_dash container, opus  (48000Hz), 26.82KiB
140-0        m4a        audio only tiny  129k , m4a_dash container, mp4a.40.2 (44100Hz), 949.77KiB
140-1        m4a        audio only tiny  129k , m4a_dash container, mp4a.40.2 (44100Hz), 949.77KiB
278          webm       256x144    144p   34k , webm_dash container, vp9, 30fps, video only, 251.25KiB
160          mp4        256x144    144p   34k , mp4_dash container, avc1.4d400c, 30fps, video only, 253.46KiB
242          webm       426x240    240p   41k , webm_dash container, vp9, 30fps, video only, 301.54KiB
133          mp4        426x240    240p  113k , mp4_dash container, avc1.4d4015, 30fps, video only, 834.73KiB
243          webm       640x360    360p  108k , webm_dash container, vp9, 30fps, video only, 796.99KiB
134          mp4        640x360    360p  229k , mp4_dash container, avc1.4d401e, 30fps, video only, 1.64MiB
135          mp4        854x480    480p  451k , mp4_dash container, avc1.4d401f, 30fps, video only, 3.23MiB
244          webm       854x480    480p  473k , webm_dash container, vp9, 30fps, video only, 3.39MiB
136          mp4        1280x720   720p 1352k , mp4_dash container, avc1.4d401f, 30fps, video only, 9.67MiB
247          webm       1280x720   720p 2106k , webm_dash container, vp9, 30fps, video only, 15.07MiB
302          webm       1280x720   720p60 2631k , webm_dash container, vp9, 60fps, video only, 18.82MiB
298          mp4        1280x720   720p60 3467k , mp4_dash container, avc1.4d4020, 60fps, video only, 24.80MiB
303          webm       1920x1080  1080p60 4397k , webm_dash container, vp9, 60fps, video only, 31.45MiB
299          mp4        1920x1080  1080p60 5753k , mp4_dash container, avc1.64002a, 60fps, video only, 41.15MiB
308          webm       2560x1440  1440p60 13230k , webm_dash container, vp9, 60fps, video only, 94.63MiB
315          webm       3840x2160  2160p60 26523k , webm_dash container, vp9, 60fps, video only, 189.71MiB
18           mp4        640x360    360p  357k , avc1.42001E, 30fps, mp4a.40.2 (44100Hz)
22           mp4        1280x720   720p 1480k , avc1.64001F, 30fps, mp4a.40.2 (44100Hz) (best)

And right thereafter:

youtube-dl -v --ignore-config -vF "7t0SqerlBA0"
[debug] System config: []
[debug] User config: []
[debug] Custom config: []
[debug] Command-line args: ['-v', '--ignore-config', '-vF', '7t0SqerlBA0']
[debug] Encodings: locale UTF-8, fs utf-8, out utf-8, pref UTF-8
[debug] youtube-dl version 2023.07.25
[debug] Lazy loading extractors enabled
[debug] Single file build
[debug] Python 3.8.10 (CPython x86_64 64bit) - Linux-5.4.0-155-generic-x86_64-with-glibc2.29 - OpenSSL 1.1.1f  31 Mar 2020 - glibc 2.31
[debug] exe versions: ffmpeg N-111355-g68e9d2835f-Nico-20230707, ffprobe N-111355-g68e9d2835f-Nico-20230707, phantomjs 2.1.1, rtmpdump 2.4
[debug] Proxy map: {}
[youtube] 7t0SqerlBA0: Downloading webpage
[youtube] Confirming age
[youtube] 7t0SqerlBA0: Downloading API JSON
[debug] [youtube] Decrypted nsig DnjCI5qHrrYYj8JVd2 => _OFJGD7-eFKezw
[debug] [youtube] Decrypted nsig E0Wd6RsIOXis2I6jAt => yn4zNrTMBPF-Vg
[info] Available formats for 7t0SqerlBA0:
format code  extension  resolution note
249-0        webm       audio only tiny    3k , webm_dash container, opus  (48000Hz), 26.82KiB
250-0        webm       audio only tiny    3k , webm_dash container, opus  (48000Hz), 26.82KiB
251-0        webm       audio only tiny    3k , webm_dash container, opus  (48000Hz), 26.82KiB
249-1        webm       audio only tiny    3k , webm_dash container, opus  (48000Hz), 26.82KiB
250-1        webm       audio only tiny    3k , webm_dash container, opus  (48000Hz), 26.82KiB
251-1        webm       audio only tiny    3k , webm_dash container, opus  (48000Hz), 26.82KiB
140-0        m4a        audio only tiny  129k , m4a_dash container, mp4a.40.2 (44100Hz), 949.77KiB
140-1        m4a        audio only tiny  129k , m4a_dash container, mp4a.40.2 (44100Hz), 949.77KiB
278          webm       256x144    144p   34k , webm_dash container, vp9, 30fps, video only, 251.25KiB
160          mp4        256x144    144p   34k , mp4_dash container, avc1.4d400c, 30fps, video only, 253.46KiB
242          webm       426x240    240p   41k , webm_dash container, vp9, 30fps, video only, 301.54KiB
133          mp4        426x240    240p  113k , mp4_dash container, avc1.4d4015, 30fps, video only, 834.73KiB
243          webm       640x360    360p  108k , webm_dash container, vp9, 30fps, video only, 796.99KiB
134          mp4        640x360    360p  229k , mp4_dash container, avc1.4d401e, 30fps, video only, 1.64MiB
135          mp4        854x480    480p  451k , mp4_dash container, avc1.4d401f, 30fps, video only, 3.23MiB
244          webm       854x480    480p  473k , webm_dash container, vp9, 30fps, video only, 3.39MiB
136          mp4        1280x720   720p 1352k , mp4_dash container, avc1.4d401f, 30fps, video only, 9.67MiB
247          webm       1280x720   720p 2106k , webm_dash container, vp9, 30fps, video only, 15.07MiB
302          webm       1280x720   720p60 2631k , webm_dash container, vp9, 60fps, video only, 18.82MiB
298          mp4        1280x720   720p60 3467k , mp4_dash container, avc1.4d4020, 60fps, video only, 24.80MiB
303          webm       1920x1080  1080p60 4397k , webm_dash container, vp9, 60fps, video only, 31.45MiB
299          mp4        1920x1080  1080p60 5753k , mp4_dash container, avc1.64002a, 60fps, video only, 41.15MiB
308          webm       2560x1440  1440p60 13230k , webm_dash container, vp9, 60fps, video only, 94.63MiB
315          webm       3840x2160  2160p60 26523k , webm_dash container, vp9, 60fps, video only, 189.71MiB
18           mp4        640x360    360p  357k , avc1.42001E, 30fps, mp4a.40.2 (44100Hz)
22           mp4        1280x720   720p 1480k , avc1.64001F, 30fps, mp4a.40.2 (44100Hz) (best)

@nicolaasjan commented on GitHub (Jul 28, 2023): > Any ideas, anyone? > > Since I have no **reliable way of reproducing** these errors, no `New issue` has been submitted... No, but I experienced the same... First try: ```shell youtube-dl -v --ignore-config -vF "7t0SqerlBA0" [debug] System config: [] [debug] User config: [] [debug] Custom config: [] [debug] Command-line args: ['-v', '--ignore-config', '-vF', '7t0SqerlBA0'] [debug] Encodings: locale UTF-8, fs utf-8, out utf-8, pref UTF-8 [debug] youtube-dl version 2023.07.25 [debug] Lazy loading extractors enabled [debug] Single file build [debug] Python 3.8.10 (CPython x86_64 64bit) - Linux-5.4.0-155-generic-x86_64-with-glibc2.29 - OpenSSL 1.1.1f 31 Mar 2020 - glibc 2.31 [debug] exe versions: ffmpeg N-111355-g68e9d2835f-Nico-20230707, ffprobe N-111355-g68e9d2835f-Nico-20230707, phantomjs 2.1.1, rtmpdump 2.4 [debug] Proxy map: {} [youtube] 7t0SqerlBA0: Downloading webpage [youtube] 7t0SqerlBA0: Downloading API JSON [youtube] Confirming age WARNING: unable to extract player URL; please report this issue on https://yt-dl.org/bug . Make sure you are using the latest version; type youtube-dl -U to update. Be sure to call youtube-dl with the --verbose flag and include its complete output. WARNING: Cannot extract signature timestamp without player_url. [youtube] 7t0SqerlBA0: Downloading API JSON WARNING: unable to extract player URL; please report this issue on https://yt-dl.org/bug . Make sure you are using the latest version; type youtube-dl -U to update. Be sure to call youtube-dl with the --verbose flag and include its complete output. WARNING: [youtube] Unable to decode n-parameter: download likely to be throttled (expected string or bytes-like object Traceback (most recent call last): File "/usr/local/bin/youtube-dl.orig/youtube_dl/extractor/youtube.py", line 1679, in _n_descramble self._player_cache[player_id] = self._extract_n_function(video_id, player_url) File "/usr/local/bin/youtube-dl.orig/youtube_dl/extractor/youtube.py", line 1644, in _extract_n_function player_id = self._extract_player_info(player_url) File "/usr/local/bin/youtube-dl.orig/youtube_dl/extractor/youtube.py", line 1476, in _extract_player_info id_m = re.search(player_re, player_url) File "/usr/lib/python3.8/re.py", line 201, in search return _compile(pattern, flags).search(string) TypeError: expected string or bytes-like object ) [youtube] 7t0SqerlBA0: Downloading API JSON [info] Available formats for 7t0SqerlBA0: format code extension resolution note 249-0 webm audio only tiny 3k , webm_dash container, opus (48000Hz), 26.82KiB 250-0 webm audio only tiny 3k , webm_dash container, opus (48000Hz), 26.82KiB 251-0 webm audio only tiny 3k , webm_dash container, opus (48000Hz), 26.82KiB 249-1 webm audio only tiny 3k , webm_dash container, opus (48000Hz), 26.82KiB 250-1 webm audio only tiny 3k , webm_dash container, opus (48000Hz), 26.82KiB 251-1 webm audio only tiny 3k , webm_dash container, opus (48000Hz), 26.82KiB 140-0 m4a audio only tiny 129k , m4a_dash container, mp4a.40.2 (44100Hz), 949.77KiB 140-1 m4a audio only tiny 129k , m4a_dash container, mp4a.40.2 (44100Hz), 949.77KiB 278 webm 256x144 144p 34k , webm_dash container, vp9, 30fps, video only, 251.25KiB 160 mp4 256x144 144p 34k , mp4_dash container, avc1.4d400c, 30fps, video only, 253.46KiB 242 webm 426x240 240p 41k , webm_dash container, vp9, 30fps, video only, 301.54KiB 133 mp4 426x240 240p 113k , mp4_dash container, avc1.4d4015, 30fps, video only, 834.73KiB 243 webm 640x360 360p 108k , webm_dash container, vp9, 30fps, video only, 796.99KiB 134 mp4 640x360 360p 229k , mp4_dash container, avc1.4d401e, 30fps, video only, 1.64MiB 135 mp4 854x480 480p 451k , mp4_dash container, avc1.4d401f, 30fps, video only, 3.23MiB 244 webm 854x480 480p 473k , webm_dash container, vp9, 30fps, video only, 3.39MiB 136 mp4 1280x720 720p 1352k , mp4_dash container, avc1.4d401f, 30fps, video only, 9.67MiB 247 webm 1280x720 720p 2106k , webm_dash container, vp9, 30fps, video only, 15.07MiB 302 webm 1280x720 720p60 2631k , webm_dash container, vp9, 60fps, video only, 18.82MiB 298 mp4 1280x720 720p60 3467k , mp4_dash container, avc1.4d4020, 60fps, video only, 24.80MiB 303 webm 1920x1080 1080p60 4397k , webm_dash container, vp9, 60fps, video only, 31.45MiB 299 mp4 1920x1080 1080p60 5753k , mp4_dash container, avc1.64002a, 60fps, video only, 41.15MiB 308 webm 2560x1440 1440p60 13230k , webm_dash container, vp9, 60fps, video only, 94.63MiB 315 webm 3840x2160 2160p60 26523k , webm_dash container, vp9, 60fps, video only, 189.71MiB 18 mp4 640x360 360p 357k , avc1.42001E, 30fps, mp4a.40.2 (44100Hz) 22 mp4 1280x720 720p 1480k , avc1.64001F, 30fps, mp4a.40.2 (44100Hz) (best) ``` And right thereafter: ```shell youtube-dl -v --ignore-config -vF "7t0SqerlBA0" [debug] System config: [] [debug] User config: [] [debug] Custom config: [] [debug] Command-line args: ['-v', '--ignore-config', '-vF', '7t0SqerlBA0'] [debug] Encodings: locale UTF-8, fs utf-8, out utf-8, pref UTF-8 [debug] youtube-dl version 2023.07.25 [debug] Lazy loading extractors enabled [debug] Single file build [debug] Python 3.8.10 (CPython x86_64 64bit) - Linux-5.4.0-155-generic-x86_64-with-glibc2.29 - OpenSSL 1.1.1f 31 Mar 2020 - glibc 2.31 [debug] exe versions: ffmpeg N-111355-g68e9d2835f-Nico-20230707, ffprobe N-111355-g68e9d2835f-Nico-20230707, phantomjs 2.1.1, rtmpdump 2.4 [debug] Proxy map: {} [youtube] 7t0SqerlBA0: Downloading webpage [youtube] Confirming age [youtube] 7t0SqerlBA0: Downloading API JSON [debug] [youtube] Decrypted nsig DnjCI5qHrrYYj8JVd2 => _OFJGD7-eFKezw [debug] [youtube] Decrypted nsig E0Wd6RsIOXis2I6jAt => yn4zNrTMBPF-Vg [info] Available formats for 7t0SqerlBA0: format code extension resolution note 249-0 webm audio only tiny 3k , webm_dash container, opus (48000Hz), 26.82KiB 250-0 webm audio only tiny 3k , webm_dash container, opus (48000Hz), 26.82KiB 251-0 webm audio only tiny 3k , webm_dash container, opus (48000Hz), 26.82KiB 249-1 webm audio only tiny 3k , webm_dash container, opus (48000Hz), 26.82KiB 250-1 webm audio only tiny 3k , webm_dash container, opus (48000Hz), 26.82KiB 251-1 webm audio only tiny 3k , webm_dash container, opus (48000Hz), 26.82KiB 140-0 m4a audio only tiny 129k , m4a_dash container, mp4a.40.2 (44100Hz), 949.77KiB 140-1 m4a audio only tiny 129k , m4a_dash container, mp4a.40.2 (44100Hz), 949.77KiB 278 webm 256x144 144p 34k , webm_dash container, vp9, 30fps, video only, 251.25KiB 160 mp4 256x144 144p 34k , mp4_dash container, avc1.4d400c, 30fps, video only, 253.46KiB 242 webm 426x240 240p 41k , webm_dash container, vp9, 30fps, video only, 301.54KiB 133 mp4 426x240 240p 113k , mp4_dash container, avc1.4d4015, 30fps, video only, 834.73KiB 243 webm 640x360 360p 108k , webm_dash container, vp9, 30fps, video only, 796.99KiB 134 mp4 640x360 360p 229k , mp4_dash container, avc1.4d401e, 30fps, video only, 1.64MiB 135 mp4 854x480 480p 451k , mp4_dash container, avc1.4d401f, 30fps, video only, 3.23MiB 244 webm 854x480 480p 473k , webm_dash container, vp9, 30fps, video only, 3.39MiB 136 mp4 1280x720 720p 1352k , mp4_dash container, avc1.4d401f, 30fps, video only, 9.67MiB 247 webm 1280x720 720p 2106k , webm_dash container, vp9, 30fps, video only, 15.07MiB 302 webm 1280x720 720p60 2631k , webm_dash container, vp9, 60fps, video only, 18.82MiB 298 mp4 1280x720 720p60 3467k , mp4_dash container, avc1.4d4020, 60fps, video only, 24.80MiB 303 webm 1920x1080 1080p60 4397k , webm_dash container, vp9, 60fps, video only, 31.45MiB 299 mp4 1920x1080 1080p60 5753k , mp4_dash container, avc1.64002a, 60fps, video only, 41.15MiB 308 webm 2560x1440 1440p60 13230k , webm_dash container, vp9, 60fps, video only, 94.63MiB 315 webm 3840x2160 2160p60 26523k , webm_dash container, vp9, 60fps, video only, 189.71MiB 18 mp4 640x360 360p 357k , avc1.42001E, 30fps, mp4a.40.2 (44100Hz) 22 mp4 1280x720 720p 1480k , avc1.64001F, 30fps, mp4a.40.2 (44100Hz) (best) ```

deekerman commented

https://github.com/yt-dlp/yt-dlp/issues/7594

Author

Owner

@pukkandan commented on GitHub (Jul 28, 2023):

Off-topic, but re: https://github.com/ytdl-org/youtube-dl/issues/31585#issuecomment-1654843333

@pukkandan commented on GitHub (Jul 28, 2023): Off-topic, but re: https://github.com/ytdl-org/youtube-dl/issues/31585#issuecomment-1654843333 https://github.com/yt-dlp/yt-dlp/issues/7594

deekerman commented

Author

Owner

@dirkf commented on GitHub (Jul 28, 2023):

... is human intervention required to trigger the Build workflow

It is if upstream had to be rebased manually, since then the workflow thinks it has nothing to do.

[Edit: or if the flake8 rules have been secretly updated!]

yt-dlp/yt-dlp#7594

Indeed.

@dirkf commented on GitHub (Jul 28, 2023): > ... is human intervention required to trigger the Build workflow It is if upstream had to be rebased manually, since then the workflow thinks it has nothing to do. [Edit: or if the _flake8_ rules have been secretly updated!] >yt-dlp/yt-dlp#7594 Indeed.

deekerman commented

Author

Owner

@Vangelis66 commented on GitHub (Jul 31, 2023):

From the maintainer's reply here:

The redirect fix was only committed on 20 July, so the problem with
fetching the new version isn't fixed until 20230721 or later.

... FWIW, the Windows build of ytdl-nightly v2023.07.21 stills fails to properly update (via the internal update mechanism) to the latest Windows build v2023.07.30:

youtube-dl -vU

[debug] System config: []
[debug] User config: []
[debug] Custom config: []
[debug] Command-line args: ['-vU']
[debug] Encodings: locale cp1253, fs mbcs, out cp737, pref cp1253
[debug] youtube-dl version 2023.07.21 [069471f38] (single file build)
[debug] ** This version was built from the latest master code at https://github.com/ytdl-org/youtube-dl.
[debug] ** For support, visit the main site.
[debug] Python 3.4.4 (CPython x86 32bit) - Windows-Vista-6.0.6003-SP2 - OpenSSL 1.0.2d 9 Jul 2015
[debug] exe versions: none
[debug] Proxy map: {}
Latest version: 2023.07.30, Current version: 2023.07.21
Current Build Hash aed7e8206cb770af358d831510b51464b730f828ec4bfd9398fb05ecd9712536
Updating to version 2023.07.30 ...
ERROR: Unable to download latest version; Visit  https://github.com/ytdl-org/ytdl-nightly/releases/latest

As reported previously, the detection of the availability of a newer build is successful, not so the actual fetching of that binary 😭 ...

Am I simply being "thick" 😄 , or should one wait for the ytdl-nightly build after the 2023.07.30 one (current) for the internal update routine to resume to normal?

Thanks 😉

@Vangelis66 commented on GitHub (Jul 31, 2023): From the maintainer's reply [here](https://github.com/ytdl-org/youtube-dl/issues/31585#issuecomment-1646947972): > The **redirect fix** was only committed on **20 July**, so the problem with > **fetching the new version** isn't fixed until **20230721 or later.** ... FWIW, the **Windows** build of `ytdl-nightly` v[2023.07.21](https://github.com/ytdl-org/ytdl-nightly/releases/tag/2023.07.21) stills **fails to properly update** (via the _internal update mechanism_) to the **latest** Windows build v[2023.07.30](https://github.com/ytdl-org/ytdl-nightly/releases/tag/2023.07.30): ```console youtube-dl -vU [debug] System config: [] [debug] User config: [] [debug] Custom config: [] [debug] Command-line args: ['-vU'] [debug] Encodings: locale cp1253, fs mbcs, out cp737, pref cp1253 [debug] youtube-dl version 2023.07.21 [069471f38] (single file build) [debug] ** This version was built from the latest master code at https://github.com/ytdl-org/youtube-dl. [debug] ** For support, visit the main site. [debug] Python 3.4.4 (CPython x86 32bit) - Windows-Vista-6.0.6003-SP2 - OpenSSL 1.0.2d 9 Jul 2015 [debug] exe versions: none [debug] Proxy map: {} Latest version: 2023.07.30, Current version: 2023.07.21 Current Build Hash aed7e8206cb770af358d831510b51464b730f828ec4bfd9398fb05ecd9712536 Updating to version 2023.07.30 ... ERROR: Unable to download latest version; Visit https://github.com/ytdl-org/ytdl-nightly/releases/latest ``` As reported [previously](https://github.com/ytdl-org/youtube-dl/issues/31585#issuecomment-1646714738), the detection of the availability of a newer build is successful, not so the **actual fetching of that binary** 😭 ... Am I simply being "thick" 😄 , or should one wait for the `ytdl-nightly` build **after the** `2023.07.30` **one** (current) for the **internal update routine** to resume to normal? Thanks :wink:

deekerman commented

Author

Owner

@Vangelis66 commented on GitHub (Jul 31, 2023):

or should one wait for the ytdl-nightly build after the 2023.07.30 one
(current) for the internal update routine to resume to normal?

... Indeed one 😜 should:

youtube-dl -vU

[debug] System config: []
[debug] User config: []
[debug] Custom config: []
[debug] Command-line args: ['-vU']
[debug] Encodings: locale cp1253, fs mbcs, out cp737, pref cp1253
[debug] youtube-dl version 2023.07.30 [abef53466] (single file build)
[debug] ** This version was built from the latest master code at https://github.com/ytdl-org/youtube-dl.
[debug] ** For support, visit the main site.
[debug] Python 3.4.4 (CPython x86 32bit) - Windows-Vista-6.0.6003-SP2 - OpenSSL 1.0.2d 9 Jul 2015
[debug] exe versions: none
[debug] Proxy map: {}
Latest version: 2023.08.01, Current version: 2023.07.30
Current Build Hash 9af257800ddbb0a77c68a7eb07fe1b28cff9a7fbcd80662ac45c0f17aa972d59
Updating to version 2023.08.01 ...
Updated youtube-dl to version 2023.08.01

👍 🎉

@Vangelis66 commented on GitHub (Jul 31, 2023): > or should one wait for the `ytdl-nightly` build **after the** `2023.07.30` **one** > (current) for the **internal update routine** to resume to normal? ... Indeed one 😜 should: ```console youtube-dl -vU [debug] System config: [] [debug] User config: [] [debug] Custom config: [] [debug] Command-line args: ['-vU'] [debug] Encodings: locale cp1253, fs mbcs, out cp737, pref cp1253 [debug] youtube-dl version 2023.07.30 [abef53466] (single file build) [debug] ** This version was built from the latest master code at https://github.com/ytdl-org/youtube-dl. [debug] ** For support, visit the main site. [debug] Python 3.4.4 (CPython x86 32bit) - Windows-Vista-6.0.6003-SP2 - OpenSSL 1.0.2d 9 Jul 2015 [debug] exe versions: none [debug] Proxy map: {} Latest version: 2023.08.01, Current version: 2023.07.30 Current Build Hash 9af257800ddbb0a77c68a7eb07fe1b28cff9a7fbcd80662ac45c0f17aa972d59 Updating to version 2023.08.01 ... Updated youtube-dl to version 2023.08.01 ``` 👍 🎉

deekerman commented

Author

Owner

@Vangelis66 commented on GitHub (Aug 1, 2023):

... Speaking of the internal update routine, but this time on the "release"/"stable" channel, i.e. on Windows builds compiled from code on the yt-dl master branch, I'm witnessing breakage 😞 , at least with the builds kindly ❤️ provided by my Dutch friend 😜 @nicolaasjan:

Latest build 2023.08.01, compiled from 2efc8de4d2:

youtube-dl -vU => 

[debug] System config: []
[debug] User config: []
[debug] Custom config: []
[debug] Command-line args: ['-vU']
[debug] Encodings: locale cp1253, fs mbcs, out cp737, pref cp1253
[debug] youtube-dl version 2023.08.01
[debug] Lazy loading extractors enabled
[debug] Single file build
[debug] Python 3.4.4 (CPython x86 32bit) - Windows-Vista-6.0.6003-SP2 - OpenSSL 1.0.2d 9 Jul 2015
[debug] exe versions: none
[debug] Proxy map: {}
Traceback (most recent call last):
  File "C:\Users\nico\Desktop\youtube-dl_source\youtube_dl\update.py", line 48, in update_self
  File "c:\Program Files (x86)\Python3.4.4\lib\urllib\request.py", line 470, in open
  File "c:\Program Files (x86)\Python3.4.4\lib\urllib\request.py", line 580, in http_response
  File "c:\Program Files (x86)\Python3.4.4\lib\urllib\request.py", line 508, in error
  File "c:\Program Files (x86)\Python3.4.4\lib\urllib\request.py", line 442, in_call_chain
  File "c:\Program Files (x86)\Python3.4.4\lib\urllib\request.py", line 588, in http_error_default
urllib.error.HTTPError: HTTP Error 403: Forbidden

ERROR: can't find the current version. Please try again later.

This is a recent breakage that also affects previous master branch builds, including my own-compiled py2.7/py3.4 ones:

[debug] System config: []
[debug] User config: []
[debug] Custom config: []
[debug] Command-line args: [u'-vU']
[debug] Encodings: locale cp1253, fs mbcs, out cp737, pref cp1253
[debug] youtube-dl version 2023.07.25.1411
[debug] Lazy loading extractors enabled
[debug] Single file build
[debug] Python 2.7.18 (CPython x86 32bit) - Windows-Vista-6.0.6003-SP2 - OpenSSL 1.0.2t  10 Sep 2019
[debug] exe versions: none
[debug] Proxy map: {}
Traceback (most recent call last):
  File "youtube_dl\update.pyo", line 48, in update_self
  File "urllib2.pyo", line 435, in open
  File "urllib2.pyo", line 548, in http_response
  File "urllib2.pyo", line 473, in error
  File "urllib2.pyo", line 407, in _call_chain
  File "urllib2.pyo", line 556, in http_error_default
HTTPError: HTTP Error 403: Forbidden

ERROR: can't find the current version. Please try again later.

... and:

[debug] System config: []
[debug] User config: []
[debug] Custom config: []
[debug] Command-line args: ['-vU']
[debug] Encodings: locale cp1253, fs mbcs, out cp737, pref cp1253
[debug] youtube-dl version 2023.07.25.1411
[debug] Lazy loading extractors enabled
[debug] Single file build
[debug] Python 3.4.10 (CPython x86 32bit) - Windows-Vista-6.0.6003-SP2 - OpenSSL 1.0.2k  26 Jan 2017
[debug] exe versions: none
[debug] Proxy map: {}
Traceback (most recent call last):
  File "C:\Python3410-32\yt-dl_pycompile\youtube_dl\update.py", line 48, in update_self
    newversion = opener.open(VERSION_URL).read().decode('utf-8').strip()
  File "C:\Python3410-32\lib\urllib\request.py", line 470, in open
    response = meth(req, response)
  File "C:\Python3410-32\lib\urllib\request.py", line 580, in http_response
      'http', request, response, code, msg, hdrs)
  File "C:\Python3410-32\lib\urllib\request.py", line 508, in error
    return self._call_chain(*args)
  File "C:\Python3410-32\lib\urllib\request.py", line 442, in _call_chain
    result = func(*args)
  File "C:\Python3410-32\lib\urllib\request.py", line 588, in http_error_default
    raise HTTPError(req.full_url, code, msg, hdrs, fp)
urllib.error.HTTPError: HTTP Error 403: Forbidden

ERROR: can't find the current version. Please try again later.

Both these builds yesterday would report they were "up-to-date":

youtube-dl is up to date (2023.07.25.1411)

@dirkf, any guess what went "belly up" this time ?

Regards.

@Vangelis66 commented on GitHub (Aug 1, 2023): ... Speaking of **the internal update routine**, but this time on the **"release"/"stable" channel**, i.e. on **Windows builds compiled from code on the** `yt-dl` `master` **branch**, I'm witnessing breakage 😞 , at least with the builds kindly ❤️ provided by my Dutch friend 😜 @nicolaasjan: Latest build `2023.08.01`, compiled from 2efc8de4d2299e08e0c84d674d7fc7f3fa669487: ```console youtube-dl -vU => [debug] System config: [] [debug] User config: [] [debug] Custom config: [] [debug] Command-line args: ['-vU'] [debug] Encodings: locale cp1253, fs mbcs, out cp737, pref cp1253 [debug] youtube-dl version 2023.08.01 [debug] Lazy loading extractors enabled [debug] Single file build [debug] Python 3.4.4 (CPython x86 32bit) - Windows-Vista-6.0.6003-SP2 - OpenSSL 1.0.2d 9 Jul 2015 [debug] exe versions: none [debug] Proxy map: {} Traceback (most recent call last): File "C:\Users\nico\Desktop\youtube-dl_source\youtube_dl\update.py", line 48, in update_self File "c:\Program Files (x86)\Python3.4.4\lib\urllib\request.py", line 470, in open File "c:\Program Files (x86)\Python3.4.4\lib\urllib\request.py", line 580, in http_response File "c:\Program Files (x86)\Python3.4.4\lib\urllib\request.py", line 508, in error File "c:\Program Files (x86)\Python3.4.4\lib\urllib\request.py", line 442, in_call_chain File "c:\Program Files (x86)\Python3.4.4\lib\urllib\request.py", line 588, in http_error_default urllib.error.HTTPError: HTTP Error 403: Forbidden ERROR: can't find the current version. Please try again later. ``` This is a **recent breakage** that also affects **previous** `master` branch builds, including **my own-compiled** py2.7/py3.4 ones: ```console [debug] System config: [] [debug] User config: [] [debug] Custom config: [] [debug] Command-line args: [u'-vU'] [debug] Encodings: locale cp1253, fs mbcs, out cp737, pref cp1253 [debug] youtube-dl version 2023.07.25.1411 [debug] Lazy loading extractors enabled [debug] Single file build [debug] Python 2.7.18 (CPython x86 32bit) - Windows-Vista-6.0.6003-SP2 - OpenSSL 1.0.2t 10 Sep 2019 [debug] exe versions: none [debug] Proxy map: {} Traceback (most recent call last): File "youtube_dl\update.pyo", line 48, in update_self File "urllib2.pyo", line 435, in open File "urllib2.pyo", line 548, in http_response File "urllib2.pyo", line 473, in error File "urllib2.pyo", line 407, in _call_chain File "urllib2.pyo", line 556, in http_error_default HTTPError: HTTP Error 403: Forbidden ERROR: can't find the current version. Please try again later. ``` ... and: ```console [debug] System config: [] [debug] User config: [] [debug] Custom config: [] [debug] Command-line args: ['-vU'] [debug] Encodings: locale cp1253, fs mbcs, out cp737, pref cp1253 [debug] youtube-dl version 2023.07.25.1411 [debug] Lazy loading extractors enabled [debug] Single file build [debug] Python 3.4.10 (CPython x86 32bit) - Windows-Vista-6.0.6003-SP2 - OpenSSL 1.0.2k 26 Jan 2017 [debug] exe versions: none [debug] Proxy map: {} Traceback (most recent call last): File "C:\Python3410-32\yt-dl_pycompile\youtube_dl\update.py", line 48, in update_self newversion = opener.open(VERSION_URL).read().decode('utf-8').strip() File "C:\Python3410-32\lib\urllib\request.py", line 470, in open response = meth(req, response) File "C:\Python3410-32\lib\urllib\request.py", line 580, in http_response 'http', request, response, code, msg, hdrs) File "C:\Python3410-32\lib\urllib\request.py", line 508, in error return self._call_chain(*args) File "C:\Python3410-32\lib\urllib\request.py", line 442, in _call_chain result = func(*args) File "C:\Python3410-32\lib\urllib\request.py", line 588, in http_error_default raise HTTPError(req.full_url, code, msg, hdrs, fp) urllib.error.HTTPError: HTTP Error 403: Forbidden ERROR: can't find the current version. Please try again later. ``` **Both** these builds **yesterday** would report they were "up-to-date": `youtube-dl is up to date (2023.07.25.1411)` @dirkf, any guess what went "belly up" this time ? Regards.

deekerman commented

Author

Owner

@nicolaasjan commented on GitHub (Aug 1, 2023):

Same error with version built with PyInstaller on Windows 7 with Python 3.8.17:

[debug] System config: []
[debug] User config: ['--no-mtime', '-o', '~/Desktop/%(title)s.%(ext)s', '-f', 'bestvideo[height<=1080][ext=mp4][vcodec^=avc]+bestaudio[ext=m4a]/best[ext=mp4]/best', '--embed-thumbnail', '--add-metadata']
[debug] Custom config: []
[debug] Command-line args: ['-vU']
[debug] Encodings: locale cp1252, fs utf-8, out cp1252, pref cp1252
[debug] youtube-dl version 2023.08.01
[debug] Lazy loading extractors enabled
[debug] Single file build
[debug] Python 3.8.17 (CPython AMD64 64bit) - Windows-7-6.1.7601-SP1 - OpenSSL 1.1.1u  30 May 2023
[debug] exe versions: ffmpeg N-111642-g8c67e1347-2023-07-28-nonfree, ffprobe N-111642-g8c67e1347-2023-07-28-nonfree
[debug] Proxy map: {}
Traceback (most recent call last):
  File "youtube_dl\update.py", line 48, in update_self
  File "urllib\request.py", line 531, in open
  File "urllib\request.py", line 640, in http_response
  File "urllib\request.py", line 569, in error
  File "urllib\request.py", line 502, in _call_chain
  File "urllib\request.py", line 649, in http_error_default
urllib.error.HTTPError: HTTP Error 403: Forbidden

ERROR: can't find the current version. Please try again later.

@nicolaasjan commented on GitHub (Aug 1, 2023): Same error with version built with PyInstaller on Windows 7 with Python 3.8.17: ```shell [debug] System config: [] [debug] User config: ['--no-mtime', '-o', '~/Desktop/%(title)s.%(ext)s', '-f', 'bestvideo[height<=1080][ext=mp4][vcodec^=avc]+bestaudio[ext=m4a]/best[ext=mp4]/best', '--embed-thumbnail', '--add-metadata'] [debug] Custom config: [] [debug] Command-line args: ['-vU'] [debug] Encodings: locale cp1252, fs utf-8, out cp1252, pref cp1252 [debug] youtube-dl version 2023.08.01 [debug] Lazy loading extractors enabled [debug] Single file build [debug] Python 3.8.17 (CPython AMD64 64bit) - Windows-7-6.1.7601-SP1 - OpenSSL 1.1.1u 30 May 2023 [debug] exe versions: ffmpeg N-111642-g8c67e1347-2023-07-28-nonfree, ffprobe N-111642-g8c67e1347-2023-07-28-nonfree [debug] Proxy map: {} Traceback (most recent call last): File "youtube_dl\update.py", line 48, in update_self File "urllib\request.py", line 531, in open File "urllib\request.py", line 640, in http_response File "urllib\request.py", line 569, in error File "urllib\request.py", line 502, in _call_chain File "urllib\request.py", line 649, in http_error_default urllib.error.HTTPError: HTTP Error 403: Forbidden ERROR: can't find the current version. Please try again later. ```

deekerman commented

Author

Owner

@dirkf commented on GitHub (Aug 1, 2023):

You would have to alias yt-dl.org to ytdl-org.github.io for the moment as the DNS for the former server needs to be changed. But as there's no update to fetch it's not critical.

@dirkf commented on GitHub (Aug 1, 2023): You would have to alias yt-dl.org to ytdl-org.github.io for the moment as the DNS for the former server needs to be changed. But as there's no update to fetch it's not critical.

deekerman commented

Author

Owner

@Vangelis66 commented on GitHub (Aug 1, 2023):

as the DNS for the former server needs to be changed

https://yt-dl.org/ =>

Access denied

Due to a ruling of the Hamburg Regional Court, access to this website is blocked.

Sounds all pretty serious and a direct attack of the German authorities against yt-dl itself: https://openjur.de/u/2466945.ppdf

to ytdl-org.github.io for the moment

While
https://ytdl-org.github.io/youtube-dl/download.html
does load, the Windows binaries are still linked via yt-dl.org 😞 ...
Might be worth switching everything in the code to GitHub directly, e.g:

https://api.github.com/repos/ytdl-org/youtube-dl/releases/latest

, but I know nothing 😜 ...

youtube-dl -vU =>

[debug] System config: []
[debug] User config: []
[debug] Custom config: []
[debug] Command-line args: ['-vU']
[debug] Encodings: locale cp1253, fs mbcs, out cp737, pref cp1253
[debug] youtube-dl version 2023.08.01
[debug] Lazy loading extractors enabled
[debug] Single file build
[debug] Python 3.4.4 (CPython x86 32bit) - Windows-Vista-6.0.6003-SP2 - OpenSSL 1.0.2d 9 Jul 2015
[debug] exe versions: none
[debug] Proxy map: {}
Latest version: 2021.12.17, Current version: 2023.08.01
youtube-dl is up to date (2023.08.01)

@Vangelis66 commented on GitHub (Aug 1, 2023): > as the DNS for the former server needs to be changed https://yt-dl.org/ => > Access denied > > Due to a [ruling](https://openjur.de/u/2466945.html) of the Hamburg Regional Court, access to this website is blocked. Sounds all **pretty serious** and a direct attack of the **German authorities** against `yt-dl` itself: https://openjur.de/u/2466945.ppdf > to `ytdl-org.github.io` for the moment While https://ytdl-org.github.io/youtube-dl/download.html does load, the Windows binaries are still linked via `yt-dl.org` 😞 ... Might be worth switching everything in the code to GitHub **directly**, e.g: `https://api.github.com/repos/ytdl-org/youtube-dl/releases/latest` , but _I_ know _nothing_ 😜 ... ```console youtube-dl -vU => [debug] System config: [] [debug] User config: [] [debug] Custom config: [] [debug] Command-line args: ['-vU'] [debug] Encodings: locale cp1253, fs mbcs, out cp737, pref cp1253 [debug] youtube-dl version 2023.08.01 [debug] Lazy loading extractors enabled [debug] Single file build [debug] Python 3.4.4 (CPython x86 32bit) - Windows-Vista-6.0.6003-SP2 - OpenSSL 1.0.2d 9 Jul 2015 [debug] exe versions: none [debug] Proxy map: {} Latest version: 2021.12.17, Current version: 2023.08.01 youtube-dl is up to date (2023.08.01) ```

deekerman commented

Author

Owner

@nicolaasjan commented on GitHub (Aug 2, 2023):

Since when is yt-dl.org blocked?
The ruling is from March 31, 2023.

@nicolaasjan commented on GitHub (Aug 2, 2023): Since _when_ is `yt-dl.org` blocked? The ruling is from March 31, 2023.

deekerman commented

http://web.archive.org/web/20230727220633/https://yt-dl.org/

Author

Owner

@Vangelis66 commented on GitHub (Aug 2, 2023):

Since when is yt-dl.org blocked?

... I can only assume, going by the "update check" breakage I reported 😉 , that the court order was finally enforced (by the German hoster hosting yt-dl.org) on Aug 1st, 2023 😭 ...

FWIW, this doesn't seem to have arrived "out of the blue" 😞 :

We would also like to heartily thank our main website hoster Uberspace who is currently being sued in Germany for hosting our essentially business card website and who have already spent thousands of Euros in their legal defense.

I don't want to sound pessimistic, but, since Germany is a member of the EU, what comes next for "us", EU-based, users of yt-dl?

@Vangelis66 commented on GitHub (Aug 2, 2023): > Since when is `yt-dl.org` blocked? ... I can only assume, going by the "update check" breakage I reported 😉 , that the court order was **finally enforced** (by the **German hoster** hosting `yt-dl.org`) on **Aug 1st, 2023** 😭 ... FWIW, this doesn't seem to have arrived "out of the blue" 😞 : http://web.archive.org/web/20230727220633/https://yt-dl.org/ > **We would also like to heartily thank** [our main website](http://web.archive.org/web/20230727220633/https://yt-dl.org/) **hoster** [Uberspace](https://uberspace.de/en/) **who is currently being sued in Germany for hosting our essentially business card website and who have already spent thousands of Euros in their legal defense.** I don't want to sound pessimistic, but, since **Germany** is a member of the **EU**, what comes next for "us", **EU-based**, users of `yt-dl`?

deekerman commented

Author

Owner

@pukkandan commented on GitHub (Aug 2, 2023):

imo RIAA went against the web hoster, and that too, in a local court because they couldn't do anything to GitHub. I wouldn't read too much into the takedown. While it sucks that the site is down, that's probably the full extend of this for the time being.

@pukkandan commented on GitHub (Aug 2, 2023): imo RIAA went against the web hoster, and that too, in a local court because they couldn't do anything to GitHub. I wouldn't read too much into the takedown. While it sucks that the site is down, that's probably the full extend of this for the time being.

deekerman commented

Author

Owner

@DmytroUsenko commented on GitHub (Aug 2, 2023):

What's the issue with opening the mirror on any non-eu hosing? see no
problem

ср, 2 авг. 2023 г. в 18:32, Vangelis66 @.***>:

Since when is yt-dl.org blocked?

... I can only assume, going by the "update check" breakage I reported 😉
, that the court order was finally enforced (to the German hoster
hosting yt-dl.org) on Aug 1st, 2023 😭 ...

FWIW, this doesn't seem to have arrived "out of the blue" 😞 :

http://web.archive.org/web/20230727220633/https://yt-dl.org/

We would also like to heartily thank our main website
http://web.archive.org/web/20230727220633/https://yt-dl.org/ hoster
Uberspace https://uberspace.de/en/ who is currently being sued in
Germany for hosting our essentially business card website and who have
already spent thousands of Euros in their legal defense.

I don't want to be pessimistic, but, since Germany is a member of the EU,
what comes next for "us", EU-based, users of yt-dl?

—
Reply to this email directly, view it on GitHub
https://github.com/ytdl-org/youtube-dl/issues/31585#issuecomment-1662426241,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/ABY467NWLXHIKMVBMWJZIMLXTJXINANCNFSM6AAAAAAVARTTYU
.
You are receiving this because you commented.Message ID:
@.***>

@DmytroUsenko commented on GitHub (Aug 2, 2023): What's the issue with opening the mirror on any non-eu hosing? see no problem ср, 2 авг. 2023 г. в 18:32, Vangelis66 ***@***.***>: > Since when is yt-dl.org blocked? > > ... I can only assume, going by the "update check" breakage I reported 😉 > , that the court order was *finally enforced* (to the *German hoster* > hosting yt-dl.org) on *Aug 1st, 2023* 😭 ... > > FWIW, this doesn't seem to have arrived "out of the blue" 😞 : > > http://web.archive.org/web/20230727220633/https://yt-dl.org/ > > *We would also like to heartily thank* our main website > <http://web.archive.org/web/20230727220633/https://yt-dl.org/> *hoster* > Uberspace <https://uberspace.de/en/> *who is currently being sued in > Germany for hosting our essentially business card website and who have > already spent thousands of Euros in their legal defense.* > > I don't want to be pessimistic, but, since Germany is a member of the *EU*, > what comes next for "us", *EU-based*, users of yt-dl? > > — > Reply to this email directly, view it on GitHub > <https://github.com/ytdl-org/youtube-dl/issues/31585#issuecomment-1662426241>, > or unsubscribe > <https://github.com/notifications/unsubscribe-auth/ABY467NWLXHIKMVBMWJZIMLXTJXINANCNFSM6AAAAAAVARTTYU> > . > You are receiving this because you commented.Message ID: > ***@***.***> >

deekerman commented

Author

Owner

@Vangelis66 commented on GitHub (Aug 2, 2023):

What's the issue with opening ~~the~~ a mirror on any non-EU hosting?
See no problem...

IIANM, yt-dl.org was set up by the previous team of devs (dsftw, remitamine, e.a.); apart from full access to the GitHub ytdl-org organisation granted to the current maintainer (dirkf), I'm unsure what other "privileges" were passed on to him from the previous devs (administration/ownership of the yt-dl.org domain name?) ...

My initial report above served the purpose of making the "breakage" (more) widely known and that "breakage" has been acknowledged by the maintainer 😉 ; the exact way this new "issue" is going to be mitigated by dirkf is probably his own call 😄 ; despite my meanderings, this is still the "New Release?" thread and the internal update feature of youtube-dl will have to be addressed sooner rather than later; when that new release finally arrives, I bet most of the users still stranded on 2021.12.17 will try youtube-dl -U to update it 😉 ...

@Vangelis66 commented on GitHub (Aug 2, 2023): > What's the issue with opening ~the~ a mirror on any non-EU hosting? > See no problem... IIANM, `yt-dl.org` was set up by the previous team of devs (**dsftw**, **remitamine**, e.a.); apart from **full access** to the **GitHub** `ytdl-org` organisation granted to the current maintainer (**dirkf**), I'm unsure what other "privileges" were passed on to him from the previous devs (administration/ownership of the `yt-dl.org` domain name?) ... My initial report [above](https://github.com/ytdl-org/youtube-dl/issues/31585#issuecomment-1660649283) served the purpose of making the "breakage" (more) widely known and that "breakage" has been [acknowledged](https://github.com/ytdl-org/youtube-dl/issues/31585#issuecomment-1660941891) by the maintainer 😉 ; the exact way this new "issue" is going to be mitigated by **dirkf** is probably his own call 😄 ; despite my meanderings, this is still the "**New Release?**" thread and the **internal update feature** of `youtube-dl` will have to be addressed sooner rather than later; when that _new release_ finally arrives, I bet most of the users still stranded on `2021.12.17` will try `youtube-dl -U` to update it 😉 ...

deekerman commented

github.com/ytdl-org/youtube-dl@2efc8de4d2/youtube_dl/update.py (L37)

Author

Owner

@Vangelis66 commented on GitHub (Aug 2, 2023):

You would have to alias yt-dl.org to ytdl-org.github.io for the moment

Indeed, prior to 20230801, below redirection was taking place when an "update check" was invoked:

(that screenshot was taken by me on 20230719, but I had forgotten saving it - early signs of dementia, probably? ...)

So, I suppose,

should now become:

    UPDATE_URL = 'https://ytdl-org.github.io/youtube-dl/update/'

Then, indeed:

youtube-dl -vU

[debug] System config: []
[debug] User config: []
[debug] Custom config: []
[debug] Command-line args: ['-vU']
[debug] Encodings: locale cp1253, fs mbcs, out cp737, pref cp1253
[debug] youtube-dl version 2023.08.01
[debug] Lazy loading extractors enabled
[debug] Single file build
[debug] Python 3.4.4 (CPython x86 32bit) - Windows-Vista-6.0.6003-SP2 - OpenSSL 1.0.2d 9 Jul 2015
[debug] exe versions: none
[debug] Proxy map: {}
youtube-dl is up to date (2023.08.01)

@Vangelis66 commented on GitHub (Aug 2, 2023): > You would have to alias `yt-dl.org` to `ytdl-org.github.io` for the moment Indeed, prior to **20230801**, **below redirection** was taking place when an "update check" was invoked: ![ytdl-redir](https://github.com/ytdl-org/youtube-dl/assets/9669492/e01cdef0-4b1f-4f0c-a346-9a6833948030) (that screenshot was taken by me on **20230719**, but I had forgotten saving it - early signs of dementia, probably? ...) So, I suppose, https://github.com/ytdl-org/youtube-dl/blob/2efc8de4d2299e08e0c84d674d7fc7f3fa669487/youtube_dl/update.py#L37 should now become: ```python UPDATE_URL = 'https://ytdl-org.github.io/youtube-dl/update/' ``` Then, indeed: ```console youtube-dl -vU [debug] System config: [] [debug] User config: [] [debug] Custom config: [] [debug] Command-line args: ['-vU'] [debug] Encodings: locale cp1253, fs mbcs, out cp737, pref cp1253 [debug] youtube-dl version 2023.08.01 [debug] Lazy loading extractors enabled [debug] Single file build [debug] Python 3.4.4 (CPython x86 32bit) - Windows-Vista-6.0.6003-SP2 - OpenSSL 1.0.2d 9 Jul 2015 [debug] exe versions: none [debug] Proxy map: {} youtube-dl is up to date (2023.08.01)

deekerman commented

Author

Owner

@xaur commented on GitHub (Sep 15, 2023):

the internal update feature of youtube-dl will have to be addressed sooner rather than later; when that new release finally arrives, I bet most of the users still stranded on 2021.12.17 will try youtube-dl -U to update it 😉

Sorry I am not up to speed on what is blocking the release, but if it is the update feature it may be worth to release without it, and until the update backend is fixed it could return something like "Not available yet, please update manually from: (link...)". Personally I never used the auto update and updated manually, while having a signed release is much better than not having it.

@xaur commented on GitHub (Sep 15, 2023): > the internal update feature of youtube-dl will have to be addressed sooner rather than later; when that new release finally arrives, I bet most of the users still stranded on 2021.12.17 will try youtube-dl -U to update it 😉 Sorry I am not up to speed on what is blocking the release, but if it is the update feature it may be worth to release without it, and until the update backend is fixed it could return something like "Not available yet, please update manually from: (link...)". Personally I never used the auto update and updated manually, while having a signed release is much better than not having it.

deekerman commented

Author

Owner

@ReenigneArcher commented on GitHub (Sep 15, 2023):

I would also like to see the library updated in PyPi so @dependabot could keep it updated, which does not work when specifying a git tag/commit for pip requirements.txt.

@ReenigneArcher commented on GitHub (Sep 15, 2023): I would also like to see the library updated in PyPi so @dependabot could keep it updated, which does not work when specifying a git tag/commit for pip requirements.txt.

deekerman commented

Brotli-1.0.9-cp34-cp34m-win32.zip

Author

Owner

@Vangelis66 commented on GitHub (Sep 20, 2023):

With the somewhat recently implemented support for the brotli decoder, I have been locally compiling py2.7+py3.4 youtube-dl.exe builds with that optional dependency bundled in...

For brotli-1.0.9, PyPI holds wheels for py2.7, but NOT for py3.4, which was well "dead" at the time (2020) 1.0.9 was released! Despite that, the 1.0.9 source is still compatible with py3.4 and, as such, I was, in the end, successful at locally compiling a win32 wheel of brotli-1.0.9 (Visual Studio 2010 Express required):

Earlier this month, Google ( 😡 ) released brotli-1.1.0, but their devs, inadvertently or otherwise 😠 , managed to break compatibility with any CPython version that doesn't natively support f-strings (< py3.6); OTOH, the PyPI entry for 1.1.0 still advertises compatibility with py2.7/3.4; this discrepancy has been reported here, with no input from the brotli devs 👎 as of yet...

In that tracker, I was given some hints by a kind person as to how to modify the setup.py source file of brotli-1.1.0 and restore compatibility with py<3.6; most sadly, I admit I'm too thick 😊 to follow those coding guidelines 😢 ; @dirkf, could you be extremely kind 😄 and share a "transpiled" version of

https://raw.githubusercontent.com/google/brotli/v1.1/setup.py

? I intend to, at least, build a py3.4 wheel of 1.1.0 with your help, someone else would have to build (if there's any need for it 😜 ) the py2.7-based wheel of it...

Thanks in advance 😄 ...

@Vangelis66 commented on GitHub (Sep 20, 2023): With the somewhat recently implemented support for the `brotli` decoder, I have been locally compiling py**2.7**+py**3.4** `youtube-dl.exe` builds with that **optional dependency** bundled in... For `brotli-1.0.9`, PyPI holds [wheels](https://files.pythonhosted.org/packages/f7/fb/97060f469f19d5e68377b3e6871802442e01bee90201dc8ed37f6b9e9322/Brotli-1.0.9-cp27-cp27m-win32.whl) for **py2.7**, but NOT for **py3.4**, which was well "dead" at the time (2020) `1.0.9` was released! Despite that, the `1.0.9` source is **still compatible with py3.4** and, as such, I was, in the end, successful at locally compiling a **win32** wheel of `brotli-1.0.9` (**Visual Studio 2010 Express** required): [Brotli-1.0.9-cp34-cp34m-win32.zip](https://github.com/ytdl-org/youtube-dl/files/12678867/Brotli-1.0.9-cp34-cp34m-win32.zip) Earlier this month, **Google** ( 😡 ) released `brotli-1.1.0`, but their devs, inadvertently or otherwise 😠 , managed to **break compatibility with _any_ CPython version that doesn't natively support** `f-strings` (< py3.**6**); OTOH, the [PyPI entry for 1.1.0](https://pypi.org/project/Brotli/1.1.0/) still advertises compatibility with py2.7/3.4; this discrepancy has been reported [here](https://github.com/google/brotli/issues/1074), with no input from the `brotli` devs 👎 as of yet... In that tracker, I was given some hints by a kind person as to how to modify the `setup.py` source file of `brotli-1.1.0` and restore compatibility with **py<3.6**; most sadly, I admit I'm too thick 😊 to follow those coding guidelines 😢 ; @dirkf, could you be extremely kind 😄 and share a "transpiled" version of https://raw.githubusercontent.com/google/brotli/v1.1/setup.py ? I intend to, at least, **build a py3.4 wheel of 1.1.0** with your help, someone else would have to build (if there's any need for it 😜 ) the **py2.7**-based wheel of it... Thanks in advance 😄 ...

deekerman commented

Author

Owner

@dirkf commented on GitHub (Sep 21, 2023):

This `setup.py` doesn't use `f''` and also supplies an explicit exception class at l.14 which the _flake8_ checker likes to see.

# Copyright 2015 The Brotli Authors. All rights reserved.
#
# Distributed under MIT license.
# See file LICENSE for detail or copy at https://opensource.org/licenses/MIT

import os
import platform
import re
import unittest

try:
    from setuptools import Extension
    from setuptools import setup
except ImportError:
    from distutils.core import Extension
    from distutils.core import setup
from distutils.command.build_ext import build_ext
from distutils import errors
from distutils import dep_util
from distutils import log


CURR_DIR = os.path.abspath(os.path.dirname(os.path.realpath(__file__)))


def read_define(path, macro):
  """ Return macro value from the given file. """
  with open(path, 'r') as f:
    for line in f:
      m = re.match(r'#define\s{0}\s+(.+)'.format(macro), line)
      if m:
        return m.group(1)
  return ''


def get_version():
  """ Return library version string from 'common/version.h' file. """
  version_file_path = os.path.join(CURR_DIR, 'c', 'common', 'version.h')
  major = read_define(version_file_path, 'BROTLI_VERSION_MAJOR')
  minor = read_define(version_file_path, 'BROTLI_VERSION_MINOR')
  patch = read_define(version_file_path, 'BROTLI_VERSION_PATCH')
  if not major or not minor or not patch:
    return ''
  return '{0}.{1}.{2}'.format(major, minor, patch)


def get_test_suite():
  test_loader = unittest.TestLoader()
  test_suite = test_loader.discover('python', pattern='*_test.py')
  return test_suite


class BuildExt(build_ext):

  def get_source_files(self):
    filenames = build_ext.get_source_files(self)
    for ext in self.extensions:
      filenames.extend(ext.depends)
    return filenames

  def build_extension(self, ext):
    if ext.sources is None or not isinstance(ext.sources, (list, tuple)):
      raise errors.DistutilsSetupError(
        "in 'ext_modules' option (extension '%s'), "
        "'sources' must be present and must be "
        "a list of source filenames" % ext.name)

    ext_path = self.get_ext_fullpath(ext.name)
    depends = ext.sources + ext.depends
    if not (self.force or dep_util.newer_group(depends, ext_path, 'newer')):
      log.debug("skipping '%s' extension (up-to-date)", ext.name)
      return
    else:
      log.info("building '%s' extension", ext.name)

    c_sources = []
    for source in ext.sources:
      if source.endswith('.c'):
        c_sources.append(source)
    extra_args = ext.extra_compile_args or []

    objects = []

    macros = ext.define_macros[:]
    if platform.system() == 'Darwin':
      macros.append(('OS_MACOSX', '1'))
    elif self.compiler.compiler_type == 'mingw32':
      # On Windows Python 2.7, pyconfig.h defines "hypot" as "_hypot",
      # This clashes with GCC's cmath, and causes compilation errors when
      # building under MinGW: http://bugs.python.org/issue11566
      macros.append(('_hypot', 'hypot'))
    for undef in ext.undef_macros:
      macros.append((undef,))

    objs = self.compiler.compile(
        c_sources,
        output_dir=self.build_temp,
        macros=macros,
        include_dirs=ext.include_dirs,
        debug=self.debug,
        extra_postargs=extra_args,
        depends=ext.depends)
    objects.extend(objs)

    self._built_objects = objects[:]
    if ext.extra_objects:
      objects.extend(ext.extra_objects)
    extra_args = ext.extra_link_args or []
    # when using GCC on Windows, we statically link libgcc and libstdc++,
    # so that we don't need to package extra DLLs
    if self.compiler.compiler_type == 'mingw32':
        extra_args.extend(['-static-libgcc', '-static-libstdc++'])

    ext_path = self.get_ext_fullpath(ext.name)
    # Detect target language, if not provided
    language = ext.language or self.compiler.detect_language(c_sources)

    self.compiler.link_shared_object(
        objects,
        ext_path,
        libraries=self.get_libraries(ext),
        library_dirs=ext.library_dirs,
        runtime_library_dirs=ext.runtime_library_dirs,
        extra_postargs=extra_args,
        export_symbols=self.get_export_symbols(ext),
        debug=self.debug,
        build_temp=self.build_temp,
        target_lang=language)


NAME = 'Brotli'

VERSION = get_version()

URL = 'https://github.com/google/brotli'

DESCRIPTION = 'Python bindings for the Brotli compression library'

AUTHOR = 'The Brotli Authors'

LICENSE = 'MIT'

PLATFORMS = ['Posix', 'MacOS X', 'Windows']

CLASSIFIERS = [
    'Development Status :: 4 - Beta',
    'Environment :: Console',
    'Intended Audience :: Developers',
    'License :: OSI Approved :: MIT License',
    'Operating System :: MacOS :: MacOS X',
    'Operating System :: Microsoft :: Windows',
    'Operating System :: POSIX :: Linux',
    'Programming Language :: C',
    'Programming Language :: C++',
    'Programming Language :: Python',
    'Programming Language :: Python :: 2',
    'Programming Language :: Python :: 2.7',
    'Programming Language :: Python :: 3',
    'Programming Language :: Python :: 3.3',
    'Programming Language :: Python :: 3.4',
    'Programming Language :: Python :: 3.5',
    'Programming Language :: Unix Shell',
    'Topic :: Software Development :: Libraries',
    'Topic :: Software Development :: Libraries :: Python Modules',
    'Topic :: System :: Archiving',
    'Topic :: System :: Archiving :: Compression',
    'Topic :: Text Processing :: Fonts',
    'Topic :: Utilities',
]

PACKAGE_DIR = {'': 'python'}

PY_MODULES = ['brotli']

EXT_MODULES = [
    Extension(
        '_brotli',
        sources=[
            'python/_brotli.c',
            'c/common/constants.c',
            'c/common/context.c',
            'c/common/dictionary.c',
            'c/common/platform.c',
            'c/common/shared_dictionary.c',
            'c/common/transform.c',
            'c/dec/bit_reader.c',
            'c/dec/decode.c',
            'c/dec/huffman.c',
            'c/dec/state.c',
            'c/enc/backward_references.c',
            'c/enc/backward_references_hq.c',
            'c/enc/bit_cost.c',
            'c/enc/block_splitter.c',
            'c/enc/brotli_bit_stream.c',
            'c/enc/cluster.c',
            'c/enc/command.c',
            'c/enc/compound_dictionary.c',
            'c/enc/compress_fragment.c',
            'c/enc/compress_fragment_two_pass.c',
            'c/enc/dictionary_hash.c',
            'c/enc/encode.c',
            'c/enc/encoder_dict.c',
            'c/enc/entropy_encode.c',
            'c/enc/fast_log.c',
            'c/enc/histogram.c',
            'c/enc/literal_cost.c',
            'c/enc/memory.c',
            'c/enc/metablock.c',
            'c/enc/static_dict.c',
            'c/enc/utf8_util.c',
        ],
        depends=[
            'c/common/constants.h',
            'c/common/context.h',
            'c/common/dictionary.h',
            'c/common/platform.h',
            'c/common/shared_dictionary_internal.h',
            'c/common/transform.h',
            'c/common/version.h',
            'c/dec/bit_reader.h',
            'c/dec/huffman.h',
            'c/dec/prefix.h',
            'c/dec/state.h',
            'c/enc/backward_references.h',
            'c/enc/backward_references_hq.h',
            'c/enc/backward_references_inc.h',
            'c/enc/bit_cost.h',
            'c/enc/bit_cost_inc.h',
            'c/enc/block_encoder_inc.h',
            'c/enc/block_splitter.h',
            'c/enc/block_splitter_inc.h',
            'c/enc/brotli_bit_stream.h',
            'c/enc/cluster.h',
            'c/enc/cluster_inc.h',
            'c/enc/command.h',
            'c/enc/compound_dictionary.h',
            'c/enc/compress_fragment.h',
            'c/enc/compress_fragment_two_pass.h',
            'c/enc/dictionary_hash.h',
            'c/enc/encoder_dict.h',
            'c/enc/entropy_encode.h',
            'c/enc/entropy_encode_static.h',
            'c/enc/fast_log.h',
            'c/enc/find_match_length.h',
            'c/enc/hash.h',
            'c/enc/hash_composite_inc.h',
            'c/enc/hash_forgetful_chain_inc.h',
            'c/enc/hash_longest_match64_inc.h',
            'c/enc/hash_longest_match_inc.h',
            'c/enc/hash_longest_match_quickly_inc.h',
            'c/enc/hash_rolling_inc.h',
            'c/enc/hash_to_binary_tree_inc.h',
            'c/enc/histogram.h',
            'c/enc/histogram_inc.h',
            'c/enc/literal_cost.h',
            'c/enc/memory.h',
            'c/enc/metablock.h',
            'c/enc/metablock_inc.h',
            'c/enc/params.h',
            'c/enc/prefix.h',
            'c/enc/quality.h',
            'c/enc/ringbuffer.h',
            'c/enc/static_dict.h',
            'c/enc/static_dict_lut.h',
            'c/enc/utf8_util.h',
            'c/enc/write_bits.h',
        ],
        include_dirs=[
            'c/include',
        ]),
]

TEST_SUITE = 'setup.get_test_suite'

CMD_CLASS = {
    'build_ext': BuildExt,
}

setup(
    name=NAME,
    description=DESCRIPTION,
    version=VERSION,
    url=URL,
    author=AUTHOR,
    license=LICENSE,
    platforms=PLATFORMS,
    classifiers=CLASSIFIERS,
    package_dir=PACKAGE_DIR,
    py_modules=PY_MODULES,
    ext_modules=EXT_MODULES,
    test_suite=TEST_SUITE,
    cmdclass=CMD_CLASS)

@dirkf commented on GitHub (Sep 21, 2023): <details> <summary>This `setup.py` doesn't use `f''` and also supplies an explicit exception class at l.14 which the _flake8_ checker likes to see. </summary> ```py # Copyright 2015 The Brotli Authors. All rights reserved. # # Distributed under MIT license. # See file LICENSE for detail or copy at https://opensource.org/licenses/MIT import os import platform import re import unittest try: from setuptools import Extension from setuptools import setup except ImportError: from distutils.core import Extension from distutils.core import setup from distutils.command.build_ext import build_ext from distutils import errors from distutils import dep_util from distutils import log CURR_DIR = os.path.abspath(os.path.dirname(os.path.realpath(__file__))) def read_define(path, macro): """ Return macro value from the given file. """ with open(path, 'r') as f: for line in f: m = re.match(r'#define\s{0}\s+(.+)'.format(macro), line) if m: return m.group(1) return '' def get_version(): """ Return library version string from 'common/version.h' file. """ version_file_path = os.path.join(CURR_DIR, 'c', 'common', 'version.h') major = read_define(version_file_path, 'BROTLI_VERSION_MAJOR') minor = read_define(version_file_path, 'BROTLI_VERSION_MINOR') patch = read_define(version_file_path, 'BROTLI_VERSION_PATCH') if not major or not minor or not patch: return '' return '{0}.{1}.{2}'.format(major, minor, patch) def get_test_suite(): test_loader = unittest.TestLoader() test_suite = test_loader.discover('python', pattern='*_test.py') return test_suite class BuildExt(build_ext): def get_source_files(self): filenames = build_ext.get_source_files(self) for ext in self.extensions: filenames.extend(ext.depends) return filenames def build_extension(self, ext): if ext.sources is None or not isinstance(ext.sources, (list, tuple)): raise errors.DistutilsSetupError( "in 'ext_modules' option (extension '%s'), " "'sources' must be present and must be " "a list of source filenames" % ext.name) ext_path = self.get_ext_fullpath(ext.name) depends = ext.sources + ext.depends if not (self.force or dep_util.newer_group(depends, ext_path, 'newer')): log.debug("skipping '%s' extension (up-to-date)", ext.name) return else: log.info("building '%s' extension", ext.name) c_sources = [] for source in ext.sources: if source.endswith('.c'): c_sources.append(source) extra_args = ext.extra_compile_args or [] objects = [] macros = ext.define_macros[:] if platform.system() == 'Darwin': macros.append(('OS_MACOSX', '1')) elif self.compiler.compiler_type == 'mingw32': # On Windows Python 2.7, pyconfig.h defines "hypot" as "_hypot", # This clashes with GCC's cmath, and causes compilation errors when # building under MinGW: http://bugs.python.org/issue11566 macros.append(('_hypot', 'hypot')) for undef in ext.undef_macros: macros.append((undef,)) objs = self.compiler.compile( c_sources, output_dir=self.build_temp, macros=macros, include_dirs=ext.include_dirs, debug=self.debug, extra_postargs=extra_args, depends=ext.depends) objects.extend(objs) self._built_objects = objects[:] if ext.extra_objects: objects.extend(ext.extra_objects) extra_args = ext.extra_link_args or [] # when using GCC on Windows, we statically link libgcc and libstdc++, # so that we don't need to package extra DLLs if self.compiler.compiler_type == 'mingw32': extra_args.extend(['-static-libgcc', '-static-libstdc++']) ext_path = self.get_ext_fullpath(ext.name) # Detect target language, if not provided language = ext.language or self.compiler.detect_language(c_sources) self.compiler.link_shared_object( objects, ext_path, libraries=self.get_libraries(ext), library_dirs=ext.library_dirs, runtime_library_dirs=ext.runtime_library_dirs, extra_postargs=extra_args, export_symbols=self.get_export_symbols(ext), debug=self.debug, build_temp=self.build_temp, target_lang=language) NAME = 'Brotli' VERSION = get_version() URL = 'https://github.com/google/brotli' DESCRIPTION = 'Python bindings for the Brotli compression library' AUTHOR = 'The Brotli Authors' LICENSE = 'MIT' PLATFORMS = ['Posix', 'MacOS X', 'Windows'] CLASSIFIERS = [ 'Development Status :: 4 - Beta', 'Environment :: Console', 'Intended Audience :: Developers', 'License :: OSI Approved :: MIT License', 'Operating System :: MacOS :: MacOS X', 'Operating System :: Microsoft :: Windows', 'Operating System :: POSIX :: Linux', 'Programming Language :: C', 'Programming Language :: C++', 'Programming Language :: Python', 'Programming Language :: Python :: 2', 'Programming Language :: Python :: 2.7', 'Programming Language :: Python :: 3', 'Programming Language :: Python :: 3.3', 'Programming Language :: Python :: 3.4', 'Programming Language :: Python :: 3.5', 'Programming Language :: Unix Shell', 'Topic :: Software Development :: Libraries', 'Topic :: Software Development :: Libraries :: Python Modules', 'Topic :: System :: Archiving', 'Topic :: System :: Archiving :: Compression', 'Topic :: Text Processing :: Fonts', 'Topic :: Utilities', ] PACKAGE_DIR = {'': 'python'} PY_MODULES = ['brotli'] EXT_MODULES = [ Extension( '_brotli', sources=[ 'python/_brotli.c', 'c/common/constants.c', 'c/common/context.c', 'c/common/dictionary.c', 'c/common/platform.c', 'c/common/shared_dictionary.c', 'c/common/transform.c', 'c/dec/bit_reader.c', 'c/dec/decode.c', 'c/dec/huffman.c', 'c/dec/state.c', 'c/enc/backward_references.c', 'c/enc/backward_references_hq.c', 'c/enc/bit_cost.c', 'c/enc/block_splitter.c', 'c/enc/brotli_bit_stream.c', 'c/enc/cluster.c', 'c/enc/command.c', 'c/enc/compound_dictionary.c', 'c/enc/compress_fragment.c', 'c/enc/compress_fragment_two_pass.c', 'c/enc/dictionary_hash.c', 'c/enc/encode.c', 'c/enc/encoder_dict.c', 'c/enc/entropy_encode.c', 'c/enc/fast_log.c', 'c/enc/histogram.c', 'c/enc/literal_cost.c', 'c/enc/memory.c', 'c/enc/metablock.c', 'c/enc/static_dict.c', 'c/enc/utf8_util.c', ], depends=[ 'c/common/constants.h', 'c/common/context.h', 'c/common/dictionary.h', 'c/common/platform.h', 'c/common/shared_dictionary_internal.h', 'c/common/transform.h', 'c/common/version.h', 'c/dec/bit_reader.h', 'c/dec/huffman.h', 'c/dec/prefix.h', 'c/dec/state.h', 'c/enc/backward_references.h', 'c/enc/backward_references_hq.h', 'c/enc/backward_references_inc.h', 'c/enc/bit_cost.h', 'c/enc/bit_cost_inc.h', 'c/enc/block_encoder_inc.h', 'c/enc/block_splitter.h', 'c/enc/block_splitter_inc.h', 'c/enc/brotli_bit_stream.h', 'c/enc/cluster.h', 'c/enc/cluster_inc.h', 'c/enc/command.h', 'c/enc/compound_dictionary.h', 'c/enc/compress_fragment.h', 'c/enc/compress_fragment_two_pass.h', 'c/enc/dictionary_hash.h', 'c/enc/encoder_dict.h', 'c/enc/entropy_encode.h', 'c/enc/entropy_encode_static.h', 'c/enc/fast_log.h', 'c/enc/find_match_length.h', 'c/enc/hash.h', 'c/enc/hash_composite_inc.h', 'c/enc/hash_forgetful_chain_inc.h', 'c/enc/hash_longest_match64_inc.h', 'c/enc/hash_longest_match_inc.h', 'c/enc/hash_longest_match_quickly_inc.h', 'c/enc/hash_rolling_inc.h', 'c/enc/hash_to_binary_tree_inc.h', 'c/enc/histogram.h', 'c/enc/histogram_inc.h', 'c/enc/literal_cost.h', 'c/enc/memory.h', 'c/enc/metablock.h', 'c/enc/metablock_inc.h', 'c/enc/params.h', 'c/enc/prefix.h', 'c/enc/quality.h', 'c/enc/ringbuffer.h', 'c/enc/static_dict.h', 'c/enc/static_dict_lut.h', 'c/enc/utf8_util.h', 'c/enc/write_bits.h', ], include_dirs=[ 'c/include', ]), ] TEST_SUITE = 'setup.get_test_suite' CMD_CLASS = { 'build_ext': BuildExt, } setup( name=NAME, description=DESCRIPTION, version=VERSION, url=URL, author=AUTHOR, license=LICENSE, platforms=PLATFORMS, classifiers=CLASSIFIERS, package_dir=PACKAGE_DIR, py_modules=PY_MODULES, ext_modules=EXT_MODULES, test_suite=TEST_SUITE, cmdclass=CMD_CLASS) ``` </details>

deekerman commented

Author

Owner

@parkr commented on GitHub (Sep 21, 2023):

Sorry, isn't the brotli update from 1.0.9 to 1.1.0 out of scope? Also py2.7 and py3.4 are severely out of date receiving no security updates or anything. If supporting decade-old Python versions is blocking this release, perhaps this is a good opportunity to reevaluate the support strategy for old Python versions?

@parkr commented on GitHub (Sep 21, 2023): Sorry, isn't the brotli update from 1.0.9 to 1.1.0 out of scope? Also py2.7 and py3.4 are severely out of date receiving no security updates or anything. If supporting decade-old Python versions is blocking this release, perhaps this is a good opportunity to reevaluate the support strategy for old Python versions?

deekerman commented

Author

Owner

@Vangelis66 commented on GitHub (Sep 21, 2023):

If supporting decade-old Python versions is blocking this release

... It isn't!

isn't the brotli update from 1.0.9 to 1.1.0 out of scope?

How so? brotli-1.1.0 is advertised as still compatible with py2.7/3.4 in PyPI, and it appears it is (or was intended to still be), minus this setup.py file cock-up caused by the Google devs (or their AI) ...

Also py2.7 and py3.4 are severely out of date, receiving no security updates or anything.

... But there exists H/W that can't (for a variety of reasons) be updated to more "modern" versions of Python and it's those cases that the current development strives to also cater to - this has been discussed/explained/analysed to death elsewhere in this repo...

Please, direct your current frustration elsewhere 😉 ; since, it seems, you won't feel comfortable (and more secure?) with nothing but the PSF-supported versions of Python, "downstream" (i.e. yt-dlp) are just perfecting their support for the coming release of py3.12-final.; you can always migrate to that (yt-dlp) ...

Kind regards.

@Vangelis66 commented on GitHub (Sep 21, 2023): > If supporting decade-old Python versions **is blocking this release** ... **It isn't**! > isn't the brotli update from 1.0.9 to 1.1.0 **out of scope**? How so? `brotli-1.1.0` is advertised as still compatible with py2.7/3.4 in PyPI, and it appears **it is** (or _was intended_ to still be), minus this `setup.py` file cock-up caused by the **Google** devs (or their **AI**) ... > Also py2.7 and py3.4 are severely out of date, receiving no security updates or anything. ... But there exists H/W that can't (for a variety of reasons) be updated to more "modern" versions of Python and it's those cases that the current development strives to also cater to - this has been discussed/explained/analysed to death elsewhere in this repo... Please, direct your current frustration elsewhere 😉 ; since, it seems, you won't feel comfortable (and more secure?) with nothing but the **PSF-supported** versions of Python, "downstream" (i.e. `yt-dlp`) are just [perfecting their support](https://github.com/yt-dlp/yt-dlp/commit/35f9a306e6934793cff100200cd03f288ec33f11) for the coming release of **py3.12-final.**; you can always migrate to that (`yt-dlp`) ... Kind regards.

deekerman commented

Author

Owner

@Vangelis66 commented on GitHub (Sep 21, 2023):

This setup.py doesn't use f'' and also supplies an explicit exception class at l.14, which the flake8 checker likes to see.

I intend to, at least, build a py3.4 wheel of 1.1.0 with your help,

... It turns out I was too optimistic to begin with 😞 , basically fueled by this comment in the brotli issue tracker:

I can confirm that removing the two f-strings fixes the issue and that the package can then be installed with Python 2.

@nonatomiclabs, are you on Windows and if yes, what version?
Do you have vcpython27 (a download no longer available from M$) installed?
Can you share a build of Brotli-1.1.0-cp27-cp27m-win32.whl ?

I tried building Brotli-1.1.0-cp34-cp34m-win32.whl with the patched setup.py file (kindly provided by dirkf), but it emerged that the C code inside the brotli-1.1.0 source has become incompatible with the VS2010 (MS Visual C++ 10.0) compiler required to build the module under CPython 3.4 on Windows:

Failed attempt at installing brotli-1.1.0(mod) under py3.4, on Win x86

python -m pip install "Brotli-1.1.0-mod.tar.gz" => 

Processing c:\python3410-32\Brotli-1.1.0-mod.tar.gz
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
  Installing backend dependencies ... done
    Preparing wheel metadata ... done
Building wheels for collected packages: Brotli
  Building wheel for Brotli (PEP 517) ... error
  ERROR: Complete output from command 'C:\Python3410-32\python.exe' 'C:\Python3410-32\lib\site-packages\pip\_vendor\pep517\_in_process.py' build_wheel 'C:\Users\<redacted>\AppData\Local\Temp\tmppbrr9dw6':
  ERROR: running bdist_wheel
  running build
  running build_py
  creating bin
  creating bin\lib.win32-3.4
  copying python\brotli.py -> bin\lib.win32-3.4
  running build_ext
  building '_brotli' extension
  creating bin\temp.win32-3.4
  creating bin\temp.win32-3.4\Release
  creating bin\temp.win32-3.4\Release\python
  creating bin\temp.win32-3.4\Release\c
  creating bin\temp.win32-3.4\Release\c\common
  creating bin\temp.win32-3.4\Release\c\dec
  creating bin\temp.win32-3.4\Release\c\enc
  C:\Program Files\Microsoft Visual Studio 10.0\VC\BIN\cl.exe /c /nologo /Ox /MD /W3 /GS- /DNDEBUG -Ic/include -IC:\Python3410-32\include -IC:\Python3410-32\include /Tcpython/_brotli.c /Fobin\temp.win32-3.4\Release\python/_brotli.obj
  _brotli.c
  python/_brotli.c(72) : error C2054: expected '(' to follow 'inline'
  python/_brotli.c(74) : error C2085: 'BlocksOutputBuffer_InitAndGrow' : not in formal parameter list
  python/_brotli.c(74) : error C2143: syntax error : missing ';' before '{'
  python/_brotli.c(108) : error C2054: expected '(' to follow 'inline'
  python/_brotli.c(110) : error C2085: 'BlocksOutputBuffer_Grow' : not in formal parameter list
  python/_brotli.c(110) : error C2143: syntax error : missing ';' before '{'
  python/_brotli.c(155) : error C2054: expected '(' to follow 'inline'
  python/_brotli.c(157) : error C2085: 'BlocksOutputBuffer_Finish' : not in formal parameter list
  python/_brotli.c(157) : error C2143: syntax error : missing ';' before '{'
  python/_brotli.c(203) : error C2054: expected '(' to follow 'inline'
  python/_brotli.c(204) : error C2085: 'BlocksOutputBuffer_OnError' : not in formal parameter list
  python/_brotli.c(204) : error C2143: syntax error : missing ';' before '{'
  python/_brotli.c(224) : error C2143: syntax error : missing ';' before 'type'
  python/_brotli.c(225) : error C2065: 'mode_value' : undeclared identifier
  python/_brotli.c(229) : error C2065: 'mode_value' : undeclared identifier
  python/_brotli.c(291) : error C2059: syntax error : '.'
  python/_brotli.c(294) : warning C4013: 'BlocksOutputBuffer_InitAndGrow' undefined; assuming extern returning int
  python/_brotli.c(310) : warning C4013: 'BlocksOutputBuffer_Grow' undefined; assuming extern returning int
  python/_brotli.c(320) : warning C4013: 'BlocksOutputBuffer_Finish' undefined; assuming extern returning int
  python/_brotli.c(320) : warning C4047: '=' : 'PyObject *' differs in levels of indirection from 'int'
  python/_brotli.c(326) : warning C4013: 'BlocksOutputBuffer_OnError' undefined; assuming extern returning int
  python/_brotli.c(604) : error C2059: syntax error : '.'
  python/_brotli.c(634) : warning C4047: '=' : 'PyObject *' differs in levels of indirection from 'int'
  python/_brotli.c(856) : error C2059: syntax error : '.'
  python/_brotli.c(906) : warning C4047: '=' : 'PyObject *' differs in levels of indirection from 'int'
  python/_brotli.c(922) : error C2061: syntax error : identifier 'brotli_methods'
  python/_brotli.c(922) : error C2059: syntax error : ';'
  python/_brotli.c(922) : error C3409: empty attribute block is not allowed
  python/_brotli.c(922) : error C2513: '/*global*/ ' : no variable declared before '='
  python/_brotli.c(940) : error C2065: 'brotli_methods' : undeclared identifier
  python/_brotli.c(940) : error C2099: initializer is not a constant
  python/_brotli.c(978) : error C2143: syntax error : missing ';' before 'type'
  python/_brotli.c(979) : error C2275: 'uint32_t' : illegal use of this type as an expression
          C:\Program Files\Microsoft Visual Studio 10.0\VC\INCLUDE\stdint.h(23): see declaration of 'uint32_t'
  python/_brotli.c(979) : error C2146: syntax error : missing ';' before identifier 'decoderVersion'
  python/_brotli.c(979) : error C2065: 'decoderVersion' : undeclared identifier
  python/_brotli.c(980) : error C2065: 'version' : undeclared identifier
  python/_brotli.c(980) : warning C4047: 'function' : 'char *' differs in levels of indirection from 'int'
  python/_brotli.c(980) : warning C4024: '_snprintf' : different types for formal and actual parameter 1
  python/_brotli.c(980) : error C2065: 'version' : undeclared identifier
  python/_brotli.c(981) : error C2065: 'decoderVersion' : undeclared identifier
  python/_brotli.c(981) : error C2065: 'decoderVersion' : undeclared identifier
  python/_brotli.c(981) : error C2065: 'decoderVersion' : undeclared identifier
  python/_brotli.c(982) : error C2065: 'version' : undeclared identifier
  python/_brotli.c(982) : warning C4047: 'function' : 'const char *' differs in levels of indirection from 'int'
  python/_brotli.c(982) : warning C4024: 'PyModule_AddStringConstant' : different types for formal and actual parameter 3
  error: command 'C:\\Program Files\\Microsoft Visual Studio 10.0\\VC\\BIN\\cl.exe' failed with exit status 2
  ----------------------------------------
  ERROR: Failed building wheel for Brotli
  Running setup.py clean for Brotli
Failed to build Brotli
ERROR: Could not build wheels for Brotli which use PEP 517 and cannot be installed directly

Thus, I'm even more perplexed by the fact someone else did report installation success under py2.7 which, on Windows, requires the VS2008 (MS Visual C++ 9.0) based VCPython27 compiler (???) ...

So, for me, when on py<3.6, brotli-1.0.9 from Aug 2020 is the only choice if installing this yt-dl optional dependency; FTR, brotli support was introduced in e7926ae and 2efc8de; BTW, the ncompress module is out of reach for py<3.8 and/or the 32-bit OS architecture 😞 ...

@Vangelis66 commented on GitHub (Sep 21, 2023): > This `setup.py` doesn't use `f''` and also supplies an explicit exception class at l.14, which the _flake8_ checker likes to see. > I intend to, at least, **build a py3.4 wheel of 1.1.0** with your help, ... It turns out I was **too optimistic** to begin with 😞 , basically fueled by this comment in the `brotli` issue tracker: > I can confirm that removing the two `f-strings` fixes the issue and that **the package can then be installed with Python 2.** @nonatomiclabs, are you on **Windows** and if yes, what version? Do you have **vcpython27** (a download no longer available from **M$**) installed? Can you share a build of `Brotli-1.1.0-cp27-cp27m-win32.whl` ? I tried building `Brotli-1.1.0-cp34-cp34m-win32.whl` with the patched `setup.py` file (kindly provided by **dirkf**), but it emerged that the **C code** inside the `brotli-1.1.0` source **has become incompatible** with the **VS2010** (MS Visual C++ 10.0) compiler required to build the module under **CPython 3.4 on Windows**: <details> <summary>Failed attempt at installing brotli-1.1.0(mod) under py3.4, on Win x86</summary> ```shell python -m pip install "Brotli-1.1.0-mod.tar.gz" => Processing c:\python3410-32\Brotli-1.1.0-mod.tar.gz Installing build dependencies ... done Getting requirements to build wheel ... done Installing backend dependencies ... done Preparing wheel metadata ... done Building wheels for collected packages: Brotli Building wheel for Brotli (PEP 517) ... error ERROR: Complete output from command 'C:\Python3410-32\python.exe' 'C:\Python3410-32\lib\site-packages\pip\_vendor\pep517\_in_process.py' build_wheel 'C:\Users\<redacted>\AppData\Local\Temp\tmppbrr9dw6': ERROR: running bdist_wheel running build running build_py creating bin creating bin\lib.win32-3.4 copying python\brotli.py -> bin\lib.win32-3.4 running build_ext building '_brotli' extension creating bin\temp.win32-3.4 creating bin\temp.win32-3.4\Release creating bin\temp.win32-3.4\Release\python creating bin\temp.win32-3.4\Release\c creating bin\temp.win32-3.4\Release\c\common creating bin\temp.win32-3.4\Release\c\dec creating bin\temp.win32-3.4\Release\c\enc C:\Program Files\Microsoft Visual Studio 10.0\VC\BIN\cl.exe /c /nologo /Ox /MD /W3 /GS- /DNDEBUG -Ic/include -IC:\Python3410-32\include -IC:\Python3410-32\include /Tcpython/_brotli.c /Fobin\temp.win32-3.4\Release\python/_brotli.obj _brotli.c python/_brotli.c(72) : error C2054: expected '(' to follow 'inline' python/_brotli.c(74) : error C2085: 'BlocksOutputBuffer_InitAndGrow' : not in formal parameter list python/_brotli.c(74) : error C2143: syntax error : missing ';' before '{' python/_brotli.c(108) : error C2054: expected '(' to follow 'inline' python/_brotli.c(110) : error C2085: 'BlocksOutputBuffer_Grow' : not in formal parameter list python/_brotli.c(110) : error C2143: syntax error : missing ';' before '{' python/_brotli.c(155) : error C2054: expected '(' to follow 'inline' python/_brotli.c(157) : error C2085: 'BlocksOutputBuffer_Finish' : not in formal parameter list python/_brotli.c(157) : error C2143: syntax error : missing ';' before '{' python/_brotli.c(203) : error C2054: expected '(' to follow 'inline' python/_brotli.c(204) : error C2085: 'BlocksOutputBuffer_OnError' : not in formal parameter list python/_brotli.c(204) : error C2143: syntax error : missing ';' before '{' python/_brotli.c(224) : error C2143: syntax error : missing ';' before 'type' python/_brotli.c(225) : error C2065: 'mode_value' : undeclared identifier python/_brotli.c(229) : error C2065: 'mode_value' : undeclared identifier python/_brotli.c(291) : error C2059: syntax error : '.' python/_brotli.c(294) : warning C4013: 'BlocksOutputBuffer_InitAndGrow' undefined; assuming extern returning int python/_brotli.c(310) : warning C4013: 'BlocksOutputBuffer_Grow' undefined; assuming extern returning int python/_brotli.c(320) : warning C4013: 'BlocksOutputBuffer_Finish' undefined; assuming extern returning int python/_brotli.c(320) : warning C4047: '=' : 'PyObject *' differs in levels of indirection from 'int' python/_brotli.c(326) : warning C4013: 'BlocksOutputBuffer_OnError' undefined; assuming extern returning int python/_brotli.c(604) : error C2059: syntax error : '.' python/_brotli.c(634) : warning C4047: '=' : 'PyObject *' differs in levels of indirection from 'int' python/_brotli.c(856) : error C2059: syntax error : '.' python/_brotli.c(906) : warning C4047: '=' : 'PyObject *' differs in levels of indirection from 'int' python/_brotli.c(922) : error C2061: syntax error : identifier 'brotli_methods' python/_brotli.c(922) : error C2059: syntax error : ';' python/_brotli.c(922) : error C3409: empty attribute block is not allowed python/_brotli.c(922) : error C2513: '/*global*/ ' : no variable declared before '=' python/_brotli.c(940) : error C2065: 'brotli_methods' : undeclared identifier python/_brotli.c(940) : error C2099: initializer is not a constant python/_brotli.c(978) : error C2143: syntax error : missing ';' before 'type' python/_brotli.c(979) : error C2275: 'uint32_t' : illegal use of this type as an expression C:\Program Files\Microsoft Visual Studio 10.0\VC\INCLUDE\stdint.h(23): see declaration of 'uint32_t' python/_brotli.c(979) : error C2146: syntax error : missing ';' before identifier 'decoderVersion' python/_brotli.c(979) : error C2065: 'decoderVersion' : undeclared identifier python/_brotli.c(980) : error C2065: 'version' : undeclared identifier python/_brotli.c(980) : warning C4047: 'function' : 'char *' differs in levels of indirection from 'int' python/_brotli.c(980) : warning C4024: '_snprintf' : different types for formal and actual parameter 1 python/_brotli.c(980) : error C2065: 'version' : undeclared identifier python/_brotli.c(981) : error C2065: 'decoderVersion' : undeclared identifier python/_brotli.c(981) : error C2065: 'decoderVersion' : undeclared identifier python/_brotli.c(981) : error C2065: 'decoderVersion' : undeclared identifier python/_brotli.c(982) : error C2065: 'version' : undeclared identifier python/_brotli.c(982) : warning C4047: 'function' : 'const char *' differs in levels of indirection from 'int' python/_brotli.c(982) : warning C4024: 'PyModule_AddStringConstant' : different types for formal and actual parameter 3 error: command 'C:\\Program Files\\Microsoft Visual Studio 10.0\\VC\\BIN\\cl.exe' failed with exit status 2 ---------------------------------------- ERROR: Failed building wheel for Brotli Running setup.py clean for Brotli Failed to build Brotli ERROR: Could not build wheels for Brotli which use PEP 517 and cannot be installed directly ``` </details> Thus, I'm even more perplexed by the fact someone else did report installation success under **py2.7** which, on Windows, requires the **VS2008** (MS Visual C++ 9.0) based **VCPython27** compiler (???) ... So, for me, when on **py<3.6**, `brotli-1.0.9` from Aug 2020 is **the only choice** if installing this `yt-dl` **optional** dependency; FTR, `brotli` support was introduced in e7926ae and 2efc8de; BTW, the `ncompress` module is out of reach for **py<3.8** and/or the **32-bit** OS architecture 😞 ...

deekerman commented

Author

Owner

@dirkf commented on GitHub (Sep 22, 2023):

Probably the way to make the C compile is to diff the latest Brotli against the one that works with VC2010 and reapply any substantive code changes, ignoring compiler directives, etc. Since it SHOULD (RFC7231) be possible to negotiate to no content-coding, the only reason for yt-dl to support br is that offering it might help to bypass fingerprinting, with the consequence that the actual encoding has to be supported (but Accept-Encoding: br;q=0, gzip;q=1.0 ?). It would be enough to have a native Python decoder for this.

The compress content-coding is in the RFCs but not seen in the wild. gzip and br compress faster/better. The benefit of the latter might only be significant for a giant corporation with a massive network infrastructure.

@dirkf commented on GitHub (Sep 22, 2023): Probably the way to make the C compile is to diff the latest Brotli against the one that works with VC2010 and reapply any substantive code changes, ignoring compiler directives, etc. Since it `SHOULD` (RFC7231) be possible to negotiate to no content-coding, the only reason for yt-dl to support `br` is that offering it might help to bypass fingerprinting, with the consequence that the actual encoding has to be supported (but `Accept-Encoding: br;q=0, gzip;q=1.0` ?). It would be enough to have a native Python decoder for this. The `compress` content-coding is in the RFCs but not seen in the wild. `gzip` and `br` compress faster/better. The benefit of the latter might only be significant for a giant corporation with a massive network infrastructure.

deekerman commented

github.com/google/brotli@acc265655d/python/bro.py (L4-L6)

Author

Owner

@Vangelis66 commented on GitHub (Sep 22, 2023):

... The following code was committed by Google on July 19th 2023 (a little more than two months ago):

How magnanimous of them, I say 👍 ; still, on Aug 31st, tag 1.1.0 shows up as being incompatible with py2.7 😞 ...

Probably the way to make the C compile is
to diff the latest Brotli against the one that works with VC2010
and reapply any substantive code changes,
ignoring compiler directives, etc.

Going back in time, I found that the last brotli code snapshot that will compile successfully with Visual Studio 2010 is google-brotli-1.0.9-51-git-20221222-g509d441:

(py3.4.10 x86 building environment)

python -m pip install "https://codeload.github.com/google/brotli/legacy.tar.gz/509d4419bd" => 

Collecting https://codeload.github.com/google/brotli/legacy.tar.gz/509d4419bd
  Downloading https://codeload.github.com/google/brotli/legacy.tar.gz/509d4419bd

     / 1.1MB 939kB/s
Building wheels for collected packages: Brotli
  Building wheel for Brotli (setup.py) ... done
  Stored in directory: C:\Users\<redacted>\AppData\Local\Temp\pip-ephem-wheel-cache-b009j45f\wheels\bb\80\3b\d13a072b35f01f199e1e2e4aa6d1a0df65909db7732a499fb1
Successfully built Brotli
Installing collected packages: Brotli
  Found existing installation: brotli 1.0.9
    Uninstalling brotli-1.0.9:
      Successfully uninstalled brotli-1.0.9
Successfully installed Brotli-1.0.9

The "breaking" change is google-brotli-1.0.9-52-git-20221229-gc8df4b3:

Python: use a new output buffer code

That one changed both files
python/_brotli.cc (also renamed to python/_brotli.c),
setup.py

Well, C/C++ is Greek to me (😜 ), but it looks like this "new output buffer code" was essentially a performance optimisation (and security patch?) feature for the upcoming 1.1.0 release, so I see no reason to revert it (then, it wouldn't be brotli-1.1.0, would it?); if only that new C code had been written in a syntax palatable to VC++ 10.0 😞 ; py3.5+, under Windows, use at minimum Visual Studio 2015 (VC++ 14.0) to compile C/C++ extensions, so the Google devs must have targeted such a VS version when authoring that "new" code (either unwittingly or intentionally) ...

@Vangelis66 commented on GitHub (Sep 22, 2023): ... The following code was committed by **Google** on July 19th 2023 (a little more than two months ago): https://github.com/google/brotli/blob/acc265655d6dde4ff1bce65308e9e152fbf3381a/python/bro.py#L4-L6 How magnanimous of them, I say 👍 ; still, on Aug 31st, tag `1.1.0` shows up as being incompatible with **py2.7** 😞 ... > Probably the way to make the C compile is > to diff the latest Brotli against the one that works with VC2010 > and reapply any substantive code changes, > ignoring compiler directives, etc. Going back in time, I found that the last `brotli` code snapshot that will compile successfully with **Visual Studio 2010** is `google-brotli-1.0.9-51-git-20221222-g509d441`: (py**3.4**.10 **x86** building environment) ```console python -m pip install "https://codeload.github.com/google/brotli/legacy.tar.gz/509d4419bd" => Collecting https://codeload.github.com/google/brotli/legacy.tar.gz/509d4419bd Downloading https://codeload.github.com/google/brotli/legacy.tar.gz/509d4419bd / 1.1MB 939kB/s Building wheels for collected packages: Brotli Building wheel for Brotli (setup.py) ... done Stored in directory: C:\Users\<redacted>\AppData\Local\Temp\pip-ephem-wheel-cache-b009j45f\wheels\bb\80\3b\d13a072b35f01f199e1e2e4aa6d1a0df65909db7732a499fb1 Successfully built Brotli Installing collected packages: Brotli Found existing installation: brotli 1.0.9 Uninstalling brotli-1.0.9: Successfully uninstalled brotli-1.0.9 Successfully installed Brotli-1.0.9 ``` The "breaking" change is `google-brotli-1.0.9-52-git-20221229-gc8df4b3`: [Python: use a new output buffer code](https://github.com/google/brotli/commit/c8df4b3049ff1283fc4525defbcd003188f88963) That one changed both files `python/_brotli.cc` (also renamed to `python/_brotli.c`), `setup.py` Well, **C/C++** is Greek to me (😜 ), but it looks like this "**new output buffer code**" was essentially a **performance optimisation** (and security patch?) feature for the upcoming `1.1.0` release, so I see no reason to revert it (then, it wouldn't be `brotli-1.1.0`, would it?); if only that new C code had been written **in a syntax palatable to VC++ 10.0** 😞 ; py3.**5**+, under Windows, use at minimum **Visual Studio 2015** (VC++ 14.0) to compile C/C++ extensions, so the **Google** devs must have targeted such a VS version when authoring that "new" code (either unwittingly or intentionally) ...

deekerman commented

Author

Owner

@dirkf commented on GitHub (Sep 23, 2023):

G changed the module from C++ to C. In principle this would have been a Good Thing, as the author hoped, but the inline keyword in C99+ wasn't supported in VC2005 and apparently not until after VC2012 (C++ inline support differed).

This fragment somewhere at the top of the file might help:

/* 1700 == VC2012 (https://dev.to/yumetodo/list-of-mscver-and-mscfullver-8nd) */
#if defined(_MSC_VER) && (_MSC_VER <= 1700) 
  #define inline __inline
#endif

Or modify setup.py to detect VC <= 2012 and add the equivalent of gcc's -Dinline=__inline for VC to the compilation parameters in that case.

@dirkf commented on GitHub (Sep 23, 2023): G changed the module from C++ to C. In principle this would have been a Good Thing, as the author hoped, but the `inline` keyword in C99+ wasn't supported in VC2005 and apparently not until after VC2012 (C++ `inline` support differed). This fragment somewhere at the top of the file might help: ```c /* 1700 == VC2012 (https://dev.to/yumetodo/list-of-mscver-and-mscfullver-8nd) */ #if defined(_MSC_VER) && (_MSC_VER <= 1700) #define inline __inline #endif ``` Or modify `setup.py` to detect VC <= 2012 and add the equivalent of `gcc`'s `-Dinline=__inline` for VC to the compilation parameters in that case.

deekerman commented

Author

Owner

@Vangelis66 commented on GitHub (Sep 24, 2023):

This fragment somewhere at the top of the file might help:

/* 1700 == VC2012 (https://dev.to/yumetodo/list-of-mscver-and-mscfullver-8nd) */
#if defined(_MSC_VER) && (_MSC_VER <= 1700) 
  #define inline __inline
#endif

Indebted for your continuous help on this ❤️ ; file _brotli.c was patched as you suggested, the inline-related error lines are now gone, but compilation still fails due to lots of syntax errors:

Failed attempt at building brotli-v1.0.9-52-gc8df4b3-mod under py3.4, on Win x86

python -m pip wheel "google-brotli-v1.0.9-52-gc8df4b3.tar.gz" => 

Processing c:\python3410-32\google-brotli-v1.0.9-52-gc8df4b3.tar.gz
Building wheels for collected packages: Brotli
  Building wheel for Brotli (setup.py) ... error
  ERROR: Complete output from command 'C:\Python3410-32\python.exe' -u -c 'import setuptools, tokenize;__file__='"'"'C:\\Users\\<redacted>\\AppData\\Local\\Temp\\pip-req-build-a3le9hya\\setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' bdist_wheel -d 'C:\Users\<redacted>\AppData\Local\Temp\pip-wheel-z7_iyuty':
  ERROR: running bdist_wheel
  running build
  running build_py
  creating bin
  creating bin\lib.win32-3.4
  copying python\brotli.py -> bin\lib.win32-3.4
  running build_ext
  building '_brotli' extension
  creating bin\temp.win32-3.4
  creating bin\temp.win32-3.4\Release
  creating bin\temp.win32-3.4\Release\python
  creating bin\temp.win32-3.4\Release\c
  creating bin\temp.win32-3.4\Release\c\common
  creating bin\temp.win32-3.4\Release\c\dec
  creating bin\temp.win32-3.4\Release\c\enc
  C:\Program Files\Microsoft Visual Studio 10.0\VC\BIN\cl.exe /c /nologo /Ox /MD /W3 /GS- /DNDEBUG -Ic/include -IC:\Python3410-32\include -IC:\Python3410-32\include /Tcpython/_brotli.c /Fobin\temp.win32-3.4\Release\python/_brotli.obj
  _brotli.c
  python/_brotli.c(229) : error C2143: syntax error : missing ';' before 'type'
  python/_brotli.c(230) : error C2065: 'mode_value' : undeclared identifier
  python/_brotli.c(234) : error C2065: 'mode_value' : undeclared identifier
  python/_brotli.c(296) : error C2059: syntax error : '.'
  python/_brotli.c(609) : error C2059: syntax error : '.'
  python/_brotli.c(861) : error C2059: syntax error : '.'
  python/_brotli.c(927) : error C2061: syntax error : identifier 'brotli_methods'
  python/_brotli.c(927) : error C2059: syntax error : ';'
  python/_brotli.c(927) : error C3409: empty attribute block is not allowed
  python/_brotli.c(927) : error C2513: '/*global*/ ' : no variable declared before '='
  python/_brotli.c(945) : error C2065: 'brotli_methods' : undeclared identifier
  python/_brotli.c(945) : error C2099: initializer is not a constant
  python/_brotli.c(983) : error C2143: syntax error : missing ';' before 'type'
  python/_brotli.c(984) : error C2275: 'uint32_t' : illegal use of this type as an expression
          C:\Program Files\Microsoft Visual Studio 10.0\VC\INCLUDE\stdint.h(23): see declaration of 'uint32_t'
  python/_brotli.c(984) : error C2146: syntax error : missing ';' before identifier 'decoderVersion'
  python/_brotli.c(984) : error C2065: 'decoderVersion' : undeclared identifier
  python/_brotli.c(985) : error C2065: 'version' : undeclared identifier
  python/_brotli.c(985) : warning C4047: 'function' : 'char *' differs in levels of indirection from 'int'
  python/_brotli.c(985) : warning C4024: '_snprintf' : different types for formal and actual parameter 1
  python/_brotli.c(985) : error C2065: 'version' : undeclared identifier
  python/_brotli.c(986) : error C2065: 'decoderVersion' : undeclared identifier
  python/_brotli.c(986) : error C2065: 'decoderVersion' : undeclared identifier
  python/_brotli.c(986) : error C2065: 'decoderVersion' : undeclared identifier
  python/_brotli.c(987) : error C2065: 'version' : undeclared identifier
  python/_brotli.c(987) : warning C4047: 'function' : 'const char *' differs in levels of indirection from 'int'
  python/_brotli.c(987) : warning C4024: 'PyModule_AddStringConstant' : different types for formal and actual parameter 3
  error: command 'C:\\Program Files\\Microsoft Visual Studio 10.0\\VC\\BIN\\cl.exe' failed with exit status 2
  ----------------------------------------
  ERROR: Failed building wheel for Brotli
  Running setup.py clean for Brotli
Failed to build Brotli
ERROR: Failed to build one or more wheels

If you or someone else can come up with more ideas, chime in - remember, I have to be spoon-fed in this case, as C/C++/Python are totally alien to me 😊 ...

FWIW, I'm not pursuing this anymore with determination 😉 ; the previous Brotli tag 1.0.9 still works as expected with yt-dl built on py3.4 (read more about the test case here):

Brotli TEST

youtube-dl -v "https://www.extra.cz/cauky-lidi-70-dil-babis-predstavil-pohadky-prymulanek-nebo-andrejovy-nove-saty-ac867" -o brotli_TEST.mp4 => 

[debug] System config: []
[debug] User config: []
[debug] Custom config: []
[debug] Command-line args: ['-v', 'https://www.extra.cz/cauky-lidi-70-dil-babis-predstavil-pohadky-prymulanek-nebo-andrejovy-nove-saty-ac867', '-o', 'brotli_TEST.mp4']
[debug] Encodings: locale cp1253, fs mbcs, out cp737, pref cp1253
[debug] youtube-dl version 2023.09.03.2215
[debug] Lazy loading extractors enabled
[debug] Single file build
[debug] Python 3.4.10 (CPython x86 32bit) - Windows-Vista-6.0.6003-SP2 - OpenSSL 1.0.2k  26 Jan 2017
[debug] exe versions: none
[debug] Proxy map: {}
[generic] cauky-lidi-70-dil-babis-predstavil-pohadky-prymulanek-nebo-andrejovy-nove-saty-ac867: Requesting header
WARNING: Could not send HEAD request to https://www.extra.cz/cauky-lidi-70-dil-babis-predstavil-pohadky-prymulanek-nebo-andrejovy-nove-saty-ac867: HTTP Error 405: Method Not Allowed
[generic] cauky-lidi-70-dil-babis-predstavil-pohadky-prymulanek-nebo-andrejovy-nove-saty-ac867: Downloading webpage
WARNING: Falling back on generic information extractor.
[generic] cauky-lidi-70-dil-babis-predstavil-pohadky-prymulanek-nebo-andrejovy-nove-saty-ac867: Extracting information
[debug] Default format spec: best/bestvideo+bestaudio
[debug] Invoking downloader on 'https://stream.eom.cz/storage-prod/videos/D1yA3uml-57xeMHzN.mp4'
[download] Destination: brotli_TEST.mp4
[download]   8.1% of 86.81MiB at 881.40KiB/s ETA 01:40

@Vangelis66 commented on GitHub (Sep 24, 2023): >This fragment somewhere at the top of the file might help: > > ```C > /* 1700 == VC2012 (https://dev.to/yumetodo/list-of-mscver-and-mscfullver-8nd) */ > #if defined(_MSC_VER) && (_MSC_VER <= 1700) > #define inline __inline > #endif > ``` Indebted for your continuous help on this ❤️ ; file `_brotli.c` was patched as you suggested, the `inline`-related error lines are now gone, but compilation still fails due to lots of `syntax error`s: <details> <summary>Failed attempt at building brotli-v1.0.9-52-gc8df4b3-mod under py3.4, on Win x86</summary> ```shell python -m pip wheel "google-brotli-v1.0.9-52-gc8df4b3.tar.gz" => Processing c:\python3410-32\google-brotli-v1.0.9-52-gc8df4b3.tar.gz Building wheels for collected packages: Brotli Building wheel for Brotli (setup.py) ... error ERROR: Complete output from command 'C:\Python3410-32\python.exe' -u -c 'import setuptools, tokenize;__file__='"'"'C:\\Users\\<redacted>\\AppData\\Local\\Temp\\pip-req-build-a3le9hya\\setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' bdist_wheel -d 'C:\Users\<redacted>\AppData\Local\Temp\pip-wheel-z7_iyuty': ERROR: running bdist_wheel running build running build_py creating bin creating bin\lib.win32-3.4 copying python\brotli.py -> bin\lib.win32-3.4 running build_ext building '_brotli' extension creating bin\temp.win32-3.4 creating bin\temp.win32-3.4\Release creating bin\temp.win32-3.4\Release\python creating bin\temp.win32-3.4\Release\c creating bin\temp.win32-3.4\Release\c\common creating bin\temp.win32-3.4\Release\c\dec creating bin\temp.win32-3.4\Release\c\enc C:\Program Files\Microsoft Visual Studio 10.0\VC\BIN\cl.exe /c /nologo /Ox /MD /W3 /GS- /DNDEBUG -Ic/include -IC:\Python3410-32\include -IC:\Python3410-32\include /Tcpython/_brotli.c /Fobin\temp.win32-3.4\Release\python/_brotli.obj _brotli.c python/_brotli.c(229) : error C2143: syntax error : missing ';' before 'type' python/_brotli.c(230) : error C2065: 'mode_value' : undeclared identifier python/_brotli.c(234) : error C2065: 'mode_value' : undeclared identifier python/_brotli.c(296) : error C2059: syntax error : '.' python/_brotli.c(609) : error C2059: syntax error : '.' python/_brotli.c(861) : error C2059: syntax error : '.' python/_brotli.c(927) : error C2061: syntax error : identifier 'brotli_methods' python/_brotli.c(927) : error C2059: syntax error : ';' python/_brotli.c(927) : error C3409: empty attribute block is not allowed python/_brotli.c(927) : error C2513: '/*global*/ ' : no variable declared before '=' python/_brotli.c(945) : error C2065: 'brotli_methods' : undeclared identifier python/_brotli.c(945) : error C2099: initializer is not a constant python/_brotli.c(983) : error C2143: syntax error : missing ';' before 'type' python/_brotli.c(984) : error C2275: 'uint32_t' : illegal use of this type as an expression C:\Program Files\Microsoft Visual Studio 10.0\VC\INCLUDE\stdint.h(23): see declaration of 'uint32_t' python/_brotli.c(984) : error C2146: syntax error : missing ';' before identifier 'decoderVersion' python/_brotli.c(984) : error C2065: 'decoderVersion' : undeclared identifier python/_brotli.c(985) : error C2065: 'version' : undeclared identifier python/_brotli.c(985) : warning C4047: 'function' : 'char *' differs in levels of indirection from 'int' python/_brotli.c(985) : warning C4024: '_snprintf' : different types for formal and actual parameter 1 python/_brotli.c(985) : error C2065: 'version' : undeclared identifier python/_brotli.c(986) : error C2065: 'decoderVersion' : undeclared identifier python/_brotli.c(986) : error C2065: 'decoderVersion' : undeclared identifier python/_brotli.c(986) : error C2065: 'decoderVersion' : undeclared identifier python/_brotli.c(987) : error C2065: 'version' : undeclared identifier python/_brotli.c(987) : warning C4047: 'function' : 'const char *' differs in levels of indirection from 'int' python/_brotli.c(987) : warning C4024: 'PyModule_AddStringConstant' : different types for formal and actual parameter 3 error: command 'C:\\Program Files\\Microsoft Visual Studio 10.0\\VC\\BIN\\cl.exe' failed with exit status 2 ---------------------------------------- ERROR: Failed building wheel for Brotli Running setup.py clean for Brotli Failed to build Brotli ERROR: Failed to build one or more wheels ``` </details> If you or someone else can come up with more ideas, chime in - remember, I have to be spoon-fed in this case, as C/C++/Python are totally alien to me 😊 ... FWIW, I'm not pursuing this anymore with determination 😉 ; the previous **Brotli** tag `1.0.9` still works as expected with `yt-dl` built on py3.**4** (read more about the test case [here](https://github.com/yt-dlp/yt-dlp/issues/5855)): <details> <summary>Brotli TEST</summary> ```console youtube-dl -v "https://www.extra.cz/cauky-lidi-70-dil-babis-predstavil-pohadky-prymulanek-nebo-andrejovy-nove-saty-ac867" -o brotli_TEST.mp4 => [debug] System config: [] [debug] User config: [] [debug] Custom config: [] [debug] Command-line args: ['-v', 'https://www.extra.cz/cauky-lidi-70-dil-babis-predstavil-pohadky-prymulanek-nebo-andrejovy-nove-saty-ac867', '-o', 'brotli_TEST.mp4'] [debug] Encodings: locale cp1253, fs mbcs, out cp737, pref cp1253 [debug] youtube-dl version 2023.09.03.2215 [debug] Lazy loading extractors enabled [debug] Single file build [debug] Python 3.4.10 (CPython x86 32bit) - Windows-Vista-6.0.6003-SP2 - OpenSSL 1.0.2k 26 Jan 2017 [debug] exe versions: none [debug] Proxy map: {} [generic] cauky-lidi-70-dil-babis-predstavil-pohadky-prymulanek-nebo-andrejovy-nove-saty-ac867: Requesting header WARNING: Could not send HEAD request to https://www.extra.cz/cauky-lidi-70-dil-babis-predstavil-pohadky-prymulanek-nebo-andrejovy-nove-saty-ac867: HTTP Error 405: Method Not Allowed [generic] cauky-lidi-70-dil-babis-predstavil-pohadky-prymulanek-nebo-andrejovy-nove-saty-ac867: Downloading webpage WARNING: Falling back on generic information extractor. [generic] cauky-lidi-70-dil-babis-predstavil-pohadky-prymulanek-nebo-andrejovy-nove-saty-ac867: Extracting information [debug] Default format spec: best/bestvideo+bestaudio [debug] Invoking downloader on 'https://stream.eom.cz/storage-prod/videos/D1yA3uml-57xeMHzN.mp4' [download] Destination: brotli_TEST.mp4 [download] 8.1% of 86.81MiB at 881.40KiB/s ETA 01:40 ``` </details>

deekerman commented

Author

Owner

@dirkf commented on GitHub (Sep 25, 2023):

Two further issues for C90:

all declarations must precede statements within a {block};
initializers can't use the index/member syntax {.list=NULL}

That should resolve everything except the problem at l.927 and the consequent errors at l.945:

python/_brotli.c(927) : error C2061: syntax error : identifier 'brotli_methods'

Maybe it's a weird side-effect of other errors, since all the syntax cases here are used elsewhere without error, except the expression METH_VARARGS | METH_KEYWORDS which surely should be valid.

This patch against the 1.1 code includes changes at l.920 and l.951 that would help if the compiler really couldn't use that expression, but should be omitted in a first test.

--- <unnamed>
+++ <unnamed>
@@ -1,3 +1,8 @@
+/* 1700 == VC2012 (https://dev.to/yumetodo/list-of-mscver-and-mscfullver-8nd) */
+#if defined(_MSC_VER) && (_MSC_VER <= 1700) 
+  #define inline __inline
+#endif
+
 #define PY_SSIZE_T_CLEAN 1
 #include <Python.h>
 #include <bytesobject.h>
@@ -24,6 +29,13 @@
     /* Number of whole allocated size. */
     Py_ssize_t allocated;
 } BlocksOutputBuffer;
+
+/* initialize a new BlocksOutputBuffer in C<99 */
+static inline BlocksOutputBuffer BlocksOutputBuffer_New() {
+      BlocksOutputBuffer buffer;
+      buffer.list = NULL;
+      return buffer;
+}
 
 static const char unable_allocate_msg[] = "Unable to allocate output buffer.";
 
@@ -219,19 +231,20 @@
   if (!PyInt_Check(o)) {
     PyErr_SetString(BrotliError, "Invalid mode");
     return 0;
-  }
-
-  int mode_value = -1;
-  if (!as_bounded_int(o, &mode_value, 0, 255)) {
-    PyErr_SetString(BrotliError, "Invalid mode");
-    return 0;
-  }
-  *mode = (BrotliEncoderMode) mode_value;
-  if (*mode != BROTLI_MODE_GENERIC &&
-      *mode != BROTLI_MODE_TEXT &&
-      *mode != BROTLI_MODE_FONT) {
-    PyErr_SetString(BrotliError, "Invalid mode");
-    return 0;
+  } else { /* declarations precede statements in C<99 */
+
+      int mode_value = -1;
+      if (!as_bounded_int(o, &mode_value, 0, 255)) {
+        PyErr_SetString(BrotliError, "Invalid mode");
+        return 0;
+      }
+      *mode = (BrotliEncoderMode) mode_value;
+      if (*mode != BROTLI_MODE_GENERIC &&
+          *mode != BROTLI_MODE_TEXT &&
+          *mode != BROTLI_MODE_FONT) {
+        PyErr_SetString(BrotliError, "Invalid mode");
+        return 0;
+      }
   }
 
   return 1;
@@ -288,7 +301,7 @@
 
   size_t available_out;
   uint8_t* next_out;
-  BlocksOutputBuffer buffer = {.list=NULL};
+  BlocksOutputBuffer buffer = BlocksOutputBuffer_New();
   PyObject *ret;
 
   if (BlocksOutputBuffer_InitAndGrow(&buffer, &available_out, &next_out) < 0) {
@@ -601,7 +614,7 @@
 
   size_t available_out;
   uint8_t* next_out;
-  BlocksOutputBuffer buffer = {.list=NULL};
+  BlocksOutputBuffer buffer = BlocksOutputBuffer_New();
   PyObject *ret;
 
   if (BlocksOutputBuffer_InitAndGrow(&buffer, &available_out, &next_out) < 0) {
@@ -853,7 +866,7 @@
 
   uint8_t* next_out;
   size_t available_out;
-  BlocksOutputBuffer buffer = {.list=NULL};
+  BlocksOutputBuffer buffer = BlocksOutputBuffer_New();
   PyObject *ret;
 
   static const char *kwlist[] = {"string", NULL};
@@ -920,7 +933,7 @@
 }
 
 static PyMethodDef brotli_methods[] = {
-  {"decompress", (PyCFunction)brotli_decompress, METH_VARARGS | METH_KEYWORDS, brotli_decompress__doc__},
+  {"decompress", (PyCFunction)brotli_decompress, METH_VARARGS /* | METH_KEYWORDS */, brotli_decompress__doc__},
   {NULL, NULL, 0, NULL}
 };
 
@@ -951,7 +964,9 @@
 #endif
 
 PyMODINIT_FUNC INIT_BROTLI(void) {
-  PyObject *m = CREATE_BROTLI;
+  PyObject *m;
+  brotli_methods[0].ml_flags |= METH_KEYWORDS
+  m = CREATE_BROTLI;
 
   BrotliError = PyErr_NewException((char*) "brotli.error", NULL, NULL);
   if (BrotliError != NULL) {
@@ -975,11 +990,13 @@
   PyModule_AddIntConstant(m, "MODE_TEXT", (int) BROTLI_MODE_TEXT);
   PyModule_AddIntConstant(m, "MODE_FONT", (int) BROTLI_MODE_FONT);
 
-  char version[16];
-  uint32_t decoderVersion = BrotliDecoderVersion();
-  snprintf(version, sizeof(version), "%d.%d.%d",
-      decoderVersion >> 24, (decoderVersion >> 12) & 0xFFF, decoderVersion & 0xFFF);
-  PyModule_AddStringConstant(m, "__version__", version);
+  do { /* declarations precede statements in C<99 */
+      char version[16];
+      uint32_t decoderVersion = BrotliDecoderVersion();
+      snprintf(version, sizeof(version), "%d.%d.%d",
+          decoderVersion >> 24, (decoderVersion >> 12) & 0xFFF, decoderVersion & 0xFFF);
+      PyModule_AddStringConstant(m, "__version__", version);
+  } while (0);
 
   RETURN_BROTLI;
 }

@dirkf commented on GitHub (Sep 25, 2023): Two further issues for C90: * all declarations must precede statements within a {block}; * initializers can't use the index/member syntax `{.list=NULL}` That should resolve everything except the problem at l.927 and the consequent errors at l.945: > python/_brotli.c(927) : error C2061: syntax error : identifier 'brotli_methods' Maybe it's a weird side-effect of other errors, since all the syntax cases here are used elsewhere without error, except the expression `METH_VARARGS | METH_KEYWORDS` which surely should be valid. <details><summary>This patch against the 1.1 code includes changes at l.920 and l.951 that would help if the compiler really couldn't use that expression, but should be omitted in a first test.</summary> ```diff --- <unnamed> +++ <unnamed> @@ -1,3 +1,8 @@ +/* 1700 == VC2012 (https://dev.to/yumetodo/list-of-mscver-and-mscfullver-8nd) */ +#if defined(_MSC_VER) && (_MSC_VER <= 1700) + #define inline __inline +#endif + #define PY_SSIZE_T_CLEAN 1 #include <Python.h> #include <bytesobject.h> @@ -24,6 +29,13 @@ /* Number of whole allocated size. */ Py_ssize_t allocated; } BlocksOutputBuffer; + +/* initialize a new BlocksOutputBuffer in C<99 */ +static inline BlocksOutputBuffer BlocksOutputBuffer_New() { + BlocksOutputBuffer buffer; + buffer.list = NULL; + return buffer; +} static const char unable_allocate_msg[] = "Unable to allocate output buffer."; @@ -219,19 +231,20 @@ if (!PyInt_Check(o)) { PyErr_SetString(BrotliError, "Invalid mode"); return 0; - } - - int mode_value = -1; - if (!as_bounded_int(o, &mode_value, 0, 255)) { - PyErr_SetString(BrotliError, "Invalid mode"); - return 0; - } - *mode = (BrotliEncoderMode) mode_value; - if (*mode != BROTLI_MODE_GENERIC && - *mode != BROTLI_MODE_TEXT && - *mode != BROTLI_MODE_FONT) { - PyErr_SetString(BrotliError, "Invalid mode"); - return 0; + } else { /* declarations precede statements in C<99 */ + + int mode_value = -1; + if (!as_bounded_int(o, &mode_value, 0, 255)) { + PyErr_SetString(BrotliError, "Invalid mode"); + return 0; + } + *mode = (BrotliEncoderMode) mode_value; + if (*mode != BROTLI_MODE_GENERIC && + *mode != BROTLI_MODE_TEXT && + *mode != BROTLI_MODE_FONT) { + PyErr_SetString(BrotliError, "Invalid mode"); + return 0; + } } return 1; @@ -288,7 +301,7 @@ size_t available_out; uint8_t* next_out; - BlocksOutputBuffer buffer = {.list=NULL}; + BlocksOutputBuffer buffer = BlocksOutputBuffer_New(); PyObject *ret; if (BlocksOutputBuffer_InitAndGrow(&buffer, &available_out, &next_out) < 0) { @@ -601,7 +614,7 @@ size_t available_out; uint8_t* next_out; - BlocksOutputBuffer buffer = {.list=NULL}; + BlocksOutputBuffer buffer = BlocksOutputBuffer_New(); PyObject *ret; if (BlocksOutputBuffer_InitAndGrow(&buffer, &available_out, &next_out) < 0) { @@ -853,7 +866,7 @@ uint8_t* next_out; size_t available_out; - BlocksOutputBuffer buffer = {.list=NULL}; + BlocksOutputBuffer buffer = BlocksOutputBuffer_New(); PyObject *ret; static const char *kwlist[] = {"string", NULL}; @@ -920,7 +933,7 @@ } static PyMethodDef brotli_methods[] = { - {"decompress", (PyCFunction)brotli_decompress, METH_VARARGS | METH_KEYWORDS, brotli_decompress__doc__}, + {"decompress", (PyCFunction)brotli_decompress, METH_VARARGS /* | METH_KEYWORDS */, brotli_decompress__doc__}, {NULL, NULL, 0, NULL} }; @@ -951,7 +964,9 @@ #endif PyMODINIT_FUNC INIT_BROTLI(void) { - PyObject *m = CREATE_BROTLI; + PyObject *m; + brotli_methods[0].ml_flags |= METH_KEYWORDS + m = CREATE_BROTLI; BrotliError = PyErr_NewException((char*) "brotli.error", NULL, NULL); if (BrotliError != NULL) { @@ -975,11 +990,13 @@ PyModule_AddIntConstant(m, "MODE_TEXT", (int) BROTLI_MODE_TEXT); PyModule_AddIntConstant(m, "MODE_FONT", (int) BROTLI_MODE_FONT); - char version[16]; - uint32_t decoderVersion = BrotliDecoderVersion(); - snprintf(version, sizeof(version), "%d.%d.%d", - decoderVersion >> 24, (decoderVersion >> 12) & 0xFFF, decoderVersion & 0xFFF); - PyModule_AddStringConstant(m, "__version__", version); + do { /* declarations precede statements in C<99 */ + char version[16]; + uint32_t decoderVersion = BrotliDecoderVersion(); + snprintf(version, sizeof(version), "%d.%d.%d", + decoderVersion >> 24, (decoderVersion >> 12) & 0xFFF, decoderVersion & 0xFFF); + PyModule_AddStringConstant(m, "__version__", version); + } while (0); RETURN_BROTLI; } ``` </details>

deekerman commented

Author

Owner

@Neustradamus commented on GitHub (Jan 22, 2024):

Dear @ytdl-org team,

I think it is time to create a new release build with all recent improvements.

The latest released "2021.12.17" from 2021-12-16, more two years and one month:

Thanks in advance.

@Neustradamus commented on GitHub (Jan 22, 2024): Dear @ytdl-org team, I think it is time to create a new release build with all recent improvements. The latest released "2021.12.17" from 2021-12-16, more two years and one month: - https://github.com/ytdl-org/youtube-dl/releases - https://github.com/ytdl-org/youtube-dl/tags Thanks in advance.

deekerman commented

Author

Owner

@kloczek commented on GitHub (Jun 7, 2024):

gentle ping .. any update? 🤔

@kloczek commented on GitHub (Jun 7, 2024): gentle ping .. any update? 🤔

deekerman commented

Author

Owner

@dirkf commented on GitHub (Jun 8, 2024):

Feel free to treat any recent nightly release as stable.

@dirkf commented on GitHub (Jun 8, 2024): Feel free to treat any recent nightly release as stable.

deekerman commented

Author

Owner

@rofl0r commented on GitHub (Jun 8, 2024):

where are those nightlies ? download page on website links to 2021 release.

@rofl0r commented on GitHub (Jun 8, 2024): where are those nightlies ? download page on website links to 2021 release.

deekerman commented

https://github.com/ytdl-org/youtube-dl/issues/31585#issuecomment-1617156161

Author

Owner

@dirkf commented on GitHub (Jun 8, 2024):

Thanks GitHub for hiding the posts:

@dirkf commented on GitHub (Jun 8, 2024): Thanks GitHub for hiding the posts: https://github.com/ytdl-org/youtube-dl/issues/31585#issuecomment-1617156161

deekerman commented

Author

Owner

@ReenigneArcher commented on GitHub (Jun 8, 2024):

Feel free to treat any recent nightly release as stable.

It would still be nice to get updated versions to PyPI. Or at a minimum create periodic tags in the repo. That way dependabot and/or renovate-bot can properly track this.

@ReenigneArcher commented on GitHub (Jun 8, 2024): > Feel free to treat any recent nightly release as stable. It would still be nice to get updated versions to PyPI. Or at a minimum create periodic tags in the repo. That way dependabot and/or renovate-bot can properly track this.

deekerman commented

Author

Owner

@rofl0r commented on GitHub (Jun 8, 2024):

https://github.com/ytdl-org/youtube-dl/issues/31585#issuecomment-1617156161

finally good news. will these tarballs stay available, or are they wiped whenever a new nightly is created ?

@rofl0r commented on GitHub (Jun 8, 2024): > https://github.com/ytdl-org/youtube-dl/issues/31585#issuecomment-1617156161 finally good news. will these tarballs stay available, or are they wiped whenever a new nightly is created ?

deekerman commented

Author

Owner

@kloczek commented on GitHub (Jun 8, 2024):

It would be really nice to have regular release with version git tag in git repo ..

@kloczek commented on GitHub (Jun 8, 2024): It would be really nice to have regular release with version git tag in git repo ..

deekerman commented

Author

Owner

@dirkf commented on GitHub (Jun 8, 2024):

A nightly release stays up unless some terrible bug is found that means downloading its assets would be a bad idea. In that sense they're all better than the 2021 release. Ofc you can still check out the bad tag and run the broken release if you want.

@dirkf commented on GitHub (Jun 8, 2024): A nightly release stays up unless some terrible bug is found that means downloading its assets would be a bad idea. In that sense they're all better than the 2021 release. Ofc you can still check out the bad tag and run the broken release if you want.

deekerman commented

Author

Owner

@rofl0r commented on GitHub (Jun 8, 2024):

A nightly release stays up unless some terrible bug is found that means downloading its assets would be a bad idea.

it's not about bugs but about a reproducible build. i'm maintaining a distro and a package build script contains an upstream URL for the source tarball, a checksum of the tarball, and a few script instructions to build and install it. that means if a nightly tarball is only available for half a week the distro package recipe is basically useless.

@rofl0r commented on GitHub (Jun 8, 2024): > A nightly release stays up unless some terrible bug is found that means downloading its assets would be a bad idea. it's not about bugs but about a reproducible build. i'm maintaining a distro and a package build script contains an upstream URL for the source tarball, a checksum of the tarball, and a few script instructions to build and install it. that means if a nightly tarball is only available for half a week the distro package recipe is basically useless.

deekerman commented

Author

Owner

@dirkf commented on GitHub (Jun 8, 2024):

Check out the nightly release pages. I think you'll see that what you want is available, so long as GitHub keeps hosting the pages. Obvs don't build in a brand-new release in case some anomaly ((c) Post Office Ltd) has got through the CI and local tests.

@dirkf commented on GitHub (Jun 8, 2024): Check out the nightly release pages. I think you'll see that what you want is available, so long as GitHub keeps hosting the pages. Obvs don't build in a brand-new release in case some anomaly ((c) Post Office Ltd) has got through the CI and local tests.

deekerman commented

Author

Owner

@rofl0r commented on GitHub (Jun 8, 2024):

yeah, looks good. the nightlies from 2023 are still available. i guess it'd be good to make a link on the homepage or make a "final" tag here where you refer to the nightlies repo, so everyone seeing the new "release" here gets the news.

@rofl0r commented on GitHub (Jun 8, 2024): yeah, looks good. the nightlies from 2023 are still available. i guess it'd be good to make a link on the homepage or make a "final" tag here where you refer to the nightlies repo, so everyone seeing the new "release" here gets the news.

deekerman commented

Author

Owner

@Mis012 commented on GitHub (Jul 14, 2024):

hm, couldn't find this by searching the issues...

It seems that some if not all of the issues mentioned are with making a binary release, but a new tag without any artifacts would be more than sufficient for having something for linux distros to upgrade to if they don't want to chase nightlies. People who use the prebuilts are presumably going to use nightlies anyway, they don't really care whether something is an official release.

@Mis012 commented on GitHub (Jul 14, 2024): hm, couldn't find this by searching the issues... It seems that some if not all of the issues mentioned are with making a binary release, but a new tag without any artifacts would be more than sufficient for having something for linux distros to upgrade to if they don't want to chase nightlies. People who use the prebuilts are presumably going to use nightlies anyway, they don't really care whether something is an official release.

deekerman commented

Author

Owner

@ddelange commented on GitHub (Jul 14, 2024):

easiest installation currently (does not require git) is to go:

pip install https://github.com/ytdl-org/youtube-dl/archive/refs/heads/master.zip

or to add

youtube-dl @ https://github.com/ytdl-org/youtube-dl/archive/refs/heads/master.zip

to your requirements.txt or your library requirements

@ddelange commented on GitHub (Jul 14, 2024): easiest installation currently (does not require `git`) is to go: ``` pip install https://github.com/ytdl-org/youtube-dl/archive/refs/heads/master.zip ``` or to add ``` youtube-dl @ https://github.com/ytdl-org/youtube-dl/archive/refs/heads/master.zip ``` to your requirements.txt or your library requirements

deekerman commented

Author

Owner

@Mis012 commented on GitHub (Jul 14, 2024):

I don't have an issue installing youtube-dl, it's just that the README says that "nothing can be done" when a distro has outdated package, but making a release tag is definitely something that has to be done if the distro only packages release tags

I'm pretty sure that at least in the single case of Tumbleweed, the reason the package is stuck on the 2021 version is that from their point of view there is no newer version.

as a Linux user, I obviously don't have issues with using git since I can just install git using the package manager ~~like god intended~~ (and I already have it installed anyway), obviously on windows installing git is kinda annoying though so is installing python fwiw

@Mis012 commented on GitHub (Jul 14, 2024): I don't have an issue installing youtube-dl, it's just that the README says that "nothing can be done" when a distro has outdated package, but making a release tag is definitely something that has to be done if the distro only packages release tags I'm pretty sure that at least in the single case of Tumbleweed, the reason the package is stuck on the 2021 version is that from their point of view there is no newer version. as a Linux user, I obviously don't have issues with using git since I can just install git using the package manager ~~like god intended~~ (and I already have it installed anyway), obviously on windows installing git is kinda annoying though so is installing python fwiw

deekerman commented

Author

Owner

@dirkf commented on GitHub (Jul 14, 2024):

There's nothing to stop anyone packaging a nightly build (it has tags, etc) but it'll immediately be obsolete because of web site turnover.

This is why the hassle of porting back the build process from the nightly repo hasn't seemed worthwhile.

... obviously on windows installing git is kinda annoying though so is installing python fwiw

Unlike the POSIX-ish single file bundled executable, the single file Windows build contains a Python executable and only needs the user to install ffmpeg for a good experience. The POSIX-ish targets are expected to have or be able to install a python program.

@dirkf commented on GitHub (Jul 14, 2024): There's nothing to stop anyone [packaging a nightly build](https://github.com/ytdl-org/youtube-dl/issues/31585#issuecomment-2156112550) (it has tags, etc) but it'll immediately be obsolete because of web site turnover. This is why the hassle of porting back the build process from the nightly repo hasn't seemed worthwhile. >... obviously on windows installing git is kinda annoying though so is installing python fwiw Unlike the POSIX-ish single file bundled executable, the single file Windows build contains a Python executable and only needs the user to install _ffmpeg_ for a good experience. The POSIX-ish targets are expected to have or be able to install a `python` program.

deekerman commented

Author

Owner

@ddelange commented on GitHub (Jul 15, 2024):

@dirkf if you set contents: write permissions, the github actions pipeline could also create releases and upload (binary) assets to them 🤔 for instance this step uploads to existing release, or creates a release (named after run_id) and uploads to it

@ddelange commented on GitHub (Jul 15, 2024): @dirkf if you set `contents: write` permissions, the github actions pipeline could also create releases and upload (binary) assets to them :thinking: for instance [this step](https://github.com/ddelange/psutil/blob/aa9fe3b071d1594611b7c94ff4544681f5b720c1/.github/workflows/build.yml#L86) uploads to existing release, or creates a release (named after `run_id`) and uploads to it

deekerman commented

Author

Owner

@dirkf commented on GitHub (Jul 15, 2024):

I'm not sure what problem that addresses. There is already an Actions workflow that creates nightly releases. A new release for every commit is not wanted. If you're suggesting cross-posting nightly releases, it's not just a matter of posting the release in the main repo: the releases are linked to the repo, so the nightly release updates from the nightly repo. It would be confusing for people to download a release and then have it update from a different source.

@dirkf commented on GitHub (Jul 15, 2024): I'm not sure what problem that addresses. There is already an Actions workflow that creates nightly releases. A new release for every commit is not wanted. If you're suggesting cross-posting nightly releases, it's not just a matter of posting the release in the main repo: the releases are linked to the repo, so the nightly release updates from the nightly repo. It would be confusing for people to download a release and then have it update from a different source.

deekerman commented

Author

Owner

@Mis012 commented on GitHub (Jul 15, 2024):

hm, indeed https://github.com/ytdl-org/ytdl-nightly/releases has source tarballs... I guess it's just not on the radar for some packagers :/

@Mis012 commented on GitHub (Jul 15, 2024): hm, indeed https://github.com/ytdl-org/ytdl-nightly/releases has source tarballs... I guess it's just not on the radar for some packagers :/

deekerman commented

Author

Owner

@Grub4K commented on GitHub (Jul 15, 2024):

If the nightly releases are the source of truth, this repository loses its value. It on the other hand also does not correlate to to a release being blocked on this repo. Nightly builds are being made and tested and using a workflow like that on this repository will require little changes. You can omit the duplication that does not match the actual development state since the two repositories are diverged. Instead, consider creating a prerelease on this repository, with future posibility of making it a regular release. Automation can be fine tuned to achieve this easily.

I will offer my help on this topic again: Feel free to contact me through email if you want any help or someone to discuss with in private

@Grub4K commented on GitHub (Jul 15, 2024): If the nightly releases are the source of truth, this repository loses its value. It on the other hand also does not correlate to to a release being blocked on this repo. Nightly builds are being made and tested and using a workflow like that on this repository will require little changes. You can omit the duplication that ***does not match the actual development state*** since the two repositories are diverged. Instead, consider creating a prerelease on this repository, with future posibility of making it a regular release. Automation can be fine tuned to achieve this easily. I will offer my help on this topic again: Feel free to contact me through [email](<mailto:contact@grub4k.xyz>) if you want any help or someone to discuss with in private

deekerman commented

Author

Owner

@dirkf commented on GitHub (Jul 15, 2024):

Technically not much would have to be done to pull the nightly workflow into the main repo, changing it to make pre-releases, but it would actually have to be done and with more detailed work, at least initially and especially when designating such a pre-release as a stable release. First off, the actions in the nightly workflow need to be brought up-to-date in GitHub's version hell actions ecosystem.

Otherwise I just suggest that, if a packager needs an up-to-date tagged commit to ingest into its workflow, the nightly repo is a good choice; a packager can install the tar.gz as if from the main repo but presumably would have to patch the build process if delivering a single-file build for which ytdl_is_updatable() would normally be true regardless of the source repo.

@dirkf commented on GitHub (Jul 15, 2024): Technically not much would have to be done to pull the nightly workflow into the main repo, changing it to make pre-releases, but it would actually have to be done and with more detailed work, at least initially and especially when designating such a pre-release as a stable release. First off, the actions in the nightly workflow need to be brought up-to-date in GitHub's version hell actions ecosystem. Otherwise I just suggest that, if a packager needs an up-to-date tagged commit to ingest into its workflow, the nightly repo is a good choice; a packager can install the tar.gz as if from the main repo but presumably would have to patch the build process if delivering a single-file build for which `ytdl_is_updatable()` would normally be true regardless of the source repo.

deekerman commented

Author

Owner

@seproDev commented on GitHub (Jul 20, 2024):

Replying to:

There have been releases since last year. — https://github.com/ytdl-org/youtube-dl/issues/32842#issuecomment-2241138911

No, nightlies tagged in a different repo are not releases. This is exactly what led to the current state:

Alpine: removed
Arch: removed
Brew: marked deprecated
Debian: installs yt-dlp instead
FreeBSD: removed
Gentoo: removed
NixOS: marked unmaintained
Ubuntu: installs yt-dlp instead

@seproDev commented on GitHub (Jul 20, 2024): Replying to: > There have been releases since last year. — https://github.com/ytdl-org/youtube-dl/issues/32842#issuecomment-2241138911 No, nightlies tagged in a different repo are not releases. This is exactly what led to the current state: - Alpine: [removed](https://gitlab.alpinelinux.org/alpine/aports/-/commit/6ffe7d0d310c249f1cd00f74a4a0b7aabeb95fe4) - Arch: [removed](https://gitlab.archlinux.org/archlinux/packaging/state/-/commit/d9639dd376d28c9627f2ee25887216f2bf66125d) - Brew: [marked deprecated](https://github.com/Homebrew/homebrew-core/commit/556b8ead35aa52c527a10d59f87e2b84ecd80bf1) - Debian: [installs yt-dlp instead](https://salsa.debian.org/debian/youtube-dl/-/commit/2a825917bfe302784b03b8531b9f80b4c45adae4) - FreeBSD: [removed](https://cgit.freebsd.org/ports/commit/www?id=336dff981d65c244979d7f8ceb8bcde06033420a) - Gentoo: [removed](https://gitweb.gentoo.org/repo/gentoo.git/commit/?id=a91f98e856723f42c8636180f2f3a5bf829ff899) - NixOS: [marked unmaintained](https://github.com/NixOS/nixpkgs/commit/9f82aef9b451e6ca6488a332ed407b98caf38129) - Ubuntu: [installs yt-dlp instead](https://packages.ubuntu.com/mantic/youtube-dl)

deekerman commented

Author

Owner

@bashonly commented on GitHub (Jul 20, 2024):

Most downstream packagers are not going to be OK with arbitrarily selecting a pre-release from a repo other than the actual upstream repo; some have policies that forbid doing this. E.g. https://github.com/ytdl-org/youtube-dl/issues/31585#issuecomment-1476323879

@bashonly commented on GitHub (Jul 20, 2024): Most downstream packagers are not going to be OK with arbitrarily selecting a pre-release from a repo other than the actual upstream repo; some have policies that forbid doing this. E.g. https://github.com/ytdl-org/youtube-dl/issues/31585#issuecomment-1476323879

deekerman commented

Author

Owner

@ReenigneArcher commented on GitHub (Jul 20, 2024):

Replying to:

There have been releases since last year. — #32842 (comment)

No, nightlies tagged in a different repo are not releases. This is exactly what led to the current state:

* Alpine: [removed](https://gitlab.alpinelinux.org/alpine/aports/-/commit/6ffe7d0d310c249f1cd00f74a4a0b7aabeb95fe4)

* Arch: [removed](https://gitlab.archlinux.org/archlinux/packaging/state/-/commit/d9639dd376d28c9627f2ee25887216f2bf66125d)

* Brew: [marked deprecated](https://github.com/Homebrew/homebrew-core/commit/556b8ead35aa52c527a10d59f87e2b84ecd80bf1)

* Debian: [installs yt-dlp instead](https://salsa.debian.org/debian/youtube-dl/-/commit/2a825917bfe302784b03b8531b9f80b4c45adae4)

* FreeBSD: [removed](https://cgit.freebsd.org/ports/commit/www?id=336dff981d65c244979d7f8ceb8bcde06033420a)

* Gentoo: [removed](https://gitweb.gentoo.org/repo/gentoo.git/commit/?id=a91f98e856723f42c8636180f2f3a5bf829ff899)

* NixOS: [marked unmaintained](https://github.com/NixOS/nixpkgs/commit/9f82aef9b451e6ca6488a332ed407b98caf38129)

* Ubuntu: [installs yt-dlp instead](https://packages.ubuntu.com/mantic/youtube-dl)

Also, Kodi is struggling a lot to keep this updated https://github.com/xbmc/repo-scripts/commits/matrix/script.module.youtube.dl

@ReenigneArcher commented on GitHub (Jul 20, 2024): > Replying to: > > > There have been releases since last year. — [#32842 (comment)](https://github.com/ytdl-org/youtube-dl/issues/32842#issuecomment-2241138911) > > No, nightlies tagged in a different repo are not releases. This is exactly what led to the current state: > > * Alpine: [removed](https://gitlab.alpinelinux.org/alpine/aports/-/commit/6ffe7d0d310c249f1cd00f74a4a0b7aabeb95fe4) > > * Arch: [removed](https://gitlab.archlinux.org/archlinux/packaging/state/-/commit/d9639dd376d28c9627f2ee25887216f2bf66125d) > > * Brew: [marked deprecated](https://github.com/Homebrew/homebrew-core/commit/556b8ead35aa52c527a10d59f87e2b84ecd80bf1) > > * Debian: [installs yt-dlp instead](https://salsa.debian.org/debian/youtube-dl/-/commit/2a825917bfe302784b03b8531b9f80b4c45adae4) > > * FreeBSD: [removed](https://cgit.freebsd.org/ports/commit/www?id=336dff981d65c244979d7f8ceb8bcde06033420a) > > * Gentoo: [removed](https://gitweb.gentoo.org/repo/gentoo.git/commit/?id=a91f98e856723f42c8636180f2f3a5bf829ff899) > > * NixOS: [marked unmaintained](https://github.com/NixOS/nixpkgs/commit/9f82aef9b451e6ca6488a332ed407b98caf38129) > > * Ubuntu: [installs yt-dlp instead](https://packages.ubuntu.com/mantic/youtube-dl) Also, Kodi is struggling a lot to keep this updated https://github.com/xbmc/repo-scripts/commits/matrix/script.module.youtube.dl

deekerman commented

Author

Owner

@dirkf commented on GitHub (Jul 20, 2024):

There are no pre-releases. A nightly release may be retrospectively marked pre-production and even have artefacts removed if it includes a significant bug that is fixed in a later release or by falling back to a previous one. But the proposal is to select a nightly release for packaging that is not brand new so as to avoid any possible lemon. As packagers have to tweak the updating and version code, it doesn't really matter whether they use the main repo or the nightly repo, since this is the only difference in the executable code.

The complaint linked above referred to the ytdl-patched/youtube-dl (superseded by the nightly build repo here). Unlike ytdl-patched/youtube-dl, the nightly build repo is clearly an adjunct of the development repo.

Maybe a nightly tag (s/t like 2024.07.11-nightly, on the understanding that this does not signify any sort of pre-release, despite SemVer) could be back-propagated to the equivalent main repo commit. After rebasing the equivalent commits have different SHAs (or am I gitting something wrong?): this might confuse someone who is not familiar with the code. The equivalent main repo commit is already shown in the --verbose output.

@dirkf commented on GitHub (Jul 20, 2024): There are no pre-releases. A nightly release may be retrospectively marked `pre-production` and even have artefacts removed if it includes a significant bug that is fixed in a later release or by falling back to a previous one. But the proposal is to select a nightly release for packaging that is not brand new so as to avoid any possible lemon. As packagers have to tweak the updating and version code, it doesn't really matter whether they use the main repo or the nightly repo, since this is the only difference in the executable code. The complaint [linked above](https://github.com/ytdl-org/youtube-dl/issues/31585#issuecomment-1476323879) referred to the ytdl-patched/youtube-dl (superseded by the nightly build repo here). Unlike ytdl-patched/youtube-dl, the nightly build repo is clearly an adjunct of the development repo. Maybe a nightly tag (s/t like `2024.07.11-nightly`, on the understanding that this does not signify any sort of pre-release, despite [SemVer](https://semver.org/)) could be back-propagated to the equivalent main repo commit. After rebasing the equivalent commits have different SHAs (or am I gitting something wrong?): this might confuse someone who is not familiar with the code. The equivalent main repo commit is already shown in the `--verbose` output.

deekerman commented

Author

Owner

@Grub4K commented on GitHub (Jul 20, 2024):

The complaints linked are about tagging a regular release in THIS repo. None other. The project is this repository, and like I said before, rebasing or adding a nightly commit means they are DISJOINTED, and can no longer be seen as the same project / state.

Creating a regular, stable git tag on this repository is trivial. Downstream packagers can do the remaining work. No binaries are required.

Using the nightly build workflow on this repository to create binaries is optional, albeit also trivial. Interestingly doing both on workflow_dispatch when state is deemed stable is already equivalent to a regular release.

Trying to reason against doing either is an interesting choice though

@Grub4K commented on GitHub (Jul 20, 2024): The complaints linked are about tagging a regular release in ***THIS*** repo. None other. The project is this repository, and like I said before, rebasing or adding a nightly commit means they are ***DISJOINTED***, and can no longer be seen as the same project / state. Creating a regular, stable git tag on this repository is trivial. Downstream packagers can do the remaining work. No binaries are required. Using the nightly build workflow on this repository to create binaries is optional, albeit also trivial. Interestingly doing both on `workflow_dispatch` when state is deemed stable is already equivalent to a regular release. Trying to reason against doing either is an interesting choice though

deekerman commented

Author

Owner

@dirkf commented on GitHub (Jul 21, 2024):

So there is now https://github.com/ytdl-org/youtube-dl/tag/2024.07.11-nightly and so also https://github.com/ytdl-org/youtube-dl/archive/refs/tags/2024.07.11-nightly.tar.gz. Let's see if that helps. I have avoided, once GH allowed me to delete it, having an actual release page here since it isn't one.

Otherwise, or anyway, I stand by the previous comments

the nightly repo is an "official" adjunct that would be perfectly adequate for packaging
Mike McQuaid was specifically (and not unreasonably) rejecting an "unofficial" repo.

... doing both on workflow_dispatch when state is deemed stable is already equivalent to a regular release.

For packagers? I don't see how that applies otherwise.

@dirkf commented on GitHub (Jul 21, 2024): So there is now https://github.com/ytdl-org/youtube-dl/tag/2024.07.11-nightly and so also https://github.com/ytdl-org/youtube-dl/archive/refs/tags/2024.07.11-nightly.tar.gz. Let's see if that helps. I have avoided, once GH allowed me to delete it, having an actual release page here since it isn't one. Otherwise, or anyway, I stand by the previous comments * the nightly repo is an "official" adjunct that would be perfectly adequate for packaging * Mike McQuaid was specifically (and not unreasonably) rejecting an "unofficial" repo. >... doing both on workflow_dispatch when state is deemed stable is already equivalent to a regular release. For packagers? I don't see how that applies otherwise.

deekerman commented

Author

Owner

@Grub4K commented on GitHub (Jul 23, 2024):

For packagers? I don't see how that applies otherwise.

Tried to clarify in #32876. Running that workflow when the repository is deemed stable will produce a stable release.

@Grub4K commented on GitHub (Jul 23, 2024): > For packagers? I don't see how that applies otherwise. Tried to clarify in #32876. Running that workflow when the repository is deemed stable will produce a stable release.

deekerman commented

Author

Owner

@hudsonm62 commented on GitHub (Aug 5, 2024):

Will the pypi packages be updated with any missed Nightly releases? Hasn't been updated since 2021.
I want to use a requirements file, kept updated via dependabot or renovate, but I obviously don't want the version from all the way back then

@hudsonm62 commented on GitHub (Aug 5, 2024): Will the [pypi packages](https://pypi.org/project/youtube_dl/#history) be updated with any missed Nightly releases? Hasn't been updated since 2021. I want to use a requirements file, kept updated via dependabot or renovate, but I obviously don't want the version from all the way back then

deekerman commented

Author

Owner

@ReenigneArcher commented on GitHub (Aug 5, 2024):

@hudsonm62 If you want @dependabot to keep it updated, you can do what I do. Add it as a submodule.

gitmodules file: https://github.com/LizardByte/Themerr-plex/blob/master/.gitmodules
the submodule: https://github.com/LizardByte/Themerr-plex/tree/master/third-party
dependabot config: github.com/LizardByte/Themerr-plex@c948caa739/.github/dependabot.yml (L43)
requirements file: github.com/LizardByte/Themerr-plex@c948caa739/requirements.txt (L16)

@ReenigneArcher commented on GitHub (Aug 5, 2024): @hudsonm62 If you want @dependabot to keep it updated, you can do what I do. Add it as a submodule. - gitmodules file: https://github.com/LizardByte/Themerr-plex/blob/master/.gitmodules - the submodule: https://github.com/LizardByte/Themerr-plex/tree/master/third-party - dependabot config: https://github.com/LizardByte/Themerr-plex/blob/c948caa73991e05cebe8b24b88bce5ccb6bede4c/.github/dependabot.yml#L43 - requirements file: https://github.com/LizardByte/Themerr-plex/blob/c948caa73991e05cebe8b24b88bce5ccb6bede4c/requirements.txt#L16

deekerman commented

Author

Owner

@hudsonm62 commented on GitHub (Aug 6, 2024):

ah good idea, thank you sir I shall look into this for my use case
totally forgot submodules existed tbh

@hudsonm62 commented on GitHub (Aug 6, 2024): ah good idea, thank you sir I shall look into this for my use case totally forgot submodules existed tbh

deekerman commented