starred/youtube-dl

Fork 0

mirror of https://github.com/ytdl-org/youtube-dl.git synced 2026-03-02 23:07:04 -05:00

--download archive on OSX 10.9.4 chokes on fnctl.flock #2947

New issue

Closed

opened 2026-02-20 23:11:36 -05:00 by deekerman · 3 comments

deekerman commented

2026-02-20 23:11:36 -05:00

Owner

Originally created by @jringloff on GitHub (Aug 30, 2014).

I am a total hack (I don't know python from lisp) but it appears that the locking method used by youtube-dl for the --download-archive option don't jive with OSX 10.9.4

imac:john(60:1060)$ youtube-dl -vicwn --download-archive archive "http://www.youtube.com/playlist id here/videos"
[debug] System config: []
[debug] User config: []
[debug] Command-line args: ['-vicwn', '--download-archive', 'archive', 'http://www.youtube.com/playlist id here/videos']
[debug] Encodings: locale UTF-8, fs utf-8, out UTF-8, pref UTF-8
[debug] youtube-dl version 2014.08.10
[debug] Python version 2.7.6 - Darwin-13.3.0-x86_64-i386-64bit
[debug] Proxy map: {}
[download] Downloading playlist: playlist id here
[youtube:user] playlist id here: Downloading video ids from 1 to 51
[youtube:user] playlist playlist id here: Downloading 4 videos
[download] Downloading video #1 of 4
ERROR: [Errno 45] Operation not supported
Traceback (most recent call last):
File "/Users/john/unix/bin/youtube-dl/youtube_dl/YoutubeDL.py", line 527, in extract_info
return self.process_ie_result(ie_result, download, extra_info)
File "/Users/john/unix/bin/youtube-dl/youtube_dl/YoutubeDL.py", line 638, in process_ie_result
reason = self._match_entry(entry)
File "/Users/john/unix/bin/youtube-dl/youtube_dl/YoutubeDL.py", line 484, in _match_entry
if self.in_download_archive(info_dict):
File "/Users/john/unix/bin/youtube-dl/youtube_dl/YoutubeDL.py", line 1123, in in_download_archive
with locked_file(fn, 'r', encoding='utf-8') as archive_file:
File "/Users/john/unix/bin/youtube-dl/youtube_dl/utils.py", line 1130, in enter
_lock_file(self.f, exclusive)
File "/Users/john/unix/bin/youtube-dl/youtube_dl/utils.py", line 1115, in _lock_file
fcntl.lockf(f, fcntl.LOCK_EX if exclusive else fcntl.LOCK_SH)
IOError: [Errno 45] Operation not supported

In utils.py changing from a fcntl.lockf to a flock seems to the trick:

from

else:
    import fcntl

    def _lock_file(f, exclusive):
        fcntl.lockf(f, fcntl.LOCK_EX if exclusive else fcntl.LOCK_SH)

    def _unlock_file(f):
        fcntl.lockf(f, fcntl.LOCK_UN)

else:
    import fcntl

    def _lock_file(f, exclusive):
        fcntl.flock(f, fcntl.LOCK_EX if exclusive else fcntl.LOCK_SH)

    def _unlock_file(f):
        fcntl.flock(f, fcntl.LOCK_UN)

Originally created by @jringloff on GitHub (Aug 30, 2014). I am a total hack (I don't know python from lisp) but it appears that the locking method used by youtube-dl for the --download-archive option don't jive with OSX 10.9.4 imac:john(60:1060)$ youtube-dl -vicwn --download-archive archive "http://www.youtube.com/playlist id here/videos" [debug] System config: [] [debug] User config: [] [debug] Command-line args: ['-vicwn', '--download-archive', 'archive', 'http://www.youtube.com/playlist id here/videos'] [debug] Encodings: locale UTF-8, fs utf-8, out UTF-8, pref UTF-8 [debug] youtube-dl version 2014.08.10 [debug] Python version 2.7.6 - Darwin-13.3.0-x86_64-i386-64bit [debug] Proxy map: {} [download] Downloading playlist: playlist id here [youtube:user] playlist id here: Downloading video ids from 1 to 51 [youtube:user] playlist playlist id here: Downloading 4 videos [download] Downloading video #1 of 4 ERROR: [Errno 45] Operation not supported Traceback (most recent call last): File "/Users/john/unix/bin/youtube-dl/youtube_dl/YoutubeDL.py", line 527, in extract_info return self.process_ie_result(ie_result, download, extra_info) File "/Users/john/unix/bin/youtube-dl/youtube_dl/YoutubeDL.py", line 638, in process_ie_result reason = self._match_entry(entry) File "/Users/john/unix/bin/youtube-dl/youtube_dl/YoutubeDL.py", line 484, in _match_entry if self.in_download_archive(info_dict): File "/Users/john/unix/bin/youtube-dl/youtube_dl/YoutubeDL.py", line 1123, in in_download_archive with locked_file(fn, 'r', encoding='utf-8') as archive_file: File "/Users/john/unix/bin/youtube-dl/youtube_dl/utils.py", line 1130, in __enter__ _lock_file(self.f, exclusive) File "/Users/john/unix/bin/youtube-dl/youtube_dl/utils.py", line 1115, in _lock_file fcntl.lockf(f, fcntl.LOCK_EX if exclusive else fcntl.LOCK_SH) IOError: [Errno 45] Operation not supported In utils.py changing from a fcntl.lockf to a flock seems to the trick: from ``` else: import fcntl def _lock_file(f, exclusive): fcntl.lockf(f, fcntl.LOCK_EX if exclusive else fcntl.LOCK_SH) def _unlock_file(f): fcntl.lockf(f, fcntl.LOCK_UN) ``` to ``` else: import fcntl def _lock_file(f, exclusive): fcntl.flock(f, fcntl.LOCK_EX if exclusive else fcntl.LOCK_SH) def _unlock_file(f): fcntl.flock(f, fcntl.LOCK_UN) ```

deekerman closed this issue

2026-02-20 23:11:37 -05:00

deekerman commented

2026-02-20 23:11:38 -05:00

Author

Owner

@jringloff commented on GitHub (Aug 30, 2014):

Of course this might break some other non-windows flavors. so I am sure more logic needs to be applied.

@jringloff commented on GitHub (Aug 30, 2014): Of course this might break some other non-windows flavors. so I am sure more logic needs to be applied.

deekerman commented

2026-02-20 23:11:40 -05:00

Author

Owner

@phihag commented on GitHub (Aug 31, 2014):

Thank you very much for the detailed report!

I don't know OSX development, but this seems weird since lockf is actually documented on OSX. But I'm not emotionally attached to it, and flock should emulate where it's not available anyways, so I've just switch to flock.

@phihag commented on GitHub (Aug 31, 2014): Thank you very much for the detailed report! I don't know OSX development, but this seems weird since `lockf` is actually [documented on OSX](https://developer.apple.com/library/mac/documentation/Darwin/Reference/Manpages/man3/lockf.3.html). But I'm not emotionally attached to it, and `flock` should emulate where it's not available anyways, so I've just switch to `flock`.

deekerman commented

2026-02-20 23:11:40 -05:00

Author

Owner

@jringloff commented on GitHub (Sep 1, 2014):

I could be missing a library or have something broken. But I thought I would bug it for validation. I will also update my mac ports and see if that has an impact using the version I was using. lockf seems the better option, so while flock may be a fix, it may not be the best fix, particularly if it turns out to be environmental (just my system).

@jringloff commented on GitHub (Sep 1, 2014): I could be missing a library or have something broken. But I thought I would bug it for validation. I will also update my mac ports and see if that has an impact using the version I was using. lockf seems the better option, so while flock may be a fix, it may not be the best fix, particularly if it turns out to be environmental (just my system).

deekerman referenced this issue

2026-02-21 03:38:17 -05:00

Changes to NBC News site #9671

No milestone

No project

No assignees

1 participant

Notifications

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference

starred/youtube-dl#2947

No description provided.

Rows
Columns