The first and last parts of large videos are not forced to download bad enough #1968

Closed
opened 2026-02-21 15:57:16 -05:00 by deekerman · 18 comments
Owner

Originally created by @aplanx on GitHub (Dec 23, 2014).

Sequential downloading. This is the key, not the future, the present of torrent file sharing. This is the function that makes qBittorrent special, and worth downloading.

This function demolish the need of paying streaming sites for HD contents if we want to watch movies, and want to do it right now.

Unfortunately, the thrilling it is when 1-2 GB videos were generally shared, the awful it becomes when it comes to larger 4-5 GB or even larger files.

In case the first and last part of a media file is not retrieved, streaming is not possible, even if half of the movie is already, sequentially downloaded. And when it comes to large files, this is exactly what happens.

I'm positive that the meaning of "firsts and last parts" are defined by a simple value, say the first and last 10 megabytes, should either be expressed by percentage or forcing the swarm to provide that very first and last part we need to actually start streaming.

Please, somebody, please either change the first value to a higher number – just to make sure – or a percentage value or code something better that won't let the swarm upload any other parts of a file but the so-called "first and last".

Thank you. Seriously.

Originally created by @aplanx on GitHub (Dec 23, 2014). Sequential downloading. This is the key, not the future, the present of torrent file sharing. This is the function that makes qBittorrent special, and worth downloading. This function demolish the need of paying streaming sites for HD contents if we want to watch movies, and want to do it right now. Unfortunately, the thrilling it is when 1-2 GB videos were generally shared, the awful it becomes when it comes to larger 4-5 GB or even larger files. In case the first and last part of a media file is not retrieved, streaming is not possible, even if half of the movie is already, sequentially downloaded. And when it comes to large files, this is exactly what happens. I'm positive that the meaning of "firsts and last parts" are defined by a simple value, say the first and last 10 megabytes, should either be expressed by percentage or forcing the swarm to provide that very first and last part we need to actually start streaming. Please, somebody, please either change the first value to a higher number – just to make sure – or a percentage value or code something better that won't let the swarm upload any other parts of a file but the so-called "first and last". Thank you. Seriously.
Author
Owner

@aplanx commented on GitHub (Dec 23, 2014):

I mean, there are over half a thousand issues defined on this forum, but only THIS one worth a second to worry about. This function makes qBittorrent outstanding, and if this is not working, qBittorrent simply lost all its charm. First this, anything else later

Thank you.

@aplanx commented on GitHub (Dec 23, 2014): I mean, there are over half a thousand issues defined on this forum, but only THIS one worth a second to worry about. This function makes qBittorrent outstanding, and if this is not working, qBittorrent simply lost all its charm. First this, anything else later Thank you.
Author
Owner

@chrishirst commented on GitHub (Dec 23, 2014):

I'm positive that the meaning of "firsts and last parts" are defined by a simple value, say the first and last 10 megabytes,

Nope! "first and last" means piece 0 and piece n-1 where n is the number of pieces in the torrent, and the size in bytes of each piece is determined at creation time according to the size of the payload.

https://wiki.vuze.com/w/Torrent_Piece_Size
http://torrentfreak.com/how-to-make-the-best-torrents-081121/
https://wiki.theory.org/BitTorrentSpecification

And the place for topics such as this, is the support forum at http://forum.qbittorrent.org/

@chrishirst commented on GitHub (Dec 23, 2014): > I'm positive that the meaning of "firsts and last parts" are defined by a simple value, say the first and last 10 megabytes, Nope! "first and last" means piece 0 and piece n-1 where n is the number of pieces in the torrent, and the size in bytes of each piece is determined at creation time according to the size of the payload. https://wiki.vuze.com/w/Torrent_Piece_Size http://torrentfreak.com/how-to-make-the-best-torrents-081121/ https://wiki.theory.org/BitTorrentSpecification And the place for topics such as this, is the support forum at http://forum.qbittorrent.org/
Author
Owner

@Belove0 commented on GitHub (Dec 23, 2014):

Hi @aplanx, if I understand you correctly -- just as to your general use case -- you want to use qBittorrent to stream media files from a torrent swarm?

If so, that is not a feature qBittorrent has or has had. It would require a feature to download files in piece order (after last pieces if needed), but there is no option for that.

  • Download in Sequential Order downloads or attempts to download one file at a time, in the order they are in the torrent, as you may know.
    • Within each file, the pieces are downloaded in a non-sequential order, so there will usually be lots of gaps, and you can't expect a media file to play back correctly until that specific file has downloaded completely.
    • For a torrent with many files though, it enables you to open or play files one at a time after each has completed.
  • Download first and last piece first, as you probably know, has the purpose of downloading portions of files that commonly contain identifying metadata first -- the beginning and end.
    • In the case of media files this could include information about the codecs used and an index for seeking within the file, for example.
    • The main qBittorrent developer, @sledgehammer999, has previously commented with an apparent interest in improving this feature to have it better estimate the amount of data needed for different file types, and is seeking feedback in that regard. See https://github.com/qbittorrent/qBittorrent/issues/1813#issuecomment-48115417.
@Belove0 commented on GitHub (Dec 23, 2014): Hi @aplanx, if I understand you correctly -- just as to your general use case -- you want to use qBittorrent to stream media files from a torrent swarm? If so, that is not a feature qBittorrent has or has had. It would require a feature to download files in **piece** order (after last pieces if needed), but there is no option for that. - Download in Sequential Order downloads or attempts to download one file at a time, in the order they are in the torrent, as you may know. - Within each file, the pieces are downloaded in a non-sequential order, so there will usually be lots of gaps, and you can't expect a media file to play back correctly until that specific file has downloaded completely. - For a torrent with many files though, it enables you to open or play files one at a time after each has completed. - Download first and last piece first, as you probably know, has the purpose of downloading portions of files that commonly contain identifying metadata first -- the beginning and end. - In the case of media files this could include information about the codecs used and an index for seeking within the file, for example. - The main qBittorrent developer, @sledgehammer999, has previously commented with an apparent interest in improving this feature to have it better estimate the amount of data needed for different file types, and is seeking feedback in that regard. See https://github.com/qbittorrent/qBittorrent/issues/1813#issuecomment-48115417.
Author
Owner

@aplanx commented on GitHub (Dec 24, 2014):

@AsaRossoff no, buddy, give it a go, and you'll see that it works exactly as you described not.

Download in sequential order exactly means that the pieces within each separate files are downloaded in sequential order.

If you download an .avi, for example, and hit on "Download first and last piece first" and "Download in sequential order" the metadata necessary to start playing in a media player, for example VLC, will download first, and you CAN stream torrent videos, music, you name it.

Please, do take a look!

@aplanx commented on GitHub (Dec 24, 2014): @AsaRossoff no, buddy, give it a go, and you'll see that it works exactly as you described not. Download in sequential order exactly means that the pieces within each separate files are downloaded in sequential order. If you download an .avi, for example, and hit on "Download first and last piece first" and "Download in sequential order" the metadata necessary to start playing in a media player, for example VLC, will download first, and you CAN stream torrent videos, music, you name it. Please, do take a look!
Author
Owner

@chrishirst commented on GitHub (Dec 24, 2014):

"First and last" & "sequential order" ARE mutually exclusive terms.

@chrishirst commented on GitHub (Dec 24, 2014): "First and last" & "sequential order" ARE mutually exclusive terms.
Author
Owner

@chrishirst commented on GitHub (Dec 24, 2014):

For aplanx;

Sequential downloading. This is the key, not the future, the present of torrent file sharing.

You are incorrect on that, "sequential downloading" is the anathema of a good bittorrent swarm

@chrishirst commented on GitHub (Dec 24, 2014): For aplanx; > Sequential downloading. This is the key, not the future, the present of torrent file sharing. You are incorrect on that, "sequential downloading" is the anathema of a good bittorrent swarm
Author
Owner

@aplanx commented on GitHub (Dec 24, 2014):

yes, they are if you are a computer – otherwise you can make them work, for e.g. by starting with the first command and executing the second one afterwards.

@aplanx commented on GitHub (Dec 24, 2014): yes, they are if you are a computer – otherwise you can make them work, for e.g. by starting with the first command and executing the second one afterwards.
Author
Owner

@chrishirst commented on GitHub (Dec 24, 2014):

yes, they are if you are a computer

What???????

@chrishirst commented on GitHub (Dec 24, 2014): > yes, they are if you are a computer What???????
Author
Owner

@aplanx commented on GitHub (Dec 24, 2014):

if you order a computer do both, a computer won't solve the problem if you don't detail how to actually do it. a human being should be capable of solving the two "mutually exclusive" commands...

@aplanx commented on GitHub (Dec 24, 2014): if you order a computer do both, a computer won't solve the problem if you don't detail how to actually do it. a human being should be capable of solving the two "mutually exclusive" commands...
Author
Owner

@pmzqla commented on GitHub (Dec 24, 2014):

I should look again at the code to be sure of this, but I remember that qBittorrent takes care of downloading the first and the last piece of previewable files when sequential download is enabled.

@pmzqla commented on GitHub (Dec 24, 2014): I should look again at the code to be sure of this, but I remember that qBittorrent takes care of downloading the first and the last piece of previewable files when sequential download is enabled.
Author
Owner

@aplanx commented on GitHub (Dec 24, 2014):

@pmzqla yes, it generally does, of course, but sometimes it happens that other parts are requested – they light up green and soon get blue – or what's worse, the first and last piece is probably not large enough to cover the pieces of the meta data of the file. And this leads to the problem you can't preview those files.

There's a slight chance that these might be of help:

I thought of downloading the first and last 5 to 10 percent of all pieces or creating an option for downloading in reversed sequential order would be a good guess. (All other related functions are launched manually, any ways, so this might actually work.)

Hope this can help, and you can do something about it! 👍

@aplanx commented on GitHub (Dec 24, 2014): @pmzqla yes, it generally does, of course, but sometimes it happens that other parts are requested – they light up green and soon get blue – or what's worse, the first and last piece is probably not large enough to cover the pieces of the meta data of the file. And this leads to the problem you can't preview those files. There's a slight chance that these might be of help: I thought of downloading the first and last 5 to 10 percent of all pieces or creating an option for downloading in reversed sequential order would be a good guess. (All other related functions are launched manually, any ways, so this might actually work.) Hope this can help, and you can do something about it! :+1:
Author
Owner

@pmzqla commented on GitHub (Dec 24, 2014):

I'm positive that the meaning of "firsts and last parts" are defined by a simple value, say the first and last 10 megabytes, should either be expressed by percentage or forcing the swarm to provide that very first and last part we need to actually start streaming.

When the download "first and last piece" option is enabled, qBittorrent gives a high priority to the first and last piece of a file. Each piece has a fixed size determined when the torrent is created and that's what determine how many bytes are downloaded when the said option is activated.

Since pieces can be shared between two contiguous files, it could happen that the last piece or the first piece of a file doesn't contain all the information needed for the preview. I don't know how much is needed to preview files, but a simple solution to this problem would be downloading the first two pieces and the last two pieces of a file.

@pmzqla commented on GitHub (Dec 24, 2014): > I'm positive that the meaning of "firsts and last parts" are defined by a simple value, say the first and last 10 megabytes, should either be expressed by percentage or forcing the swarm to provide that very first and last part we need to actually start streaming. When the download "first and last piece" option is enabled, qBittorrent gives a high priority to the first and last piece of a file. Each piece has a fixed size determined when the torrent is created and that's what determine how many bytes are downloaded when the said option is activated. Since pieces can be shared between two contiguous files, it could happen that the last piece or the first piece of a file doesn't contain all the information needed for the preview. I don't know how much is needed to preview files, but a simple solution to this problem would be downloading the first two pieces and the last two pieces of a file.
Author
Owner

@pmzqla commented on GitHub (Dec 24, 2014):

I thought of downloading the first and last 5 to 10 percent of all pieces or creating an option for downloading in reversed sequential order would be a good guess.

Sequential download is a feature of libtorrent, qBittorrent simply tells libtorrent when to do it. Although possible (I think), I wouldn't implement the reverse sequential download in qBittorrent directly.


I've just tried to download some big torrents and for only one of them I had issues with the preview. It wasn't the biggest one (4,8 GiB, some others were 10GiB big) and the piece size was 8MiB. Aside from the mp4, there was a 58B txt.
Forcing the download of the last two pieces was enough to preview it.

Given that I had troubles with just one of the torrents I tried (they were not many though), I'd say that forcing the download of two pieces at the end of the file should cover almost all the cases, but doing more tests would be better.

@sledgehammer999
What about something like this?

diff --git a/src/qtlibtorrent/qtorrenthandle.cpp b/src/qtlibtorrent/qtorrenthandle.cpp
index 4c06b68..ecba313 100644
--- a/src/qtlibtorrent/qtorrenthandle.cpp
+++ b/src/qtlibtorrent/qtorrenthandle.cpp
@@ -716,6 +716,12 @@ void QTorrentHandle::prioritize_first_last_piece(int file_index, bool b) const
     QPair<int, int> extremities = get_file_extremity_pieces(*tf, file_index);
     piece_priority(extremities.first, prio);
     piece_priority(extremities.second, prio);
+    if (extremities.first != extremities.second) {
+        // In some cases, the first piece and the last piece are
+        // not enough for the preview.
+        piece_priority(extremities.first + 1, prio);
+        piece_priority(extremities.second - 1, prio);
+    }
 }

 void QTorrentHandle::prioritize_first_last_piece(bool b) const
@pmzqla commented on GitHub (Dec 24, 2014): > I thought of downloading the first and last 5 to 10 percent of all pieces or creating an option for downloading in reversed sequential order would be a good guess. Sequential download is a feature of libtorrent, qBittorrent simply tells libtorrent when to do it. Although possible (I think), I wouldn't implement the reverse sequential download in qBittorrent directly. <br> I've just tried to download some big torrents and for only one of them I had issues with the preview. It wasn't the biggest one (4,8 GiB, some others were 10GiB big) and the piece size was 8MiB. Aside from the mp4, there was a 58B txt. Forcing the download of the last two pieces was enough to preview it. Given that I had troubles with just one of the torrents I tried (they were not many though), I'd say that forcing the download of two pieces at the end of the file should cover almost all the cases, but doing more tests would be better. @sledgehammer999 What about something like this? ``` diff diff --git a/src/qtlibtorrent/qtorrenthandle.cpp b/src/qtlibtorrent/qtorrenthandle.cpp index 4c06b68..ecba313 100644 --- a/src/qtlibtorrent/qtorrenthandle.cpp +++ b/src/qtlibtorrent/qtorrenthandle.cpp @@ -716,6 +716,12 @@ void QTorrentHandle::prioritize_first_last_piece(int file_index, bool b) const QPair<int, int> extremities = get_file_extremity_pieces(*tf, file_index); piece_priority(extremities.first, prio); piece_priority(extremities.second, prio); + if (extremities.first != extremities.second) { + // In some cases, the first piece and the last piece are + // not enough for the preview. + piece_priority(extremities.first + 1, prio); + piece_priority(extremities.second - 1, prio); + } } void QTorrentHandle::prioritize_first_last_piece(bool b) const ```
Author
Owner

@aplanx commented on GitHub (Dec 24, 2014):

@pmzqla

Sir, I'm in support of any improvements to the matter! And, in my personal opinion, doubling the number of the last pieces downloaded may significantly increase the chance that you get the meta data! 👍

@aplanx commented on GitHub (Dec 24, 2014): @pmzqla Sir, I'm in support of any improvements to the matter! And, in my personal opinion, doubling the number of the last pieces downloaded may significantly increase the chance that you get the meta data! :+1:
Author
Owner

@sledgehammer999 commented on GitHub (Dec 26, 2014):

I link to my (old) comment: https://github.com/qbittorrent/qBittorrent/issues/1813#issuecomment-48115417
If none of you has any idea on how to answer, then I suppose setting the first 2 pieces and the last 2 pieces to download is pretty reasonable.
Also "download in sequential order" seems to interfere with "download first/last piece" even though we try to instruct libtorrent otherwise in the code. This probably will be fixed for good when #182 is implemented.

@sledgehammer999 commented on GitHub (Dec 26, 2014): I link to my (old) comment: https://github.com/qbittorrent/qBittorrent/issues/1813#issuecomment-48115417 If none of you has any idea on how to answer, then I suppose setting the first 2 pieces and the last 2 pieces to download is pretty reasonable. Also "download in sequential order" seems to interfere with "download first/last piece" even though we try to instruct libtorrent otherwise in the code. This probably will be fixed for good when #182 is implemented.
Author
Owner

@pmzqla commented on GitHub (Dec 26, 2014):

I think using a size treshold as implemented here is better.

Here the analysis of three mp4s. The last one is the one I couldn't preview. Sizes are in bytes.

File size Metadata size Metadata offset
1 397 530 313 2 995 298 24
1 031 830 439 4 635 322 24
5 166 376 123 11 638 990 5 154 737 161

So yeah, metadata can be pretty big.

@pmzqla commented on GitHub (Dec 26, 2014): I think using a size treshold as implemented <a href="https://github.com/qbittorrent/qBittorrent/pull/189">here</a> is better. Here the analysis of three mp4s. The last one is the one I couldn't preview. Sizes are in bytes. | File size | Metadata size | Metadata offset | | --- | --- | --- | | 1 397 530 313 | 2 995 298 | 24 | | 1 031 830 439 | 4 635 322 | 24 | | 5 166 376 123 | 11 638 990 | 5 154 737 161 | So yeah, metadata can be pretty big.
Author
Owner

@ngosang commented on GitHub (Apr 24, 2016):

Already implemented in #3816.
@sledgehammer999 close.

@ngosang commented on GitHub (Apr 24, 2016): Already implemented in #3816. @sledgehammer999 close.
Author
Owner

@thalieht commented on GitHub (Feb 13, 2017):

@ngosang close

@thalieht commented on GitHub (Feb 13, 2017): @ngosang close
Sign in to join this conversation.
No milestone
No project
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
starred/qBittorrent#1968
No description provided.