qbt with lt 1.1 has worse torrent rechecking/creating speed on Windows #7391

Author

Owner

@Seeker2 commented on GitHub (Jun 7, 2018):

There may have been a change to make force rechecking less "hard" on drives and CPUs.

@Seeker2 commented on GitHub (Jun 7, 2018): There may have been a change to make force rechecking less "hard" on drives and CPUs.

deekerman commented

Author

Owner

@Chocobo1 commented on GitHub (Jun 7, 2018):

@arvidn
Ping, you might want to see this.

@Chocobo1 commented on GitHub (Jun 7, 2018): @arvidn Ping, you might want to see this.

deekerman commented

Author

Owner

@arvidn commented on GitHub (Jun 7, 2018):

@airium what is the percent figure in the table?

My first guess at the factor dominating this throughput is the settings_pack::checking_mem_usage configuration option. This essentially caps the buffer size of pieces in-flight, being checked. It's specified in blocks of 16kiB, and defaults to 256 blocks (i.e. 4 MiB). If the bandwidth delay product exceeds this, you will experience degraded performance. In libtorrent 1.1.x, this delay includes the communication between the main thread and the disk thread(s), which likely explains the degraded performance; the bandwidth delay product is larger.

Perhaps the default should be 8 MiB. @airium would you mind doubling or quadrupling this setting and see how it affects the checking throughput?

@arvidn commented on GitHub (Jun 7, 2018): @airium what is the percent figure in the table? My first guess at the factor dominating this throughput is the `settings_pack::checking_mem_usage` configuration option. This essentially caps the buffer size of pieces in-flight, being checked. It's specified in blocks of 16kiB, and defaults to 256 blocks (i.e. 4 MiB). If the bandwidth delay product exceeds this, you will experience degraded performance. In libtorrent 1.1.x, this delay includes the communication between the main thread and the disk thread(s), which likely explains the degraded performance; the bandwidth delay product is larger. Perhaps the default should be 8 MiB. @airium would you mind doubling or quadrupling this setting and see how it affects the checking throughput?

deekerman commented

Author

Owner

@airium commented on GitHub (Jun 7, 2018):

@arvidn

what is the percent figure in the table?

It is simply the disk utilisation percentage shown in Task Manager. I just logged it but this value might not really indicate something. Empirically, a 100% does not necessarily say the disk throughput has been exhausted.

would you mind doubling or quadrupling this setting and see how it affects the checking throughput?

Of course, I will upload the result maybe later today.

@airium commented on GitHub (Jun 7, 2018): @arvidn > what is the percent figure in the table? It is simply the disk utilisation percentage shown in Task Manager. I just logged it but this value might not really indicate something. Empirically, a 100% does not necessarily say the disk throughput has been exhausted. ![tim 20180607100954](https://user-images.githubusercontent.com/38249940/41089987-11479aaa-6a3b-11e8-8719-1aea9529edc3.jpg) > would you mind doubling or quadrupling this setting and see how it affects the checking throughput? Of course, I will upload the result maybe later today.

deekerman commented

Author

Owner

@airium commented on GitHub (Jun 9, 2018):

@arvidn Sorry for my delay. Here is the result I put on Google Sheet.
https://docs.google.com/spreadsheets/d/1q5eL3j4HVGU_wZel23M7bvcRomJRdAnNylo0yqYHKCQ/edit?usp=sharing
Data in the table are all re-run. I also add some enties of official deluge 1.3.15 build for comparison and it is on top of libtorrent 1.0.11. It seems for qbt 4.1.1+1.1.7 the piece size doesn't really matter though a slight difference is witnessed. But for libtorrent 1.0.11, both qbt and deluge see some improvement on smaller piece size.

@airium commented on GitHub (Jun 9, 2018): @arvidn Sorry for my delay. Here is the result I put on Google Sheet. https://docs.google.com/spreadsheets/d/1q5eL3j4HVGU_wZel23M7bvcRomJRdAnNylo0yqYHKCQ/edit?usp=sharing Data in the table are all re-run. I also add some enties of official deluge 1.3.15 build for comparison and it is on top of libtorrent 1.0.11. It seems for qbt 4.1.1+1.1.7 the piece size doesn't really matter though a slight difference is witnessed. But for libtorrent 1.0.11, both qbt and deluge see some improvement on smaller piece size.

deekerman commented

Author

Owner

@airium commented on GitHub (Jun 9, 2018):

Also sorry I don't have time to fill every entry in the table but if you want some specific entries just tell me.

@airium commented on GitHub (Jun 9, 2018): Also sorry I don't have time to fill every entry in the table but if you want some specific entries just tell me.

deekerman commented

Author

Owner

@arvidn commented on GitHub (Jun 9, 2018):

I'm primarily interested in whether settings_pack::checking_mem_usage improves checking throughput or not. I don't see this being reflected in the table.

@arvidn commented on GitHub (Jun 9, 2018): I'm primarily interested in whether `settings_pack::checking_mem_usage` improves checking throughput or not. I don't see this being reflected in the table.

deekerman commented

Author

Owner

@airium commented on GitHub (Jun 9, 2018):

@arvidn Oh sorry I misunderstood your reply. Now I will change settings_pack::checking_mem_usage.
Stand by and I will renew the result in one hour.

@airium commented on GitHub (Jun 9, 2018): @arvidn Oh sorry I misunderstood your reply. Now I will change `settings_pack::checking_mem_usage`. Stand by and I will renew the result in one hour.

deekerman commented

Author

Owner

@airium commented on GitHub (Jun 9, 2018):

@arvidn So this time I alter settings_pack::checking_mem_usage to
SET(checking_mem_usage, 1024, 0) i.e. 4x of the default value
SET(checking_mem_usage, 32, 0) i.e. 1/8 of the default
SET(checking_mem_usage, 256000, 0) i.e. 1000x of the default, as an aggressive one

However, the 3 builds with 32, 1024 and 256000 show no observable difference on torrent rechecking/creaing speed compared with the default build with 256.

Here are my builds: https://drive.google.com/open?id=16cerOCIJOAW2ksQhu8PgslDJxWLZvUx_

@airium commented on GitHub (Jun 9, 2018): @arvidn So this time I alter `settings_pack::checking_mem_usage` to `SET(checking_mem_usage, 1024, 0)` i.e. 4x of the default value `SET(checking_mem_usage, 32, 0)` i.e. 1/8 of the default `SET(checking_mem_usage, 256000, 0)` i.e. 1000x of the default, as an aggressive one However, the 3 builds with 32, 1024 and 256000 show **no observable difference** on torrent rechecking/creaing speed compared with the default build with 256. Here are my builds: [https://drive.google.com/open?id=16cerOCIJOAW2ksQhu8PgslDJxWLZvUx_](https://drive.google.com/open?id=16cerOCIJOAW2ksQhu8PgslDJxWLZvUx_)

deekerman commented

Author

Owner

@khnielsen commented on GitHub (Jun 12, 2018):

I am actually seeing the same thing here, it is not only limited to checking speed but also actual torrent speeds are also much faster and more stable when libtorrent 1.0.11 is used rather than 1.1.X.

Arvidn I have been emailing you about this, as I can reproduce this not only on Windows but also on Linux (Using deluge), 1.0.11 will outperform the newer versions in all of my testings.

I have tried nearly everything I can think of, nearly all options using "ltconfig" on my linux installs and I have simply not been able to get 1.1.X to go anywhere near the speeds of 1.0.11, thats in terms of network throughput but also in terms of creating and checking torrents.

@khnielsen commented on GitHub (Jun 12, 2018): I am actually seeing the same thing here, it is not only limited to checking speed but also actual torrent speeds are also much faster and more stable when libtorrent 1.0.11 is used rather than 1.1.X. Arvidn I have been emailing you about this, as I can reproduce this not only on Windows but also on Linux (Using deluge), 1.0.11 will outperform the newer versions in all of my testings. I have tried nearly everything I can think of, nearly all options using "ltconfig" on my linux installs and I have simply not been able to get 1.1.X to go anywhere near the speeds of 1.0.11, thats in terms of network throughput but also in terms of creating and checking torrents.

deekerman commented

Author

Owner

@arvidn commented on GitHub (Jun 13, 2018):

@khnielsen would you be able to enable session stats logging (in the alert mask) and post the logs here?

@arvidn commented on GitHub (Jun 13, 2018): @khnielsen would you be able to enable session stats logging (in the alert mask) and post the logs here?

deekerman commented

Author

Owner

@khnielsen commented on GitHub (Jun 13, 2018):

I would if you could provide some documentation on how to do that, I am a bit lost on how to proceed, my findings on linux is not supported by anything but testing and messing around with it, but I have very little knowledge of how to get the proper logs for you.

@khnielsen commented on GitHub (Jun 13, 2018): I would if you could provide some documentation on how to do that, I am a bit lost on how to proceed, my findings on linux is not supported by anything but testing and messing around with it, but I have very little knowledge of how to get the proper logs for you.

deekerman commented

Author

Owner

@Chocobo1 commented on GitHub (Jun 16, 2018):

@airium
I'm not sure if this is relevant but have you tried bumping settings_pack::aio_threads?
https://github.com/arvidn/libtorrent/issues/3005

@Chocobo1 commented on GitHub (Jun 16, 2018): @airium I'm not sure if this is relevant but have you tried bumping `settings_pack::aio_threads`? https://github.com/arvidn/libtorrent/issues/3005

deekerman commented

Author

Owner

@arvidn commented on GitHub (Jun 16, 2018):

I run some tests to see what I can find on MacOS with a spinning disk. creating and checking torrents are slightly different. For checking, there currently is logic to concentrate the checking jobs to a single thread (as long as there are 4 or more disk threads).

There's an effect where spreading out the checking over more threads slows it down, presumably because of disk I/O requests not being perfectly sequential. It's possible that having aio_threads < 4 primarily makes it worse by spreading out the hashing across 3 threads.

However, when creating a torrent, the hashing happens in a single thread.

In my testing, the CPU used for hashing in insignificant compared to the time of the disk read operations. This is on a spinning disk on macOS. People that experience this problem may have a system with different characteristics, like an SSD for instance.

@arvidn commented on GitHub (Jun 16, 2018): I run some tests to see what I can find on MacOS with a spinning disk. creating and checking torrents are slightly different. For checking, there currently is logic to concentrate the checking jobs to a single thread (as long as there are 4 or more disk threads). There's an effect where spreading out the checking over more threads slows it down, presumably because of disk I/O requests not being perfectly sequential. It's possible that having `aio_threads` < 4 primarily makes it worse by spreading out the hashing across 3 threads. However, when creating a torrent, the hashing happens in a single thread. In my testing, the CPU used for hashing in insignificant compared to the time of the disk read operations. This is on a spinning disk on macOS. People that experience this problem may have a system with different characteristics, like an SSD for instance.

deekerman commented

Author

Owner

@arvidn commented on GitHub (Jun 16, 2018):

looking closer at the code, I think the main reason for the poor performance is probably that libtorrent reads 16kiB at a time from the disk. I think the original motivation for this is to keep the code simple and to not use excessive amounts of memory when the piece size is large.

I will try to improve this.

@arvidn commented on GitHub (Jun 16, 2018): looking closer at the code, I think the main reason for the poor performance is probably that libtorrent reads 16kiB at a time from the disk. I think the original motivation for this is to keep the code simple and to not use excessive amounts of memory when the piece size is large. I will try to improve this.

deekerman commented

Author

Owner

@arvidn commented on GitHub (Jun 16, 2018):

anyone wants to give this patch a try? https://github.com/arvidn/libtorrent/pull/3112

@arvidn commented on GitHub (Jun 16, 2018): anyone wants to give this patch a try? https://github.com/arvidn/libtorrent/pull/3112

deekerman commented

Author

Owner

@arvidn commented on GitHub (Jun 17, 2018):

iirc, especially windows is very sensitive to small read calls. This reads entire pieces at a time, rather than 16kiB, but it still falls back to the generic (16 kiB at a time) logic in case it fails to allocate enough space in the cache

@arvidn commented on GitHub (Jun 17, 2018): iirc, especially windows is very sensitive to small read calls. This reads entire pieces at a time, rather than 16kiB, but it still falls back to the generic (16 kiB at a time) logic in case it fails to allocate enough space in the cache

deekerman commented

Author

Owner

@airium commented on GitHub (Jun 18, 2018):

Sorry for my delay in response.

I'm not sure if this is relevant but have you tried bumping settings_pack::aio_threads?

@Chocobo1 Thank you for your advice, but actually this setting does not matter. You could find my test packages in the link below, in which the ones with "aio_thread_1" and "aio_thread_64" are.

anyone wants to give this patch a try?

@arvidn I tried this one and it works! I noticed the creating/rechecking speed on SSD is almost 2x of that of current RC_1_1, close to the same speed of R_1_0. As for the previously noticed strange observation, i.e. the abnormally slow speed on my 3.5" HDD, now I identified it is a my-side fault, so nothing needed for it.
Here is the test package, the one with "fix_hash_perf_1.1": test builds

@airium commented on GitHub (Jun 18, 2018): Sorry for my delay in response. >I'm not sure if this is relevant but have you tried bumping settings_pack::aio_threads? @Chocobo1 Thank you for your advice, but actually this setting does not matter. You could find my test packages in the link below, in which the ones with "aio_thread_1" and "aio_thread_64" are. >anyone wants to give this patch a try? @arvidn I tried this one and it works! I noticed the creating/rechecking speed on SSD is almost 2x of that of current `RC_1_1`, close to the same speed of `R_1_0`. As for the previously noticed strange observation, i.e. the abnormally slow speed on my 3.5" HDD, now I identified it is a my-side fault, so nothing needed for it. Here is the test package, the one with "fix_hash_perf_1.1": [test builds](https://drive.google.com/open?id=16cerOCIJOAW2ksQhu8PgslDJxWLZvUx_)

deekerman commented

Author

Owner

@arvidn commented on GitHub (Jun 18, 2018):

thanks for testing and making these builds! This will be part of the 1.1.8 release

@arvidn commented on GitHub (Jun 18, 2018): thanks for testing and making these builds! This will be part of the 1.1.8 release

deekerman commented

Author

Owner

@airium commented on GitHub (Jun 18, 2018):

@arvidn btw, before I close this issue, do you have any immediate idea to further improve this performance? As we can see uTorrent still outperforms libtorrent in some cases (e.g. 500MB/s vs 200-300MB/s on SATA3 SSD), I am still of interest in beating uT and plan to look into how libtorrent implements this part, maybe you could offer a start point.

@airium commented on GitHub (Jun 18, 2018): @arvidn btw, before I close this issue, do you have any immediate idea to further improve this performance? As we can see uTorrent still outperforms libtorrent in some cases (e.g. 500MB/s vs 200-300MB/s on SATA3 SSD), I am still of interest in beating uT and plan to look into how libtorrent implements this part, maybe you could offer a start point.

deekerman commented

Author

Owner

@arvidn commented on GitHub (Jun 18, 2018):

the only other thing I can think of would be to bump the settings_pack::checking_mem_usage

@arvidn commented on GitHub (Jun 18, 2018): the only other thing I can think of would be to bump the `settings_pack::checking_mem_usage`

deekerman commented

Author

Owner

@khnielsen commented on GitHub (Jun 18, 2018):

What kind of items changed in the past arvid? I can concur that the patch improves performance but from my testings it is still a ~250MB\s 1.1.7 with patch vs 420MB\s 1.0.11 which is a massive difference.

I would assume that newer versions include better performance, and not something that goes the other way - From my testings then 1.0.11 is generally outperforming the 1.1.x branch in basically everything, from connecting to peers, network throughput on 10G interfaces to hashing speed.

@khnielsen commented on GitHub (Jun 18, 2018): What kind of items changed in the past arvid? I can concur that the patch improves performance but from my testings it is still a ~250MB\s 1.1.7 with patch vs 420MB\s 1.0.11 which is a massive difference. I would assume that newer versions include better performance, and not something that goes the other way - From my testings then 1.0.11 is generally outperforming the 1.1.x branch in basically everything, from connecting to peers, network throughput on 10G interfaces to hashing speed.

deekerman commented

Author

Owner

@arvidn commented on GitHub (Jun 18, 2018):

@khnielsen a lot of things changed, perhaps the most important one was to support more than one disk I/O thread.

would you like to build and run performance regression tests, to avoid changes that negatively impact performance in the future?
You can try the master branch today, and there's even a new-disk-io branch which overhauls the disk I/O subsystem in an attempt to make it simpler and faster.

@arvidn commented on GitHub (Jun 18, 2018): @khnielsen a lot of things changed, perhaps the most important one was to support more than one disk I/O thread. would you like to build and run performance regression tests, to avoid changes that negatively impact performance in the future? You can try the `master` branch today, and there's even a `new-disk-io` branch which overhauls the disk I/O subsystem in an attempt to make it simpler and faster.

deekerman commented

Author

Owner

@airium commented on GitHub (Jun 18, 2018):

@arvidn I have some new findings by chance. Now on the top of branch hash-job-performance-1.1, a large increase on aio_threads and checking_memory_usage both at once can effectively boost disk throughput in torrent rechecking. As an example, now when I increase aio_threads to 64 (=16x default) and checking_memory_usage to 25600 (=100x default), the speed on SM951 reaches 1.5+GB/s and X400 550MB/s, both at their theoretical max bandwidth, fully utilised.

Screenshots first:

*columns are on different torrents as 1.7GB/s is very quick to check any torrent, noticeably the CPU usage is correspondingly high

*stable ~550MB/s on SATA3 SSD

However, a single increase on one of the two has no effect, i.e. only increase aio_threads or only increase checking_memory_usage. Apart from this, an identical change (4->64 and 256->25600) but without the patch shows an weaker improvement (~250MB/s -> 300~400MB/s). My further test shows that samller increase does not show a similar effect, e.g. only doubling aio_threads and checking_memory_usage to 8 and 512 respectively gives nearly no improvement. At last, the change has no effect on creating (maybe because creating torrent only uses 1 thread as said)

Windows binaries for test: the same google drive link

I also noticed that in the past hours a little more io related patch is introduced, but I have no time to test either it or other changes in next days. Sorry in advance.

And just a note here: I also find some other problems during the whole tests. One is that libtorrent 1.1 executes all queued torrent rechecking jobs simultaneously, not likely on 1.0. This might not be expected especially on HDD as HDDs are slower in the case. Another one is that, when creating torrent, clicking on "Cancel" actually does not stop the job immediately. Hard disk is still being read until the job finishes (but gives no torrent). These 2 problems should be discussed as new issues.

@airium commented on GitHub (Jun 18, 2018): @arvidn I have some new findings by chance. Now on the top of branch `hash-job-performance-1.1`, a large increase on `aio_threads` and `checking_memory_usage` both at once can effectively boost disk throughput in torrent rechecking. As an example, now when I increase `aio_threads` to 64 (=16x default) and `checking_memory_usage` to 25600 (=100x default), the speed on SM951 reaches 1.5+GB/s and X400 550MB/s, both at their theoretical max bandwidth, fully utilised. Screenshots first: ![nvme](https://user-images.githubusercontent.com/38249940/41556048-7a1c989c-7331-11e8-857a-e592a19eaa54.jpg) **_*columns are on different torrents as 1.7GB/s is very quick to check any torrent, noticeably the CPU usage is correspondingly high_** ![sata3](https://user-images.githubusercontent.com/38249940/41556062-84c8e7fa-7331-11e8-81e7-0d4def64dd5e.jpg) **_*stable ~550MB/s on SATA3 SSD_** However, **a single increase on one of the two has no effect**, i.e. only increase `aio_threads` or only increase `checking_memory_usage`. Apart from this, an identical change (4->64 and 256->25600) but **without the patch shows an weaker improvement** (\~250MB/s -> 300\~400MB/s). My further test shows that **samller increase does not show a similar effect**, e.g. only doubling `aio_threads` and `checking_memory_usage` to 8 and 512 respectively gives nearly no improvement. At last, the change has **no effect on creating** (maybe because creating torrent only uses 1 thread as said) Windows binaries for test: [the same google drive link](https://drive.google.com/open?id=16cerOCIJOAW2ksQhu8PgslDJxWLZvUx_) I also noticed that in the past hours a little more io related patch is introduced, but I have no time to test either it or other changes in next days. Sorry in advance. And just a note here: I also find some other problems during the whole tests. One is that libtorrent 1.1 executes all queued torrent rechecking jobs simultaneously, not likely on 1.0. This might not be expected especially on HDD as HDDs are slower in the case. Another one is that, when creating torrent, clicking on "Cancel" actually does not stop the job immediately. Hard disk is still being read until the job finishes (but gives no torrent). These 2 problems should be discussed as new issues.

deekerman commented

Author

Owner

@arvidn commented on GitHub (Jun 19, 2018):

One is that libtorrent 1.1 executes all queued torrent rechecking jobs simultaneously, not likely on 1.0

It's not supposed to do this by default. It suggests the active_checking setting is greater than 1. Another possible explanation is that force started torrents also apply to checking. They will be force checked so to speak.

@arvidn commented on GitHub (Jun 19, 2018): > One is that libtorrent 1.1 executes all queued torrent rechecking jobs simultaneously, not likely on 1.0 It's not supposed to do this by default. It suggests the `active_checking` setting is greater than 1. Another possible explanation is that *force started* torrents also apply to checking. They will be force checked so to speak.

deekerman commented

Author

Owner

@arvidn commented on GitHub (Jun 19, 2018):

@ssiloti I can't recall the motivation for concentrating hash jobs into specific threads. I imagine it would have benefits of making disk I/O more sequential when checking a whole torrent (by serialising them all in a single thread). But I can also imagine a benefit being that it prevents hash jobs from starving out other disk jobs, just because there are so many of them.

I get the feeling though that in this case, with an SSD, the bottleneck is not the disk I/O, but the CPU of SHA-1. Does that sound reasonable? I can't think of a good architecture that would satisfy both the I/O bound and CPU bound case.

As for the checking_memory_usage setting, I believe this is a direct consequence of moving out the loop over the pieces into the network thread (out of the disk thread as it was in 1.0). Perhaps there should be some more sophisticated logic to determine that number of outstanding hash jobs, based on the bandwidth delay product. Or maybe not, maybe the limit should just be raised 100x. As long hash jobs are posted to dedicated threads and a dedicated job queue, it's not like there any significant cost of having a long backlog of hash jobs in there. They don't allocate any memory up-front.

@arvidn commented on GitHub (Jun 19, 2018): @ssiloti I can't recall the motivation for concentrating hash jobs into specific threads. I imagine it would have benefits of making disk I/O more sequential when checking a whole torrent (by serialising them all in a single thread). But I can also imagine a benefit being that it prevents hash jobs from starving out other disk jobs, just because there are so many of them. I get the feeling though that in this case, with an SSD, the bottleneck is not the disk I/O, but the CPU of SHA-1. Does that sound reasonable? I can't think of a good architecture that would satisfy both the I/O bound and CPU bound case. As for the `checking_memory_usage` setting, I believe this is a direct consequence of moving out the loop over the pieces into the network thread (out of the disk thread as it was in 1.0). Perhaps there should be some more sophisticated logic to determine that number of outstanding hash jobs, based on the bandwidth delay product. Or maybe not, maybe the limit should just be raised 100x. As long hash jobs are posted to dedicated threads and a dedicated job queue, it's not like there any significant cost of having a long backlog of hash jobs in there. They don't allocate any memory up-front.

deekerman commented

Author

Owner

@Chocobo1 commented on GitHub (Jun 19, 2018):

It's not supposed to do this by default. It suggests the active_checking setting is greater than 1.

Just want to add that qbt doesn't touch that parameter, it should be at the default 1.

@Chocobo1 commented on GitHub (Jun 19, 2018): >It's not supposed to do this by default. It suggests the active_checking setting is greater than 1. Just want to add that qbt doesn't touch that parameter, it should be at the default `1`.

deekerman commented

Author

Owner

@ssiloti commented on GitHub (Jun 20, 2018):

I think the splitting of hash jobs into their own threads was before my time, but preventing them from starving other jobs seems like the most plausible justification.

A 6700K CPU is capable of computing SHA1 at well over 2GB/s with four hash threads, so it shouldn't be a bottleneck even with the NVMe drive as long as aio_threads is set to at least 16.

I suspect the biggest drag on performance here is insufficient pipelining of hash jobs combined with insufficient parallelism. SSDs, especially high speed NVMe drives, really need to have multiple requests outstanding to hit their peak performance, so it makes sense that @airium needed both an increased number of active jobs and multiple hash threads to get the most out of those drives.

I think increasing aio_threads is probably a good idea, at least on master where we have a scalable thread pool. We should also probably increase the minimum number of active hash jobs when checking to 4 or 8 or maybe even 16. I think increasing checking_memory_usage 100x is probably overkill, our latency should not be high enough to need that many active jobs. In fact I wonder if we even need the checking_memory_usage setting. Calculating the number of active checking jobs by applying a fixed multiplier to aio_threads should Just Work for any reasonable situation I can think of.

@ssiloti commented on GitHub (Jun 20, 2018): I think the splitting of hash jobs into their own threads was before my time, but preventing them from starving other jobs seems like the most plausible justification. A 6700K CPU is capable of computing SHA1 at well over 2GB/s with four hash threads, so it shouldn't be a bottleneck even with the NVMe drive as long as `aio_threads` is set to at least 16. I suspect the biggest drag on performance here is insufficient pipelining of hash jobs combined with insufficient parallelism. SSDs, especially high speed NVMe drives, _really_ need to have multiple requests outstanding to hit their peak performance, so it makes sense that @airium needed both an increased number of active jobs and multiple hash threads to get the most out of those drives. I think increasing `aio_threads` is probably a good idea, at least on master where we have a scalable thread pool. We should also probably increase the minimum number of active hash jobs when checking to 4 or 8 or maybe even 16. I think increasing `checking_memory_usage` 100x is probably overkill, our latency should not be high enough to need that many active jobs. In fact I wonder if we even need the `checking_memory_usage` setting. Calculating the number of active checking jobs by applying a fixed multiplier to `aio_threads` should Just Work for any reasonable situation I can think of.

deekerman commented

Author

Owner

@ssiloti commented on GitHub (Jun 20, 2018):

Actually nevermind about increasing the minimum active jobs and removing checking_memory_usage, I think it does make sense the way it is. I still think the number of active jobs should be multiplied by the number of hash threads though.

@ssiloti commented on GitHub (Jun 20, 2018): Actually nevermind about increasing the minimum active jobs and removing `checking_memory_usage`, I think it does make sense the way it is. I still think the number of active jobs should be multiplied by the number of hash threads though.

deekerman commented

Author

Owner

@arvidn commented on GitHub (Jun 24, 2018):

could someone make a new test build with latest RC_1_1 of libtorrent? I believe the situation have been improved now.

hash disk jobs now read whole pieces at a time from disk
read and write jobs are coalesced by default on windows (as the preadv/writev emulation seems to be slow)
the default checking_memory usage has been increased
the minimum number of outstanding disk jobs now scales with the number of hash threads

@arvidn commented on GitHub (Jun 24, 2018): could someone make a new test build with latest `RC_1_1` of libtorrent? I believe the situation have been improved now. 1. hash disk jobs now read whole pieces at a time from disk 2. read and write jobs are coalesced by default on windows (as the preadv/writev emulation seems to be slow) 3. the default checking_memory usage has been increased 4. the minimum number of outstanding disk jobs now scales with the number of hash threads

deekerman commented

Author

Owner

@airium commented on GitHub (Jun 29, 2018):

Sorry I was busy in my life. Now here is my compilation of qb 4.1.1 + lt 1.1.8 both at release: Google Drive
As far as I noticed, this one makes no difference with the previous one on hash-job-performance-1.1. In other words, the performance on disk reading / torrent rechecking / creating is similar to that of 1.0.11 (maybe still 15% lower than the latter, but much better than 1.1.7-)

This time I remind of using the statistic panel of qbittorrent. I found that on 1.1.8, during rechecking it shows queued I/O jobs at 3 and total_buffer_size at 4MB, then gives 200-300MB/s on SATA3 SSD. While older versions with default 1.1.7 settings show queued I/O jobs at 1 and total_buffer_size at 4MB, with 70-140MB/s. And the one I compilated with aio_threads = 64 or checking_memory_usage = 25600 makes it to 84 and sereral hundred MB, giving 500+MB/s (and amazingly 2.2GB/s on NVMe SSD if also give a cache size larger than 64MB). Maybe this gives some hints for further improvement (e.g. there is just too less IO jobs). So I am curious about libtorrent's internal mechasim that leads to different queued I/O jobs, but I cannot spare time looking into it.

And this time I still encounter the concurrent rechecking problem. To be specific, when queued multiple torrents to recheck, normally it should run one by one while the others are stalled at the queued for checking status. However, currently with libtorrent 1.1.x no one enters this queued state. I noticed #9120 discussed the rechecking behaviour, maybe a better policy may be only allowing concurrent rechecking on different hard disk while for torrents on the same disk they can only be run one by one. I don't know if this is feasible by libtorrent. A similar symptom should be noted here, that moving torrent are also executed concurrently. That is, if one moves serveral torrents, now all these torrents are moved in the same time instead of one by one, which is especially not expected when moving to HDDs. I have no idea if this problem comes from libtorrent or qbittorrent.

At last, some of my friends reported crash issue with 1.1.8 against either deluge and qbittorrent. Maybe I also happened to encounter once yesterday on one my seedbox, but we didn't collect any trace info.

@airium commented on GitHub (Jun 29, 2018): Sorry I was busy in my life. Now here is my compilation of qb 4.1.1 + lt 1.1.8 both at release: [Google Drive](https://drive.google.com/open?id=1EzwqwG9vD4OHUOX2D_3NWl1wKjpeIpy0) As far as I noticed, this one makes no difference with the previous one on `hash-job-performance-1.1`. In other words, the performance on disk reading / torrent rechecking / creating is similar to that of 1.0.11 (maybe still 15% lower than the latter, but much better than 1.1.7-) This time I remind of using the `statistic` panel of qbittorrent. I found that on 1.1.8, during rechecking it shows `queued I/O jobs` at 3 and `total_buffer_size` at 4MB, then gives 200-300MB/s on SATA3 SSD. While older versions with default 1.1.7 settings show `queued I/O jobs` at 1 and `total_buffer_size` at 4MB, with 70-140MB/s. And the one I compilated with `aio_threads = 64` or `checking_memory_usage = 25600` makes it to 84 and sereral hundred MB, giving 500+MB/s (and amazingly 2.2GB/s on NVMe SSD if also give a cache size larger than 64MB). Maybe this gives some hints for further improvement (e.g. there is just too less IO jobs). So I am curious about libtorrent's internal mechasim that leads to different `queued I/O jobs`, but I cannot spare time looking into it. And this time I still encounter the concurrent rechecking problem. To be specific, when queued multiple torrents to recheck, normally it should run one by one while the others are stalled at the `queued for checking` status. However, currently with libtorrent 1.1.x no one enters this queued state. I noticed #9120 discussed the rechecking behaviour, maybe a better policy may be only allowing concurrent rechecking on different hard disk while for torrents on the same disk they can only be run one by one. I don't know if this is feasible by libtorrent. A similar symptom should be noted here, that **moving torrent are also executed concurrently**. That is, if one moves serveral torrents, now all these torrents are moved in the same time instead of one by one, which is especially not expected when moving to HDDs. I have no idea if this problem comes from libtorrent or qbittorrent. At last, some of my friends reported crash issue with 1.1.8 against either deluge and qbittorrent. Maybe I also happened to encounter once yesterday on one my seedbox, but we didn't collect any trace info.

deekerman commented

Author

Owner

@seiferflo commented on GitHub (Jul 10, 2018):

I just want to inform I tested the last build of airium and it's day and night. Before, I was having many error / checking issues constantly, mainly with big files >30GB to a NAS. Now I've got almost no issue.
Download speed in much better as well, almost maxed out, though I need to keep playing with cache / connections per torrent as I get speed drop after a couple of hours or with more than 2 files.

Also when I exit qbittorent, it really stops now. Before it used to keep running for 5min in the background... Launching qbittorent again led to "checking" on all file.

So thanks for fixing all this, I was about to drop qbittorrent. Nice one !

@seiferflo commented on GitHub (Jul 10, 2018): I just want to inform I tested the last build of airium and it's day and night. Before, I was having many error / checking issues constantly, mainly with big files >30GB to a NAS. Now I've got almost no issue. Download speed in much better as well, almost maxed out, though I need to keep playing with cache / connections per torrent as I get speed drop after a couple of hours or with more than 2 files. Also when I exit qbittorent, it really stops now. Before it used to keep running for 5min in the background... Launching qbittorent again led to "checking" on all file. So thanks for fixing all this, I was about to drop qbittorrent. Nice one !

deekerman commented