too much memory consumption in the latest version #10259

https://imgur.com/T5VOMu3

Author

Owner

@electrifying commented on GitHub (Apr 24, 2020):

How much RAM do you have? Do you have disk cache set to auto in the advanced settings?

16 gb ram ddr4 3200mz

Do you have disk cache set to auto in the advanced settings?

all settings are by default, I did not change anything.

@electrifying commented on GitHub (Apr 24, 2020): > How much RAM do you have? Do you have disk cache set to auto in the advanced settings? 16 gb ram ddr4 3200mz > Do you have disk cache set to auto in the advanced settings? all settings are by default, I did not change anything. https://imgur.com/T5VOMu3

deekerman commented

Author

Owner

@FranciscoPombal commented on GitHub (Apr 24, 2020):

16 gb ram ddr4 3200mz

Assuming you do in fact have disk cache set to auto, this is expected usage then, it's just disk cache.

Here is the relevant libtorrent code, that determines how much RAM is used for disk cache when set to auto: github.com/arvidn/libtorrent@c537f6277f/src/disk_buffer_pool.cpp (L298). With 16 GB RAM, the auto setting results in up to 400-something MiB RAM being used for disk cache.

Depending on your connection speed and system specs, you can manually set it to a lower value, but set it too low and you'll have this issue: https://github.com/qbittorrent/qBittorrent/issues/12336. Values from and 128 MiB and up are reasonable. You might even get away with 64 MiB, if your connection is slow, and/or your I/O subsystem very fast.

You can monitor the amount of RAM usage that's due to the disk cache in View->Statistics -> Total buffer size.

@FranciscoPombal commented on GitHub (Apr 24, 2020): > 16 gb ram ddr4 3200mz Assuming you do in fact have disk cache set to auto, this is expected usage then, it's just disk cache. Here is the relevant libtorrent code, that determines how much RAM is used for disk cache when set to auto: https://github.com/arvidn/libtorrent/blob/c537f6277f4de2d80cadc0eb40733ac8705f4cad/src/disk_buffer_pool.cpp#L298. With 16 GB RAM, the auto setting results in up to 400-something MiB RAM being used for disk cache. Depending on your connection speed and system specs, you can manually set it to a lower value, but set it too low and you'll have this issue: https://github.com/qbittorrent/qBittorrent/issues/12336. Values from and 128 MiB and up are reasonable. You might even get away with 64 MiB, if your connection is slow, and/or your I/O subsystem very fast. You can monitor the amount of RAM usage that's due to the disk cache in View->Statistics -> Total buffer size.

deekerman commented

Author

Owner

@Ryrynz commented on GitHub (Apr 25, 2020):

Perhaps Libtorrent could be a bit smarter about when it scales up the cache? In this case it would appear that disk performance with qBitorrent would be perfectly acceptable at around 100MiB, using up to five times the RAM seems a tad excessive @arvidn is this what users should expect now?

@Ryrynz commented on GitHub (Apr 25, 2020): Perhaps Libtorrent could be a bit smarter about when it scales up the cache? In this case it would appear that disk performance with qBitorrent would be perfectly acceptable at around 100MiB, using up to five times the RAM seems a tad excessive @arvidn is this what users should expect now?

deekerman commented

Author

Owner

@ghost commented on GitHub (Apr 26, 2020):

I think we should go back to hard limits for defaults instead of auto. As some people have been using qBt for ages with little to no RAM usage and now suddenly they're seeing a +400 MB RAM usage out of nowhere.

@ghost commented on GitHub (Apr 26, 2020): I think we should go back to hard limits for defaults instead of auto. As some people have been using qBt for ages with little to no RAM usage and now suddenly they're seeing a +400 MB RAM usage out of nowhere.

deekerman commented

Author

Owner

@Ryrynz commented on GitHub (Apr 26, 2020):

Perhaps at least until there's logic that can better scale the cache with the disk load, this is probably a good idea.

@Ryrynz commented on GitHub (Apr 26, 2020): Perhaps at least until there's logic that can better scale the cache with the disk load, this is probably a good idea.

deekerman commented

Author

Owner

@ghost commented on GitHub (Apr 26, 2020):

Current logic is using excessive amounts of RAM on 16 GiB or 32 GiB systems. Nobody wants a client to be consuming ~500 MiB or even ~800MiB RAM respectively with only 3-4 torrents running.

@ghost commented on GitHub (Apr 26, 2020): Current logic is using excessive amounts of RAM on 16 GiB or 32 GiB systems. Nobody wants a client to be consuming ~500 MiB or even ~800MiB RAM respectively with only 3-4 torrents running.

deekerman commented

Author

Owner

@Ryrynz commented on GitHub (Apr 26, 2020):

Using more RAM simply because you have more definitely isn't the way to go, I do set mine to 1 GiB but once it appears to reach that it's not reduced no matter what's happening with regards to data transfer which isn't ideal either.

@Ryrynz commented on GitHub (Apr 26, 2020): Using more RAM simply because you have more definitely isn't the way to go, I do set mine to 1 GiB but once it appears to reach that it's not reduced no matter what's happening with regards to data transfer which isn't ideal either.

deekerman commented

Author

Owner

@ghost commented on GitHub (Apr 26, 2020):

Yeah that's correct. People do have other RAM hungry stuffs running all the time. And then if you have an idle torrent client consuming 500-800 MiB for now good reason, that'd prompt some people to just close the client or switch to some less RAM hungry alternative instead of looking into settings.

@ghost commented on GitHub (Apr 26, 2020): Yeah that's correct. People do have other RAM hungry stuffs running all the time. And then if you have an idle torrent client consuming 500-800 MiB for now good reason, that'd prompt some people to just close the client or switch to some less RAM hungry alternative instead of looking into settings.

deekerman commented

Author

Owner

@p43b1 commented on GitHub (Apr 26, 2020):

please check if your issue it's like this one #12326

@p43b1 commented on GitHub (Apr 26, 2020): please check if your issue it's like this one #12326

deekerman commented

Author

Owner

@FranciscoPombal commented on GitHub (Apr 26, 2020):

@an0n666
There will be no good solution for this problem until libtorrent 2.0. As for the fact that the auto size starts to get a little too big for people with 16 or 32 GiB of RAM, the logic could be tweaked in libtorrent to account for that until 2.0 is released.

@arvidn What would be the downsides of hard-capping the cache at 256 MiB (or maybe 300 or even 400), when set to auto? If it is adequate for all but the most extreme use cases/system setups, it could be the way to go.

Alternatively, the same pattern of dividing by some constant could be used, e.g. for >= 16 GiB, divide by 60 or something, 32 and above divide by 80 or 90 or something, etc. The problem with this is obviously that the formula needs to be extended every time a bigger amount of RAM becomes commonplace. Though hopefully, this could be the last time that's needed before libtorrent 2.0 is released and qbittorrent adopts it.

Side note: I don't want to go back to hard-coded values because that:

incentivizes users to change it to ridiculous values like 1+ GiB (I think it's a psychological thing) without really understanding how that will impact the memory usage and that such values are useless.
there isn't a single acceptable value to cover the whole 1 GiB - 16 GiB spectrum. It would force us to use the lowest-common denominator and go for something like 64 MiB, which is a performance bottleneck for many, which was the entire reason why the setting was changed to auto anyway.

For now, I think the excessive RAM usage due to auto cache for 32+ GiB users (although already on 16 GiB it's a little too much) is an acceptable trade-off to ensure it's never the bottleneck for anyone, and to ensure that for most people (1-16 GiB), the value is adequate. Plus, remember that 400 MiB extra RAM usage on a 32+ GiB system makes much less difference than an extra 100 MiB in a 1 or 2 GiB system. The users who want to reduce the memory footprint can always manually set a smaller value than what auto decides for their machine.

The most we can do is add an entry to the FAQ or something.

@FranciscoPombal commented on GitHub (Apr 26, 2020): @an0n666 There will be no good solution for this problem until libtorrent 2.0. As for the fact that the auto size starts to get a little too big for people with 16 or 32 GiB of RAM, the logic could be tweaked in libtorrent to account for that until 2.0 is released. @arvidn What would be the downsides of hard-capping the cache at 256 MiB (or maybe 300 or even 400), when set to auto? If it is adequate for all but the most extreme use cases/system setups, it could be the way to go. Alternatively, the same pattern of dividing by some constant could be used, e.g. for >= 16 GiB, divide by 60 or something, 32 and above divide by 80 or 90 or something, etc. The problem with this is obviously that the formula needs to be extended every time a bigger amount of RAM becomes commonplace. Though hopefully, this could be the last time that's needed before libtorrent 2.0 is released and qbittorrent adopts it. Side note: I don't want to go back to hard-coded values because that: - incentivizes users to change it to ridiculous values like 1+ GiB (I think it's a psychological thing) without really understanding how that will impact the memory usage and that such values are useless. - there isn't a single acceptable value to cover the whole 1 GiB - 16 GiB spectrum. It would force us to use the lowest-common denominator and go for something like 64 MiB, which is a performance bottleneck for many, which was the entire reason why the setting was changed to auto anyway. For now, I think the excessive RAM usage due to auto cache for 32+ GiB users (although already on 16 GiB it's a little too much) is an acceptable trade-off to ensure it's never the bottleneck for anyone, and to ensure that for most people (1-16 GiB), the value is adequate. Plus, remember that 400 MiB extra RAM usage on a 32+ GiB system makes much less difference than an extra 100 MiB in a 1 or 2 GiB system. The users who want to reduce the memory footprint can always manually set a smaller value than what `auto` decides for their machine. The most we can do is add an entry to the FAQ or something.

deekerman commented

Author

Owner

@Ryrynz commented on GitHub (Apr 26, 2020):

A hard cap of 256MiB might be the way to go when set to Auto, should avoid complaints.
I doubt there's a significant disadvantage for the majority staying within this limit.

@Ryrynz commented on GitHub (Apr 26, 2020): A hard cap of 256MiB might be the way to go when set to Auto, should avoid complaints. I doubt there's a significant disadvantage for the majority staying within this limit.

deekerman commented

Author

Owner

@aphelion1 commented on GitHub (Apr 26, 2020):

i just want to say for me is working perfectly fine, in idle i have 7 mb ram of memory usage, downloading beetwen 300-400 mb, uploading between 100-200 mb

@aphelion1 commented on GitHub (Apr 26, 2020): i just want to say for me is working perfectly fine, in idle i have 7 mb ram of memory usage, downloading beetwen 300-400 mb, uploading between 100-200 mb

deekerman commented

Author

Owner

@nokti commented on GitHub (Apr 27, 2020):

I'm posting here since #12654 was marked as a duplicate. My problem was that the high memory usage (800 MB for a 16 GB system) was never coming down when qBit was idle. Prior to v4.2.4 memory was going up when qBit was active, then coming down when qBit became idle. That seemed like normal behavior. But now it's not the case anymore, when it's up, it stays up and keeps adding more. The only way to free memory is to turn off auto disk cache.

@nokti commented on GitHub (Apr 27, 2020): I'm posting here since #12654 was marked as a duplicate. My problem was that the high memory usage (800 MB for a 16 GB system) was never coming down when qBit was idle. Prior to v4.2.4 memory was going up when qBit was active, then coming down when qBit became idle. That seemed like normal behavior. But now it's not the case anymore, when it's up, it stays up and keeps adding more. The only way to free memory is to turn off auto disk cache.

deekerman commented

Author

Owner

@arvidn commented on GitHub (Apr 27, 2020):

auto disk cache is meant to only affect the upper limit of the disk cache size. Are you suggesting setting auto disk cache has a different behavior than setting the same limit manually?

@arvidn commented on GitHub (Apr 27, 2020): auto disk cache is meant to only affect the upper limit of the disk cache size. Are you suggesting setting auto disk cache has a different behavior than setting *the same* limit manually?

deekerman commented

Author

Owner

@FranciscoPombal commented on GitHub (Apr 27, 2020):

@nokti Are you saying that it keeps going above 800 MiB, deep into GiB territory? Or are you saying that usage never comes down from 800 MiB even if qBittorrent is doing nothing at all (no seeding/downloading/rechecking/etc at all)?

Even so, it might be libtorrent just deciding to keep the cache. Or should it evict it after a certain period of inactivity? @arvidn

@FranciscoPombal commented on GitHub (Apr 27, 2020): @nokti Are you saying that it keeps going above 800 MiB, deep into GiB territory? Or are you saying that usage never comes down from 800 MiB even if qBittorrent is doing nothing at all (no seeding/downloading/rechecking/etc at all)? Even so, it might be libtorrent just deciding to keep the cache. Or should it evict it after a certain period of inactivity? @arvidn

deekerman commented

Author

Owner

@nokti commented on GitHub (Apr 27, 2020):

@arvidn I'm not sure I understand your question correctly, so my apologies if my answer doesn't fit. In my case, setting disk cache to auto or setting manually a limit (128, 256 MB), has the same effect from what I can see in task manager regarding ram use.

With disk cache set to auto I saw it go up to around 800 MB (@FranciscoPombal this is the maximum that I've seen) when torrents were active. And staying there when no torrents were active. If I set a limit, say to 128 MB, the same behaviour happens from what I can see. RAM use starts at around 160 MB (160 MB is what qBit uses with disk cache disabled) then creeps up to around 300 MB when torrents become active. And stays there even if there are no torrents active anymore.

@nokti commented on GitHub (Apr 27, 2020): @arvidn I'm not sure I understand your question correctly, so my apologies if my answer doesn't fit. In my case, setting disk cache to auto or setting manually a limit (128, 256 MB), has the same effect from what I can see in task manager regarding ram use. With disk cache set to auto I saw it go up to around 800 MB (@FranciscoPombal this is the maximum that I've seen) when torrents were active. And staying there when no torrents were active. If I set a limit, say to 128 MB, the same behaviour happens from what I can see. RAM use starts at around 160 MB (160 MB is what qBit uses with disk cache disabled) then creeps up to around 300 MB when torrents become active. And stays there even if there are no torrents active anymore.

deekerman commented

Author

Owner

@FranciscoPombal commented on GitHub (Apr 27, 2020):

@nokti Can you post screenshots of your qbittorrent instance? Plus the settings file in plain text, and the typical speeds at which you upload + download.

@FranciscoPombal commented on GitHub (Apr 27, 2020): @nokti Can you post screenshots of your qbittorrent instance? Plus the settings file in plain text, and the typical speeds at which you upload + download.

deekerman commented

Author

Owner

@arvidn commented on GitHub (Apr 27, 2020):

@nokti ok, thanks. that makes sense.

@arvidn commented on GitHub (Apr 27, 2020): @nokti ok, thanks. that makes sense.

deekerman commented

Author

Owner

@Ryrynz commented on GitHub (Apr 28, 2020):

I have mine set to 1000MiB, currently allocated is 1238MiB according to Taskmanager. The last 30 minutes have seen 50KiB/s transfer speeds or lower, I would expect RAM to be freed up and for it to approach similar levels to me restarting it which in this case ended up being 227MiB.

@Ryrynz commented on GitHub (Apr 28, 2020): I have mine set to 1000MiB, currently allocated is 1238MiB according to Taskmanager. The last 30 minutes have seen 50KiB/s transfer speeds or lower, I would expect RAM to be freed up and for it to approach similar levels to me restarting it which in this case ended up being 227MiB.

deekerman commented

Author

Owner

@FranciscoPombal commented on GitHub (Apr 28, 2020):

I have mine set to 1000MiB

Such a great amount is at best useless, while at worst it could be hurting your performance.

@FranciscoPombal commented on GitHub (Apr 28, 2020): > I have mine set to 1000MiB Such a great amount is at best useless, while at worst it could be hurting your performance.

deekerman commented

Author

Owner

@nokti commented on GitHub (Apr 28, 2020):

@FranciscoPombal Settings are here (with all personal identifications deleted): https://pastebin.com/bj0tPM8n

Screenshots of task manager. qBit disk cache set to auto. DL speed capped at 3,000 KB/s.
Columns are: CPU/RAM/Disk/Network/GPU:

qBit at startup, no torrents active:

Downloading a 214 MB torrent (50% completed):

Same but at 99% completed:

30 minutes later, no torrent active:

I added a 2,32 GB torrent, 50% completed:

Same, 99% completed:

1 hour later, qBit is seeding only:

90 minutes later, no torrent active:

@nokti commented on GitHub (Apr 28, 2020): @FranciscoPombal Settings are here (with all personal identifications deleted): https://pastebin.com/bj0tPM8n Screenshots of task manager. qBit disk cache set to auto. DL speed capped at 3,000 KB/s. Columns are: CPU/**RAM**/Disk/Network/GPU: 1. qBit at startup, no torrents active: ![1 qBit (startup)](https://user-images.githubusercontent.com/58652818/80479093-d0b1f780-894e-11ea-9dde-ede959eb3da7.jpg) 2. Downloading a 214 MB torrent (50% completed): ![2 qbit (50% of 214MB torrent)](https://user-images.githubusercontent.com/58652818/80479126-db6c8c80-894e-11ea-94b2-98a70a34a41a.jpg) 3. Same but at 99% completed: ![3 qbit (99% of 214MB torrent)](https://user-images.githubusercontent.com/58652818/80479201-f212e380-894e-11ea-83d2-ca20ccdb4172.jpg) 4. 30 minutes later, no torrent active: ![4 qBit (30 mn later)](https://user-images.githubusercontent.com/58652818/80479227-fd660f00-894e-11ea-9f6c-8a924846197a.jpg) 5. I added a 2,32 GB torrent, 50% completed: ![5 qBit (50% of a 2,32GB torrent)](https://user-images.githubusercontent.com/58652818/80479287-14a4fc80-894f-11ea-8e35-33110fa7846a.jpg) 6. Same, 99% completed: ![5 qBit (99% of a 2,32GB torrent)](https://user-images.githubusercontent.com/58652818/80479327-25557280-894f-11ea-98eb-cf49d7703a76.jpg) 7. 1 hour later, qBit is seeding only: ![6 qBit (1 hour later)](https://user-images.githubusercontent.com/58652818/80479368-3605e880-894f-11ea-9753-aed87e0398ee.jpg) 8. 90 minutes later, no torrent active: ![7 qBit (90 mn later)](https://user-images.githubusercontent.com/58652818/80479545-72394900-894f-11ea-948f-fe0386b6bb2b.jpg)

deekerman commented

Author

Owner

@FranciscoPombal commented on GitHub (Apr 28, 2020):

@nokti
Seems reasonable, as long as most of that is disk cache, which it could very well be, since on a 16 GiB RAM system it can grow up to ~450 MiB when set to auto. How many torrents do you have?

@arvidn do you think libtorrent should periodically evict disk cache, if idle for a long time? Or is the policy to keep it indefinitely?

@FranciscoPombal commented on GitHub (Apr 28, 2020): @nokti Seems reasonable, as long as most of that is disk cache, which it could very well be, since on a 16 GiB RAM system it can grow up to ~450 MiB when set to auto. How many torrents do you have? @arvidn do you think libtorrent should periodically evict disk cache, if idle for a long time? Or is the policy to keep it indefinitely?

deekerman commented

Author

Owner

@nokti commented on GitHub (Apr 28, 2020):

How many torrents do you have?

2,655 atm

@nokti commented on GitHub (Apr 28, 2020): > How many torrents do you have? 2,655 atm

deekerman commented

Author

Owner

@arvidn commented on GitHub (Apr 28, 2020):

it would seem reasonable to expire cache blocks. I think they used to

@arvidn commented on GitHub (Apr 28, 2020): it would seem reasonable to expire cache blocks. I think they used to

deekerman commented

Author

Owner

@FranciscoPombal commented on GitHub (Apr 28, 2020):

@arvidn
Well, looking into this a bit more, there is already this setting: https://www.libtorrent.org/reference-Settings.html#cache_expiry, which qBittorrent sets to 60s by default (but it is configurable). So you're suggesting something similar to this but for the read cache, right?

@FranciscoPombal commented on GitHub (Apr 28, 2020): @arvidn Well, looking into this a bit more, there is already this setting: https://www.libtorrent.org/reference-Settings.html#cache_expiry, which qBittorrent sets to 60s by default (but it is configurable). So you're suggesting something similar to this but for the read cache, right?

deekerman commented

Author

Owner

@xavier2k6 commented on GitHub (Apr 28, 2020):

The cache expiry shows as Downloads\DiskWriteCacheTTL=xxx in the qBittorrent's config file.

So you're suggesting something similar to this but for the read cache, right?

Should it be just Downloads\DiskCacheTTL=xxx? or does it have any real bearing on how it's named?

@xavier2k6 commented on GitHub (Apr 28, 2020): The cache expiry shows as `Downloads\DiskWriteCacheTTL=xxx` in the qBittorrent's config file. > So you're suggesting something similar to this but for the read cache, right? Should it be just `Downloads\DiskCacheTTL=xxx`? or does it have any real bearing on how it's named?

deekerman commented

Author

Owner

@FranciscoPombal commented on GitHub (Apr 28, 2020):

Should it be just Downloads\DiskCacheTTL=xxx? or does it have any real bearing on how it's named?

The name in settings file doesn't really matter, but when it is too different, it becomes a little confusing, I agree.

@FranciscoPombal commented on GitHub (Apr 28, 2020): > Should it be just `Downloads\DiskCacheTTL=xxx`? or does it have any real bearing on how it's named? The name in settings file doesn't really matter, but when it is too different, it becomes a little confusing, I agree.

deekerman commented

Author

Owner

@Ryrynz commented on GitHub (Apr 28, 2020):

I have mine set to 1000MiB

Such a great amount is at best useless, while at worst it could be hurting your performance.

I read your comment about this some days ago, reason is it's a dedicated seed machine and I have 16GiB of RAM and downloads can sometimes reach around 100MiB/s so I don't see the harm in it. The downloads could overflow beyond the disk transfer rate so it makes sense I think in my case.

@Ryrynz commented on GitHub (Apr 28, 2020): > > I have mine set to 1000MiB > > Such a great amount is at best useless, while at worst it could be hurting your performance. I read your comment about this some days ago, reason is it's a dedicated seed machine and I have 16GiB of RAM and downloads can sometimes reach around 100MiB/s so I don't see the harm in it. The downloads could overflow beyond the disk transfer rate so it makes sense I think in my case.

deekerman commented

Author

Owner

@nokti commented on GitHub (Apr 29, 2020):

I think they used to

It probably did, this behaviour started with v4.2.4 I think.

@nokti commented on GitHub (Apr 29, 2020): > I think they used to It probably did, this behaviour started with v4.2.4 I think.

deekerman commented

Author

Owner

@JNavas2 commented on GitHub (Jul 6, 2020):

@FranciscoPombal

Here is the relevant libtorrent code, that determines how much RAM is used for disk cache when set to auto: github.com/arvidn/libtorrent@c537f6277f/src/disk_buffer_pool.cpp (L298). With 16 GB RAM, the auto setting results in up to 400-something MiB RAM being used for disk cache.

Ugh. So working-as-designed, with a design based on poor assumptions.
"There's a lot of memory so we might as well grab a bunch whether we actually need it or not."
"Assumption is the mother of all screw ups."

... Plus, remember that 400 MiB extra RAM usage on a 32+ GiB system makes much less difference than an extra 100 MiB in a 1 or 2 GiB system.

With respect, that's another poor assumption. I have 32 GB RAM in my system because I need it.
When (say) editing 4K video projects in Adobe CC (and other necessary support programs),
the last thing I need is excessive RAM consumption in a background task.

The most we can do is add an entry to the FAQ or something.

I think you need to do more than that. The current setting is buried
in the Advanced menu, which is intimidating to the average user.
I suggest you surface the cache setting in (say) Speed, with appropriate guidance,
and limited choices (e.g., Auto, Disabled, 128 MB, 256 MB, 512 MB).

@JNavas2 commented on GitHub (Jul 6, 2020): @FranciscoPombal > Here is the relevant libtorrent code, that determines how much RAM is used for disk cache when set to auto: https://github.com/arvidn/libtorrent/blob/c537f6277f4de2d80cadc0eb40733ac8705f4cad/src/disk_buffer_pool.cpp#L298. With 16 GB RAM, the auto setting results in up to 400-something MiB RAM being used for disk cache. Ugh. So working-as-designed, with a design based on poor assumptions. "There's a lot of memory so we might as well grab a bunch whether we actually need it or not." _"Assumption is the mother of all screw ups."_ > ... Plus, remember that 400 MiB extra RAM usage on a 32+ GiB system makes much less difference than an extra 100 MiB in a 1 or 2 GiB system. With respect, that's another poor assumption. I have 32 GB RAM in my system because I _need_ it. When (say) editing 4K video projects in Adobe CC (and other necessary support programs), the last thing I need is excessive RAM consumption in a background task. > The most we can do is add an entry to the FAQ or something. I think you need to do more than that. The current setting is buried in the Advanced menu, which is intimidating to the average user. I suggest you surface the cache setting in (say) Speed, with appropriate guidance, and limited choices (e.g., Auto, Disabled, 128 MB, 256 MB, 512 MB).

deekerman commented

Author

Owner

@FranciscoPombal commented on GitHub (Jul 6, 2020):

@JNavas2

@FranciscoPombal
Ugh. So working-as-designed, with a design based on poor assumptions.
"There's a lot of memory so we might as well grab a bunch whether we actually need it or not."
"Assumption is the mother of all screw ups."

Don't worry, this option will stop being relevant in libtorrent >= 2.0

With respect, that's another poor assumption. I have 32 GB RAM in my system because I need it.
When (say) editing 4K video projects in Adobe CC (and other necessary support programs),
the last thing I need is excessive RAM consumption in a background task.

Guess I should have added "in the general". Of course there are cases where no matter the amount of RAM, you're always running pinned. But outside of that, for most users, it is quite different to use an additional 10% (100 MiB) of your total RAM (1 GiB), than to use an additional ~1% (400 MiB vs 32 GiB). At a glance, the latter looks like a rounding error.

The most we can do is add an entry to the FAQ or something.

I think you need to do more than that. The current setting is buried
in the Advanced menu, which is intimidating to the average user.
I suggest you surface the cache setting in (say) Speed, with appropriate guidance,
and limited choices (e.g., Auto, 128 MB, 256 MB, 512 MB).

with appropriate guidance,
and limited choices (e.g., Auto, 128 MB, 256 MB, 512 MB).

this is a nice idea, actually. @arvidn what would you say is reasonable maximum? Or do you need some data before any conclusion about this?

@FranciscoPombal commented on GitHub (Jul 6, 2020): @JNavas2 > @FranciscoPombal > Ugh. So working-as-designed, with a design based on poor assumptions. > "There's a lot of memory so we might as well grab a bunch whether we actually need it or not." > _"Assumption is the mother of all screw ups."_ Don't worry, this option will stop being relevant in libtorrent >= 2.0 > With respect, that's another poor assumption. I have 32 GB RAM in my system because I _need_ it. > When (say) editing 4K video projects in Adobe CC (and other necessary support programs), > the last thing I need is excessive RAM consumption in a background task. Guess I should have added "in the general". Of course there are cases where no matter the amount of RAM, you're always running pinned. But outside of that, for most users, it is quite different to use an additional 10% (100 MiB) of your total RAM (1 GiB), than to use an additional ~1% (400 MiB vs 32 GiB). At a glance, the latter looks like a rounding error. > > > The most we can do is add an entry to the FAQ or something. > > I think you need to do more than that. The current setting is buried > in the Advanced menu, which is intimidating to the average user. > I suggest you surface the cache setting in (say) Speed, with appropriate guidance, > and limited choices (e.g., Auto, 128 MB, 256 MB, 512 MB). > with appropriate guidance, and limited choices (e.g., Auto, 128 MB, 256 MB, 512 MB). this is a nice idea, actually. @arvidn what would you say is reasonable maximum? Or do you need some data before any conclusion about this?

deekerman commented

Author

Owner

@JNavas2 commented on GitHub (Jul 6, 2020):

@FranciscoPombal

Guess I should have added "in the general". Of course there are cases where no matter the amount of RAM, you're always running pinned. But outside of that, for most users, it is quite different to use an additional 10% (100 MiB) of your total RAM (1 GiB), than to use an additional ~1% (400 MiB vs 32 GiB). At a glance, the latter looks like a rounding error.

My point is that many people who have gone to the expense of 32 GB RAM are just those cases where RAM really matters.
(I'm guessing that RAM consumption is actually less important on 16 GB systems, although I don't have data to back that up.)
And in my case, it was more like 800 MB of wasted RAM, not 400 MB, and it does really matter to me–even with 32 GB,
I still sometimes get out of memory conditions.

@JNavas2 commented on GitHub (Jul 6, 2020): @FranciscoPombal > Guess I should have added "in the general". Of course there are cases where no matter the amount of RAM, you're always running pinned. But outside of that, for most users, it is quite different to use an additional 10% (100 MiB) of your total RAM (1 GiB), than to use an additional ~1% (400 MiB vs 32 GiB). At a glance, the latter looks like a rounding error. My point is that many people who have gone to the expense of 32 GB RAM are _just those cases where RAM really matters_. (I'm guessing that RAM consumption is actually less important on 16 GB systems, although I don't have data to back that up.) And in my case, it was more like 800 MB of wasted RAM, not 400 MB, and it does really matter to me–even with 32 GB, I still sometimes get out of memory conditions.

deekerman commented

Author

Owner

@arvidn commented on GitHub (Jul 7, 2020):

@JNavas2 I agree with you (except I would be careful generalising one person's experience to "most people"). The next major version of libtorrent will not have a user-space disk cache, but rely on the page cache instead, to let the kernel decide how much it can spare.

However, in the current stable release, the auto setting could be a lot more sophisticated, but it's not obvious that it would be worth the effort, given that it's being phased out.

I'm sure the UI for this could be improved too, but that's not int my scope.

@arvidn commented on GitHub (Jul 7, 2020): @JNavas2 I agree with you (except I would be careful generalising one person's experience to "most people"). The next major version of libtorrent will not have a user-space disk cache, but rely on the page cache instead, to let the kernel decide how much it can spare. However, in the current stable release, the auto setting could be a lot more sophisticated, but it's not obvious that it would be worth the effort, given that it's being phased out. I'm sure the UI for this could be improved too, but that's not int my scope.

deekerman commented

Author

Owner

@FranciscoPombal commented on GitHub (Jul 7, 2020):

@arvidn

with appropriate guidance,
and limited choices (e.g., Auto, 128 MB, 256 MB, 512 MB).
this is a nice idea, actually. @arvidn what would you say is reasonable maximum? Or do you need some data before any conclusion about this?

Better questions: are there cases that benefit from > 512 MiB disk cache? Is there any evidence of that? In what circumstances does one benefit from such "large" values?

@FranciscoPombal commented on GitHub (Jul 7, 2020): @arvidn >>with appropriate guidance, and limited choices (e.g., Auto, 128 MB, 256 MB, 512 MB). >this is a nice idea, actually. @arvidn what would you say is reasonable maximum? Or do you need some data before any conclusion about this? Better questions: are there cases that benefit from > 512 MiB disk cache? Is there any evidence of that? In what circumstances does one benefit from such "large" values?

deekerman commented

Author

Owner

@arvidn commented on GitHub (Jul 7, 2020):

If your drive is slow, and your connection is fast, the size of the cache determines how large a torrent you can download at a high rate (and flush to disk later).

If you only start Torrent’s intermittently, such delayed flush may not be a problem.

@arvidn commented on GitHub (Jul 7, 2020): If your drive is slow, and your connection is fast, the size of the cache determines how large a torrent you can download at a high rate (and flush to disk later). If you only start Torrent’s intermittently, such delayed flush may not be a problem.

deekerman commented

Author

Owner

@JNavas2 commented on GitHub (Jul 7, 2020):

@arvidn

@JNavas2 I agree with you (except I would be careful generalising one person's experience to "most people"). ...

Although I'm not just generalizing from my own experience -- my comment was based on many years of professional industry experience and reports from industry colleagues and sources -- it's certainly not scientific, so point taken -- I've revised "most" to "many". 😊

If your drive is slow, and your connection is fast, the size of the cache determines how large a torrent you can download at a high rate (and flush to disk later).
If you only start Torrent’s intermittently, such delayed flush may not be a problem.

When it comes to writing, cache can help in 2 cases:

Files small enough to not overflow the cache
Write buffer during temporary storage access congestion

Both cases are better handled by the OS. It's not clear to me that any application write cache will actually be beneficial -- my own system does not appear to suffer any ill effects from disabling the application cache, but to be fair, my system is not typical, and I've not rigorously tested it across disparate test cases.

p.s. If my UI suggestion is adopted, I'd add Disabled to the choices.

@JNavas2 commented on GitHub (Jul 7, 2020): @arvidn > @JNavas2 I agree with you (except I would be careful generalising one person's experience to "most people"). ... Although I'm not just generalizing from my own experience -- my comment was based on many years of professional industry experience and reports from industry colleagues and sources -- it's certainly not scientific, so point taken -- I've revised "_most_" to "many". :blush: > If your drive is slow, and your connection is fast, the size of the cache determines how large a torrent you can download at a high rate (and flush to disk later). > If you only start Torrent’s intermittently, such delayed flush may not be a problem. When it comes to writing, cache can help in 2 cases: 1. Files small enough to _not overflow_ the cache 2. Write buffer during _temporary_ storage access congestion Both cases are _better handled by the OS_. It's not clear to me that _any_ application write cache will actually be beneficial -- my own system does not appear to suffer any ill effects from disabling the application cache, but to be fair, my system is not typical, and I've not rigorously tested it across disparate test cases. p.s. If my UI suggestion is adopted, I'd add Disabled to the choices.

deekerman commented

Author

Owner

@FranciscoPombal commented on GitHub (Jul 7, 2020):

@arvidn

Right, but at what point do we reach the point of diminishing returns, in most "common" cases? I would expect that most consumer-level machines/environments nowadays (i.e., HDD/SSD (SATA), sub 1 Gbps asymetrical download-biased network speed, 2-8 CPU cores, 1 or a few torrents running at a time) don't really "benefit" from caches > 512 MiB. Here's what I mean by that:

As seen in the somewhat recent issue with auto cache setting a too low value on Windows machines with > 16 GiB RAM, a too low value leads to bad performance overall, all the time - it becomes a generalized bottleneck. An "ideal" value is the minimum one that leads to good performance, with possibly a tapering off/slight degradation once the cache fills up. Higher values than the "ideal" value offer no better performance relative to the "ideal" value other than providing a greater "soak up time", where peak performance can be maintained for longer simply because the cache is a bit bigger, and thus takes longer to fill up - but, once it fills up, the performance level tapers off to the same degree it would do so as with the "ideal" value.

Most people care much more about having the minimum possible RAM usage (read: disk cache) that doesn't hobble performance, rather than a greater "soak up time". With this in mind, and the "typical consumer level setups" mentioned above, I suspect the range of "ideal values" is from 128 MiB to 512 MiB. With this in mind, I think it would be best from a UX perspective to have a drop-down that allowed the user to choose between auto, 128 MiB, 256 MiB, 512 MiB, or Custom (advanced). This is intended to help users choose the "most likely correct values". Custom (advanced) allows setting an arbitrary value and is intended to support the use cases where the user really benefits from values higher/lower than the default range. auto, as before, provides a "good enough" value that errs on the side of more "soak up time" for users with large amounts of system RAM.

What do you think? Do you think this assumption is reasonable:

I would expect that most consumer-level machines/environments nowadays (i.e., HDD/SSD (SATA), sub 1 Gbps asymetrical download-biased network speed, 2-8 CPU cores, 1 or a few torrents running at a time)

and, if so, that a drop-down with a choice between auto, 128 MiB, 256 MiB, 512 MiB, or Custom (advanced) is a good fit for such a model? Or would you use/add different values, like 1 GiB (or maybe even remove 512 MiB as being too high)?

@FranciscoPombal commented on GitHub (Jul 7, 2020): @arvidn Right, but at what point do we reach the point of diminishing returns, in most "common" cases? I would expect that most consumer-level machines/environments nowadays (i.e., HDD/SSD (SATA), sub 1 Gbps asymetrical download-biased network speed, 2-8 CPU cores, 1 or a few torrents running at a time) don't really "benefit" from caches > 512 MiB. Here's what I mean by that: As seen in the somewhat recent issue with `auto` cache setting a too low value on Windows machines with > 16 GiB RAM, a too low value leads to bad performance overall, all the time - it becomes a generalized bottleneck. An "ideal" value is the minimum one that leads to good performance, with possibly a tapering off/slight degradation once the cache fills up. Higher values than the "ideal" value offer no better performance relative to the "ideal" value other than providing a greater "soak up time", where peak performance can be maintained for longer simply because the cache is a bit bigger, and thus takes longer to fill up - but, once it fills up, the performance level tapers off to the same degree it would do so as with the "ideal" value. Most people care much more about having the minimum possible RAM usage (read: disk cache) that doesn't hobble performance, rather than a greater "soak up time". With this in mind, and the "typical consumer level setups" mentioned above, I suspect the range of "ideal values" is from 128 MiB to 512 MiB. With this in mind, I think it would be best from a UX perspective to have a drop-down that allowed the user to choose between `auto`, `128 MiB`, `256 MiB`, `512 MiB`, or `Custom (advanced)`. This is intended to help users choose the "most likely correct values". `Custom (advanced)` allows setting an arbitrary value and is intended to support the use cases where the user really benefits from values higher/lower than the default range. `auto`, as before, provides a "good enough" value that errs on the side of more "soak up time" for users with large amounts of system RAM. What do you think? Do you think this assumption is reasonable: > I would expect that most consumer-level machines/environments nowadays (i.e., HDD/SSD (SATA), sub 1 Gbps asymetrical download-biased network speed, 2-8 CPU cores, 1 or a few torrents running at a time) and, if so, that a drop-down with a choice between `auto`, `128 MiB`, `256 MiB`, `512 MiB`, or `Custom (advanced)` is a good fit for such a model? Or would you use/add different values, like `1 GiB` (or maybe even remove `512 MiB` as being too high)?

deekerman commented

Author

Owner

@FranciscoPombal commented on GitHub (Jul 7, 2020):

@JNavas2

Both cases are better handled by the OS.

I think this is consensual, which is why libtorrent >=2.0 does it like that, unlike 1.2 and below. Once 2.0 is released (there are RC releases available already) and qBittorrent supports it, this is how it will be done.

It's not clear to me that any application write cache will actually be beneficial -- my own system does not appear to suffer any ill effects from disabling the application cache, but to be fair, my system is not typical, and I've not rigorously tested it across disparate test cases.

From experience, it is beneficial to most users. You can always set it to a lower value, if you don't need the amount that auto sets for you.

@FranciscoPombal commented on GitHub (Jul 7, 2020): @JNavas2 > Both cases are _better handled by the OS_. I think this is consensual, which is why libtorrent >=2.0 does it like that, unlike 1.2 and below. Once 2.0 is released (there are RC releases available already) and qBittorrent supports it, this is how it will be done. >It's not clear to me that _any_ application write cache will actually be beneficial -- my own system does not appear to suffer any ill effects from disabling the application cache, but to be fair, my system is not typical, and I've not rigorously tested it across disparate test cases. From experience, it is beneficial to most users. You can always set it to a lower value, if you don't need the amount that `auto` sets for you.

deekerman commented

Author

Owner

@arvidn commented on GitHub (Jul 7, 2020):

and, if so, that a drop-down with a choice between auto, 128 MiB, 256 MiB, 512 MiB, or Custom (advanced) is a good fit for such a model? Or would you use/add different values, like 1 GiB (or maybe even remove 512 MiB as being too high)?

That seems reasonable. I think the lower end where a cache can be the most helpful is:

"large enough to fit all currently downloading (partially downloaded) pieces."

If the cache fits all those, it can run the SHA-1 hash over blocks in memory, without having to read it back from disk

@arvidn commented on GitHub (Jul 7, 2020): > and, if so, that a drop-down with a choice between auto, 128 MiB, 256 MiB, 512 MiB, or Custom (advanced) is a good fit for such a model? Or would you use/add different values, like 1 GiB (or maybe even remove 512 MiB as being too high)? That seems reasonable. I think the lower end where a cache can be the most helpful is: "large enough to fit all currently downloading (partially downloaded) pieces." If the cache fits all those, it can run the SHA-1 hash over blocks in memory, without having to read it back from disk

deekerman commented