mirror of
https://github.com/dani-garcia/vaultwarden.git
synced 2026-03-02 22:57:18 -05:00
Vault loading issues (attachments?) #1275
Labels
No labels
SSO
Third party
better for forum
bug
bug
documentation
duplicate
enhancement
future Vault
future Vault
future Vault
good first issue
help wanted
low priority
notes
question
troubleshooting
wontfix
No milestone
No project
No assignees
1 participant
Notifications
Due date
No due date set.
Dependencies
No dependencies set.
Reference
starred/vaultwarden#1275
Loading…
Add table
Add a link
Reference in a new issue
No description provided.
Delete branch "%!s()"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Originally created by @Decicus on GitHub (May 23, 2022).
Subject of the issue
After updating from 1.24.0 to 1.25.0, I am having issues loading my vault.
Error messages indicate issues with attachments (included error logs under "Troubleshooting data").
Other account settings seem to work fine (changing equivalent domains or updating 2FA settings still worked), so it seems to only affect the vault data.
No issues on 1.24.0. In fact, I could revert Vaultwarden back to 1.24.0 without restoring my database from backups and the vault seems to load just fine again. Syncing from the browser extension works fine too.
Deployment environment
Install method: Docker
Clients used: Web vault & official browser extension
MySQL/MariaDB or PostgreSQL version:
10.5.15-MariaDB-1:10.5.15+maria~focalSteps to reproduce
Docker Compose with the
latesttag, upgrade from 1.24.0 (pull, down, up)No other changes were done, except using
1.24.0instead of latest when reverting back.Log into web vault / sync with browser extension.
Actual behaviour
Web vault was empty with no entries. Browser extension (where previously logged in & synced) just returned a sync error.
Troubleshooting data
@tophers commented on GitHub (May 24, 2022):
Getting a similar rocket error in my build, 1.24.0 works, 1.25.0 builds but fails with same environment and configuration. Same error using teh compiled binary from the docker.io:/vaultwarden/server image.
I can open a separate issue if unrelated.
@BlackDex commented on GitHub (May 24, 2022):
Strange, that doesn't happen at all for me.
Does this happen on all accounts? On all clients?
Could you also post your
Support Stringplease?@BlackDex commented on GitHub (May 24, 2022):
@tophers that looks more like a
ROCKET_configuration issue, and not in any way related to this.Please create a new item regarding this.
@Decicus commented on GitHub (May 24, 2022):
I only have one account on my instance, otherwise I would've tested that yesterday.
I was going to test that now, but when I tried to change to
latestagain and restarting all the containers... it now seems fine? None of the errors I got yesterday and the vault loads again (I definitely tested restarting the containers yesterday, so not sure what exactly is different now).I'll be closing this, since I can't even reproduce it myself.
@jzahraoui commented on GitHub (May 25, 2022):
hello, I have the same issue. I understand completely, but it would be good to keep this issue opened until a solution is found.
for myself, after the upgrade from 1.24 to 1.25 the container won't start because of ROCKET_CLI_COLORS variable value.
details: https://github.com/SergioBenitez/Rocket/issues/1923
don't know if this help to reproduce.
after fixing that, the container start well, no errors into the log file. but when I try to access to url "/#/vault", the error message "Commands out of sync; you can't run this command now" appear into the log and no items is displayed on the web page.
IOS and android application not able to synchronize.
My log :
@BlackDex commented on GitHub (May 25, 2022):
@jzahraoui could you post your
Support Stringplease?@sunny5055 commented on GitHub (May 25, 2022):
I have exactly the same issue. reverting 1.24 resolved for now. Below is the support string.
Your environment (Generated via diagnostics page)
Config (Generated via diagnostics page)
Show Running Config
Environment settings which are overridden:
@jzahraoui commented on GitHub (May 25, 2022):
Hello I revert back to 1.24, but here is the support string. let me know if you want the support string for the 1.25
Your environment (Generated via diagnostics page)
Config (Generated via diagnostics page)
Show Running Config
Environment settings which are overridden:
@BlackDex commented on GitHub (May 26, 2022):
Ok, it looks like you all at running MariaDB.
Could you run the following and post the full output of the following queries on your databases.
Change
vaultwardento the database name you used.@PeterRob commented on GitHub (May 26, 2022):
MariaDB [vaultwarden]> SHOW CREATE TABLE
attachments;+-------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| Table | Create Table |
+-------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| attachments | CREATE TABLE
attachments(idchar(36) NOT NULL,cipher_uuidchar(36) NOT NULL,file_nametext NOT NULL,file_sizeint(11) NOT NULL,akeytext DEFAULT NULL,PRIMARY KEY (
id)) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 |
+-------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
1 row in set (0.001 sec)
MariaDB [vaultwarden]> SHOW CREATE DATABASE
vaultwarden;+-------------+-------------------------------------------------------------------------+
| Database | Create Database |
+-------------+-------------------------------------------------------------------------+
| vaultwarden | CREATE DATABASE
vaultwarden/*!40100 DEFAULT CHARACTER SET utf8mb4 */ |+-------------+-------------------------------------------------------------------------+
1 row in set (0.009 sec)
@BlackDex commented on GitHub (May 26, 2022):
I tried several times to cause this exact same issue on my environment but I'm not able to.
In my search to this issue i came across this issue: https://dba.stackexchange.com/a/183010.
So, it could be the database needs some repair/rebuild. That maybe could resolve the issue.
The answer above (https://dba.stackexchange.com/posts/285664/edit) is mentioning a memory issue, not sure if that could also be the case here.
Could you try to repair at least the attachment table, and maybe also the others if that isn't fixing it?
@PeterRob commented on GitHub (May 26, 2022):
I ran mysqlcheck on the vaultwarden database, the problem remains.
@BlackDex commented on GitHub (May 26, 2022):
@PeterRob what did you exactly run?
Did you run
mysqlcheck --all-databases --optimizefor example. Because that will rebuild/recreate innodb tables.Any other command will not do that.
@PeterRob commented on GitHub (May 26, 2022):
mysqlcheck -c vaultwarden -u root -pand testedmysqlcheck -u root -p -o vaultwarden@BlackDex commented on GitHub (May 26, 2022):
So, did you tested after the optimize?
Also, can you tell me how many attachments you have?
That is visible in the admin interface for both the users and the organizations.
@PeterRob commented on GitHub (May 26, 2022):
Yes I tested after optimisation
There are no attachments - both users and organizations.
@BlackDex commented on GitHub (May 26, 2022):
Strange. I'm really unable to reproduce this at all.
@BlackDex commented on GitHub (May 26, 2022):
Could you provide a bit more info on the host where this is running?
What kind of hardware, how many CPU cores, how much memory.
Because i tried using the exact same MariaDB, but that isn't causing an issue for me at all.
Also, what happens if you create a new user, does it happen for that user also?
What if you export the vault, import it into the new user and test the new Vaultwarden version again.
If i could cause this my self i wouldn't need to ask these questions, but unfortunately that is not the case.
@jzahraoui commented on GitHub (May 26, 2022):
is this can help : https://github.com/SergioBenitez/Rocket/issues/400 ?
@BlackDex commented on GitHub (May 26, 2022):
That is very very old, and both Rocket and Diesel are updated to newer version already.
I rather think it could have something to do with async, but that would also be a bit strange, also that is happens at the exact same spot for two different reports.
@jzahraoui commented on GitHub (May 26, 2022):
tried to investigate.
I've logged the query send to the database :
May be the se select have to much elements into the "IN" clause. counted 3213 elements. find some people saying the max is 1000 but don't have the clue.
put vaultwarden log level to trace. unfortunately not helping.
@jzahraoui commented on GitHub (May 26, 2022):
to compare the version 1.24.0 give this query trace :
@BlackDex commented on GitHub (May 26, 2022):
Yes, that has changed to improve sync.
Look at those answers its a configuration item on the server. https://stackoverflow.com/questions/4275640/mysql-in-condition-limit/4275704#4275704
There isn't a hardcoded limitation on the amount. I have tested 6000+ items during the development.
I would suggest to see if changing that
max_allowed_packetsoption would help.@jzahraoui commented on GitHub (May 26, 2022):
tried
max_allowed_packet=1Gbut it didn't help. I'm still searching.@sunny5055 commented on GitHub (May 27, 2022):
Have you tried creating an organization on 1.25 and check if you get the issue ?
@jzahraoui commented on GitHub (May 27, 2022):
yes I tried. It return the error when I try to create an organization. but when I revert back to 1.24 the organization appear.
Another thing, If I login with an other user, no errors at all, everything is working fine.
@BlackDex commented on GitHub (May 27, 2022):
@jzahraoui does that new user have more then 1000 items?
@jzahraoui commented on GitHub (May 27, 2022):
my user have 3143 items, it is not working.
the second user have only 44 items (it is working)
@BlackDex commented on GitHub (May 27, 2022):
How much memory does the system have which is running MariaDB?
Is it using the default Debian configuration?
@jzahraoui commented on GitHub (May 27, 2022):
Having 32gb of ram but running some other process. Turned some innodb parameters.
@PeterRob commented on GitHub (May 27, 2022):
I made a new user account and progressively added logins (plain logins, no attachments). Syncing the vault failed at 1,000 logins added to the account.
@BlackDex commented on GitHub (May 28, 2022):
It probably has something to do with
in_predicate_conversion_threshold. But in my setup that is also 1000, but even if i set that to 200 it will not break. Also, if i understood correctly, MariaDB uses that to try and optimize the query it self, but still allows larger values.The
max_allowed_packetvalue, if set to low, generates a totally different error, so that is probably not the issue here.It would be interesting to see what happens if you put the
in_predicate_conversion_thresholdto 4000 or something to see what that does.Also see: https://mariadb.com/kb/en/conversion-of-big-in-predicates-into-subqueries/
@PeterRob commented on GitHub (May 28, 2022):
Setting in_predicate_conversion_threshold to 4000 = success!
@BlackDex commented on GitHub (May 29, 2022):
Good to see that that is a working solution. Still strange, as i understand the documentation correctly that MariaDB only uses that value to split the IN() clause into multiple IN() clauses or something like that. And me having it set to 200, and having 6000+ entries in there does work without any issue.
I'm going to see if i can figure out how i can trigger this my self, maybe it's a bug in the specific MariaDB version used by Debian, that is something i did not try yet.
@BlackDex commented on GitHub (Jun 4, 2022):
I tried and tried, and nothing seemed to break on my testing environments what ever i tried.
Low memory, low CPU etc..
Then i went to ask for some help on IRC from the MariaDB Devs, and they mentioned this was a bug in the exact version you are using. It seems that both Ubuntu and Debian didn't updated there packages yet, or cherry-picked that fix into there builds.
It is fixed in
v10.5.16of MariaDB.For some more information see the following links:
github.com/MariaDB/server@e048289e55I'm afraid that we can't fix this on the client (Vaultwarden) side.
So, what they suggest is to set
in_predicate_conversion_threshold=0.By setting it to
0it will behave as MariaDB did before they implemented this feature.Or try to update your version to
v10.5.16or higher.Because of this, i will close this issue.