starred/anything-llm

Fork 0

mirror of https://github.com/Mintplex-Labs/anything-llm.git synced 2026-03-02 22:57:05 -05:00

[lancedb] SQLITE_BUSY when embedding many documents #29

New issue

Closed

opened 2026-02-28 04:20:46 -05:00 by deekerman · 4 comments

deekerman commented

2026-02-28 04:20:46 -05:00

Owner

Originally created by @danglingptr0x0 on GitHub (Jun 13, 2023).

When embedding a larger number of docs (> 100 (>100k tokens)), the DB seems to lock itself during the process (using lancedb as vector db):

INSERT INTO document_vectors (docId, vectorId) VALUES ('f03e0ec4-1307-4928-b982-91a8a2e34563', '5a791ec8-f1be-4276-8928-5e129accc0d6')
{ result: 0 }
node:internal/process/promises:288
            triggerUncaughtException(err, true /* fromPromise */);
            ^

Error: SQLITE_BUSY: database is locked
--> in Statement#run([
  '4f09f8e7-b1a8-4c0b-82e0-bbde80473612',
  '05-26-2023-db4e845d-9c45-4d80-9581-73c53844132c.json',
  'custom-documents/05-26-2023-db4e845d-9c45-4d80-9581-73c53844132c.json',
  2,
  '{"id":"db4e845d-9c45-4d80-9581-73c53844132c","url":"file:///home/<username>/git/anything-llm/collector/hotdir/processed/05-26-2023.md","title":"05-26-2023.md","description":"a custom file uploaded by the user.","published":"2023-06-12 09:26:29","wordCount":12758,"token_count_estimate":2849}'
], [Function (anonymous)])
    at /home/<username>/git/anything-llm/server/node_modules/sqlite/build/Statement.js:80:23
    at new Promise (<anonymous>)
    at Statement.run (/home/<username>/git/anything-llm/server/node_modules/sqlite/build/Statement.js:78:16)
    at Object.addDocuments (/home/<username>/git/anything-llm/server/models/documents.js:89:12)
    at async /home/<username>/git/anything-llm/server/endpoints/workspaces.js:35:7 {
  errno: 5,
  code: 'SQLITE_BUSY',
  __augmented: true
}

Node.js v18.16.0
[nodemon] app crashed - waiting for file changes before starting...

Cannot tell if this is caused by the larger number of docs or if the issue is caused by token count (as in, the issue would be caused even with low doc but high token count).

Originally created by @danglingptr0x0 on GitHub (Jun 13, 2023). When embedding a larger number of docs (> 100 (>100k tokens)), the DB seems to lock itself during the process (using lancedb as vector db): ``` INSERT INTO document_vectors (docId, vectorId) VALUES ('f03e0ec4-1307-4928-b982-91a8a2e34563', '5a791ec8-f1be-4276-8928-5e129accc0d6') { result: 0 } node:internal/process/promises:288 triggerUncaughtException(err, true /* fromPromise */); ^ Error: SQLITE_BUSY: database is locked --> in Statement#run([ '4f09f8e7-b1a8-4c0b-82e0-bbde80473612', '05-26-2023-db4e845d-9c45-4d80-9581-73c53844132c.json', 'custom-documents/05-26-2023-db4e845d-9c45-4d80-9581-73c53844132c.json', 2, '{"id":"db4e845d-9c45-4d80-9581-73c53844132c","url":"file:///home/<username>/git/anything-llm/collector/hotdir/processed/05-26-2023.md","title":"05-26-2023.md","description":"a custom file uploaded by the user.","published":"2023-06-12 09:26:29","wordCount":12758,"token_count_estimate":2849}' ], [Function (anonymous)]) at /home/<username>/git/anything-llm/server/node_modules/sqlite/build/Statement.js:80:23 at new Promise (<anonymous>) at Statement.run (/home/<username>/git/anything-llm/server/node_modules/sqlite/build/Statement.js:78:16) at Object.addDocuments (/home/<username>/git/anything-llm/server/models/documents.js:89:12) at async /home/<username>/git/anything-llm/server/endpoints/workspaces.js:35:7 { errno: 5, code: 'SQLITE_BUSY', __augmented: true } Node.js v18.16.0 [nodemon] app crashed - waiting for file changes before starting... ``` Cannot tell if this is caused by the larger number of docs or if the issue is caused by token count (as in, the issue would be caused even with low doc but high token count).

deekerman

2026-02-28 04:20:46 -05:00

closed this issue
added the
question

bug
labels

deekerman commented

2026-02-28 04:21:08 -05:00

Author

Owner

@timothycarambat commented on GitHub (Jun 13, 2023):

connect #19

@timothycarambat commented on GitHub (Jun 13, 2023): connect #19

deekerman commented

2026-02-28 04:21:08 -05:00

Author

Owner

@timothycarambat commented on GitHub (Jun 13, 2023):

This should be resolved now with #19!

Can you pull in an try again? Was a concurrency issue with SQLite

@timothycarambat commented on GitHub (Jun 13, 2023): This should be resolved now with #19! Can you pull in an try again? Was a concurrency issue with SQLite

deekerman commented

2026-02-28 04:21:09 -05:00

Author

Owner

@danglingptr0x0 commented on GitHub (Jun 13, 2023):

This should be resolved now with #19!

Can you pull in an try again? Was a concurrency issue with SQLite

Will do! Thank you :-)

@danglingptr0x0 commented on GitHub (Jun 13, 2023): > This should be resolved now with #19! > > Can you pull in an try again? Was a concurrency issue with SQLite Will do! Thank you :-)

deekerman commented

2026-02-28 04:21:09 -05:00

Author

Owner

@danglingptr0x0 commented on GitHub (Jun 14, 2023):

Seems to have been fixed! Closing as complete ...

Thanks

@danglingptr0x0 commented on GitHub (Jun 14, 2023): Seems to have been fixed! Closing as complete ... Thanks

No milestone

No project

No assignees

1 participant

Notifications

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference

starred/anything-llm#29

No description provided.

Rows
Columns