starred/audiobookshelf-advplyr
1
0
Fork 0
mirror of https://github.com/advplyr/audiobookshelf.git synced 2026-03-02 22:46:55 -05:00
Code Issues 853 Projects Packages Wiki Activity

[Bug]: Slow file upload/processing speed with large library (~54k items) despite optimal config #3202

New issue
Open
opened 2026-02-20 11:01:25 -05:00 by deekerman · 4 comments
deekerman commented 2026-02-20 11:01:25 -05:00
Owner
Copy link

Originally created by @piotr1alien-art on GitHub (Jan 25, 2026).

What happened?

File upload/processing caps at ~3-3,5 MB/s. Diagnostic steps show this is not due to CPU (Zlib off), disk I/O (iostat shows low %util), network, or reverse proxy. The bottleneck appears to be the high volume of small database I/O operations generated per file in a large library.

What did you expect to happen?

File upload/processing caps at min~16-18 MB/s.

Steps to reproduce the issue

  1. find /home/Audiobook/ | wc -l 54649

Audiobookshelf version

ABS v2.30.0/2.32.1, Debian 12, AB_ZLIB_LEVEL=0, SQLite DB on NVMe (WAL mode), library on SSD RAID.

How are you running audiobookshelf?

Debian/PPA

What OS is your Audiobookshelf server hosted from?

Linux

If the issue is being seen in the UI, what browsers are you seeing the problem on?

Firefox

Logs

Total DISK READ:         0.00 B/s | Total DISK WRITE:       237.84 K/s
Current DISK READ:       0.00 B/s | Current DISK WRITE:       8.92 K/s
    PID  PRIO  USER     DISK READ DISK WRITE>    COMMAND                                                                        
2826431 be/4 audioboo      4.00 M     12.48 M audiobookshelf

Additional Notes

No response

Originally created by @piotr1alien-art on GitHub (Jan 25, 2026). ### What happened? File upload/processing caps at ~3-3,5 MB/s. Diagnostic steps show this is not due to CPU (Zlib off), disk I/O (iostat shows low %util), network, or reverse proxy. The bottleneck appears to be the high volume of small database I/O operations generated per file in a large library. ### What did you expect to happen? File upload/processing caps at min~16-18 MB/s. ### Steps to reproduce the issue 1. find /home/Audiobook/ | wc -l 54649 ### Audiobookshelf version ABS v2.30.0/2.32.1, Debian 12, `AB_ZLIB_LEVEL=0`, SQLite DB on NVMe (WAL mode), library on SSD RAID. ### How are you running audiobookshelf? Debian/PPA ### What OS is your Audiobookshelf server hosted from? Linux ### If the issue is being seen in the UI, what browsers are you seeing the problem on? Firefox ### Logs ```shell Total DISK READ: 0.00 B/s | Total DISK WRITE: 237.84 K/s Current DISK READ: 0.00 B/s | Current DISK WRITE: 8.92 K/s PID PRIO USER DISK READ DISK WRITE> COMMAND 2826431 be/4 audioboo 4.00 M 12.48 M audiobookshelf ``` ### Additional Notes _No response_
deekerman commented 2026-02-20 11:01:26 -05:00
Author
Owner
Copy link

@piotr1alien-art commented on GitHub (Jan 25, 2026):

2026-01-25 14:39:46.440

FATAL

[Server] Unhandled rejection: Error at Database. (/snapshot/audiobookshelf/node_modules/sequelize/lib/dialects/sqlite/query.js:185:27) at /snapshot/audiobookshelf/node_modules/sequelize/lib/dialects/sqlite/query.js:183:50 at new Promise () at Query.run (/snapshot/audiobookshelf/node_modules/sequelize/lib/dialects/sqlite/query.js:183:12) at /snapshot/audiobookshelf/node_modules/sequelize/lib/sequelize.js:315:28 at async SQLiteQueryInterface.bulkInsert (/snapshot/audiobookshelf/node_modules/sequelize/lib/dialects/abstract/query-interface.js:346:21) at async recursiveBulkCreate (/snapshot/audiobookshelf/node_modules/sequelize/lib/model.js:1697:25) at async bookAuthor.bulkCreate (/snapshot/audiobookshelf/node_modules/sequelize/lib/model.js:1786:12) at async ApiRouter.update (/snapshot/audiobookshelf/server/controllers/AuthorController.js) { name: 'SequelizeUniqueConstraintError', errors: [ ValidationErrorItem { message: 'bookId must be unique', type: 'unique violation', path: 'bookId', value: null, origin: 'DB', instance: null, validatorKey: 'not_unique', validatorName: null, validatorArgs: [] }, ValidationErrorItem { message: 'authorId must be unique', type: 'unique violation', path: 'authorId', value: null, origin: 'DB', instance: null, validatorKey: 'not_unique', validatorName: null, validatorArgs: [] } ], parent: [Error: SQLITE_CONSTRAINT: UNIQUE constraint failed: bookAuthors.bookId, bookAuthors.authorId] { errno: 19, code: 'SQLITE_CONSTRAINT', sql: "INSERT INTO bookAuthors (id,createdAt,bookId,authorId) VALUES ('73db31a5-3de8-4551-86ef-22c1fccd5d6b','2026-01-25 13:39:46.427 +00:00','5a32c81b-5e37-4fa8-8d0c-7b40c929c245','c4c746a4-7f3e-43a3-9761-e3aa46e6c51a');" }, original: [Error: SQLITE_CONSTRAINT: UNIQUE constraint failed: bookAuthors.bookId, bookAuthors.authorId] { errno: 19, code: 'SQLITE_CONSTRAINT', sql: "INSERT INTO bookAuthors (id,createdAt,bookId,authorId) VALUES ('73db31a5-3de8-4551-86ef-22c1fccd5d6b','2026-01-25 13:39:46.427 +00:00','5a32c81b-5e37-4fa8-8d0c-7b40c929c245','c4c746a4-7f3e-43a3-9761-e3aa46e6c51a');" }, fields: [ 'bookId', 'authorId' ], sql: "INSERT INTO bookAuthors (id,createdAt,bookId,authorId) VALUES ('73db31a5-3de8-4551-86ef-22c1fccd5d6b','2026-01-25 13:39:46.427 +00:00','5a32c81b-5e37-4fa8-8d0c-7b40c929c245','c4c746a4-7f3e-43a3-9761-e3aa46e6c51a');" } promise: Promise { Error at Database. (/snapshot/audiobookshelf/node_modules/sequelize/lib/dialects/sqlite/query.js:185:27) at /snapshot/audiobookshelf/node_modules/sequelize/lib/dialects/sqlite/query.js:183:50 at new Promise () at Query.run (/snapshot/audiobookshelf/node_modules/sequelize/lib/dialects/sqlite/query.js:183:12) at /snapshot/audiobookshelf/node_modules/sequelize/lib/sequelize.js:315:28 at async SQLiteQueryInterface.bulkInsert (/snapshot/audiobookshelf/node_modules/sequelize/lib/dialects/abstract/query-interface.js:346:21) at async recursiveBulkCreate (/snapshot/audiobookshelf/node_modules/sequelize/lib/model.js:1697:25) at async bookAuthor.bulkCreate (/snapshot/audiobookshelf/node_modules/sequelize/lib/model.js:1786:12) at async ApiRouter.update (/snapshot/audiobookshelf/server/controllers/AuthorController.js) { name: 'SequelizeUniqueConstraintError', errors: [ [ValidationErrorItem], [ValidationErrorItem] ], parent: [Error: SQLITE_CONSTRAINT: UNIQUE constraint failed: bookAuthors.bookId, bookAuthors.authorId] { errno: 19, code: 'SQLITE_CONSTRAINT', sql: "INSERT INTO bookAuthors (id,createdAt,bookId,authorId) VALUES ('73db31a5-3de8-4551-86ef-22c1fccd5d6b','2026-01-25 13:39:46.427 +00:00','5a32c81b-5e37-4fa8-8d0c-7b40c929c245','c4c746a4-7f3e-43a3-9761-e3aa46e6c51a');" }, original: [Error: SQLITE_CONSTRAINT: UNIQUE constraint failed: bookAuthors.bookId, bookAuthors.authorId] { errno: 19, code: 'SQLITE_CONSTRAINT', sql: "INSERT INTO bookAuthors (id,createdAt,bookId,authorId) VALUES ('73db31a5-3de8-4551-86ef-22c1fccd5d6b','2026-01-25 13:39:46.427 +00:00','5a32c81b-5e37-4fa8-8d0c-7b40c929c245','c4c746a4-7f3e-43a3-9761-e3aa46e6c51a');" }, fields: [ 'bookId', 'authorId' ], sql: "INSERT INTO bookAuthors (id,createdAt,bookId,authorId) VALUES ('73db31a5-3de8-4551-86ef-22c1fccd5d6b','2026-01-25 13:39:46.427 +00:00','5a32c81b-5e37-4fa8-8d0c-7b40c929c245','c4c746a4-7f3e-43a3-9761-e3aa46e6c51a');" } }

@piotr1alien-art commented on GitHub (Jan 25, 2026): 2026-01-25 14:39:46.440 FATAL [Server] Unhandled rejection: Error at Database.<anonymous> (/snapshot/audiobookshelf/node_modules/sequelize/lib/dialects/sqlite/query.js:185:27) at /snapshot/audiobookshelf/node_modules/sequelize/lib/dialects/sqlite/query.js:183:50 at new Promise (<anonymous>) at Query.run (/snapshot/audiobookshelf/node_modules/sequelize/lib/dialects/sqlite/query.js:183:12) at /snapshot/audiobookshelf/node_modules/sequelize/lib/sequelize.js:315:28 at async SQLiteQueryInterface.bulkInsert (/snapshot/audiobookshelf/node_modules/sequelize/lib/dialects/abstract/query-interface.js:346:21) at async recursiveBulkCreate (/snapshot/audiobookshelf/node_modules/sequelize/lib/model.js:1697:25) at async bookAuthor.bulkCreate (/snapshot/audiobookshelf/node_modules/sequelize/lib/model.js:1786:12) at async ApiRouter.update (/snapshot/audiobookshelf/server/controllers/AuthorController.js) { name: 'SequelizeUniqueConstraintError', errors: [ ValidationErrorItem { message: 'bookId must be unique', type: 'unique violation', path: 'bookId', value: null, origin: 'DB', instance: null, validatorKey: 'not_unique', validatorName: null, validatorArgs: [] }, ValidationErrorItem { message: 'authorId must be unique', type: 'unique violation', path: 'authorId', value: null, origin: 'DB', instance: null, validatorKey: 'not_unique', validatorName: null, validatorArgs: [] } ], parent: [Error: SQLITE_CONSTRAINT: UNIQUE constraint failed: bookAuthors.bookId, bookAuthors.authorId] { errno: 19, code: 'SQLITE_CONSTRAINT', sql: "INSERT INTO `bookAuthors` (`id`,`createdAt`,`bookId`,`authorId`) VALUES ('73db31a5-3de8-4551-86ef-22c1fccd5d6b','2026-01-25 13:39:46.427 +00:00','5a32c81b-5e37-4fa8-8d0c-7b40c929c245','c4c746a4-7f3e-43a3-9761-e3aa46e6c51a');" }, original: [Error: SQLITE_CONSTRAINT: UNIQUE constraint failed: bookAuthors.bookId, bookAuthors.authorId] { errno: 19, code: 'SQLITE_CONSTRAINT', sql: "INSERT INTO `bookAuthors` (`id`,`createdAt`,`bookId`,`authorId`) VALUES ('73db31a5-3de8-4551-86ef-22c1fccd5d6b','2026-01-25 13:39:46.427 +00:00','5a32c81b-5e37-4fa8-8d0c-7b40c929c245','c4c746a4-7f3e-43a3-9761-e3aa46e6c51a');" }, fields: [ 'bookId', 'authorId' ], sql: "INSERT INTO `bookAuthors` (`id`,`createdAt`,`bookId`,`authorId`) VALUES ('73db31a5-3de8-4551-86ef-22c1fccd5d6b','2026-01-25 13:39:46.427 +00:00','5a32c81b-5e37-4fa8-8d0c-7b40c929c245','c4c746a4-7f3e-43a3-9761-e3aa46e6c51a');" } promise: Promise { <rejected> Error at Database.<anonymous> (/snapshot/audiobookshelf/node_modules/sequelize/lib/dialects/sqlite/query.js:185:27) at /snapshot/audiobookshelf/node_modules/sequelize/lib/dialects/sqlite/query.js:183:50 at new Promise (<anonymous>) at Query.run (/snapshot/audiobookshelf/node_modules/sequelize/lib/dialects/sqlite/query.js:183:12) at /snapshot/audiobookshelf/node_modules/sequelize/lib/sequelize.js:315:28 at async SQLiteQueryInterface.bulkInsert (/snapshot/audiobookshelf/node_modules/sequelize/lib/dialects/abstract/query-interface.js:346:21) at async recursiveBulkCreate (/snapshot/audiobookshelf/node_modules/sequelize/lib/model.js:1697:25) at async bookAuthor.bulkCreate (/snapshot/audiobookshelf/node_modules/sequelize/lib/model.js:1786:12) at async ApiRouter.update (/snapshot/audiobookshelf/server/controllers/AuthorController.js) { name: 'SequelizeUniqueConstraintError', errors: [ [ValidationErrorItem], [ValidationErrorItem] ], parent: [Error: SQLITE_CONSTRAINT: UNIQUE constraint failed: bookAuthors.bookId, bookAuthors.authorId] { errno: 19, code: 'SQLITE_CONSTRAINT', sql: "INSERT INTO `bookAuthors` (`id`,`createdAt`,`bookId`,`authorId`) VALUES ('73db31a5-3de8-4551-86ef-22c1fccd5d6b','2026-01-25 13:39:46.427 +00:00','5a32c81b-5e37-4fa8-8d0c-7b40c929c245','c4c746a4-7f3e-43a3-9761-e3aa46e6c51a');" }, original: [Error: SQLITE_CONSTRAINT: UNIQUE constraint failed: bookAuthors.bookId, bookAuthors.authorId] { errno: 19, code: 'SQLITE_CONSTRAINT', sql: "INSERT INTO `bookAuthors` (`id`,`createdAt`,`bookId`,`authorId`) VALUES ('73db31a5-3de8-4551-86ef-22c1fccd5d6b','2026-01-25 13:39:46.427 +00:00','5a32c81b-5e37-4fa8-8d0c-7b40c929c245','c4c746a4-7f3e-43a3-9761-e3aa46e6c51a');" }, fields: [ 'bookId', 'authorId' ], sql: "INSERT INTO `bookAuthors` (`id`,`createdAt`,`bookId`,`authorId`) VALUES ('73db31a5-3de8-4551-86ef-22c1fccd5d6b','2026-01-25 13:39:46.427 +00:00','5a32c81b-5e37-4fa8-8d0c-7b40c929c245','c4c746a4-7f3e-43a3-9761-e3aa46e6c51a');" } }
deekerman commented 2026-02-20 11:01:26 -05:00
Author
Owner
Copy link

@piotr1alien-art commented on GitHub (Jan 25, 2026):

SQLite version 3.50.1 2025-06-06 14:52:32
Enter ".help" for usage hints.
sqlite> SELECT * FROM bookAuthors WHERE bookId='5a32c81b-5e37-4fa8-8d0c-7b40c929c245' AND authorId='c4c746a4-7f3e-43a3-9761-e3aa46e6c51a';
699899b5-7fcf-4e61-8473-2e48af7d4e36|2025-04-30 12:08:42.760 +00:00|5a32c81b-5e37-4fa8-8d0c-7b40c929c245|c4c746a4-7f3e-43a3-9761-e3aa46e6c51a
sqlite> SELECT bookId, authorId, COUNT() as count
FROM bookAuthors
GROUP BY bookId, authorId
HAVING COUNT() > 1; ...> ...> ...>
sqlite> .exit

@piotr1alien-art commented on GitHub (Jan 25, 2026): SQLite version 3.50.1 2025-06-06 14:52:32 Enter ".help" for usage hints. sqlite> SELECT * FROM bookAuthors WHERE bookId='5a32c81b-5e37-4fa8-8d0c-7b40c929c245' AND authorId='c4c746a4-7f3e-43a3-9761-e3aa46e6c51a'; 699899b5-7fcf-4e61-8473-2e48af7d4e36|2025-04-30 12:08:42.760 +00:00|5a32c81b-5e37-4fa8-8d0c-7b40c929c245|c4c746a4-7f3e-43a3-9761-e3aa46e6c51a sqlite> SELECT bookId, authorId, COUNT(*) as count FROM bookAuthors GROUP BY bookId, authorId HAVING COUNT(*) > 1; ...> ...> ...> sqlite> .exit
deekerman commented 2026-02-20 11:01:26 -05:00
Author
Owner
Copy link

@nichwall commented on GitHub (Jan 25, 2026):

I'm confused, is this with scanning a brand new library of 54000 books or uploading a single book to an existing 54000 book library?

@nichwall commented on GitHub (Jan 25, 2026): I'm confused, is this with scanning a brand new library of 54000 books or uploading a single book to an existing 54000 book library?
deekerman commented 2026-02-20 11:01:26 -05:00
Author
Owner
Copy link

@piotr1alien-art commented on GitHub (Jan 25, 2026):

In my case, it's about uploading/updating a single book to an already existing large library (~54k items).

It seems like a race condition or a lack of proper "upsert" logic. I’ve caught a SequelizeUniqueConstraintError in the logs. Even though the system resources are underutilized, the server throws a FATAL error because it tries to INSERT a bookId and authorId pair that already exists in the SQLite database, instead of ignoring or updating it.

Here is the specific error from my logs: SQLITE_CONSTRAINT: UNIQUE constraint failed: bookAuthors.bookId, bookAuthors.authorId

When I checked the database manually: SELECT * FROM bookAuthors WHERE bookId='...' AND authorId='...'; The entry was already there (created months ago), but the upload process tried to recreate it with a new ID, crashing the server.

@piotr1alien-art commented on GitHub (Jan 25, 2026): In my case, it's about uploading/updating a single book to an already existing large library (~54k items). It seems like a race condition or a lack of proper "upsert" logic. I’ve caught a SequelizeUniqueConstraintError in the logs. Even though the system resources are underutilized, the server throws a FATAL error because it tries to INSERT a bookId and authorId pair that already exists in the SQLite database, instead of ignoring or updating it. Here is the specific error from my logs: SQLITE_CONSTRAINT: UNIQUE constraint failed: bookAuthors.bookId, bookAuthors.authorId When I checked the database manually: SELECT * FROM bookAuthors WHERE bookId='...' AND authorId='...'; The entry was already there (created months ago), but the upload process tried to recreate it with a new ID, crashing the server.
Sign in to join this conversation.
No Branch/Tag specified
Branches Tags
master
book_tags_genres_dedupe
episode_download_fallback
Issue-4540-SortBy-StartedDate-and-FinishedDate
episode_meta_tagging
fix_authorize_race_condition
redirect_transcode_requests
progress_updated_sort
fix_ereader_socket_event
fix_change_empty_root_password
fix_podcast_session_track_index
fix_set_token
session_modal_user
localize_durations
fix_oidc_create_user
jwt_auth_refactor
fix_scanner_deleting_single_file_books
fix_mediaprogress_updatedat_2
experimental_next_client
podcast_episode_duration
episode-timestamps-clickable
book_author_secondary_sort_title
podcast_useragents
pathexists_user_access
fix_pathexists_join
book_author_secondary_sort
clean_duplicate_mediaprogress
sanitize_html_description
trix_prevent_attachments
check_path_api_fix
fix_mediaprogress_updatedat
increase_express_json_limit
fix_dockerfile_nunicode
search_episodes
audiobook_tools_update
episode_secondary_sorts
hls_stream_url_update
new_session_track_endpoint
audiobook_tools_enhancements
watcher_rescans_update
player_track_tooltip
fix_exclude_prefixes_crash
socket_item_events
fix_podcast_episode_scanner_promise
new_stats_controller
count_cache_for_userpermissions
parsing-opf-v3
validate_migration_files
fix-quick-match-all-crash
fix-chapter-end-sleep-timer
stringify_sequelize_query
remove-col-ambiguity
fix_next_prev_edit_description
details_trim_whitespace
fix_content_url_basepath
fix_logger_fatal
progress_bar_visibility
batch-edit-populate-map-details
feed_generator_updates
bookmark-modal-updates
migrate-library-item-in-scanner
migrate-new-library-items
migrate-podcasts-new-library-item-2
migrate-podcasts-new-library-item
fix-remove-episode-from-playlist
playback-session-use-new-library-item
refactor-library-item
fix-heatmap-caption
feed-episodes-upsert
share-media-player-media-session-api
remove-old-playlist
remove_old_collection_object
plugin-implementation-demo
feed_migration
refactor-feeds-from-item
fix_remove_authors_no_books
v2.17.3-fk-constraints-migration
migrations-first-upgrade
sqlite_2
feature/nuxt-target-server
waveform
sqlite
playlists
video
v2.32.1
v2.32.0
v2.31.0
v2.30.0
v2.29.0
v2.28.0
v2.27.0
v2.26.3
v2.26.2
v2.26.1
v2.26.0
v2.25.1
v2.25.0
v2.24.0
v2.23.0
v2.22.0
v2.21.0
v2.20.0
v2.19.5
v2.19.4
v2.19.3
v2.19.2
v2.19.1
v2.19.0
v2.18.1
v2.18.0
v2.17.7
v2.17.6
v2.17.5
v2.17.4
v2.17.3
v2.17.2
v2.17.1
v2.17.0
v2.16.2
v2.16.1
v2.16.0
v2.15.1
v2.15.0
v2.14.0
v2.13.4
v2.13.3
v2.13.2
v2.13.1
v2.13.0
v2.12.3
v2.12.2
v2.12.1
v2.12.0
v2.11.0
v2.10.1
v2.10.0
v2.9.0
v2.8.1
v2.8.0
v2.7.2
v2.7.1
v2.7.0
v2.6.0
v2.5.0
v2.4.4
v2.4.3
v2.4.2
v2.4.1
v2.4.0
v2.3.5
v2.3.4
v2.3.3
v2.3.2
v2.3.1
v2.3.0
v2.2.23
v2.2.22
v2.2.21
v2.2.20
v2.2.19
v2.2.18
v2.2.17
v2.2.16
v2.2.15
v2.2.14
v2.2.13
v2.2.12
v2.2.11
v2.2.10
v2.2.9
v2.2.8
v2.2.7
v2.2.6
v2.2.5
v2.2.4
v2.2.3
v2.2.2
v2.2.1
v2.2.0
v2.1.5
v2.1.4
v2.1.3
v2.1.2
v2.1.1
v2.1.0
v2.0.24
v2.0.23
v2.0.22
v2.0.21
v2.0.20
v2.0.19
v2.0.18
v2.0.17
v2.0.16
v2.0.15
v2.0.14
v2.0.13
v2.0.12
v2.0.11
v2.0.10
v2.0.9
v2.0.8
v2.0.7
v2.0.6
v2.0.5
v2.0.4
v2.0.3
v2.0.2
v2.0.1
v1.7.2
v1.7.1
v1.7.0
v1.6.0
v1.5.5
v1.5.0
v1.4.11
v1.4.9
v1.4.7
v1.4.6
v1.4.4
v1.4.2
v1.4.1
v1.4.0
v1.3.4
v1.3.3
v1.3.1
v1.2.8
v1.2.6
v1.2.5
v1.2.4
v1.2.1
v1.1.15
v1.1.14
v1.1.13
v1.1.12
v1.1.11
v1.1.10
v1.1.9
v1.1.8
v1.0.0
0.9.61-beta.0
0.9.61-beta
Labels
Clear labels   
authentication

   
awaiting release

   
backlog

   
bug

   
chapter editor

   
config-issue

   
ebooks

   
encoding/embedding

   
enhancement

   
help wanted

   
listening sessions & progress

   
planned

   
possible plugin

   
progress sync

   
sorting/filtering/searching

   
unable to reproduce

   
upload

   
users & permissions

   
waiting
No labels
authentication
awaiting release
backlog
bug
chapter editor
config-issue
ebooks
encoding/embedding
enhancement
help wanted
listening sessions & progress
planned
possible plugin
progress sync
sorting/filtering/searching
unable to reproduce
upload
users & permissions
waiting
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
starred/audiobookshelf-advplyr#3202
No description provided.