mirror of
https://github.com/advplyr/audiobookshelf.git
synced 2026-03-02 22:46:56 -05:00
[Enhancement]: Advanced search #1516
Labels
No labels
authentication
awaiting release
backlog
bug
chapter editor
config-issue
ebooks
encoding/embedding
enhancement
help wanted
listening sessions & progress
planned
possible plugin
progress sync
sorting/filtering/searching
unable to reproduce
upload
users & permissions
waiting
No milestone
No project
No assignees
1 participant
Notifications
Due date
No due date set.
Dependencies
No dependencies set.
Reference
starred/audiobookshelf#1516
Loading…
Add table
Add a link
Reference in a new issue
No description provided.
Delete branch "%!s()"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Originally created by @fivestones on GitHub (Oct 8, 2023).
Describe the feature/enhancement
It would be great to have an advanced search, where you could use boolean logic on search fields, and search for things in certain fields.
For example, if I wanted to search for books with the tag "children's books" and the genre "non-fiction" but not the genre "picture books". Or if I wanted to show books that were of the Genre "non-fiction" but not "politics".
Maybe I want to search for books published before 1980 that are in the genre "philosophy" which have an audiobook version.
So basically, the ability to search for one or more tags, one or more genres, and search specifically in any other field, and keep certain genres or tags out of the results. Basically, if it's in the metadata, I want to be able to find it.
Calibre and Calibre-web both allow these kinds of searches, and they are very useful.
Other things that would be nice would be things only show results that have audiobooks with attached pdf files, or only show results without (or with) a cover, or books not marked as explicit, etc.
@vedosis commented on GitHub (Jun 19, 2025):
Howdy folks!
This particular set of issues (sorting/filtering/searching) has been a growing frustration for me and I'd love to start investing in the solution.
If there's an existing plan, point me to it and I'll contribute on existing asks, if not, where should we start?
It seems like this is a big enough change I think it's worth aligning on the overall architectural changes before going down a path antithetical to @advplyr's machinations.
As a general outline/process, I'm thinking it would be easy to use an ElasticSearch instance (sticking with the ABS container motif) which would give us all sorts of fun goodies "out of the box" like fuzzy search, more_like_this queries, and kNN / Embeddings for LLM friendliness.
I also think it would also be worth having an API shim between the search platform (ES) and ABS so the search tech can be changed out if es.io looses their mind (i.e. changes the licensing) or some other better tech comes along in the future and we want to swap out. We can add a config into ABS's admin interfaces to specify the Search Service's endpoint allowing evolution of the two platforms somewhat independently.
Thoughts? Maybe this is worth an RFC PR to start discussing? Start defining Search API schemata that satisfy the current open issues?
@advplyr commented on GitHub (Jun 19, 2025):
Hey, do you know of a solution that works with SQLite?
@vedosis commented on GitHub (Jun 19, 2025):
Nope. But I'll look around and see what I can dig up and return and report.
Am I correct in inferring you'd rather lean on the existing data store rather than replicating the data into a more specialized search system? (You don't want to use SQLite for application logic / metadata and a search backend for only search operations, you'd rather just to SQLite to rule them all and in the darkness bind them.)
I haven't dug too deeply into your DAL, but it's usually fairly straight forward to handle dual-writes within an ORM. Either that or we can use a periodic task that pushes changes to the search system. Because backfill and search cache purging would need to be supported, an async vacuum task wouldn't be much more of a lift to implement.
@vedosis commented on GitHub (Jun 19, 2025):
Well, looks like fuzzy searching can be handled with FTS. And it looks pretty fast. I'd have to experiment a little to see if FTS5 is already available with the container we're using and see if "fox" == "phox" and the like.
But... I'm thinking about the output and I feel like if we just use the current models without adding additional important relationship data, we're not going to get meaningful results. For example, if I ask, "what books are similar to Name of the Wind?" I would expect to see the other books from The Kingkiller Chronicle and Patrick Rothfuss as well as other fantasy books. But because we store those in relationship tables (the author's name isn't part of the book's model because we have to traverse the table relationship between the book and the author) we'll probably not see that in the output.
Which kind of brings us back to my original proposal. We need a single document to represent all the searchable data to evaluate if an entity is similar to another. I think we'd have to pick from:
sqlite-vecas a way to store vectors.It's all trade-offs for sure.
The boolean searches recommended in this specific issue are mostly a UI and query builder problem, so it doesn't matter what the backend is. I'm fairly certain we can make that do anything we want. OpenSearch has some good recommendations on how to structure those types of queries.
Open Issue Evaluation
COLLATE LOCALIZEDand allowing users to specify their target language in the system settings.Conclusion
I think this can be done in SQLite, but I think, regardless of the implementation, it should be separate document tables, not virtual tables, which means the level of effort between using SQLite and ElasticSearch is actually lower on the ElasticSearch side just because all these features come built in. If we stick to SQLite, I think it would be a good idea to have a separate SQLite db specific to search so cross library searching is easier. The biggest downside to ES is the cognitive cost of "yet another system" we have to interact with, support, build, maintain, etc., but I think that's somewhat ameliorated because of the extensive work es.io has put into the ES container and getting things to work with simple configs and that the data stored in the search cache would be considered ephemeral and highly volatile. If it gets into a weird state, blow it away and rebuild the cache.
How do you want to proceed?
@advplyr commented on GitHub (Jun 19, 2025):
One issue is that we need to be able to build Debian and Windows binaries. Docker is just the primary way of running Abs.
Can ElasticSearch be a single binary like SQLite?
@vedosis commented on GitHub (Jun 19, 2025):
It's absolutely multi-platform. It's a Java binary.
https://www.elastic.co/downloads/elasticsearch
@vedosis commented on GitHub (Jun 27, 2025):
Checking in here. Where would you like me to start?
@advplyr commented on GitHub (Jun 27, 2025):
I haven't used elastic search and haven't had a chance to look at it yet. It would be great if you could map out how you think this can be implemented.
I may be completely misunderstanding how this would work so let me know if these implementation thoughts are off-base.
titlecolumn on thelibraryItemstable up-to-date (here)@vedosis commented on GitHub (Jun 27, 2025):
No problem. I'll put that together in a bit.
Can you give me some data size counts you've seen? I can use mine, but with only 1.2k books I can't imagine I'm a good representation of a super user.
@advplyr commented on GitHub (Jun 27, 2025):
I've seen reported around 100k audiobooks which if they are multi-file could be over a million audio files. I'm not sure if we would be including filenames as something searchable. That is an outlier though.
A super-user is probably 10k audiobooks and up.
Some users have reported very large podcast collections with 100k+ episodes
@nichwall commented on GitHub (Jun 27, 2025):
What are the minimum system requirements for ElasticSearch? Taking a look at their website it looks like 8GB of RAM (among other high minimums) which is very heavy compared to audiobookshelf. A number of people use Raspberry Pi or older machines for their servers, and that seems excessive just for searching.
@Vito0912 commented on GitHub (Jun 27, 2025):
Coming from a discussion on GitHub, I also want to bring up some concerns about performance and resource use.
The last time I used ElasticSearch for a database project, it was not very light on resources. It was not as heavy as some other search engine solutions, but it was still not very good.
The 8GB mentioned above, I think, is just for their enterprise "tier." Their own guide says 1GB is the starting point, which would not be too bad. However, ABS uses less than 200MB while running for me. So 1 GB is..... But still, 8GB is recommended (altough not required) for production use that's true.
Also, I am not sure if SQLite supports it, but a basic better search could be done with FTS or vector search (I think that was also mentioned in the thread here).
I like the idea of a search engine, as I use Meilisearch (which has even more RAM usage then ES xD) in one of my projects, and in general, search engines are great. But they would make ABS much less lightweight and do not fit in ABS.
Also, search would probably become even slower because it would need to go to Elastic and then to the database.
Edit: And not to mention that such engines are really useful compared to FTS if you have several facets you can use, like descriptions, which would definitely go beyond what some systems can handle for an ABS targeted group.
Also there need to be checks to have them in sync, adding more things that can break. Just wanted to state this here as an opinion
@vedosis commented on GitHub (Jun 27, 2025):
Those are great points and I'll do some thinking about how to make it right for both use-cases.
I feel like there could be a slim version for folks who want to optimize on resource consumption,
and the deluxe version for others that have the resources to invest in a more powerful search result.
I don't think it's unreasonable to have both implementations available assuming we have a generic API between ABS core and ABS search. It could either be an alternative HTTP API that the Frontend hits or a plugin based API where alternatives can be installed.
As far as "who maintains what," I think it's reasonable for ABS to only need to include/maintain the slim version by default and perhaps the other deluxe implementations should be in other repos and therefore other deployables. That would keep the burden of maintaining a new subsystem lower.
Regardless I think some design principles are important:
How's that sound?
@Vito0912 commented on GitHub (Jun 27, 2025):
I don't think having multiple versions is a good idea (for example, there aren't even different versions for the two base paths - which I understand, but I find very unfortunate), because more things can go wrong.
It means double the work, double the possible problems, and double the debugging to figure out which version was used and reproducing of errors (and double the performance benchmarking), all for the same features and somewhat good achievable with FTS.
On top of that, there are big code changes needed between the "deluxe" and "slim" versions, like jobs to keep ES in sync.
That sounds like a completely separate fork at that point.
But I am just a contributor and wanted to share my thoughts on performance and this.
If I am being really honest, this should/sounds be a plugin, not a "core" or extra feature of ABS. In theory, this can be done through the API right now (except the search home would not be in ABS then).
There is no question that the endpoint for seaching needs a big cleanup and rewrite. But throwing a search engine, intended for handling million of million of datapoints/TB of data and used a cluster into a project like ABS will produce more errors than it solves I fear. And FTS/trigrams/vectors are very good alternatives to it
But this will my last comment on that just wanted to post my thoughts about it. I also wrote that on Discord, so this comment should not be discouraging or anything:
@advplyr commented on GitHub (Jun 27, 2025):
Is there a small alternative to elastic search?
Plugin support was put on hold in order to focus on migrating to React frontend
@Vito0912 commented on GitHub (Jun 27, 2025):
Full Text Search for SQL-based implementations. Vedosis mentioned https://www.sqlite.org/fts5.html for SQLite.
It is not as good as Elasticsearch, Meilisearch, or similar tools, but it uses fewer resources. However, it is also not very fast - at least FTS for postgres isn't it would wonder me if it is faster for Sqlite.
For postgres you can set triggers to auto pouplate the FTS/vector fields. Idk if that is possible for SQL
The issue with Elasticsearch and similar tools is that they are already highly optimized for what they are build. Provinding the best search results at a "Google level grade search". Because of this, the indexes become quite large and are best kept in RAM. So, the high resource usage is mostly by design.
@vedosis commented on GitHub (Jun 27, 2025):
There is open search which is supposed to have a Raspberry Pi compatible image, but I don't have much experience with it. I'll add it to the list along with fts5 and es just for comparison.
@Vito0912 I think I hear what your concerned about but the magic universe in my head doesn't line up with what problems you've described. But! I think I have enough input and concerns to start digging into the codebase and coming with a plan and potential maintenance impact.
Thanks for the convo! Working against voiced constraints is always more reassuring for me than operating in a vacuum.
I'll circle back with some designs in a few days.
@Vito0912 commented on GitHub (Jun 27, 2025):
Yeah, do that. Maybe I just set it up poorly last time or it is completely different from the binary. If it works even better, that would be great :) and thanks for the general input