search the library through FTS5 trigram tables

This commit is contained in:
austinried
2026-10-04 00:30:03 +00:00
parent 46dc6bda0e
commit 954d0eddd2
8 changed files with 1528 additions and 7 deletions
+5 -1
View File
@@ -122,7 +122,11 @@ Sync is not atomic. Each batch commits on its own, so a failure leaves the batch
## Search
Search is a case-insensitive substring filter on the library queries (`instr(lower(column), lower(:search)) > 0` in `LibraryDao`), scoped to the active tab and with the matching rows ordered by the tab's sort. There is no separate search index; the filter runs directly against the library tables. An FTS5 trigram shadow table was trialled and dropped: it was wired to a separate search screen that went unused, and it indexed only `title`, so it did not cover the per-field substring the inline filter does. The decision is reversible if infix search over a very large library ever needs an index, because `tokenize='trigram'` is the only SQLite built-in that can index an arbitrary substring match. A revival must index every field the filter scans (`name`/`albumArtist` for albums, `name` for artists and playlists), be kept current incrementally per sync batch rather than cleared and rebuilt, and accept a three-character minimum (shorter queries fall back to the scan); sorting and paging would still come from the normal indexed library queries.
Search is a case-insensitive substring filter on the library queries, scoped to the active tab and ordered by the tab's sort. A query of three characters or more goes through a per-entity FTS5 trigram table: `album_search` indexes `name` and `albumArtist`, `artist_search` and `playlist_search` index `name`, and the filter is `rowid IN (SELECT rowid FROM <table> WHERE <table> MATCH '"' || replace(:search, '"', '""') || '"')`. Two characters or fewer cannot form a trigram, so they fall back to the `instr(lower(column), lower(:search))` scan; an empty query skips both.
The trigram tables are external content (`content=` the base table, keyed by its rowid), so they store no second copy of the text. Room declares them as `@Fts5(contentEntity = ...)` entities and generates the insert/update/delete triggers that keep the index current per row, so every write path — batch sync upserts, star toggles, prune deletes and source cascades — maintains it with no extra code. Sorting and paging still come from the normal indexed library queries; the FTS match only narrows which rows they return, and `MIGRATION_19_20` creates the tables, triggers and a `'rebuild'` for existing libraries.
The two branches do not fold case identically: the trigram tokenizer lowercases all of Unicode, while SQLite's `lower()` (used by the fallback) is ASCII-only. So an accented query of three or more characters matches case-insensitively where its two-character prefix would not. This only ever widens the result set, and is not worth normalising in SQL.
## Networking and auth
+2 -2
View File
@@ -42,7 +42,7 @@ Not done yet in this slice (deliberately): cover-art tonal colour extraction, lo
## Phase 4 - Search, playlists, offline
- (done) Search: a substring filter on the library queries, scoped to the active tab, with a docked search field and per-tab sort/filter controls.
- (done) Search: a substring filter on the library queries, scoped to the active tab, with a docked search field and per-tab sort/filter controls. Queries of three or more characters run through per-entity FTS5 trigram tables (external content, trigger-maintained), with the `instr` scan as the below-three-character fallback.
- Playlist browsing and editing (create/add/remove), syncing changes back to the server.
- Offline mode and downloads.
@@ -61,7 +61,7 @@ Follow-up work for very large libraries, playlists and long sessions, grouped so
- **Sync pipeline.** Prune with a memory-bounded id set, remove `fetchRanks` and the write-only rank columns, fetch playlists/albums with bounded concurrency and skip unchanged ones, and raise/make configurable the page cap with a clear "library too large" failure. Independent of the read path.
- **Queue view and edits.** Bound the queue view's loaded rows (or page over the queue ordinal) and scope its ViewModel to the overlay; compact adjacent same-ref ranges after an edit and cache resolved entry lengths. The view half is independent; the edit half shares `QueueRepository` with the read-path PR.
- **Artwork.** Give `artwork_seeds` a `sourceId` with `ON DELETE CASCADE` and an LRU/TTL (or drop the table and rely on Coil's disk cache); prefetch the thumbnail, not the original, on now-playing transitions. Independent.
- **FTS5 search.** Deferred; the revival requirements are recorded in `architecture.md` (per-entity trigram tables indexing every scanned field, incremental maintenance, three-character floor).
- **FTS5 search.** Landed: per-entity external-content trigram tables (`album_search`, `artist_search`, `playlist_search`) with Room-generated content-sync triggers, a three-character floor and an `instr` fallback below it.
- **UI/flow overhead.** Subscribe only the active tab's paging flow; move the position ticker off the shared `PlaybackState`. Independent.
A `syncGen` column was considered for the sync prune and rejected: it is part of the upsert's update set, so its always-changing value defeats `upsertChanged`'s changed-only `WHERE` predicate and rewrites every row on every full sync. The prune should instead keep the two-phase diff with a memory-bounded id set (primitive long hashes) or diff a staging `seen` table in SQL.