mirror of
https://github.com/austinried/subtracks.git
synced 2026-11-15 00:42:54 +01:00
sync: batch the fallback getAlbum song fetches
The `search3`-less fallback emitted one batch per album, and `SyncService.writeSongs` opens a transaction per batch, so syncing a large library that way ran one transaction per album — thousands of fsyncs, and an invalidation round for every list observing `songs` with each. Accumulate what each `getAlbum` returns and flush at `PAGE_SIZE`, so the fallback writes the same ~500-row batches every other fetch already does. Albums are fetched concurrently and complete out of order, so the accumulator sits behind a mutex and the send happens outside it. Batches are in-flight memory, so the flush is per rows reached rather than per album page. Measured per `Song` (strings, boxed longs and genre list included): ~0.7 KiB, so a 500-row batch is ~334 KiB plus ~51 KiB of the `names`/`genres` structures `writeSongs` derives, where a whole album page (~5,500 rows) would be ~3.8 MiB. `albumSongs()` is also a bare `channelFlow`, whose channel holds 64 elements whatever their size, so 500-row batches would have made its worst case ~21 MiB; it now carries the same `.buffer(1)` that `entityBatches()` already uses on itself. The rest of the in-flight rows scale with `syncConcurrency`, so that leaves the default's batch in hand and a few MiB at the highest setting. `SubsonicSourceTest.albumSongsAreEmittedInBoundedBatches` asserts the emitted sizes, [500, 100] for 600 songs spread over 200 three-song albums, which pins the carried remainder as well as the flush: verified it fails on the pre-change code (200 batches of 3) and against a mutant that drops the remainder ([500, 99]). `docs/architecture.md`'s page cap was 20x out of date, so it is corrected alongside. `ktlintCheck`, `testDebugUnitTest` and `lintDebug` pass.
This commit is contained in:
@@ -608,6 +608,40 @@ class SubsonicSourceTest {
|
||||
assertTrue(maxInFlight.get() in 2..DEFAULT_FETCH_CONCURRENCY)
|
||||
}
|
||||
|
||||
@Test
|
||||
fun albumSongsAreEmittedInBoundedBatches() =
|
||||
runBlocking {
|
||||
val albumCount = 200
|
||||
val songsPerAlbum = 3
|
||||
server.dispatcher =
|
||||
object : Dispatcher() {
|
||||
override fun dispatch(request: RecordedRequest): MockResponse {
|
||||
val url = request.requestUrl!!
|
||||
return when (url.encodedPath) {
|
||||
"/rest/search3.view" -> {
|
||||
MockResponse().setBody(failed(10, "Required parameter is missing"))
|
||||
}
|
||||
|
||||
"/rest/getAlbumList2.view" -> {
|
||||
MockResponse().setBody(albumListOf(albumCount))
|
||||
}
|
||||
|
||||
"/rest/getAlbum.view" -> {
|
||||
MockResponse().setBody(albumWithSongs(url.queryParameter("id")!!, songsPerAlbum))
|
||||
}
|
||||
|
||||
else -> {
|
||||
MockResponse().setResponseCode(404)
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
val batches = SubsonicSource(1, client()).songs().toList().map { it.size }
|
||||
|
||||
assertEquals(listOf(500, 100), batches)
|
||||
}
|
||||
|
||||
@Test
|
||||
fun playlistEntriesWithoutIdsAreSkipped() =
|
||||
runBlocking {
|
||||
@@ -705,6 +739,13 @@ class SubsonicSourceTest {
|
||||
"<song id=\"$albumId-s1\" title=\"One\"/>" +
|
||||
"</album></subsonic-response>"
|
||||
|
||||
private fun albumWithSongs(
|
||||
albumId: String,
|
||||
count: Int,
|
||||
) = "<subsonic-response status=\"ok\"><album id=\"$albumId\">" +
|
||||
(1..count).joinToString("") { "<song id=\"$albumId-s$it\" title=\"Title $it\"/>" } +
|
||||
"</album></subsonic-response>"
|
||||
|
||||
private fun failed(
|
||||
code: Int,
|
||||
message: String,
|
||||
|
||||
Reference in New Issue
Block a user