Skip to content

StreamVault is a TV-first free IPTV player for Android TV built with Kotlin, Jetpack Compose, Room, Hilt, and Media3.

feat: Implement M3U VOD classification and XMLTV ingestion limits

David Nashash8973afc562b0
diff --git a/app/src/main/res/values/strings.xml b/app/src/main/res/values/strings.xml
index 4d10e0af..a77e4d64 100644
--- a/app/src/main/res/values/strings.xml
+++ b/app/src/main/res/values/strings.xml
@@ -340,7 +340,7 @@
<string name="settings_provider_category_controls_subtitle">Lock, unlock, hide, and restore categories for the current playlist from one place.</string>
<string name="settings_provider_category_controls_search">Search categories...</string>
<string name="settings_m3u_vod_classification_title">M3U VOD Classification</string>
- <string name="settings_m3u_vod_classification_summary">Uses the M3U URL and group-title rules to move matching entries into Movies. Refresh after changing this to rearrange the playlist.</string>
+ <string name="settings_m3u_vod_classification_summary">Uses decoded URL evidence first, then localized group and title hints. Live-stream URLs stay in Live TV. Refresh after changing this to reclassify the playlist.</string>
<string name="settings_m3u_vod_classification_refresh">Refresh Classification</string>
<string name="settings_categories_title">Category Preferences</string>
<string name="settings_categories_subtitle">Hide provider categories and keep their sort order exactly how you want it.</string>
@@ -1064,7 +1064,7 @@
<string name="setup_advanced_options_show">Show</string>
<string name="setup_advanced_options_hide">Hide</string>
<string name="setup_m3u_vod_classification_label">M3U VOD Classification</string>
- <string name="setup_m3u_vod_classification_helper">M3U only. Use playlist URL and group-title heuristics to sort matching entries into Movies instead of Live TV.</string>
+ <string name="setup_m3u_vod_classification_helper">M3U only. Classify decoded movie URLs and localized group or title hints as Movies; recognized live-stream URLs remain in Live TV.</string>
<string name="setup_xtream_live_sync_mode_label">Xtream Live Sync</string>
<string name="setup_xtream_live_sync_mode_helper">Choose how Live TV channels are fetched. Auto protects weak devices and routine refreshes while keeping categories populated.</string>
<string name="setup_xtream_live_sync_mode_auto_title">Auto</string>
diff --git a/data/src/main/java/com/streamvault/data/local/dao/CatalogSyncDao.kt b/data/src/main/java/com/streamvault/data/local/dao/CatalogSyncDao.kt
index a7dfd903..9a1fc88f 100644
--- a/data/src/main/java/com/streamvault/data/local/dao/CatalogSyncDao.kt
+++ b/data/src/main/java/com/streamvault/data/local/dao/CatalogSyncDao.kt
@@ -74,6 +74,9 @@ interface CatalogSyncDao {
@Query("SELECT COUNT(*) FROM series_import_stage WHERE provider_id = :providerId AND session_id = :sessionId")
suspend fun countSeriesStages(providerId: Long, sessionId: Long): Int
+ @Query("SELECT COUNT(*) FROM category_import_stage WHERE provider_id = :providerId AND session_id = :sessionId AND type = :type")
+ suspend fun countCategoryStages(providerId: Long, sessionId: Long, type: String): Int
+
@Query(
"""
SELECT
diff --git a/data/src/main/java/com/streamvault/data/local/dao/ProviderWorkflowDao.kt b/data/src/main/java/com/streamvault/data/local/dao/ProviderWorkflowDao.kt
index 0099ea1e..49579661 100644
--- a/data/src/main/java/com/streamvault/data/local/dao/ProviderWorkflowDao.kt
+++ b/data/src/main/java/com/streamvault/data/local/dao/ProviderWorkflowDao.kt
@@ -67,6 +67,44 @@ abstract class ProviderWorkflowDao {
generation: Long
): List<ProviderWorkflowPhaseEntity>
+ @Query(
+ """
+ SELECT checkpoint FROM provider_workflow_phases
+ WHERE provider_id = :providerId AND generation = :generation AND phase = :phase
+ """
+ )
+ abstract suspend fun getCheckpoint(
+ providerId: Long,
+ generation: Long,
+ phase: ProviderWorkflowPhase
+ ): String?
+
+ @Query(
+ """
+ UPDATE provider_workflow_phases
+ SET checkpoint = :checkpoint, updated_at = :now
+ WHERE provider_id = :providerId
+ AND generation = :generation
+ AND phase = :phase
+ AND state = 'RUNNING'
+ AND EXISTS (
+ SELECT 1 FROM provider_workflows workflow
+ WHERE workflow.provider_id = :providerId
+ AND workflow.generation = :generation
+ AND workflow.current_phase = :phase
+ AND workflow.lease_token = :token
+ )
+ """
+ )
+ abstract suspend fun updateRunningCheckpoint(
+ providerId: Long,
+ generation: Long,
+ phase: ProviderWorkflowPhase,
+ token: String,
+ checkpoint: String?,
+ now: Long
+ ): Int
+
@Insert(onConflict = OnConflictStrategy.IGNORE)
protected abstract suspend fun insertWorkflow(workflow: ProviderWorkflowEntity): Long
@@ -580,13 +618,20 @@ abstract class ProviderWorkflowDao {
} else {
ProviderWorkflowPhaseState.FAILED_PERMANENT
}
+ // A retryable failure represents an interrupted attempt, not abandoned work. Preserve
+ // the last atomically recorded page checkpoint when the caller has no newer one.
+ val retainedCheckpoint = if (retryable) {
+ checkpoint ?: getCheckpoint(lease.providerId, lease.generation, lease.phase)
+ } else {
+ checkpoint
+ }
check(
finishPhase(
lease.providerId,
lease.generation,
lease.phase,
phaseState,
- checkpoint,
+ retainedCheckpoint,
errorCode,
errorMessage,
now
diff --git a/data/src/main/java/com/streamvault/data/parser/M3uParser.kt b/data/src/main/java/com/streamvault/data/parser/M3uParser.kt
index 40d72ab0..e1b927ac 100644
--- a/data/src/main/java/com/streamvault/data/parser/M3uParser.kt
+++ b/data/src/main/java/com/streamvault/data/parser/M3uParser.kt
@@ -6,9 +6,14 @@ import com.streamvault.domain.util.ChannelNormalizer
import com.streamvault.domain.util.StreamEntryUrlPolicy
import java.io.BufferedInputStream
import java.io.BufferedReader
+import java.io.ByteArrayOutputStream
+import java.io.Closeable
import java.io.InputStream
import java.io.InputStreamReader
-import java.net.URI
+import java.nio.ByteBuffer
+import java.nio.charset.CharacterCodingException
+import java.nio.charset.Charset
+import java.nio.charset.CodingErrorAction
/**
* Robust M3U parser that handles real-world malformed playlists.
@@ -57,8 +62,8 @@ class M3uParser {
val entries: List<M3uEntry>
)
- fun parse(inputStream: InputStream): ParseResult {
- val reader = playlistReader(inputStream)
+ fun parse(inputStream: InputStream, declaredCharset: Charset? = null): ParseResult {
+ val source = playlistLines(inputStream, declaredCharset)
val entries = mutableListOf<M3uEntry>()
var header = M3uHeader()
var pendingExtinf: ParsedExtinf? = null
@@ -66,8 +71,8 @@ class M3uParser {
// lineSequence() streams the InputStream one line at a time — only a single
// String is allocated per iteration; prior lines are immediately eligible for GC.
// This keeps memory flat regardless of playlist size (e.g. 500 MB feeds).
- reader.use {
- reader.lineSequence()
+ source.use {
+ source.lines
.map { sanitizeLine(it) }
.filter { it.isNotEmpty() }
.forEach { line ->
@@ -99,9 +104,10 @@ class M3uParser {
inputStream: InputStream,
onHeader: suspend (M3uHeader) -> Unit = {},
onEntry: suspend (M3uEntry) -> Unit,
- onInvalidEntry: suspend () -> Unit = {}
+ onInvalidEntry: suspend () -> Unit = {},
+ declaredCharset: Charset? = null
) {
- val reader = playlistReader(inputStream)
+ val source = playlistLines(inputStream, declaredCharset)
var header = M3uHeader()
var pendingExtinf: ParsedExtinf? = null
@@ -110,17 +116,24 @@ class M3uParser {
// compiler cannot allow coroutine suspension inside it. A for-loop over the same
// sequence is fully coroutine-compatible and preserves the one-line-at-a-time
// memory profile.
- reader.use {
- for (rawLine in reader.lineSequence()) {
+ source.use {
+ for (rawLine in source.lines) {
val line = sanitizeLine(rawLine)
if (line.isEmpty()) continue
when {
line.startsWith("#EXTM3U", ignoreCase = true) -> {
+ if (pendingExtinf != null) {
+ onInvalidEntry()
+ pendingExtinf = null
+ }
header = parseHeader(line)
onHeader(header)
}
line.startsWith("#EXTINF", ignoreCase = true) -> {
+ if (pendingExtinf != null) {
+ onInvalidEntry()
+ }
pendingExtinf = parseExtinf(line)
}
line.startsWith("#") -> {
@@ -137,6 +150,9 @@ class M3uParser {
}
}
}
+ if (pendingExtinf != null) {
+ onInvalidEntry()
+ }
}
}
@@ -239,21 +255,85 @@ class M3uParser {
.toList()
}
- private fun playlistReader(inputStream: InputStream): BufferedReader {
+ /**
+ * Charset precedence is BOM, validated HTTP charset, then strict UTF-8 per line with a
+ * Windows-1252 fallback. Windows-1252 preserves ISO-8859-1 text in the printable 0xA0-0xFF
+ * range while also handling the legacy punctuation commonly found in provider playlists.
+ */
+ private fun playlistLines(inputStream: InputStream, declaredCharset: Charset?): PlaylistLineSource {
val buffered = inputStream as? BufferedInputStream ?: BufferedInputStream(inputStream, 64 * 1024)
buffered.mark(3)
val first = buffered.read()
val second = buffered.read()
val third = buffered.read()
buffered.reset()
- val (bomLength, charset) = when {
+ val (bomLength, bomCharset) = when {
first == 0xEF && second == 0xBB && third == 0xBF -> 3 to Charsets.UTF_8
first == 0xFF && second == 0xFE -> 2 to Charsets.UTF_16LE
first == 0xFE && second == 0xFF -> 2 to Charsets.UTF_16BE
- else -> 0 to Charsets.UTF_8
+ else -> 0 to null
}
repeat(bomLength) { buffered.read() }
- return BufferedReader(InputStreamReader(buffered, charset))
+ val charset = bomCharset ?: validatedPlaylistCharset(declaredCharset)
+ return if (charset != null) {
+ val reader = BufferedReader(InputStreamReader(buffered, charset))
+ PlaylistLineSource(reader, reader.lineSequence())
+ } else {
+ PlaylistLineSource(buffered, legacyFallbackLines(buffered))
+ }
+ }
+
+ private fun validatedPlaylistCharset(charset: Charset?): Charset? {
+ val normalized = charset?.name()?.uppercase() ?: return null
+ return charset.takeIf {
+ normalized in setOf(
+ "UTF-8",
+ "UTF-16LE",
+ "UTF-16BE",
+ "WINDOWS-1252",
+ "ISO-8859-1",
+ "US-ASCII"
+ )
+ }
+ }
+
+ private fun legacyFallbackLines(input: InputStream): Sequence<String> = sequence {
+ val line = ByteArrayOutputStream(256)
+ val utf8Decoder = Charsets.UTF_8.newDecoder()
+ .onMalformedInput(CodingErrorAction.REPORT)
+ .onUnmappableCharacter(CodingErrorAction.REPORT)
+
+ fun decodeLine(): String {
+ val bytes = line.toByteArray().let { raw ->
+ if (raw.lastOrNull() == '\r'.code.toByte()) raw.copyOf(raw.size - 1) else raw
+ }
+ return try {
+ utf8Decoder.reset().decode(ByteBuffer.wrap(bytes)).toString()
+ } catch (_: CharacterCodingException) {
+ String(bytes, WINDOWS_1252)
+ }
+ }
+
+ while (true) {
+ when (val next = input.read()) {
+ -1 -> {
+ if (line.size() > 0) yield(decodeLine())
+ break
+ }
+ '\n'.code -> {
+ yield(decodeLine())
+ line.reset()
+ }
+ else -> line.write(next)
+ }
+ }
+ }
+
+ private class PlaylistLineSource(
+ private val closeable: Closeable,
+ val lines: Sequence<String>
+ ) : Closeable {
+ override fun close() = closeable.close()
}
private fun parseStandaloneGroupTitle(line: String): String? {
@@ -456,24 +536,12 @@ class M3uParser {
"url-xml"
)
private const val MAX_HEADER_EPG_URLS = 8
+ private val WINDOWS_1252: Charset = Charset.forName("windows-1252")
- /** Exposed for callers outside M3uParser (e.g. SyncManager) to avoid duplicate logic. */
- fun isVodEntry(entry: M3uEntry): Boolean {
- val url = entry.url.lowercase()
- val path = runCatching { URI(entry.url).path }
- .getOrNull()
- ?.lowercase()
- ?: entry.url.substringBefore('#').substringBefore('?').lowercase()
- val group = entry.groupTitle.lowercase()
-
- return path.endsWith(".mp4") ||
- path.endsWith(".mkv") ||
- path.endsWith(".avi") ||
- url.contains("/movie/") ||
- group.contains("movie") ||
- group.contains("vod") ||
- group.contains("film")
- }
+ private val vodClassifier = M3uVodClassifier()
+
+ /** Exposed for importers and tests; classification rules live in one deterministic source. */
+ fun isVodEntry(entry: M3uEntry): Boolean = vodClassifier.classify(entry).isVod
val knownAttributes = setOf(
"tvg-id",
diff --git a/data/src/main/java/com/streamvault/data/parser/M3uVodClassifier.kt b/data/src/main/java/com/streamvault/data/parser/M3uVodClassifier.kt
new file mode 100644
index 00000000..7539054a
--- /dev/null
+++ b/data/src/main/java/com/streamvault/data/parser/M3uVodClassifier.kt
@@ -0,0 +1,121 @@
+package com.streamvault.data.parser
+
+import java.net.URI
+import java.util.Locale
+
+internal enum class M3uMediaKind {
+ LIVE,
+ VOD
+}
+
+internal enum class M3uVodEvidence {
+ EXPLICIT_OVERRIDE,
+ FILE_EXTENSION,
+ PATH_SEGMENT,
+ LIVE_STREAM_EXTENSION,
+ LIVE_TEXT_HINT,
+ GROUP_KEYWORD,
+ NAME_KEYWORD,
+ DEFAULT_LIVE
+}
+
+internal data class M3uVodClassification(
+ val kind: M3uMediaKind,
+ val evidence: M3uVodEvidence,
+ val confidence: Float
+) {
+ val isVod: Boolean get() = kind == M3uMediaKind.VOD
+}
+
+/**
+ * Configurable rules for providers whose naming differs from the defaults. Keywords are matched
+ * as words for whitespace-delimited languages and as phrases for languages without spaces.
+ */
+internal data class M3uVodRules(
+ val vodExtensions: Set<String> = DEFAULT_VOD_EXTENSIONS,
+ val liveExtensions: Set<String> = DEFAULT_LIVE_EXTENSIONS,
+ val vodPathSegments: Set<String> = DEFAULT_VOD_PATH_SEGMENTS,
+ val vodKeywords: Set<String> = DEFAULT_VOD_KEYWORDS,
+ val liveKeywords: Set<String> = DEFAULT_LIVE_KEYWORDS
+) {
+ companion object {
+ val DEFAULT_VOD_EXTENSIONS = setOf("mp4", "mkv", "avi", "mov", "m4v", "webm", "wmv", "flv", "mpg", "mpeg")
+ val DEFAULT_LIVE_EXTENSIONS = setOf("m3u8", "ts")
+ val DEFAULT_VOD_PATH_SEGMENTS = setOf("movie", "movies", "vod", "film", "films")
+ val DEFAULT_VOD_KEYWORDS = setOf(
+ "movie", "movies", "vod", "film", "films", "cinema",
+ "película", "películas", "pelicula", "peliculas",
+ "filme", "filmes", "kino", "фильм", "фильмы",
+ "电影", "電影", "映画", "영화", "أفلام", "סרטים"
+ )
+ val DEFAULT_LIVE_KEYWORDS = setOf(
+ "live", "live tv", "channel", "channels", "canal", "chaîne",
+ "direct", "en vivo", "ao vivo", "直播", "ライブ", "مباشر", "שידור חי"
+ )
+ }
+}
+
+/** Deterministic M3U media classifier that exposes why an entry was classified. */
+internal class M3uVodClassifier(
+ private val rules: M3uVodRules = M3uVodRules()
+) {
+ fun classify(
+ entry: M3uParser.M3uEntry,
+ override: M3uMediaKind? = null
+ ): M3uVodClassification {
+ override?.let {
+ return M3uVodClassification(it, M3uVodEvidence.EXPLICIT_OVERRIDE, 1.0f)
+ }
+
+ val path = decodedPath(entry.url)
+ val segments = path.split('/').filter(String::isNotBlank)
+ val extension = segments.lastOrNull()?.substringAfterLast('.', missingDelimiterValue = "")
+
+ if (extension in rules.vodExtensions) {
+ return M3uVodClassification(M3uMediaKind.VOD, M3uVodEvidence.FILE_EXTENSION, 1.0f)
+ }
+ if (segments.any { it in rules.vodPathSegments }) {
+ return M3uVodClassification(M3uMediaKind.VOD, M3uVodEvidence.PATH_SEGMENT, 0.95f)
+ }
+ if (extension in rules.liveExtensions) {
+ return M3uVodClassification(M3uMediaKind.LIVE, M3uVodEvidence.LIVE_STREAM_EXTENSION, 0.95f)
+ }
+
+ val group = normalize(entry.groupTitle)
+ val name = normalize(entry.name)
+ if (matchesAny(group, rules.liveKeywords) || matchesAny(name, rules.liveKeywords)) {
+ return M3uVodClassification(M3uMediaKind.LIVE, M3uVodEvidence.LIVE_TEXT_HINT, 0.8f)
+ }
+ if (matchesAny(group, rules.vodKeywords)) {
+ return M3uVodClassification(M3uMediaKind.VOD, M3uVodEvidence.GROUP_KEYWORD, 0.7f)
+ }
+ if (matchesAny(name, rules.vodKeywords)) {
+ return M3uVodClassification(M3uMediaKind.VOD, M3uVodEvidence.NAME_KEYWORD, 0.55f)
+ }
+ return M3uVodClassification(M3uMediaKind.LIVE, M3uVodEvidence.DEFAULT_LIVE, 1.0f)
+ }
+
+ private fun decodedPath(url: String): String = runCatching { URI(url).path }
+ .getOrNull()
+ ?.lowercase(Locale.ROOT)
+ ?: url.substringBefore('#').substringBefore('?').lowercase(Locale.ROOT)
+
+ private fun normalize(value: String): String = value
+ .lowercase(Locale.ROOT)
+ .replace(Regex("[^\\p{L}\\p{N}]+"), " ")
+ .trim()
+
+ private fun matchesAny(value: String, keywords: Set<String>): Boolean {
+ if (value.isEmpty()) return false
+ val paddedValue = " $value "
+ return keywords.any { keyword ->
+ val normalizedKeyword = normalize(keyword)
+ normalizedKeyword.isNotEmpty() && (
+ paddedValue.contains(" $normalizedKeyword ") ||
+ (normalizedKeyword.none(Char::isWhitespace) &&
+ normalizedKeyword.any { it.code > 127 } &&
+ value.contains(normalizedKeyword))
+ )
+ }
+ }
+}
diff --git a/data/src/main/java/com/streamvault/data/parser/XmltvIngestionLimits.kt b/data/src/main/java/com/streamvault/data/parser/XmltvIngestionLimits.kt
new file mode 100644
index 00000000..76735a52
--- /dev/null
+++ b/data/src/main/java/com/streamvault/data/parser/XmltvIngestionLimits.kt
@@ -0,0 +1,29 @@
+package com.streamvault.data.parser
+
+import com.streamvault.data.remote.NetworkTimeoutConfig
+import java.io.IOException
+
+data class XmltvIngestionLimits(
+ val maxRawBytes: Long = NetworkTimeoutConfig.EPG_MAX_RAW_SIZE_BYTES,
+ val maxDecompressedBytes: Long = NetworkTimeoutConfig.EPG_MAX_SIZE_BYTES,
+ val maxChannels: Int = 100_000,
+ val maxProgrammes: Int = 1_000_000,
+ val maxFieldChars: Int = 65_536,
+ val maxCategoriesPerProgramme: Int = 64,
+ val maxXmlDepth: Int = 32
+)
+
+enum class XmltvLimitKind(val label: String) {
+ RAW_BYTES("raw download bytes"),
+ DECOMPRESSED_BYTES("decompressed XML bytes"),
+ CHANNELS("channels"),
+ PROGRAMMES("programmes"),
+ FIELD_LENGTH("field length"),
+ CATEGORIES_PER_PROGRAMME("categories per programme"),
+ XML_DEPTH("XML depth")
+}
+
+class XmltvLimitExceeded(
+ val kind: XmltvLimitKind,
+ val maximum: Long
+) : IOException("XMLTV ${kind.label} limit exceeded (maximum $maximum)")
diff --git a/data/src/main/java/com/streamvault/data/parser/XmltvParser.kt b/data/src/main/java/com/streamvault/data/parser/XmltvParser.kt
index 3f4368d1..b232b7a6 100644
--- a/data/src/main/java/com/streamvault/data/parser/XmltvParser.kt
+++ b/data/src/main/java/com/streamvault/data/parser/XmltvParser.kt
@@ -350,7 +350,8 @@ class XmltvParser {
inputStream: InputStream,
timezoneId: String? = null,
onChannel: suspend (XmltvChannel) -> Unit,
- onProgramme: suspend (XmltvProgramme) -> Unit
+ onProgramme: suspend (XmltvProgramme) -> Unit,
+ limits: XmltvIngestionLimits = XmltvIngestionLimits()
) {
val parser = newPullParser(inputStream)
val parsingZoneId = resolveParsingZoneId(timezoneId)
@@ -384,6 +385,12 @@ class XmltvParser {
while (eventType != XmlPullParser.END_DOCUMENT) {
when (eventType) {
XmlPullParser.START_TAG -> {
+ if (parser.depth > limits.maxXmlDepth) {
+ throw XmltvLimitExceeded(XmltvLimitKind.XML_DEPTH, limits.maxXmlDepth.toLong())
+ }
+ for (index in 0 until parser.attributeCount) {
+ requireXmltvField(parser.getAttributeValue(index), limits)
+ }
when (parser.name) {
"channel" -> {
inChannel = true
@@ -443,6 +450,7 @@ class XmltvParser {
}
}
XmlPullParser.TEXT -> {
+ requireXmltvField(parser.text, limits)
if (inChannel) {
when (channelTag) {
"display-name" -> {
@@ -459,7 +467,15 @@ class XmltvParser {
"sub-title" -> currentSubtitle = parser.text?.trim()?.takeIf { it.isNotEmpty() }
"desc" -> currentDescription = parser.text
"episode-num" -> currentEpisodeNum = parser.text?.trim()?.takeIf { it.isNotEmpty() }
- "category" -> parser.text?.trim()?.takeIf { it.isNotEmpty() }?.let(currentCategories::add)
+ "category" -> parser.text?.trim()?.takeIf { it.isNotEmpty() }?.let { category ->
+ if (currentCategories.size >= limits.maxCategoriesPerProgramme) {
+ throw XmltvLimitExceeded(
+ XmltvLimitKind.CATEGORIES_PER_PROGRAMME,
+ limits.maxCategoriesPerProgramme.toLong()
+ )
+ }
+ currentCategories.add(category)
+ }
"rating" -> currentRating = parser.text?.trim()?.takeIf { it.isNotEmpty() }
}
currentTag = null
@@ -468,6 +484,9 @@ class XmltvParser {
XmlPullParser.END_TAG -> {
if (parser.name == "channel" && inChannel) {
if (channelId != null && channelDisplayName != null) {
+ if (channelCount >= limits.maxChannels) {
+ throw XmltvLimitExceeded(XmltvLimitKind.CHANNELS, limits.maxChannels.toLong())
+ }
onChannel(
XmltvChannel(
id = channelId,
@@ -484,6 +503,11 @@ class XmltvParser {
}
if (parser.name == "programme" && inProgramme) {
if (isValidProgramme(currentChannelId, currentTitle, currentStart, currentEnd)) {
+ if (programmeCount >= limits.maxProgrammes) {
+ throw XmltvLimitExceeded(XmltvLimitKind.PROGRAMMES, limits.maxProgrammes.toLong())
+ }
+ val genre = currentCategories.distinct().joinToString(" / ").takeIf { it.isNotBlank() }
+ requireXmltvField(genre, limits)
onProgramme(
XmltvProgramme(
channelId = currentChannelId!!,
@@ -495,7 +519,7 @@ class XmltvParser {
lang = currentLang,
rating = currentRating,
imageUrl = currentImageUrl,
- genre = currentCategories.distinct().joinToString(" / ").takeIf { it.isNotBlank() },
+ genre = genre,
category = currentCategories.firstOrNull(),
episodeInfo = currentEpisodeNum
)
@@ -518,6 +542,12 @@ class XmltvParser {
}
}
+ private fun requireXmltvField(value: String?, limits: XmltvIngestionLimits) {
+ if (value != null && value.length > limits.maxFieldChars) {
+ throw XmltvLimitExceeded(XmltvLimitKind.FIELD_LENGTH, limits.maxFieldChars.toLong())
+ }
+ }
+
/**
* Wraps the input stream with [GZIPInputStream] when gzip content is detected.
*
diff --git a/data/src/main/java/com/streamvault/data/remote/NetworkTimeoutConfig.kt b/data/src/main/java/com/streamvault/data/remote/NetworkTimeoutConfig.kt
index 98bdfa3b..2a68dc22 100644
--- a/data/src/main/java/com/streamvault/data/remote/NetworkTimeoutConfig.kt
+++ b/data/src/main/java/com/streamvault/data/remote/NetworkTimeoutConfig.kt
@@ -6,6 +6,7 @@ object NetworkTimeoutConfig {
const val WRITE_TIMEOUT_SECONDS = 30L
// EPG files can be large and served from slow hosts — allow more time per read.
const val EPG_READ_TIMEOUT_SECONDS = 120L
+ const val EPG_MAX_RAW_SIZE_BYTES = 64L * 1_048_576 // 64 MB before decompression
const val EPG_MAX_SIZE_BYTES = 200L * 1_048_576 // 200 MB
const val XTREAM_SEGMENTED_READ_TIMEOUT_SECONDS = 45L
const val XTREAM_SEGMENTED_WRITE_TIMEOUT_SECONDS = 45L
diff --git a/data/src/main/java/com/streamvault/data/remote/jellyfin/JellyfinProvider.kt b/data/src/main/java/com/streamvault/data/remote/jellyfin/JellyfinProvider.kt
index 5118b9ef..4d192b8f 100644
--- a/data/src/main/java/com/streamvault/data/remote/jellyfin/JellyfinProvider.kt
+++ b/data/src/main/java/com/streamvault/data/remote/jellyfin/JellyfinProvider.kt
@@ -39,6 +39,8 @@ class JellyfinProvider @Inject constructor(
const val PAGE_SIZE = 100
const val MAX_PAGE_BYTES = 4L * 1024L * 1024L
const val MAX_EPISODES_PER_SERIES = 10_000
+ const val MAX_ITEM_FIELD_CHARS = 65_536
+ const val MAX_ITEM_COLLECTION_ENTRIES = 128
}
private val itemsResponseType = object : TypeToken<JellyfinItemsResponseDto>() {}.type
@@ -125,17 +127,26 @@ class JellyfinProvider @Inject constructor(
suspend fun fetchEpisodes(provider: Provider, seriesRemoteId: String, seriesLocalId: Long): Result<List<EpisodeEntity>> = try {
val episodes = ArrayList<EpisodeEntity>()
+ val seenEpisodeIds = HashSet<Long>()
var startIndex = 0
var expectedTotal: Int? = null
do {
val page = kotlinx.coroutines.withContext(kotlinx.coroutines.Dispatchers.IO) {
fetchSeriesEpisodesPage(provider, seriesRemoteId, startIndex)
}
+ if (page.totalRecordCount > MAX_EPISODES_PER_SERIES) {
+ throw JellyfinCatalogLimitException("Jellyfin series exceeds $MAX_EPISODES_PER_SERIES episodes")
+ }
expectedTotal = expectedTotal ?: page.totalRecordCount
if (page.totalRecordCount != expectedTotal || page.items.isEmpty() && startIndex < expectedTotal) {
throw JellyfinPaginationException("Jellyfin episode catalog changed or ended early")
}
- episodes += page.items.map { item -> buildEpisodeEntity(item, provider, seriesLocalId) }
+ page.items.map { item -> buildEpisodeEntity(item, provider, seriesLocalId) }.forEach { episode ->
+ if (!seenEpisodeIds.add(episode.episodeId)) {
+ throw JellyfinPaginationException("Jellyfin episode pagination repeated an item")
+ }
+ episodes += episode
+ }
if (episodes.size > MAX_EPISODES_PER_SERIES) {
throw JellyfinCatalogLimitException("Jellyfin series exceeds $MAX_EPISODES_PER_SERIES episodes")
}
@@ -347,22 +358,66 @@ class JellyfinProvider @Inject constructor(
if (body.contentLength() > MAX_PAGE_BYTES) throw JellyfinResponseTooLargeException(body.contentLength(), MAX_PAGE_BYTES)
val reader = com.google.gson.stream.JsonReader(InputStreamReader(BoundedInputStream(body.byteStream(), MAX_PAGE_BYTES), body.contentType()?.charset(Charsets.UTF_8) ?: Charsets.UTF_8))
var total: Int? = null
+ var returnedStartIndex: Int? = null
val items = ArrayList<JellyfinItemDto>(PAGE_SIZE)
reader.use {
it.beginObject()
while (it.hasNext()) when (it.nextName()) {
"TotalRecordCount" -> total = it.nextInt()
- "Items" -> { it.beginArray(); while (it.hasNext()) { if (items.size >= PAGE_SIZE) throw JellyfinPaginationException("Jellyfin ignored requested page limit $PAGE_SIZE"); gson.fromJson<JellyfinItemDto>(it, JellyfinItemDto::class.java)?.takeIf { dto -> !dto.id.isNullOrBlank() }?.let(items::add) }; it.endArray() }
+ "StartIndex" -> returnedStartIndex = it.nextInt()
+ "Items" -> { it.beginArray(); while (it.hasNext()) { if (items.size >= PAGE_SIZE) throw JellyfinPaginationException("Jellyfin ignored requested page limit $PAGE_SIZE"); gson.fromJson<JellyfinItemDto>(it, JellyfinItemDto::class.java)?.also(::validateCatalogItem)?.takeIf { dto -> !dto.id.isNullOrBlank() }?.let(items::add) }; it.endArray() }
else -> it.skipValue()
}
it.endObject()
}
val count = total ?: throw JellyfinPaginationException("Jellyfin response omitted TotalRecordCount")
- if (count < 0 || startIndex > count || items.isEmpty() && startIndex < count) throw JellyfinPaginationException("Invalid Jellyfin pagination response")
+ if (
+ count < 0 ||
+ startIndex > count ||
+ returnedStartIndex?.let { it != startIndex } == true ||
+ items.size > count - startIndex ||
+ items.isEmpty() && startIndex < count
+ ) throw JellyfinPaginationException("Invalid Jellyfin pagination response")
return JellyfinPage(startIndex, count, items)
}
}
+ private fun validateCatalogItem(item: JellyfinItemDto) {
+ val scalarFields = listOf(
+ item.id,
+ item.name,
+ item.overview,
+ item.premiereDate,
+ item.dateCreated,
+ item.dateLastMediaAdded,
+ item.primaryMediaSource?.id,
+ item.primaryMediaSource?.container
+ )
+ if (scalarFields.any { it != null && it.length > MAX_ITEM_FIELD_CHARS }) {
+ throw JellyfinItemLimitException("Jellyfin item field exceeded $MAX_ITEM_FIELD_CHARS characters")
+ }
+ val collections = listOf(
+ item.providerIds.orEmpty().entries.map { it.key to it.value },
+ item.genres.orEmpty(),
+ item.imageTags.orEmpty().entries.map { it.key to it.value },
+ item.backdropImageTags.orEmpty(),
+ item.mediaSources.orEmpty()
+ )
+ if (collections.any { it.size > MAX_ITEM_COLLECTION_ENTRIES }) {
+ throw JellyfinItemLimitException("Jellyfin item collection exceeded $MAX_ITEM_COLLECTION_ENTRIES entries")
+ }
+ val collectionStrings = buildList {
+ item.providerIds.orEmpty().forEach { (key, value) -> add(key); add(value) }
+ addAll(item.genres.orEmpty())
+ item.imageTags.orEmpty().forEach { (key, value) -> add(key); add(value) }
+ addAll(item.backdropImageTags.orEmpty())
+ item.mediaSources.orEmpty().forEach { source -> add(source.id); add(source.container) }
+ }
+ if (collectionStrings.any { it != null && it.length > MAX_ITEM_FIELD_CHARS }) {
+ throw JellyfinItemLimitException("Jellyfin item field exceeded $MAX_ITEM_FIELD_CHARS characters")
+ }
+ }
+
private fun buildUrl(baseUrl: String, path: String, query: Map<String, String>): String {
val normalized = baseUrl.trimEnd('/')
val qs = query.entries.joinToString("&") { (k, v) -> "${enc(k)}=${enc(v)}" }
@@ -423,6 +478,7 @@ data class JellyfinPage<T>(val startIndex: Int, val totalRecordCount: Int, val i
class JellyfinResponseTooLargeException(val observedBytes: Long, val maxAllowedBytes: Long) : IOException("Jellyfin response exceeded safe page budget ($observedBytes B > $maxAllowedBytes B)")
class JellyfinPaginationException(message: String) : IOException(message)
class JellyfinCatalogLimitException(message: String) : IOException(message)
+class JellyfinItemLimitException(message: String) : IOException(message)
private class BoundedInputStream(delegate: InputStream, private val limit: Long) : FilterInputStream(delegate) {
private var count = 0L
diff --git a/data/src/main/java/com/streamvault/data/repository/EpgSourceRepositoryImpl.kt b/data/src/main/java/com/streamvault/data/repository/EpgSourceRepositoryImpl.kt
index 12275fe2..3acf6b3e 100644
--- a/data/src/main/java/com/streamvault/data/repository/EpgSourceRepositoryImpl.kt
+++ b/data/src/main/java/com/streamvault/data/repository/EpgSourceRepositoryImpl.kt
@@ -22,6 +22,9 @@ import com.streamvault.domain.model.EpgMatchType
import com.streamvault.domain.model.EpgOverrideCandidate
import com.streamvault.domain.model.EpgSourceType
import com.streamvault.data.parser.XmltvParser
+import com.streamvault.data.parser.XmltvIngestionLimits
+import com.streamvault.data.parser.XmltvLimitExceeded
+import com.streamvault.data.parser.XmltvLimitKind
import com.streamvault.data.util.ProviderInputSanitizer
import com.streamvault.data.util.runSuspendCatching
import com.streamvault.data.util.UrlSecurityPolicy
@@ -50,7 +53,6 @@ import kotlinx.coroutines.withContext
import okhttp3.OkHttpClient
import okhttp3.Request
import java.io.FilterInputStream
-import java.io.IOException
import java.io.InputStream
import java.util.concurrent.TimeUnit
import com.streamvault.data.remote.NetworkTimeoutConfig
@@ -62,21 +64,42 @@ import javax.inject.Singleton
internal fun limitEpgInput(
input: InputStream,
maxBytes: Long = NetworkTimeoutConfig.EPG_MAX_SIZE_BYTES,
+ kind: XmltvLimitKind = XmltvLimitKind.DECOMPRESSED_BYTES,
): InputStream = object : FilterInputStream(input) {
private var bytesRead = 0L
override fun read(): Int {
- if (bytesRead >= maxBytes) throw IOException("EPG response too large (>200 MB)")
+ if (bytesRead >= maxBytes) {
+ return if (super.read() == -1) -1 else throw XmltvLimitExceeded(kind, maxBytes)
+ }
return super.read().also { if (it >= 0) bytesRead++ }
}
override fun read(bytes: ByteArray, off: Int, len: Int): Int {
- if (bytesRead >= maxBytes) throw IOException("EPG response too large (>200 MB)")
+ if (len == 0) return 0
+ if (bytesRead >= maxBytes) {
+ return if (super.read() == -1) -1 else throw XmltvLimitExceeded(kind, maxBytes)
+ }
val remaining = (maxBytes - bytesRead).coerceAtMost(len.toLong()).toInt()
return super.read(bytes, off, remaining).also { if (it > 0) bytesRead += it }
}
}
+internal fun openLimitedXmltvInput(
+ rawInput: InputStream,
+ url: String,
+ xmltvParser: XmltvParser,
+ limits: XmltvIngestionLimits = XmltvIngestionLimits()
+): InputStream {
+ val rawLimited = limitEpgInput(rawInput, limits.maxRawBytes, XmltvLimitKind.RAW_BYTES)
+ val decompressed = xmltvParser.maybeDecompressGzip(url, rawLimited)
+ return limitEpgInput(
+ decompressed,
+ limits.maxDecompressedBytes,
+ XmltvLimitKind.DECOMPRESSED_BYTES
+ )
+}
+
internal fun shouldRateLimitEpgRefresh(
lastSuccessfulRefreshAt: Long,
now: Long,
@@ -104,7 +127,7 @@ class EpgSourceRepositoryImpl @Inject constructor(
companion object {
private const val TAG = "EpgSourceRepo"
- private const val MAX_EPG_SIZE_BYTES = NetworkTimeoutConfig.EPG_MAX_SIZE_BYTES
+ private const val MAX_EPG_RAW_SIZE_BYTES = NetworkTimeoutConfig.EPG_MAX_RAW_SIZE_BYTES
private const val CHANNEL_BATCH_SIZE = 500
private const val PROGRAMME_BATCH_SIZE = 500
private const val MIN_REFRESH_INTERVAL_MS = 5L * 60L * 1000L // 5 minutes
@@ -294,6 +317,9 @@ class EpgSourceRepositoryImpl @Inject constructor(
val requestProfile = HttpRequestProfile(ownerTag = "epg-source:$sourceId")
val request = Request.Builder()
.url(source.url)
+ // Preserve the actual wire representation so the smaller raw-byte
+ // ceiling is enforced before our own gzip detection/decompression.
+ .header("Accept-Encoding", "identity")
.apply {
source.etag?.let { header("If-None-Match", it) }
source.lastModifiedHeader?.let { header("If-Modified-Since", it) }
@@ -325,11 +351,12 @@ class EpgSourceRepositoryImpl @Inject constructor(
}
val contentLength = response.header("Content-Length")?.toLongOrNull() ?: -1L
- if (contentLength > MAX_EPG_SIZE_BYTES) {
+ if (contentLength > MAX_EPG_RAW_SIZE_BYTES) {
ownedResponse.close()
- val err = "File too large (${contentLength / 1_048_576}MB)"
+ val exception = XmltvLimitExceeded(XmltvLimitKind.RAW_BYTES, MAX_EPG_RAW_SIZE_BYTES)
+ val err = xmltvLimitMessage(exception)
epgSourceDao.updateRefreshError(sourceId, err)
- return@withLock Result.error(err)
+ return@withLock Result.error(err, exception)
}
val bodyStream = response.body?.byteStream() ?: run {
@@ -362,10 +389,9 @@ class EpgSourceRepositoryImpl @Inject constructor(
prepareStagingSource(source, stagingId)
rawInputStream.use { raw ->
- val rawLimited = limitEpgInput(raw)
- xmltvParser.maybeDecompressGzip(source.url, rawLimited).use { decompressed ->
+ openLimitedXmltvInput(raw, source.url, xmltvParser).use { decompressed ->
xmltvParser.parseStreamingWithChannels(
- inputStream = limitEpgInput(decompressed),
+ inputStream = decompressed,
timezoneId = sourceTimezoneId,
onChannel = { xmltvChannel ->
channelBatch.add(
@@ -446,15 +472,11 @@ class EpgSourceRepositoryImpl @Inject constructor(
epgSourceDao.delete(stagingId)
}
}
- val isOversizeError = e is IOException && e.message?.contains("too large", ignoreCase = true) == true
- val statusMessage = if (isOversizeError) {
- "EPG response exceeded 200 MB limit"
- } else {
- e.message ?: "Unknown error"
- }
+ val limitError = e as? XmltvLimitExceeded
+ val statusMessage = limitError?.let(::xmltvLimitMessage) ?: (e.message ?: "Unknown error")
epgSourceDao.updateRefreshError(sourceId, statusMessage)
- if (isOversizeError) {
- Result.error("EPG response exceeded 200 MB limit", e)
+ if (limitError != null) {
+ Result.error(statusMessage, e)
} else {
Result.error("Failed to refresh EPG source: ${e.message}", e)
}
@@ -462,6 +484,9 @@ class EpgSourceRepositoryImpl @Inject constructor(
}
}
+ private fun xmltvLimitMessage(error: XmltvLimitExceeded): String =
+ "EPG ${error.kind.label} exceeded safety limit"
+
override suspend fun refreshAllForProvider(providerId: Long): Result<Unit> {
val assignments = providerEpgSourceDao.getEnabledForProviderSync(providerId)
val errors = mutableListOf<String>()
diff --git a/data/src/main/java/com/streamvault/data/sync/DatabaseMaintenanceManager.kt b/data/src/main/java/com/streamvault/data/sync/DatabaseMaintenanceManager.kt
index cac9b770..48f39806 100644
--- a/data/src/main/java/com/streamvault/data/sync/DatabaseMaintenanceManager.kt
+++ b/data/src/main/java/com/streamvault/data/sync/DatabaseMaintenanceManager.kt
@@ -29,7 +29,18 @@ class DatabaseMaintenanceManager @Inject constructor(
private val syncManager: SyncManager
) {
- suspend fun runDailyMaintenance(now: Long = System.currentTimeMillis()): MaintenanceReport = withContext(Dispatchers.IO) {
+ suspend fun runDailyMaintenance(now: Long = System.currentTimeMillis()): MaintenanceRunResult {
+ var completedReport: MaintenanceReport? = null
+ syncManager.runWhenNoSyncActive {
+ completedReport = runDailyMaintenanceWhenAdmitted(now)
+ true
+ }
+ return completedReport
+ ?.let(MaintenanceRunResult::Completed)
+ ?: MaintenanceRunResult.DeferredForActiveSync
+ }
+
+ private suspend fun runDailyMaintenanceWhenAdmitted(now: Long): MaintenanceReport = withContext(Dispatchers.IO) {
val oldProgramThreshold = now - programRetentionMillis()
searchHistoryDao.pruneOlderThan(now - SEARCH_HISTORY_RETENTION_MILLIS)
val deletedExpiredReminders = programReminderDao.deleteExpired(now - PROGRAM_REMINDER_RETENTION_MILLIS)
@@ -43,13 +54,7 @@ class DatabaseMaintenanceManager @Inject constructor(
val statsBeforeVacuum = collectStorageStats()
val vacuumNeeded = shouldVacuum(statsBeforeVacuum)
val vacuumRan = if (vacuumNeeded) {
- syncManager.runWhenNoSyncActive {
- runVacuum()
- }.also { ran ->
- if (!ran) {
- Log.i(TAG, "VACUUM skipped: a provider sync is currently active")
- }
- }
+ runVacuum()
} else {
false
}
@@ -69,6 +74,11 @@ class DatabaseMaintenanceManager @Inject constructor(
)
}
+ sealed interface MaintenanceRunResult {
+ data class Completed(val report: MaintenanceReport) : MaintenanceRunResult
+ data object DeferredForActiveSync : MaintenanceRunResult
+ }
+
@WorkerThread
private fun collectStorageStats(): DatabaseStorageStats {
val sqliteDb = database.openHelper.writableDatabase
diff --git a/data/src/main/java/com/streamvault/data/sync/JellyfinCatalogCheckpoint.kt b/data/src/main/java/com/streamvault/data/sync/JellyfinCatalogCheckpoint.kt
new file mode 100644
index 00000000..4bcf3b87
--- /dev/null
+++ b/data/src/main/java/com/streamvault/data/sync/JellyfinCatalogCheckpoint.kt
@@ -0,0 +1,101 @@
+package com.streamvault.data.sync
+
+import com.streamvault.data.remote.jellyfin.JellyfinCatalogLimitException
+import com.streamvault.data.remote.jellyfin.JellyfinPaginationException
+
+internal enum class JellyfinCatalogPhase {
+ MOVIES,
+ SERIES,
+ READY
+}
+
+internal data class JellyfinCatalogCheckpoint(
+ val movieSessionId: Long,
+ val seriesSessionId: Long,
+ val phase: JellyfinCatalogPhase,
+ val movieStartIndex: Int = 0,
+ val movieTotal: Int? = null,
+ val seriesStartIndex: Int = 0,
+ val seriesTotal: Int? = null
+) {
+ fun afterMoviePage(reportedTotal: Int, pageItemCount: Int, stagedCount: Int): JellyfinCatalogCheckpoint {
+ val expectedTotal = movieTotal ?: reportedTotal
+ validatePage("movie", movieStartIndex, expectedTotal, reportedTotal, pageItemCount, stagedCount, 200_000)
+ return copy(movieStartIndex = movieStartIndex + pageItemCount, movieTotal = expectedTotal)
+ }
+
+ fun afterSeriesPage(reportedTotal: Int, pageItemCount: Int, stagedCount: Int): JellyfinCatalogCheckpoint {
+ val expectedTotal = seriesTotal ?: reportedTotal
+ validatePage("series", seriesStartIndex, expectedTotal, reportedTotal, pageItemCount, stagedCount, 100_000)
+ return copy(seriesStartIndex = seriesStartIndex + pageItemCount, seriesTotal = expectedTotal)
+ }
+
+ fun isConsistent(movieStageCount: Int, seriesStageCount: Int): Boolean {
+ val movieComplete = movieTotal != null && movieStartIndex == movieTotal
+ val seriesComplete = seriesTotal != null && seriesStartIndex == seriesTotal
+ return movieStageCount == movieStartIndex &&
+ seriesStageCount == seriesStartIndex &&
+ movieTotal?.let { movieStartIndex <= it } != false &&
+ seriesTotal?.let { seriesStartIndex <= it } != false &&
+ (phase == JellyfinCatalogPhase.MOVIES || movieComplete) &&
+ (phase != JellyfinCatalogPhase.READY || seriesComplete)
+ }
+
+ fun encode(): String = listOf(
+ VERSION,
+ movieSessionId,
+ seriesSessionId,
+ phase.name,
+ movieStartIndex,
+ movieTotal ?: UNKNOWN_TOTAL,
+ seriesStartIndex,
+ seriesTotal ?: UNKNOWN_TOTAL
+ ).joinToString("|")
+
+ companion object {
+ private const val VERSION = "jellyfin-v1"
+ private const val UNKNOWN_TOTAL = -1
+
+ fun decode(value: String?): JellyfinCatalogCheckpoint? = runCatching {
+ val fields = value?.split('|') ?: return null
+ if (fields.size != 8 || fields[0] != VERSION) return null
+ JellyfinCatalogCheckpoint(
+ movieSessionId = fields[1].toLong().takeIf { it > 0L } ?: return null,
+ seriesSessionId = fields[2].toLong().takeIf { it > 0L } ?: return null,
+ phase = JellyfinCatalogPhase.valueOf(fields[3]),
+ movieStartIndex = fields[4].toInt().takeIf { it >= 0 } ?: return null,
+ movieTotal = fields[5].toInt().takeIf { it >= 0 },
+ seriesStartIndex = fields[6].toInt().takeIf { it >= 0 } ?: return null,
+ seriesTotal = fields[7].toInt().takeIf { it >= 0 }
+ )
+ }.getOrNull()
+ }
+}
+
+private fun validatePage(
+ label: String,
+ startIndex: Int,
+ expectedTotal: Int,
+ reportedTotal: Int,
+ pageItemCount: Int,
+ stagedCount: Int,
+ maximumTotal: Int
+) {
+ if (reportedTotal != expectedTotal) {
+ throw JellyfinPaginationException("Jellyfin $label total changed during pagination")
+ }
+ if (reportedTotal > maximumTotal) {
+ throw JellyfinCatalogLimitException("Jellyfin $label catalog exceeds $maximumTotal items")
+ }
+ val nextStart = startIndex + pageItemCount
+ if (
+ pageItemCount < 0 ||
+ nextStart > expectedTotal ||
+ nextStart <= startIndex && nextStart < expectedTotal
+ ) {
+ throw JellyfinPaginationException("Jellyfin $label pagination did not advance")
+ }
+ if (stagedCount != nextStart) {
+ throw JellyfinPaginationException("Jellyfin $label pagination repeated or omitted items")
+ }
+}
diff --git a/data/src/main/java/com/streamvault/data/sync/ProviderWorkflowCommitFence.kt b/data/src/main/java/com/streamvault/data/sync/ProviderWorkflowCommitFence.kt
index 21210c69..a850584f 100644
--- a/data/src/main/java/com/streamvault/data/sync/ProviderWorkflowCommitFence.kt
+++ b/data/src/main/java/com/streamvault/data/sync/ProviderWorkflowCommitFence.kt
@@ -32,8 +32,10 @@ class ProviderWorkflowSupersededException(
*/
@Singleton
class ProviderWorkflowCommitFence @Inject constructor(
- private val workflowDao: ProviderWorkflowDao? = null
+ private val workflowDao: ProviderWorkflowDao?
) {
+ internal constructor() : this(null)
+
suspend fun assertCanCommit(providerId: Long) {
val execution = coroutineContext[ProviderWorkflowExecutionContext] ?: return
val lease = execution.lease
diff --git a/data/src/main/java/com/streamvault/data/sync/ProviderWorkflowRunner.kt b/data/src/main/java/com/streamvault/data/sync/ProviderWorkflowRunner.kt
index 8f585fbd..9f4816e6 100644
--- a/data/src/main/java/com/streamvault/data/sync/ProviderWorkflowRunner.kt
+++ b/data/src/main/java/com/streamvault/data/sync/ProviderWorkflowRunner.kt
@@ -5,6 +5,10 @@ import com.streamvault.data.local.dao.ProviderWorkflowDao
import com.streamvault.data.local.dao.ProviderWorkflowLease
import com.streamvault.data.local.entity.ProviderWorkflowPhase
import com.streamvault.data.local.entity.ProviderWorkflowReason
+import com.streamvault.data.remote.jellyfin.JellyfinCatalogLimitException
+import com.streamvault.data.remote.jellyfin.JellyfinItemLimitException
+import com.streamvault.data.remote.jellyfin.JellyfinPaginationException
+import com.streamvault.data.remote.jellyfin.JellyfinResponseTooLargeException
import java.io.IOException
import java.util.UUID
import javax.inject.Inject
@@ -41,6 +45,10 @@ enum class ProviderWorkflowDisposition {
object ProviderWorkFailureClassifier {
fun isRetryable(error: Throwable?): Boolean = when (error) {
+ is JellyfinPaginationException,
+ is JellyfinCatalogLimitException,
+ is JellyfinResponseTooLargeException,
+ is JellyfinItemLimitException -> false
is IOException -> true
is SQLiteException -> error.message.orEmpty().contains("locked", ignoreCase = true) ||
error.message.orEmpty().contains("busy", ignoreCase = true)
diff --git a/data/src/main/java/com/streamvault/data/sync/SyncCatalogStore.kt b/data/src/main/java/com/streamvault/data/sync/SyncCatalogStore.kt
index 50c482cc..897a60e7 100644
--- a/data/src/main/java/com/streamvault/data/sync/SyncCatalogStore.kt
+++ b/data/src/main/java/com/streamvault/data/sync/SyncCatalogStore.kt
@@ -23,6 +23,8 @@ import com.streamvault.domain.model.ContentType
import java.net.URI
import java.security.MessageDigest
import java.util.concurrent.atomic.AtomicLong
+import kotlinx.coroutines.NonCancellable
+import kotlinx.coroutines.withContext
import kotlin.random.Random
internal class SyncCatalogStore(
@@ -366,23 +368,27 @@ internal class SyncCatalogStore(
}
suspend fun stageChannelBatch(providerId: Long, sessionId: Long, channels: List<ChannelEntity>) {
- requireStageCapacity(
- currentCount = catalogSyncDao.countChannelStages(providerId, sessionId),
- incomingCount = channels.distinctBy { it.streamId }.size,
- maximum = sizeLimits.maxChannelsPerProvider,
- label = "channel"
- )
- insertStageRows(buildChannelStages(providerId, sessionId, channels), catalogSyncDao::insertChannelStages)
+ providerTransaction(providerId) {
+ requireStageCapacity(
+ currentCount = catalogSyncDao.countChannelStages(providerId, sessionId),
+ incomingCount = channels.distinctBy { it.streamId }.size,
+ maximum = sizeLimits.maxChannelsPerProvider,
+ label = "channel"
+ )
+ insertStageRows(buildChannelStages(providerId, sessionId, channels), catalogSyncDao::insertChannelStages)
+ }
}
suspend fun stageMovieBatch(providerId: Long, sessionId: Long, movies: List<MovieEntity>) {
- requireStageCapacity(
- currentCount = catalogSyncDao.countMovieStages(providerId, sessionId),
- incomingCount = movies.distinctBy { it.streamId }.size,
- maximum = sizeLimits.maxMoviesPerProvider,
- label = "movie"
- )
- insertStageRows(buildMovieStages(providerId, sessionId, movies), catalogSyncDao::insertMovieStages)
+ providerTransaction(providerId) {
+ requireStageCapacity(
+ currentCount = catalogSyncDao.countMovieStages(providerId, sessionId),
+ incomingCount = movies.distinctBy { it.streamId }.size,
+ maximum = sizeLimits.maxMoviesPerProvider,
+ label = "movie"
+ )
+ insertStageRows(buildMovieStages(providerId, sessionId, movies), catalogSyncDao::insertMovieStages)
+ }
}
suspend fun stageSeriesBatch(providerId: Long, sessionId: Long, series: List<SeriesEntity>) {
@@ -404,7 +410,17 @@ internal class SyncCatalogStore(
syncFingerprint = categoryFingerprint(category)
)
}
- insertStageRows(rows, catalogSyncDao::insertCategoryStages)
+ providerTransaction(providerId) {
+ rows.groupBy { it.type }.forEach { (type, typeRows) ->
+ requireStageCapacity(
+ currentCount = catalogSyncDao.countCategoryStages(providerId, sessionId, type.name),
+ incomingCount = typeRows.size,
+ maximum = sizeLimits.maxM3uCategoriesPerType,
+ label = "${type.name.lowercase()} category"
+ )
+ }
+ insertStageRows(rows, catalogSyncDao::insertCategoryStages)
+ }
}
suspend fun finalizeStagedImport(
@@ -438,6 +454,37 @@ internal class SyncCatalogStore(
}
}
+ /** Promotes one complete Jellyfin Movies+Series snapshot in a single Room transaction. */
+ suspend fun applyStagedJellyfinCatalog(
+ providerId: Long,
+ movieSessionId: Long,
+ seriesSessionId: Long,
+ movieCategories: List<CategoryEntity>,
+ seriesCategories: List<CategoryEntity>,
+ afterCatalogApply: suspend () -> Unit = {}
+ ) {
+ try {
+ providerTransaction(providerId) {
+ stageCategories(providerId, movieSessionId, movieCategories)
+ applyCategories(providerId, movieSessionId, "MOVIE")
+ applyMovies(providerId, movieSessionId)
+ catalogSyncDao.rebuildMovieFts()
+
+ stageCategories(providerId, seriesSessionId, seriesCategories)
+ applyCategories(providerId, seriesSessionId, "SERIES")
+ applySeries(providerId, seriesSessionId)
+ catalogSyncDao.rebuildSeriesFts()
+ afterCatalogApply()
+ }
+ movieDao.restoreWatchProgress(providerId)
+ } finally {
+ withContext(NonCancellable) {
+ clearSession(providerId, movieSessionId)
+ if (seriesSessionId != movieSessionId) clearSession(providerId, seriesSessionId)
+ }
+ }
+ }
+
suspend fun discardStagedImport(providerId: Long, sessionId: Long) {
clearSession(providerId, sessionId)
}
@@ -491,10 +538,12 @@ internal class SyncCatalogStore(
}
suspend fun clearProviderStaging(providerId: Long) {
- catalogSyncDao.clearProviderChannelStages(providerId)
- catalogSyncDao.clearProviderMovieStages(providerId)
- catalogSyncDao.clearProviderSeriesStages(providerId)
- catalogSyncDao.clearProviderCategoryStages(providerId)
+ transactionRunner.inTransaction {
+ catalogSyncDao.clearProviderChannelStages(providerId)
+ catalogSyncDao.clearProviderMovieStages(providerId)
+ catalogSyncDao.clearProviderSeriesStages(providerId)
+ catalogSyncDao.clearProviderCategoryStages(providerId)
+ }
}
private suspend fun applyCategories(
@@ -619,10 +668,14 @@ internal class SyncCatalogStore(
}
private suspend fun clearSession(providerId: Long, sessionId: Long) {
- catalogSyncDao.clearChannelStages(providerId, sessionId)
- catalogSyncDao.clearMovieStages(providerId, sessionId)
- catalogSyncDao.clearSeriesStages(providerId, sessionId)
- catalogSyncDao.clearCategoryStages(providerId, sessionId)
+ withContext(NonCancellable) {
+ transactionRunner.inTransaction {
+ catalogSyncDao.clearChannelStages(providerId, sessionId)
+ catalogSyncDao.clearMovieStages(providerId, sessionId)
+ catalogSyncDao.clearSeriesStages(providerId, sessionId)
+ catalogSyncDao.clearCategoryStages(providerId, sessionId)
+ }
+ }
}
private fun buildChannelStages(
diff --git a/data/src/main/java/com/streamvault/data/sync/SyncManager.kt b/data/src/main/java/com/streamvault/data/sync/SyncManager.kt
index 7ee83027..f20eb611 100644
--- a/data/src/main/java/com/streamvault/data/sync/SyncManager.kt
+++ b/data/src/main/java/com/streamvault/data/sync/SyncManager.kt
@@ -11,6 +11,7 @@ import com.streamvault.data.local.dao.MovieCategoryHydrationDao
import com.streamvault.data.local.dao.MovieDao
import com.streamvault.data.local.dao.ProgramDao
import com.streamvault.data.local.dao.ProviderDao
+import com.streamvault.data.local.dao.ProviderWorkflowDao
import com.streamvault.data.local.dao.SeriesCategoryHydrationDao
import com.streamvault.data.local.dao.SeriesDao
import com.streamvault.data.local.dao.TmdbIdentityDao
@@ -35,6 +36,10 @@ import com.streamvault.data.remote.stalker.StalkerPlaybackMode
import com.streamvault.data.remote.stalker.StalkerProvider
import com.streamvault.data.remote.stalker.StalkerProviderProfile
import com.streamvault.data.remote.jellyfin.JellyfinProvider
+import com.streamvault.data.remote.jellyfin.JellyfinCatalogLimitException
+import com.streamvault.data.remote.jellyfin.JellyfinItemLimitException
+import com.streamvault.data.remote.jellyfin.JellyfinPaginationException
+import com.streamvault.data.remote.jellyfin.JellyfinResponseTooLargeException
import com.streamvault.data.remote.stalker.StalkerTrafficCoordinator
import com.streamvault.data.remote.dto.XtreamCategory
import com.streamvault.data.remote.dto.XtreamSeriesItem
@@ -73,6 +78,7 @@ import dagger.hilt.android.qualifiers.ApplicationContext
import kotlinx.coroutines.async
import kotlinx.coroutines.Dispatchers
import kotlinx.coroutines.CancellationException
+import kotlinx.coroutines.NonCancellable
import kotlinx.coroutines.awaitAll
import kotlinx.coroutines.coroutineScope
import kotlinx.coroutines.flow.Flow
@@ -80,6 +86,7 @@ import kotlinx.coroutines.flow.StateFlow
import kotlinx.coroutines.flow.first
import kotlinx.coroutines.flow.map
import kotlinx.coroutines.withContext
+import kotlin.coroutines.coroutineContext
import kotlinx.coroutines.sync.Mutex
import kotlinx.coroutines.sync.Semaphore
import kotlinx.coroutines.sync.withLock
@@ -212,7 +219,8 @@ class SyncManager @Inject constructor(
private val transactionRunner: DatabaseTransactionRunner,
private val preferencesRepository: com.streamvault.data.preferences.PreferencesRepository,
private val syncProgressBus: SyncProgressBus,
- private val providerWorkflowCommitFence: ProviderWorkflowCommitFence
+ private val providerWorkflowCommitFence: ProviderWorkflowCommitFence,
+ private val providerWorkflowDao: ProviderWorkflowDao? = null
) {
private val syncStateTracker = SyncStateTracker()
private val syncErrorSanitizer = SyncErrorSanitizer()
@@ -3740,21 +3748,27 @@ class SyncManager @Inject constructor(
.first()
.mapTo(mutableSetOf()) { it.epgSourceId }
stats.header.tvgUrls.forEachIndexed { priority, url ->
- val source = existingSourcesByUrl[url] ?: when (
- val addResult = epgSourceRepository.addSource("Playlist EPG ${priority + 1}", url)
- ) {
- is Result.Success -> addResult.data
- is Result.Error -> {
- warnings += "Could not add playlist EPG source: ${addResult.message}"
- null
+ try {
+ val source = existingSourcesByUrl[url] ?: when (
+ val addResult = epgSourceRepository.addSource("Playlist EPG ${priority + 1}", url)
+ ) {
+ is Result.Success -> addResult.data
+ is Result.Error -> {
+ warnings += "Could not add playlist EPG source ${priority + 1}: ${addResult.message}"
+ null
+ }
+ Result.Loading -> null
}
- Result.Loading -> null
- }
- if (source != null && assignedSourceIds.add(source.id)) {
- when (val assignment = epgSourceRepository.assignSourceToProvider(provider.id, source.id, priority)) {
- is Result.Error -> warnings += "Could not assign playlist EPG source: ${assignment.message}"
- else -> Unit
+ if (source != null && assignedSourceIds.add(source.id)) {
+ when (val assignment = epgSourceRepository.assignSourceToProvider(provider.id, source.id, priority)) {
+ is Result.Error -> warnings += "Could not assign playlist EPG source ${priority + 1}: ${assignment.message}"
+ else -> Unit
+ }
}
+ } catch (e: CancellationException) {
+ throw e
+ } catch (e: Exception) {
+ warnings += "Could not configure playlist EPG source ${priority + 1}: ${sanitizeThrowableMessage(e)}"
}
}
}
@@ -4101,89 +4115,152 @@ class SyncManager @Inject constructor(
onProgress: ((String) -> Unit)?,
afterCatalogApply: suspend () -> Unit = {}
): SyncOutcome {
- val warnings = mutableListOf<String>()
+ val decryptedPassword = credentialCrypto.decryptIfNeeded(provider.password)
+ val decryptedProvider = provider.copy(password = decryptedPassword)
+ android.util.Log.d("JellyfinSync", "Starting Jellyfin sync for provider=${provider.id}")
+
+ val restoredCheckpoint = if (force) null else loadJellyfinCheckpoint(provider.id)
+ val canResume = restoredCheckpoint != null &&
+ isJellyfinCheckpointConsistent(provider.id, restoredCheckpoint)
+ var checkpoint = if (canResume) {
+ requireNotNull(restoredCheckpoint)
+ } else {
+ withContext(NonCancellable) { syncCatalogStore.clearProviderStaging(provider.id) }
+ JellyfinCatalogCheckpoint(
+ movieSessionId = syncCatalogStore.newSessionId(),
+ seriesSessionId = syncCatalogStore.newSessionId(),
+ phase = JellyfinCatalogPhase.MOVIES
+ ).also { saveJellyfinCheckpoint(provider.id, it) }
+ }
+
try {
- val decryptedPassword = credentialCrypto.decryptIfNeeded(provider.password)
- val decryptedProvider = provider.copy(password = decryptedPassword)
- android.util.Log.d("JellyfinSync", "Starting Jellyfin sync for provider=${provider.id}")
- progress(provider.id, onProgress, "Loading Jellyfin movies...")
- val movieSession = syncCatalogStore.newSessionId()
- val seriesSession = syncCatalogStore.newSessionId()
- syncCatalogStore.clearProviderStaging(provider.id)
- try {
- var movieStart = 0
- var movieTotal: Int? = null
+ if (checkpoint.phase == JellyfinCatalogPhase.MOVIES) {
+ progress(provider.id, onProgress, "Loading Jellyfin movies...")
do {
- when (val result = jellyfinProvider.fetchMoviesPage(decryptedProvider, movieStart)) {
+ when (val result = jellyfinProvider.fetchMoviesPage(decryptedProvider, checkpoint.movieStartIndex)) {
is com.streamvault.domain.model.Result.Error -> throw result.exception ?: IllegalStateException(result.message)
is com.streamvault.domain.model.Result.Success -> {
val page = result.data
- movieTotal = movieTotal ?: page.totalRecordCount
- if (page.totalRecordCount != movieTotal || page.totalRecordCount > 200_000) throw IllegalStateException("Jellyfin movie catalog is unstable or exceeds the supported limit")
- syncCatalogStore.stageMovieBatch(provider.id, movieSession, page.items)
- movieStart = page.nextStartIndex
+ syncCatalogStore.stageMovieBatch(provider.id, checkpoint.movieSessionId, page.items)
+ checkpoint = checkpoint.afterMoviePage(
+ reportedTotal = page.totalRecordCount,
+ pageItemCount = page.items.size,
+ stagedCount = catalogSyncDao.countMovieStages(provider.id, checkpoint.movieSessionId)
+ )
+ saveJellyfinCheckpoint(provider.id, checkpoint)
}
- is com.streamvault.domain.model.Result.Loading -> Unit
+ is com.streamvault.domain.model.Result.Loading ->
+ throw JellyfinPaginationException("Jellyfin movie page did not complete")
}
- } while (movieStart < (movieTotal ?: 0))
- if (catalogSyncDao.countMovieStages(provider.id, movieSession) != (movieTotal ?: 0)) {
- throw IllegalStateException("Jellyfin movie pagination returned duplicate or invalid items")
- }
+ } while (checkpoint.movieStartIndex < (checkpoint.movieTotal ?: 0))
+ checkpoint = checkpoint.copy(phase = JellyfinCatalogPhase.SERIES)
+ saveJellyfinCheckpoint(provider.id, checkpoint)
+ }
+ if (checkpoint.phase == JellyfinCatalogPhase.SERIES) {
progress(provider.id, onProgress, "Loading Jellyfin series...")
- var seriesStart = 0
- var seriesTotal: Int? = null
do {
- when (val result = jellyfinProvider.fetchSeriesPage(decryptedProvider, seriesStart)) {
+ when (val result = jellyfinProvider.fetchSeriesPage(decryptedProvider, checkpoint.seriesStartIndex)) {
is com.streamvault.domain.model.Result.Error -> throw result.exception ?: IllegalStateException(result.message)
is com.streamvault.domain.model.Result.Success -> {
val page = result.data
- seriesTotal = seriesTotal ?: page.totalRecordCount
- if (page.totalRecordCount != seriesTotal || page.totalRecordCount > 100_000) throw IllegalStateException("Jellyfin series catalog is unstable or exceeds the supported limit")
- syncCatalogStore.stageSeriesBatch(provider.id, seriesSession, page.items)
- seriesStart = page.nextStartIndex
+ syncCatalogStore.stageSeriesBatch(provider.id, checkpoint.seriesSessionId, page.items)
+ checkpoint = checkpoint.afterSeriesPage(
+ reportedTotal = page.totalRecordCount,
+ pageItemCount = page.items.size,
+ stagedCount = catalogSyncDao.countSeriesStages(provider.id, checkpoint.seriesSessionId)
+ )
+ saveJellyfinCheckpoint(provider.id, checkpoint)
}
- is com.streamvault.domain.model.Result.Loading -> Unit
+ is com.streamvault.domain.model.Result.Loading ->
+ throw JellyfinPaginationException("Jellyfin series page did not complete")
}
- } while (seriesStart < (seriesTotal ?: 0))
- if (catalogSyncDao.countSeriesStages(provider.id, seriesSession) != (seriesTotal ?: 0)) {
- throw IllegalStateException("Jellyfin series pagination returned duplicate or invalid items")
- }
+ } while (checkpoint.seriesStartIndex < (checkpoint.seriesTotal ?: 0))
+ checkpoint = checkpoint.copy(phase = JellyfinCatalogPhase.READY)
+ saveJellyfinCheckpoint(provider.id, checkpoint)
+ }
- progress(provider.id, onProgress, "Importing Jellyfin library...")
- syncCatalogStore.applyStagedMovieCatalog(
- provider.id,
- movieSession,
- listOf(com.streamvault.data.local.entity.CategoryEntity(
+ progress(provider.id, onProgress, "Importing Jellyfin library...")
+ syncCatalogStore.applyStagedJellyfinCatalog(
+ providerId = provider.id,
+ movieSessionId = checkpoint.movieSessionId,
+ seriesSessionId = checkpoint.seriesSessionId,
+ movieCategories = listOf(com.streamvault.data.local.entity.CategoryEntity(
providerId = provider.id,
categoryId = 1L,
name = "Movies",
type = com.streamvault.domain.model.ContentType.MOVIE
)),
- afterCatalogApply
- )
- syncCatalogStore.applyStagedSeriesCatalog(
- provider.id,
- seriesSession,
- listOf(com.streamvault.data.local.entity.CategoryEntity(
+ seriesCategories = listOf(com.streamvault.data.local.entity.CategoryEntity(
providerId = provider.id,
categoryId = 2L,
name = "Series",
type = com.streamvault.domain.model.ContentType.SERIES
)),
- afterCatalogApply
- )
- return SyncOutcome(warnings = warnings)
- } catch (error: Exception) {
- syncCatalogStore.clearProviderStaging(provider.id)
- throw error
+ afterCatalogApply = afterCatalogApply
+ )
+ saveJellyfinCheckpoint(provider.id, null)
+ return SyncOutcome()
+ } catch (cancelled: CancellationException) {
+ if (!hasDurableWorkflowContext(provider.id)) {
+ withContext(NonCancellable) { syncCatalogStore.clearProviderStaging(provider.id) }
}
- } catch (e: Exception) {
- warnings.add("Jellyfin sync failed: ${e.message.orEmpty()}")
- return SyncOutcome(partial = true, warnings = warnings)
+ throw cancelled
+ } catch (error: Exception) {
+ if (!isResumableJellyfinFailure(error) || !hasDurableWorkflowContext(provider.id)) {
+ withContext(NonCancellable) {
+ syncCatalogStore.clearProviderStaging(provider.id)
+ saveJellyfinCheckpoint(provider.id, null)
+ }
+ }
+ throw error
+ }
+ }
+
+ private suspend fun loadJellyfinCheckpoint(providerId: Long): JellyfinCatalogCheckpoint? {
+ val lease = coroutineContext[ProviderWorkflowExecutionContext]?.lease ?: return null
+ if (lease.providerId != providerId) return null
+ return JellyfinCatalogCheckpoint.decode(
+ providerWorkflowDao?.getCheckpoint(providerId, lease.generation, lease.phase)
+ )
+ }
+
+ private suspend fun saveJellyfinCheckpoint(providerId: Long, checkpoint: JellyfinCatalogCheckpoint?) {
+ val lease = coroutineContext[ProviderWorkflowExecutionContext]?.lease ?: return
+ if (lease.providerId != providerId) throw ProviderWorkflowSupersededException(providerId, lease.generation)
+ val workflowDao = providerWorkflowDao ?: return
+ if (workflowDao.updateRunningCheckpoint(
+ providerId = providerId,
+ generation = lease.generation,
+ phase = lease.phase,
+ token = lease.token,
+ checkpoint = checkpoint?.encode(),
+ now = System.currentTimeMillis()
+ ) != 1
+ ) {
+ throw ProviderWorkflowSupersededException(providerId, lease.generation)
}
}
+ private suspend fun isJellyfinCheckpointConsistent(
+ providerId: Long,
+ checkpoint: JellyfinCatalogCheckpoint
+ ): Boolean {
+ val movieCount = catalogSyncDao.countMovieStages(providerId, checkpoint.movieSessionId)
+ val seriesCount = catalogSyncDao.countSeriesStages(providerId, checkpoint.seriesSessionId)
+ return checkpoint.isConsistent(movieCount, seriesCount)
+ }
+
+ private suspend fun hasDurableWorkflowContext(providerId: Long): Boolean =
+ coroutineContext[ProviderWorkflowExecutionContext]?.lease?.providerId == providerId && providerWorkflowDao != null
+
+ private fun isResumableJellyfinFailure(error: Exception): Boolean =
+ error is IOException &&
+ error !is JellyfinPaginationException &&
+ error !is JellyfinCatalogLimitException &&
+ error !is JellyfinResponseTooLargeException &&
+ error !is JellyfinItemLimitException
+
/**
* Returned by [syncProviderEpg] so callers can distinguish between warning-only
* degradations and transient network/IO failures that WorkManager should retry.
diff --git a/data/src/main/java/com/streamvault/data/sync/SyncManagerM3uImporter.kt b/data/src/main/java/com/streamvault/data/sync/SyncManagerM3uImporter.kt
index 44dd91e7..f7c78f78 100644
--- a/data/src/main/java/com/streamvault/data/sync/SyncManagerM3uImporter.kt
+++ b/data/src/main/java/com/streamvault/data/sync/SyncManagerM3uImporter.kt
@@ -4,6 +4,8 @@ import android.util.Log
import com.streamvault.data.local.entity.ChannelEntity
import com.streamvault.data.local.entity.MovieEntity
import com.streamvault.data.parser.M3uParser
+import com.streamvault.data.parser.M3uMediaKind
+import com.streamvault.data.parser.M3uVodClassifier
import com.streamvault.data.remote.http.HttpRequestProfile
import com.streamvault.data.remote.http.useCancellableResponse
import com.streamvault.data.remote.http.safeRequestIdentitySummary
@@ -15,6 +17,8 @@ import com.streamvault.domain.model.ContentType
import com.streamvault.domain.model.Provider
import com.streamvault.domain.sync.Section
import com.streamvault.domain.sync.SyncProgress
+import kotlinx.coroutines.NonCancellable
+import kotlinx.coroutines.withContext
import okhttp3.OkHttpClient
import okhttp3.Request
import okhttp3.Response
@@ -34,7 +38,8 @@ internal class SyncManagerM3uImporter(
private val retryTransient: suspend (suspend () -> Unit) -> Unit,
private val progress: (Long, ((String) -> Unit)?, String) -> Unit,
private val emitProgress: (Long, SyncProgress) -> Unit,
- private val sizeLimits: CatalogSizeLimits = CatalogSizeLimits()
+ private val sizeLimits: CatalogSizeLimits = CatalogSizeLimits(),
+ private val vodClassifier: M3uVodClassifier = M3uVodClassifier()
) {
suspend fun importPlaylist(
provider: Provider,
@@ -78,6 +83,15 @@ internal class SyncManagerM3uImporter(
val warnings = mutableListOf<String>()
var insecureStreamCount = 0
+ fun enforceInvalidEntryRatio() {
+ val candidateCount = parsedCount + invalidEntryCount
+ if (candidateCount >= 100 &&
+ invalidEntryCount * 100 > candidateCount * sizeLimits.maxM3uInvalidEntryRatioPercent
+ ) {
+ throw CatalogAdmissionExceeded("M3U invalid-entry ratio limit exceeded")
+ }
+ }
+
try {
openPlaylistStream(provider) { streamed ->
streamed.contentLength
@@ -102,6 +116,9 @@ internal class SyncManagerM3uImporter(
m3uParser.parseStreaming(
inputStream = input,
onHeader = { parsedHeader ->
+ requireM3uFieldBounds(
+ parsedHeader.tvgUrls + listOfNotNull(parsedHeader.userAgent)
+ )
val validEpgUrls = parsedHeader.tvgUrls.filter { UrlSecurityPolicy.validateOptionalEpgUrl(it) == null }
if (validEpgUrls.size != parsedHeader.tvgUrls.size) {
warnings += "Ignored unsupported EPG URL from playlist header."
@@ -113,7 +130,26 @@ internal class SyncManagerM3uImporter(
if (parsedCount > sizeLimits.maxM3uEntries) {
throw CatalogAdmissionExceeded("M3U entry limit exceeded")
}
- requireM3uFieldBounds(entry.name, entry.url, entry.groupTitle)
+ enforceInvalidEntryRatio()
+ requireM3uFieldBounds(
+ listOf(
+ entry.name,
+ entry.groupTitle,
+ entry.tvgId,
+ entry.tvgName,
+ entry.tvgLogo,
+ entry.tvgLanguage,
+ entry.tvgCountry,
+ entry.catchUp,
+ entry.catchUpSource,
+ entry.timeshift,
+ entry.url,
+ entry.userAgent,
+ entry.rating,
+ entry.year,
+ entry.genre
+ )
+ )
if (parsedCount >= nextMilestone) {
progress(provider.id, onProgress, "Imported $parsedCount playlist entries...")
// D14 — emission M3U etape Imported : current = nombre d'entrees
@@ -138,7 +174,11 @@ internal class SyncManagerM3uImporter(
val safeLogoUrl = UrlSecurityPolicy.sanitizeImportedAssetUrl(entry.tvgLogo)
val safeCatchUpSource = UrlSecurityPolicy.sanitizeImportedAssetUrl(entry.catchUpSource)
- if (provider.m3uVodClassificationEnabled && M3uParser.isVodEntry(entry)) {
+ val mediaClassification = vodClassifier.classify(
+ entry = entry,
+ override = if (provider.m3uVodClassificationEnabled) null else M3uMediaKind.LIVE
+ )
+ if (mediaClassification.isVod) {
if (!includeMovies) return@parseStreaming
val groupTitle = entry.groupTitle.ifBlank { "Uncategorized" }
val stableStreamId = stableId(
@@ -219,13 +259,9 @@ internal class SyncManagerM3uImporter(
},
onInvalidEntry = {
invalidEntryCount++
- val candidateCount = parsedCount + invalidEntryCount
- if (candidateCount >= 100 &&
- invalidEntryCount * 100 > candidateCount * sizeLimits.maxM3uInvalidEntryRatioPercent
- ) {
- throw CatalogAdmissionExceeded("M3U invalid-entry ratio limit exceeded")
- }
- }
+ enforceInvalidEntryRatio()
+ },
+ declaredCharset = streamed.declaredCharset
)
}
}
@@ -236,8 +272,12 @@ internal class SyncManagerM3uImporter(
// empty stage with includeLive=true runs stale deletion and wipes the entire
// live-TV catalog — even though the absence of entries may reflect a server
// error or a filtered playlist rather than a legitimate empty provider.
- val effectiveLive = includeLive && liveCount > 0
- val effectiveMovies = includeMovies && movieCount > 0
+ // A successful full-playlist refresh must also commit an empty opposite section.
+ // Otherwise changing the user's VOD-classification override leaves stale rows in
+ // Movies or Live TV. Section-only retries retain the empty-catalog safeguard.
+ val isSuccessfulFullRefresh = includeLive && includeMovies && parsedCount > 0
+ val effectiveLive = includeLive && (liveCount > 0 || isSuccessfulFullRefresh)
+ val effectiveMovies = includeMovies && (movieCount > 0 || isSuccessfulFullRefresh)
syncCatalogStore.finalizeStagedImport(
providerId = provider.id,
sessionId = sessionId,
@@ -248,7 +288,9 @@ internal class SyncManagerM3uImporter(
afterCatalogApply = afterCatalogApply
)
} finally {
- syncCatalogStore.discardStagedImport(provider.id, sessionId)
+ withContext(NonCancellable) {
+ syncCatalogStore.discardStagedImport(provider.id, sessionId)
+ }
}
if (insecureStreamCount > 0) {
@@ -290,7 +332,8 @@ internal class SyncManagerM3uImporter(
inputStream = input,
contentEncoding = response.header("Content-Encoding"),
contentLength = body.contentLength().takeIf { it >= 0L },
- sourceName = urlStr
+ sourceName = urlStr,
+ declaredCharset = body.contentType()?.charset(null)
)
)
}
@@ -400,7 +443,7 @@ internal class SyncManagerM3uImporter(
return value.orEmpty().lowercase().replace(Regex("\\s+"), " ").trim()
}
- private fun requireM3uFieldBounds(vararg fields: String?) {
+ private fun requireM3uFieldBounds(fields: Iterable<String?>) {
if (fields.any { it != null && it.length > sizeLimits.maxM3uFieldLength }) {
throw CatalogAdmissionExceeded("M3U field length limit exceeded")
}
diff --git a/data/src/main/java/com/streamvault/data/sync/SyncManagerSupport.kt b/data/src/main/java/com/streamvault/data/sync/SyncManagerSupport.kt
index 99761557..1f137af6 100644
--- a/data/src/main/java/com/streamvault/data/sync/SyncManagerSupport.kt
+++ b/data/src/main/java/com/streamvault/data/sync/SyncManagerSupport.kt
@@ -7,6 +7,7 @@ import com.streamvault.data.remote.dto.XtreamCategory
import com.streamvault.data.util.AdultContentClassifier
import com.streamvault.domain.model.ContentType
import java.io.InputStream
+import java.nio.charset.Charset
import java.security.MessageDigest
internal data class SyncOutcome(
@@ -104,7 +105,8 @@ internal data class StreamedPlaylist(
val inputStream: InputStream,
val contentEncoding: String? = null,
val contentLength: Long? = null,
- val sourceName: String? = null
+ val sourceName: String? = null,
+ val declaredCharset: Charset? = null
)
internal class FallbackCategoryCollector(
diff --git a/data/src/main/java/com/streamvault/data/sync/SyncWorker.kt b/data/src/main/java/com/streamvault/data/sync/SyncWorker.kt
index 0f68d340..e03e36fa 100644
--- a/data/src/main/java/com/streamvault/data/sync/SyncWorker.kt
+++ b/data/src/main/java/com/streamvault/data/sync/SyncWorker.kt
@@ -2,6 +2,7 @@ package com.streamvault.data.sync
import android.content.Context
import android.database.sqlite.SQLiteException
+import android.database.sqlite.SQLiteFullException
import android.util.Log
import androidx.work.CoroutineWorker
import androidx.work.WorkerParameters
@@ -28,36 +29,28 @@ class SyncWorker(
Log.d("SyncWorker", "Starting data maintenance...")
return try {
val entryPoint = EntryPointAccessors.fromApplication(applicationContext, SyncWorkerEntryPoint::class.java)
- val report = entryPoint.databaseMaintenanceManager().runDailyMaintenance()
- entryPoint.preferencesRepository().setLastMaintenanceSnapshot(
- DatabaseMaintenanceSnapshot(
- ranAt = System.currentTimeMillis(),
- deletedPrograms = report.deletedPrograms,
- deletedExternalProgrammes = report.deletedExternalProgrammes,
- deletedOrphanEpisodes = report.deletedOrphanEpisodes,
- deletedStaleFavorites = report.deletedStaleFavorites,
- vacuumRan = report.vacuumRan,
- mainDbBytes = report.statsAfterVacuum.mainDbBytes,
- walBytes = report.statsAfterVacuum.walBytes,
- reclaimableBytes = report.statsAfterVacuum.reclaimableBytes,
- channelRows = report.tableStats.channels,
- movieRows = report.tableStats.movies,
- seriesRows = report.tableStats.series,
- episodeRows = report.tableStats.episodes,
- programRows = report.tableStats.programs,
- epgProgrammeRows = report.tableStats.epgProgrammes,
- playbackHistoryRows = report.tableStats.playbackHistory,
- favoriteRows = report.tableStats.favorites
+ when (
+ val outcome = completeMaintenanceRun(
+ runMaintenance = entryPoint.databaseMaintenanceManager()::runDailyMaintenance,
+ persistSnapshot = entryPoint.preferencesRepository()::setLastMaintenanceSnapshot
)
- )
- Log.d(
- "SyncWorker",
- "Maintenance complete: oldPrograms=${report.deletedPrograms}, orphanEpisodes=${report.deletedOrphanEpisodes}, " +
- "staleFavorites=${report.deletedStaleFavorites}, vacuumRan=${report.vacuumRan}, " +
- "dbBytes=${report.statsAfterVacuum.mainDbBytes}, walBytes=${report.statsAfterVacuum.walBytes}, " +
- "reclaimableBytes=${report.statsAfterVacuum.reclaimableBytes}"
- )
- Result.success()
+ ) {
+ is MaintenanceWorkerOutcome.Completed -> {
+ val report = outcome.report
+ Log.d(
+ "SyncWorker",
+ "Maintenance complete: oldPrograms=${report.deletedPrograms}, orphanEpisodes=${report.deletedOrphanEpisodes}, " +
+ "staleFavorites=${report.deletedStaleFavorites}, vacuumRan=${report.vacuumRan}, " +
+ "dbBytes=${report.statsAfterVacuum.mainDbBytes}, walBytes=${report.statsAfterVacuum.walBytes}, " +
+ "reclaimableBytes=${report.statsAfterVacuum.reclaimableBytes}"
+ )
+ Result.success()
+ }
+ MaintenanceWorkerOutcome.DeferredForActiveSync -> {
+ Log.i("SyncWorker", "Deferring maintenance because provider sync work is active")
+ Result.retry()
+ }
+ }
} catch (cancelled: kotlinx.coroutines.CancellationException) {
throw cancelled
} catch (e: Exception) {
@@ -66,12 +59,55 @@ class SyncWorker(
}
}
- private fun shouldRetry(error: Throwable): Boolean {
- return when (error) {
- is java.io.IOException -> true
- is SQLiteException -> error.message.orEmpty().contains("locked", ignoreCase = true) ||
- error.message.orEmpty().contains("busy", ignoreCase = true)
- else -> false
- }
+ private fun shouldRetry(error: Throwable): Boolean = shouldRetryMaintenance(error)
+}
+
+internal sealed interface MaintenanceWorkerOutcome {
+ data class Completed(val report: DatabaseMaintenanceManager.MaintenanceReport) : MaintenanceWorkerOutcome
+ data object DeferredForActiveSync : MaintenanceWorkerOutcome
+}
+
+internal suspend fun completeMaintenanceRun(
+ runMaintenance: suspend () -> DatabaseMaintenanceManager.MaintenanceRunResult,
+ persistSnapshot: suspend (DatabaseMaintenanceSnapshot) -> Unit,
+ nowMillis: () -> Long = System::currentTimeMillis
+): MaintenanceWorkerOutcome = when (val result = runMaintenance()) {
+ is DatabaseMaintenanceManager.MaintenanceRunResult.Completed -> {
+ val report = result.report
+ persistSnapshot(
+ DatabaseMaintenanceSnapshot(
+ ranAt = nowMillis(),
+ deletedPrograms = report.deletedPrograms,
+ deletedExternalProgrammes = report.deletedExternalProgrammes,
+ deletedOrphanEpisodes = report.deletedOrphanEpisodes,
+ deletedStaleFavorites = report.deletedStaleFavorites,
+ vacuumRan = report.vacuumRan,
+ mainDbBytes = report.statsAfterVacuum.mainDbBytes,
+ walBytes = report.statsAfterVacuum.walBytes,
+ reclaimableBytes = report.statsAfterVacuum.reclaimableBytes,
+ channelRows = report.tableStats.channels,
+ movieRows = report.tableStats.movies,
+ seriesRows = report.tableStats.series,
+ episodeRows = report.tableStats.episodes,
+ programRows = report.tableStats.programs,
+ epgProgrammeRows = report.tableStats.epgProgrammes,
+ playbackHistoryRows = report.tableStats.playbackHistory,
+ favoriteRows = report.tableStats.favorites
+ )
+ )
+ MaintenanceWorkerOutcome.Completed(report)
+ }
+ DatabaseMaintenanceManager.MaintenanceRunResult.DeferredForActiveSync ->
+ MaintenanceWorkerOutcome.DeferredForActiveSync
+}
+
+internal fun shouldRetryMaintenance(error: Throwable): Boolean = when (error) {
+ is java.io.IOException,
+ is SQLiteFullException -> true
+ is SQLiteException -> error.message.orEmpty().let { message ->
+ message.contains("locked", ignoreCase = true) ||
+ message.contains("busy", ignoreCase = true) ||
+ message.contains("full", ignoreCase = true)
}
+ else -> false
}
diff --git a/data/src/test/java/com/streamvault/data/epg/EpgResolutionEngineTest.kt b/data/src/test/java/com/streamvault/data/epg/EpgResolutionEngineTest.kt
index bfb26c5f..fc57b729 100644
--- a/data/src/test/java/com/streamvault/data/epg/EpgResolutionEngineTest.kt
+++ b/data/src/test/java/com/streamvault/data/epg/EpgResolutionEngineTest.kt
@@ -92,6 +92,35 @@ class EpgResolutionEngineTest {
assertThat(mapping.confidence).isEqualTo(1.0f)
}
+ @Test
+ fun `resolveForProvider_accentedExactId_remainsAnExactMatch`() = runTest {
+ val channel = makeChannel(id = 1, name = "Chaîne Été", epgChannelId = "chaîne-é")
+ whenever(channelDao.getByProviderSync(PROVIDER_ID)).thenReturn(listOf(channel))
+ whenever(providerEpgSourceDao.getEnabledForProviderSync(PROVIDER_ID)).thenReturn(
+ listOf(ProviderEpgSourceEntity(id = 1, providerId = PROVIDER_ID, epgSourceId = SOURCE_1, priority = 0))
+ )
+ whenever(epgChannelDao.getBySources(listOf(SOURCE_1))).thenReturn(
+ listOf(
+ EpgChannelEntity(
+ id = 1,
+ epgSourceId = SOURCE_1,
+ xmltvChannelId = "chaîne-é",
+ displayName = "Chaîne Été",
+ normalizedName = EpgNameNormalizer.normalize("Chaîne Été")
+ )
+ )
+ )
+ whenever(channelEpgMappingDao.getForProvider(PROVIDER_ID)).thenReturn(emptyList())
+
+ val summary = engine.resolveForProvider(PROVIDER_ID)
+
+ assertThat(summary.exactIdMatches).isEqualTo(1)
+ val captor = argumentCaptor<List<ChannelEpgMappingEntity>>()
+ verify(channelEpgMappingDao).replaceForProvider(any(), captor.capture())
+ assertThat(captor.firstValue.single().xmltvChannelId).isEqualTo("chaîne-é")
+ assertThat(captor.firstValue.single().matchType).isEqualTo(EpgMatchType.EXACT_ID.name)
+ }
+
@Test
fun `resolveForProvider_normalizedNameFallback_resolvesAsNormalizedName`() = runTest {
// Channel has no matching epgChannelId but name matches after normalization
diff --git a/data/src/test/java/com/streamvault/data/local/ProviderWorkflowDaoTest.kt b/data/src/test/java/com/streamvault/data/local/ProviderWorkflowDaoTest.kt
index 39685bbc..566c18c1 100644
--- a/data/src/test/java/com/streamvault/data/local/ProviderWorkflowDaoTest.kt
+++ b/data/src/test/java/com/streamvault/data/local/ProviderWorkflowDaoTest.kt
@@ -70,6 +70,51 @@ class ProviderWorkflowDaoTest {
assertThat(workflow.generation).isEqualTo(1L)
}
+ @Test
+ fun `running checkpoint survives retryable worker loss and is fenced by lease token`() = runTest {
+ insertProvider()
+ val dao = database.providerWorkflowDao()
+ val ticket = dao.request(
+ providerId = PROVIDER_ID,
+ phase = ProviderWorkflowPhase.PRIMARY_CATALOG,
+ reason = ProviderWorkflowReason.PERIODIC,
+ now = 100L
+ )
+ val firstLease = checkNotNull(
+ dao.claim(ticket, "first-process", 101L, 1_000L, staleHeartbeatBefore = 0L)
+ )
+
+ assertThat(
+ dao.updateRunningCheckpoint(
+ PROVIDER_ID,
+ ticket.generation,
+ ticket.phase,
+ firstLease.token,
+ "jellyfin-v1|101|202|MOVIES|100|300|0|-1",
+ now = 102L
+ )
+ ).isEqualTo(1)
+ assertThat(
+ dao.updateRunningCheckpoint(
+ PROVIDER_ID,
+ ticket.generation,
+ ticket.phase,
+ "wrong-owner",
+ "corrupt",
+ now = 103L
+ )
+ ).isEqualTo(0)
+ assertThat(dao.fail(firstLease, 104L, "IO", "process stopped", retryable = true)).isTrue()
+
+ val resumedLease = checkNotNull(
+ dao.claim(ticket, "second-process", 105L, 1_000L, staleHeartbeatBefore = 0L)
+ )
+
+ assertThat(resumedLease.generation).isEqualTo(firstLease.generation)
+ assertThat(dao.getCheckpoint(PROVIDER_ID, ticket.generation, ticket.phase))
+ .isEqualTo("jellyfin-v1|101|202|MOVIES|100|300|0|-1")
+ }
+
@Test
fun `superseding request fences the old owner and records a new generation`() = runTest {
insertProvider()
diff --git a/data/src/test/java/com/streamvault/data/parser/M3uParserTest.kt b/data/src/test/java/com/streamvault/data/parser/M3uParserTest.kt
index 751c0886..4dc5f34a 100644
--- a/data/src/test/java/com/streamvault/data/parser/M3uParserTest.kt
+++ b/data/src/test/java/com/streamvault/data/parser/M3uParserTest.kt
@@ -5,6 +5,7 @@ import kotlinx.coroutines.runBlocking
import org.junit.Before
import org.junit.Test
import java.io.ByteArrayOutputStream
+import java.nio.charset.Charset
import java.util.zip.GZIPOutputStream
/**
@@ -230,6 +231,27 @@ class M3uParserTest {
assertThat(invalidEntries).isEqualTo(1)
}
+ @Test
+ fun `parseStreaming_reportsReplacedAndUnterminatedCandidates`() = runBlocking {
+ val entries = mutableListOf<M3uParser.M3uEntry>()
+ var invalidEntries = 0
+
+ parser.parseStreaming(
+ inputStream = """
+ #EXTM3U
+ #EXTINF:-1,Replaced candidate
+ #EXTINF:-1,Valid candidate
+ https://stream.example.com/valid.ts
+ #EXTINF:-1,Unterminated candidate
+ """.trimIndent().byteInputStream(),
+ onEntry = entries::add,
+ onInvalidEntry = { invalidEntries++ }
+ )
+
+ assertThat(entries.map { it.name }).containsExactly("Valid candidate")
+ assertThat(invalidEntries).isEqualTo(2)
+ }
+
@Test
fun `parse_utf16Bom_decodesPlaylist`() {
val playlist = "#EXTM3U\n#EXTINF:-1 tvg-id=\"cafe\" group-title=\"Café\",Café\nhttps://stream.example.com/cafe.m3u8\n"
@@ -241,6 +263,111 @@ class M3uParserTest {
assertThat(result.entries.single().groupTitle).isEqualTo("Café")
}
+ @Test
+ fun `parse_utf16BigEndianBom_decodesPlaylist`() {
+ val playlist = "#EXTM3U\n#EXTINF:-1 tvg-id=\"cafe\" group-title=\"Café\",Café\nhttps://stream.example.com/cafe.m3u8\n"
+ val utf16 = byteArrayOf(0xFE.toByte(), 0xFF.toByte()) + playlist.toByteArray(Charsets.UTF_16BE)
+
+ val result = parser.parse(utf16.inputStream())
+
+ assertThat(result.entries.single().name).isEqualTo("Café")
+ assertThat(result.entries.single().groupTitle).isEqualTo("Café")
+ }
+
+ @Test
+ fun `parse_windows1252WithoutCharset_usesLegacyFallback`() {
+ val windows1252 = Charset.forName("windows-1252")
+ val playlist = "#EXTM3U\n#EXTINF:-1 tvg-id=\"café-tv\" group-title=\"Ciné – FR\",Café Télévision\nhttps://stream.example.com/cafe.ts\n"
+
+ val result = parser.parse(playlist.toByteArray(windows1252).inputStream())
+
+ val entry = result.entries.single()
+ assertThat(entry.name).isEqualTo("Café Télévision")
+ assertThat(entry.groupTitle).isEqualTo("Ciné – FR")
+ assertThat(entry.tvgId).isEqualTo("café-tv")
+ }
+
+ @Test
+ fun `parse_honorsValidatedIso88591Charset`() {
+ val iso88591 = Charsets.ISO_8859_1
+ val playlist = "#EXTM3U\n#EXTINF:-1 group-title=\"Télévision\",Café\nhttps://stream.example.com/cafe.ts\n"
+
+ val result = parser.parse(
+ playlist.toByteArray(iso88591).inputStream(),
+ declaredCharset = iso88591
+ )
+
+ assertThat(result.entries.single().name).isEqualTo("Café")
+ assertThat(result.entries.single().groupTitle).isEqualTo("Télévision")
+ }
+
+ @Test
+ fun `parse_bomTakesPrecedenceOverDeclaredCharset`() {
+ val playlist = "#EXTM3U\n#EXTINF:-1,Café\nhttps://stream.example.com/cafe.ts\n"
+ val utf8WithBom = byteArrayOf(0xEF.toByte(), 0xBB.toByte(), 0xBF.toByte()) +
+ playlist.toByteArray(Charsets.UTF_8)
+
+ val result = parser.parse(
+ utf8WithBom.inputStream(),
+ declaredCharset = Charset.forName("windows-1252")
+ )
+
+ assertThat(result.entries.single().name).isEqualTo("Café")
+ }
+
+ @Test
+ fun `parse_ignoresUnsupportedDeclaredCharsetAndUsesUtf8Policy`() {
+ val playlist = "#EXTM3U\n#EXTINF:-1,Chaîne Été\nhttps://stream.example.com/ete.ts\n"
+
+ val result = parser.parse(
+ playlist.toByteArray(Charsets.UTF_8).inputStream(),
+ declaredCharset = Charset.forName("UTF-32BE")
+ )
+
+ assertThat(result.entries.single().name).isEqualTo("Chaîne Été")
+ }
+
+ @Test
+ fun `parse_invalidUtf8Sequence_fallsBackWithoutReplacementCharacters`() {
+ val prefix = "#EXTM3U\n#EXTINF:-1 group-title=\"Legacy\",".toByteArray()
+ val malformedName = byteArrayOf(0xC3.toByte(), 0x28)
+ val suffix = "\nhttps://stream.example.com/legacy.ts\n".toByteArray()
+
+ val result = parser.parse((prefix + malformedName + suffix).inputStream())
+
+ assertThat(result.entries.single().name).isEqualTo("Ã(")
+ assertThat(result.entries.single().name).doesNotContain("�")
+ }
+
+ @Test
+ fun `parseStreaming_windows1252Fallback_preservesAccentedEpgIdentity`() = runBlocking {
+ val entries = mutableListOf<M3uParser.M3uEntry>()
+ val playlist = "#EXTM3U\n#EXTINF:-1 tvg-id=\"chaîne-é\" group-title=\"Télé\",Chaîne Été\nhttps://stream.example.com/ete.ts\n"
+
+ parser.parseStreaming(
+ inputStream = playlist.toByteArray(Charset.forName("windows-1252")).inputStream(),
+ onEntry = entries::add
+ )
+
+ assertThat(entries.single().tvgId).isEqualTo("chaîne-é")
+ assertThat(entries.single().name).isEqualTo("Chaîne Été")
+ }
+
+ @Test
+ fun `parse_headerGuideUrls_deduplicatesAndCapsAtEightInSourceOrder`() {
+ val urls = (1..10).map { "https://epg.example.com/$it.xml" }
+ val headerUrls = (urls.take(2) + urls[0] + urls.drop(2)).joinToString(", ")
+ val result = parser.parse(
+ """
+ #EXTM3U x-tvg-url="$headerUrls"
+ #EXTINF:-1,News
+ https://stream.example.com/news.ts
+ """.trimIndent().byteInputStream()
+ )
+
+ assertThat(result.header.tvgUrls).containsExactlyElementsIn(urls.take(8)).inOrder()
+ }
+
@Test
fun `parse_catchUpAttributes_extracted`() {
val m3u = """
@@ -355,8 +482,6 @@ class M3uParserTest {
@Test
fun `isVodEntry_movieExtension_returnsTrue`() {
- // Create a throwaway SyncManager reference to access the internal helper
- // We test via the M3uParser-level behaviour: parse a playlist and verify isVod logic
val entries = parseEntries(buildPlaylist(
"http://vod.example.com/avatar.mp4" to "Movies",
"http://vod.example.com/inception.mkv" to "VOD",
@@ -364,15 +489,7 @@ class M3uParserTest {
))
assertThat(entries).hasSize(3)
- // All three entries should be classified as VOD by SyncManager
- entries.forEach { entry ->
- val url = entry.url.lowercase()
- val group = entry.groupTitle.lowercase()
- val isVod = url.endsWith(".mp4") || url.endsWith(".mkv") || url.endsWith(".avi") ||
- url.contains("/movie/") || group.contains("movie") ||
- group.contains("vod") || group.contains("film")
- assertThat(isVod).isTrue()
- }
+ entries.forEach { entry -> assertThat(M3uParser.isVodEntry(entry)).isTrue() }
}
@Test
@@ -383,14 +500,7 @@ class M3uParserTest {
))
assertThat(entries).hasSize(2)
- entries.forEach { entry ->
- val url = entry.url.lowercase()
- val group = entry.groupTitle.lowercase()
- val isVod = url.endsWith(".mp4") || url.endsWith(".mkv") || url.endsWith(".avi") ||
- url.contains("/movie/") || group.contains("movie") ||
- group.contains("vod") || group.contains("film")
- assertThat(isVod).isFalse()
- }
+ entries.forEach { entry -> assertThat(M3uParser.isVodEntry(entry)).isFalse() }
}
// ── Helpers ────────────────────────────────────────────────────
diff --git a/data/src/test/java/com/streamvault/data/parser/M3uVodClassifierTest.kt b/data/src/test/java/com/streamvault/data/parser/M3uVodClassifierTest.kt
new file mode 100644
index 00000000..edba8e49
--- /dev/null
+++ b/data/src/test/java/com/streamvault/data/parser/M3uVodClassifierTest.kt
@@ -0,0 +1,144 @@
+package com.streamvault.data.parser
+
+import com.google.common.truth.Truth.assertThat
+import org.junit.Test
+
+class M3uVodClassifierTest {
+
+ private val classifier = M3uVodClassifier()
+
+ @Test
+ fun `file extensions ignore query fragment case and percent encoding`() {
+ val urls = listOf(
+ "https://cdn.example.com/film.MP4?token=abc",
+ "https://cdn.example.com/show.mkv#preview",
+ "https://cdn.example.com/archive.avi?token=abc#preview",
+ "https://cdn.example.com/encoded%2Emp4?token=abc"
+ )
+
+ urls.forEach { url ->
+ val result = classifier.classify(entry(url = url))
+ assertThat(result.kind).isEqualTo(M3uMediaKind.VOD)
+ assertThat(result.evidence).isEqualTo(M3uVodEvidence.FILE_EXTENSION)
+ assertThat(result.confidence).isEqualTo(1.0f)
+ }
+ }
+
+ @Test
+ fun `extensionless and encoded movie paths classify as vod`() {
+ val urls = listOf(
+ "https://provider.example.com/movie/user/pass/12345",
+ "https://provider.example.com/%6Dovie/user/pass/67890",
+ "https://provider.example.com/vod/13579?token=abc"
+ )
+
+ urls.forEach { url ->
+ val result = classifier.classify(entry(url = url))
+ assertThat(result.kind).isEqualTo(M3uMediaKind.VOD)
+ assertThat(result.evidence).isEqualTo(M3uVodEvidence.PATH_SEGMENT)
+ }
+ }
+
+ @Test
+ fun `localized group keywords classify extensionless entries as vod`() {
+ listOf("Películas", "Filmes", "Фильмы", "电影", "映画", "أفلام", "סרטים").forEach { group ->
+ val result = classifier.classify(
+ entry(url = "https://provider.example.com/item/123", group = group)
+ )
+ assertThat(result.kind).isEqualTo(M3uMediaKind.VOD)
+ assertThat(result.evidence).isEqualTo(M3uVodEvidence.GROUP_KEYWORD)
+ }
+ }
+
+ @Test
+ fun `name keyword classifies an otherwise ambiguous extensionless entry`() {
+ val result = classifier.classify(
+ entry(
+ url = "https://provider.example.com/item/123",
+ group = "General",
+ name = "Classic Cinema"
+ )
+ )
+
+ assertThat(result.kind).isEqualTo(M3uMediaKind.VOD)
+ assertThat(result.evidence).isEqualTo(M3uVodEvidence.NAME_KEYWORD)
+ assertThat(result.confidence).isEqualTo(0.55f)
+ }
+
+ @Test
+ fun `strong live evidence beats a misleading movie label`() {
+ val hls = classifier.classify(
+ entry(url = "https://live.example.com/news.m3u8", group = "Movies Live")
+ )
+ val transportStream = classifier.classify(
+ entry(url = "https://live.example.com/movie-news.ts", group = "Film Channel")
+ )
+
+ assertThat(hls.kind).isEqualTo(M3uMediaKind.LIVE)
+ assertThat(hls.evidence).isEqualTo(M3uVodEvidence.LIVE_STREAM_EXTENSION)
+ assertThat(transportStream.kind).isEqualTo(M3uMediaKind.LIVE)
+ assertThat(transportStream.evidence).isEqualTo(M3uVodEvidence.LIVE_STREAM_EXTENSION)
+ }
+
+ @Test
+ fun `live text hint beats weak group and name heuristics`() {
+ val result = classifier.classify(
+ entry(
+ url = "https://provider.example.com/item/123",
+ group = "Live Movies",
+ name = "Film Channel"
+ )
+ )
+
+ assertThat(result.kind).isEqualTo(M3uMediaKind.LIVE)
+ assertThat(result.evidence).isEqualTo(M3uVodEvidence.LIVE_TEXT_HINT)
+ }
+
+ @Test
+ fun `rules are configurable and report the matching evidence`() {
+ val customClassifier = M3uVodClassifier(
+ M3uVodRules(vodKeywords = setOf("ciné club"))
+ )
+
+ val result = customClassifier.classify(
+ entry(url = "https://provider.example.com/item/123", group = "Ciné Club Français")
+ )
+
+ assertThat(result.kind).isEqualTo(M3uMediaKind.VOD)
+ assertThat(result.evidence).isEqualTo(M3uVodEvidence.GROUP_KEYWORD)
+ assertThat(result.confidence).isEqualTo(0.7f)
+ }
+
+ @Test
+ fun `explicit override wins and repeated classification is stable`() {
+ val movieEntry = entry(url = "https://cdn.example.com/film.mp4", group = "Movies")
+
+ val overridden = classifier.classify(movieEntry, override = M3uMediaKind.LIVE)
+ val first = classifier.classify(movieEntry)
+ val second = classifier.classify(movieEntry)
+
+ assertThat(overridden.kind).isEqualTo(M3uMediaKind.LIVE)
+ assertThat(overridden.evidence).isEqualTo(M3uVodEvidence.EXPLICIT_OVERRIDE)
+ assertThat(first).isEqualTo(second)
+ }
+
+ private fun entry(
+ url: String,
+ group: String = "General",
+ name: String = "Item"
+ ) = M3uParser.M3uEntry(
+ name = name,
+ groupTitle = group,
+ tvgId = null,
+ tvgName = null,
+ tvgLogo = null,
+ tvgChno = null,
+ tvgLanguage = null,
+ tvgCountry = null,
+ catchUp = null,
+ catchUpDays = null,
+ catchUpSource = null,
+ timeshift = null,
+ url = url
+ )
+}
diff --git a/data/src/test/java/com/streamvault/data/parser/XmltvParserTest.kt b/data/src/test/java/com/streamvault/data/parser/XmltvParserTest.kt
index 8325a378..743d9e7d 100644
--- a/data/src/test/java/com/streamvault/data/parser/XmltvParserTest.kt
+++ b/data/src/test/java/com/streamvault/data/parser/XmltvParserTest.kt
@@ -487,6 +487,108 @@ class XmltvParserTest {
assertThat(programmes.single().title).isEqualTo("Valid News")
}
+ @Test
+ fun `parseStreamingWithChannels_enforcesChannelAndProgrammeCounts`() = runTest {
+ val twoChannels = """
+ <tv>
+ <channel id="one"><display-name>One</display-name></channel>
+ <channel id="two"><display-name>Two</display-name></channel>
+ </tv>
+ """.trimIndent()
+ val twoProgrammes = """
+ <tv>
+ <programme start="20250101120000 +0000" stop="20250101130000 +0000" channel="one"><title>One</title></programme>
+ <programme start="20250101130000 +0000" stop="20250101140000 +0000" channel="one"><title>Two</title></programme>
+ </tv>
+ """.trimIndent()
+
+ val channelFailure = runCatching {
+ parser.parseStreamingWithChannels(
+ inputStream = twoChannels.byteInputStream(),
+ onChannel = {},
+ onProgramme = {},
+ limits = XmltvIngestionLimits(maxChannels = 1)
+ )
+ }.exceptionOrNull()
+ val programmeFailure = runCatching {
+ parser.parseStreamingWithChannels(
+ inputStream = twoProgrammes.byteInputStream(),
+ onChannel = {},
+ onProgramme = {},
+ limits = XmltvIngestionLimits(maxProgrammes = 1)
+ )
+ }.exceptionOrNull()
+
+ assertThat(channelFailure).isInstanceOf(XmltvLimitExceeded::class.java)
+ assertThat((channelFailure as XmltvLimitExceeded).kind).isEqualTo(XmltvLimitKind.CHANNELS)
+ assertThat(programmeFailure).isInstanceOf(XmltvLimitExceeded::class.java)
+ assertThat((programmeFailure as XmltvLimitExceeded).kind).isEqualTo(XmltvLimitKind.PROGRAMMES)
+ }
+
+ @Test
+ fun `parseStreamingWithChannels_rejectsVeryLongPersistedText`() = runTest {
+ val xml = """
+ <tv>
+ <programme start="20250101120000 +0000" stop="20250101130000 +0000" channel="one">
+ <title>News</title><desc>${"x".repeat(33)}</desc>
+ </programme>
+ </tv>
+ """.trimIndent()
+
+ val failure = runCatching {
+ parser.parseStreamingWithChannels(
+ inputStream = xml.byteInputStream(),
+ onChannel = {},
+ onProgramme = {},
+ limits = XmltvIngestionLimits(maxFieldChars = 32)
+ )
+ }.exceptionOrNull()
+
+ assertThat(failure).isInstanceOf(XmltvLimitExceeded::class.java)
+ assertThat((failure as XmltvLimitExceeded).kind).isEqualTo(XmltvLimitKind.FIELD_LENGTH)
+ }
+
+ @Test
+ fun `parseStreamingWithChannels_rejectsExcessiveNesting`() = runTest {
+ val xml = "<tv><programme><desc><b>nested</b></desc></programme></tv>"
+
+ val failure = runCatching {
+ parser.parseStreamingWithChannels(
+ inputStream = xml.byteInputStream(),
+ onChannel = {},
+ onProgramme = {},
+ limits = XmltvIngestionLimits(maxXmlDepth = 3)
+ )
+ }.exceptionOrNull()
+
+ assertThat(failure).isInstanceOf(XmltvLimitExceeded::class.java)
+ assertThat((failure as XmltvLimitExceeded).kind).isEqualTo(XmltvLimitKind.XML_DEPTH)
+ }
+
+ @Test
+ fun `parseStreamingWithChannels_boundsCategoriesPerProgramme`() = runTest {
+ val xml = """
+ <tv>
+ <programme start="20250101120000 +0000" stop="20250101130000 +0000" channel="one">
+ <title>News</title><category>News</category><category>Current affairs</category>
+ </programme>
+ </tv>
+ """.trimIndent()
+
+ val failure = runCatching {
+ parser.parseStreamingWithChannels(
+ inputStream = xml.byteInputStream(),
+ onChannel = {},
+ onProgramme = {},
+ limits = XmltvIngestionLimits(maxCategoriesPerProgramme = 1)
+ )
+ }.exceptionOrNull()
+
+ assertThat(failure).isInstanceOf(XmltvLimitExceeded::class.java)
+ assertThat((failure as XmltvLimitExceeded).kind)
+ .isEqualTo(XmltvLimitKind.CATEGORIES_PER_PROGRAMME)
+ }
+
@Test
fun `maybeDecompressGzip_nonGzUrl_returnsReadableStream`() {
val original = "hello".byteInputStream()
diff --git a/data/src/test/java/com/streamvault/data/remote/jellyfin/JellyfinProviderTest.kt b/data/src/test/java/com/streamvault/data/remote/jellyfin/JellyfinProviderTest.kt
index bd6bf42a..69ae7a9d 100644
--- a/data/src/test/java/com/streamvault/data/remote/jellyfin/JellyfinProviderTest.kt
+++ b/data/src/test/java/com/streamvault/data/remote/jellyfin/JellyfinProviderTest.kt
@@ -6,12 +6,21 @@ import com.streamvault.domain.model.Provider
import com.streamvault.domain.model.ProviderType
import com.streamvault.domain.model.Result
import kotlinx.coroutines.test.runTest
+import kotlinx.coroutines.CompletableDeferred
+import kotlinx.coroutines.Dispatchers
+import kotlinx.coroutines.async
+import kotlinx.coroutines.cancelAndJoin
import okhttp3.OkHttpClient
import okhttp3.Protocol
import okhttp3.Response
import okhttp3.ResponseBody.Companion.toResponseBody
+import okhttp3.MediaType
+import okhttp3.ResponseBody
import okhttp3.MediaType.Companion.toMediaType
+import okio.Buffer
+import okio.BufferedSource
import org.junit.Test
+import java.io.IOException
class JellyfinProviderTest {
@@ -51,6 +60,110 @@ class JellyfinProviderTest {
assertThat((result as Result.Error).exception).isInstanceOf(JellyfinPaginationException::class.java)
}
+ @Test
+ fun `movie page rejects mismatched start index and early empty page`() = runTest {
+ val bodies = ArrayDeque(
+ listOf(
+ """{"StartIndex":1,"TotalRecordCount":1,"Items":[{"Id":"movie-1"}]}""",
+ """{"StartIndex":0,"TotalRecordCount":1,"Items":[]}"""
+ )
+ )
+ val client = OkHttpClient.Builder().addInterceptor { chain ->
+ Response.Builder().request(chain.request()).protocol(Protocol.HTTP_1_1).code(200).message("OK")
+ .body(bodies.removeFirst().toResponseBody("application/json".toMediaType())).build()
+ }.build()
+ val provider = JellyfinProvider(client, Gson())
+ val account = account()
+
+ val mismatchedStart = provider.fetchMoviesPage(account, 0)
+ val earlyEmpty = provider.fetchMoviesPage(account, 0)
+
+ assertThat((mismatchedStart as Result.Error).exception).isInstanceOf(JellyfinPaginationException::class.java)
+ assertThat((earlyEmpty as Result.Error).exception).isInstanceOf(JellyfinPaginationException::class.java)
+ }
+
+ @Test
+ fun `movie page rejects oversized response before decoding`() = runTest {
+ val client = OkHttpClient.Builder().addInterceptor { chain ->
+ Response.Builder().request(chain.request()).protocol(Protocol.HTTP_1_1).code(200).message("OK")
+ .body(ReportedLengthBody(4L * 1024L * 1024L + 1L, "{}".toByteArray()))
+ .build()
+ }.build()
+ val result = JellyfinProvider(client, Gson()).fetchMoviesPage(account(), 0)
+
+ assertThat((result as Result.Error).exception).isInstanceOf(JellyfinResponseTooLargeException::class.java)
+ }
+
+ @Test
+ fun `movie page rejects huge single item field`() = runTest {
+ val overview = "x".repeat(65_537)
+ val client = OkHttpClient.Builder().addInterceptor { chain ->
+ Response.Builder().request(chain.request()).protocol(Protocol.HTTP_1_1).code(200).message("OK")
+ .body(
+ """{"StartIndex":0,"TotalRecordCount":1,"Items":[{"Id":"movie-1","Overview":"$overview"}]}"""
+ .toResponseBody("application/json".toMediaType())
+ ).build()
+ }.build()
+ val result = JellyfinProvider(client, Gson()).fetchMoviesPage(account(), 0)
+
+ assertThat((result as Result.Error).exception).isInstanceOf(JellyfinItemLimitException::class.java)
+ }
+
+ @Test
+ fun `cancellation mid page cancels the in flight request`() = runTest {
+ val entered = CompletableDeferred<Unit>()
+ val client = OkHttpClient.Builder().addInterceptor { chain ->
+ entered.complete(Unit)
+ while (!chain.call().isCanceled()) Thread.sleep(2)
+ throw IOException("cancelled")
+ }.build()
+ val provider = JellyfinProvider(client, Gson())
+
+ val request = async(Dispatchers.IO) { provider.fetchMoviesPage(account(), 0) }
+ entered.await()
+ request.cancelAndJoin()
+
+ assertThat(request.isCancelled).isTrue()
+ }
+
+ @Test
+ fun `episode pagination rejects repeated pages and changing totals`() = runTest {
+ suspend fun runCase(secondTotal: Int, secondId: String): Result<List<com.streamvault.data.local.entity.EpisodeEntity>> {
+ val responses = ArrayDeque(
+ listOf(
+ """{"StartIndex":0,"TotalRecordCount":2,"Items":[{"Id":"ep-1","Name":"One"}]}""",
+ """{"StartIndex":1,"TotalRecordCount":$secondTotal,"Items":[{"Id":"$secondId","Name":"Two"}]}"""
+ )
+ )
+ val client = OkHttpClient.Builder().addInterceptor { chain ->
+ Response.Builder().request(chain.request()).protocol(Protocol.HTTP_1_1).code(200).message("OK")
+ .body(responses.removeFirst().toResponseBody("application/json".toMediaType())).build()
+ }.build()
+ return JellyfinProvider(client, Gson()).fetchEpisodes(account(), "series-1", 99L)
+ }
+
+ val repeated = runCase(secondTotal = 2, secondId = "ep-1")
+ val changed = runCase(secondTotal = 3, secondId = "ep-2")
+
+ assertThat((repeated as Result.Error).exception).isInstanceOf(JellyfinPaginationException::class.java)
+ assertThat((changed as Result.Error).exception).isInstanceOf(JellyfinPaginationException::class.java)
+ }
+
+ @Test
+ fun `episode catalog ceiling is rejected from first page metadata`() = runTest {
+ val client = OkHttpClient.Builder().addInterceptor { chain ->
+ Response.Builder().request(chain.request()).protocol(Protocol.HTTP_1_1).code(200).message("OK")
+ .body(
+ """{"StartIndex":0,"TotalRecordCount":10001,"Items":[{"Id":"ep-1","Name":"One"}]}"""
+ .toResponseBody("application/json".toMediaType())
+ ).build()
+ }.build()
+
+ val result = JellyfinProvider(client, Gson()).fetchEpisodes(account(), "series-1", 99L)
+
+ assertThat((result as Result.Error).exception).isInstanceOf(JellyfinCatalogLimitException::class.java)
+ }
+
@Test
fun `fetchMovies does not embed access token in artwork urls`() = runTest {
val client = OkHttpClient.Builder()
@@ -110,4 +223,22 @@ class JellyfinProviderTest {
assertThat(movie.posterUrl).doesNotContain("api_key")
assertThat(movie.backdropUrl).doesNotContain("api_key")
}
+
+ private fun account() = Provider(
+ name = "Jellyfin",
+ type = ProviderType.JELLYFIN,
+ serverUrl = "https://demo.example",
+ username = "alice",
+ password = "token"
+ )
+
+ private class ReportedLengthBody(
+ private val reportedLength: Long,
+ bytes: ByteArray
+ ) : ResponseBody() {
+ private val buffer = Buffer().write(bytes)
+ override fun contentType(): MediaType = "application/json".toMediaType()
+ override fun contentLength(): Long = reportedLength
+ override fun source(): BufferedSource = buffer
+ }
}
diff --git a/data/src/test/java/com/streamvault/data/remote/xtream/XtreamProviderTest.kt b/data/src/test/java/com/streamvault/data/remote/xtream/XtreamProviderTest.kt
index a08bdd63..951694d0 100644
--- a/data/src/test/java/com/streamvault/data/remote/xtream/XtreamProviderTest.kt
+++ b/data/src/test/java/com/streamvault/data/remote/xtream/XtreamProviderTest.kt
@@ -18,8 +18,13 @@ import com.streamvault.data.remote.dto.XtreamUserInfo
import com.streamvault.data.remote.dto.XtreamVodInfo
import com.streamvault.data.remote.dto.XtreamVodInfoResponse
import com.streamvault.data.remote.dto.XtreamVodMovieData
+import kotlinx.coroutines.CompletableDeferred
import kotlinx.coroutines.ExperimentalCoroutinesApi
+import kotlinx.coroutines.async
+import kotlinx.coroutines.cancelAndJoin
import kotlinx.coroutines.runBlocking
+import kotlinx.coroutines.test.runCurrent
+import kotlinx.coroutines.test.runTest
import org.junit.Test
@OptIn(ExperimentalCoroutinesApi::class)
@@ -480,6 +485,123 @@ class XtreamProviderTest {
assertThat(categoryRequests).isEqualTo(2)
}
+ @Test
+ fun `getVodStreams caches a successful empty adult category response`() = runBlocking {
+ var categoryRequests = 0
+ val provider = XtreamProvider(
+ providerId = 42,
+ api = FakeXtreamApiService(
+ vodCategoriesLoader = {
+ categoryRequests++
+ emptyList()
+ },
+ vodStreams = listOf(vodStream(categoryId = "28"))
+ ),
+ serverUrl = "https://example.com",
+ username = "user",
+ password = "pass"
+ )
+
+ provider.getVodStreams()
+ provider.getVodStreams()
+
+ assertThat(categoryRequests).isEqualTo(1)
+ }
+
+ @Test
+ fun `cancelled adult category prefetch is not cached`() = runTest {
+ val firstRequestStarted = CompletableDeferred<Unit>()
+ val keepFirstRequestOpen = CompletableDeferred<Unit>()
+ var categoryRequests = 0
+ val provider = XtreamProvider(
+ providerId = 42,
+ api = FakeXtreamApiService(
+ vodCategoriesLoader = {
+ if (categoryRequests++ == 0) {
+ firstRequestStarted.complete(Unit)
+ keepFirstRequestOpen.await()
+ }
+ listOf(XtreamCategory(categoryId = "28", categoryName = "Adults", isAdult = true))
+ },
+ vodStreams = listOf(vodStream(categoryId = "28"))
+ ),
+ serverUrl = "https://example.com",
+ username = "user",
+ password = "pass"
+ )
+
+ val firstLoad = async { provider.getVodStreams() }
+ firstRequestStarted.await()
+ firstLoad.cancelAndJoin()
+
+ val recoveredMovie = provider.getVodStreams().getOrNull().orEmpty().single()
+
+ assertThat(recoveredMovie.isAdult).isTrue()
+ assertThat(categoryRequests).isEqualTo(2)
+ }
+
+ @Test
+ fun `concurrent adult category prefetches are single flight`() = runTest {
+ val firstRequestStarted = CompletableDeferred<Unit>()
+ val releaseCategoryResponse = CompletableDeferred<Unit>()
+ var categoryRequests = 0
+ val provider = XtreamProvider(
+ providerId = 42,
+ api = FakeXtreamApiService(
+ vodCategoriesLoader = {
+ categoryRequests++
+ firstRequestStarted.complete(Unit)
+ releaseCategoryResponse.await()
+ listOf(XtreamCategory(categoryId = "28", categoryName = "Adults", isAdult = true))
+ },
+ vodStreams = listOf(vodStream(categoryId = "28"))
+ ),
+ serverUrl = "https://example.com",
+ username = "user",
+ password = "pass"
+ )
+
+ val firstLoad = async { provider.getVodStreams() }
+ firstRequestStarted.await()
+ val secondLoad = async { provider.getVodStreams() }
+ runCurrent()
+
+ assertThat(categoryRequests).isEqualTo(1)
+
+ releaseCategoryResponse.complete(Unit)
+
+ assertThat(firstLoad.await().getOrNull().orEmpty().single().isAdult).isTrue()
+ assertThat(secondLoad.await().getOrNull().orEmpty().single().isAdult).isTrue()
+ assertThat(categoryRequests).isEqualTo(1)
+ }
+
+ @Test
+ fun `successful category refresh replaces cached adult category ids`() = runBlocking {
+ var categoryRequests = 0
+ val provider = XtreamProvider(
+ providerId = 42,
+ api = FakeXtreamApiService(
+ vodCategoriesLoader = {
+ if (categoryRequests++ == 0) {
+ listOf(XtreamCategory(categoryId = "28", categoryName = "Adults", isAdult = true))
+ } else {
+ emptyList()
+ }
+ },
+ vodStreams = listOf(vodStream(categoryId = "28"))
+ ),
+ serverUrl = "https://example.com",
+ username = "user",
+ password = "pass"
+ )
+
+ assertThat(provider.getVodStreams().getOrNull().orEmpty().single().isAdult).isTrue()
+ provider.getVodCategories()
+
+ assertThat(provider.getVodStreams().getOrNull().orEmpty().single().isAdult).isFalse()
+ assertThat(categoryRequests).isEqualTo(2)
+ }
+
@Test
fun `getVodStreams honors explicit adult flag from xtream payload`() = runBlocking {
val provider = XtreamProvider(
@@ -771,6 +893,14 @@ class XtreamProviderTest {
assertThat(series?.seasons?.first()?.episodes).isEmpty()
}
+ private fun vodStream(categoryId: String): XtreamStream = XtreamStream(
+ name = "Movie",
+ streamId = 321,
+ categoryId = categoryId,
+ categoryName = "Movies",
+ containerExtension = "mp4"
+ )
+
private class FakeXtreamApiService(
private val authResponse: XtreamAuthResponse = XtreamAuthResponse(XtreamUserInfo(auth = 1), XtreamServerInfo()),
private val liveCategories: List<XtreamCategory> = emptyList(),
diff --git a/data/src/test/java/com/streamvault/data/repository/EpgSourceRepositoryImplTest.kt b/data/src/test/java/com/streamvault/data/repository/EpgSourceRepositoryImplTest.kt
index 5be820fb..30fe2e59 100644
--- a/data/src/test/java/com/streamvault/data/repository/EpgSourceRepositoryImplTest.kt
+++ b/data/src/test/java/com/streamvault/data/repository/EpgSourceRepositoryImplTest.kt
@@ -17,6 +17,9 @@ import com.streamvault.data.local.entity.EpgProgrammeEntity
import com.streamvault.data.local.entity.EpgSourceEntity
import com.streamvault.data.local.entity.ProviderEpgSourceEntity
import com.streamvault.data.parser.XmltvParser
+import com.streamvault.data.parser.XmltvIngestionLimits
+import com.streamvault.data.parser.XmltvLimitExceeded
+import com.streamvault.data.parser.XmltvLimitKind
import com.streamvault.data.preferences.PreferencesRepository
import com.streamvault.domain.model.ContentType
import com.streamvault.domain.model.EpgMatchType
@@ -26,6 +29,8 @@ import com.streamvault.domain.model.XmltvTimezonePolicy
import java.io.ByteArrayInputStream
import java.io.ByteArrayOutputStream
import java.io.IOException
+import java.io.FilterInputStream
+import java.io.InputStream
import kotlinx.coroutines.runBlocking
import kotlinx.coroutines.flow.flowOf
import kotlinx.coroutines.test.runTest
@@ -49,6 +54,7 @@ import org.mockito.kotlin.eq
import org.mockito.kotlin.inOrder
import org.mockito.kotlin.isNull
import org.mockito.kotlin.mock
+import org.mockito.kotlin.never
import org.mockito.kotlin.times
import org.mockito.kotlin.verify
import org.mockito.kotlin.verifyNoMoreInteractions
@@ -65,6 +71,74 @@ class EpgSourceRepositoryImplTest {
assertThrows(IOException::class.java) { input.read() }
}
+ @Test
+ fun `limitEpgInput accepts input exactly at the configured limit`() {
+ val input = limitEpgInput(ByteArrayInputStream(byteArrayOf(1, 2, 3)), maxBytes = 3)
+
+ assertThat(input.readBytes()).hasLength(3)
+ }
+
+ @Test
+ fun `small gzip is rejected when expansion crosses decompressed boundary`() {
+ val expanded = "<tv><desc>${"x".repeat(512)}</desc></tv>".toByteArray()
+ val compressed = gzip(expanded)
+ val limits = XmltvIngestionLimits(
+ maxRawBytes = compressed.size.toLong(),
+ maxDecompressedBytes = 128
+ )
+
+ val error = assertThrows(XmltvLimitExceeded::class.java) {
+ openLimitedXmltvInput(
+ ByteArrayInputStream(compressed),
+ "https://example.com/guide.gz",
+ XmltvParser(),
+ limits
+ ).use(InputStream::readBytes)
+ }
+
+ assertThat(error.kind).isEqualTo(XmltvLimitKind.DECOMPRESSED_BYTES)
+ }
+
+ @Test
+ fun `chunked gzip cannot bypass decompressed boundary`() {
+ val expanded = "<tv><desc>${"y".repeat(512)}</desc></tv>".toByteArray()
+ val compressed = gzip(expanded)
+ val chunked = object : FilterInputStream(ByteArrayInputStream(compressed)) {
+ override fun read(bytes: ByteArray, off: Int, len: Int): Int =
+ super.read(bytes, off, len.coerceAtMost(3))
+ }
+
+ val error = assertThrows(XmltvLimitExceeded::class.java) {
+ openLimitedXmltvInput(
+ chunked,
+ "https://example.com/chunked",
+ XmltvParser(),
+ XmltvIngestionLimits(
+ maxRawBytes = compressed.size.toLong(),
+ maxDecompressedBytes = 128
+ )
+ ).use(InputStream::readBytes)
+ }
+
+ assertThat(error.kind).isEqualTo(XmltvLimitKind.DECOMPRESSED_BYTES)
+ }
+
+ @Test
+ fun `compressed transport is rejected at the smaller raw boundary`() {
+ val compressed = gzip("<tv>${"z".repeat(512)}</tv>".toByteArray())
+
+ val error = assertThrows(XmltvLimitExceeded::class.java) {
+ openLimitedXmltvInput(
+ ByteArrayInputStream(compressed),
+ "https://example.com/guide.gz",
+ XmltvParser(),
+ XmltvIngestionLimits(maxRawBytes = compressed.size.toLong() - 1, maxDecompressedBytes = 10_000)
+ ).use(InputStream::readBytes)
+ }
+
+ assertThat(error.kind).isEqualTo(XmltvLimitKind.RAW_BYTES)
+ }
+
private val context: Context = mock()
private val contentResolver: ContentResolver = mock()
private val epgSourceDao: EpgSourceDao = mock()
@@ -246,7 +320,7 @@ class EpgSourceRepositoryImplTest {
}
@Test
- fun `refreshSource returns typed error when parser hits oversized chunked response`() = runTest {
+ fun `refreshSource returns typed error cleans staging and preserves active rows on overflow`() = runTest {
val source = EpgSourceEntity(
id = 10L,
name = "Primary",
@@ -257,15 +331,23 @@ class EpgSourceRepositoryImplTest {
whenever(contentResolver.openInputStream(Uri.parse(source.url))).thenReturn(ByteArrayInputStream("<tv/>".toByteArray()))
whenever(xmltvParser.maybeDecompressGzip(eq(source.url), any())).thenAnswer { it.arguments[1] }
whenever(providerEpgSourceDao.getProviderIdsForSourceSync(10L)).thenReturn(emptyList())
- doAnswer { throw IOException("EPG response too large (>200 MB)") }
+ doAnswer { throw XmltvLimitExceeded(XmltvLimitKind.PROGRAMMES, 1_000_000) }
.whenever(xmltvParser)
- .parseStreamingWithChannels(any(), anyOrNull(), any(), any())
+ .parseStreamingWithChannels(any(), anyOrNull(), any(), any(), any())
val result = repository.refreshSource(10L)
assertThat(result is Result.Error).isTrue()
- assertThat((result as Result.Error).message).isEqualTo("EPG response exceeded 200 MB limit")
- verify(epgSourceDao).updateRefreshError(eq(10L), eq("EPG response exceeded 200 MB limit"), any())
+ assertThat((result as Result.Error).exception).isInstanceOf(XmltvLimitExceeded::class.java)
+ assertThat(result.message).isEqualTo("EPG programmes exceeded safety limit")
+ verify(epgProgrammeDao, times(2)).deleteBySource(-10L)
+ verify(epgChannelDao, times(2)).deleteBySource(-10L)
+ verify(epgSourceDao, times(2)).delete(-10L)
+ verify(epgProgrammeDao, never()).deleteBySource(10L)
+ verify(epgChannelDao, never()).deleteBySource(10L)
+ verify(epgProgrammeDao, never()).moveToSource(any(), any())
+ verify(epgChannelDao, never()).moveToSource(any(), any())
+ verify(epgSourceDao).updateRefreshError(eq(10L), eq("EPG programmes exceeded safety limit"), any())
}
@Test
@@ -352,7 +434,7 @@ class EpgSourceRepositoryImplTest {
}
@Test
- fun `refreshSource does not force identity encoding for remote downloads`() = runTest {
+ fun `refreshSource requests identity encoding so raw bytes are bounded before decompression`() = runTest {
val source = EpgSourceEntity(
id = 10L,
name = "MyEPG",
@@ -378,7 +460,7 @@ class EpgSourceRepositoryImplTest {
val result = repository.refreshSource(10L)
assertThat(result is Result.Success).isTrue()
- assertThat(requestCaptor.firstValue.header("Accept-Encoding")).isNull()
+ assertThat(requestCaptor.firstValue.header("Accept-Encoding")).isEqualTo("identity")
}
@Test
@@ -441,7 +523,7 @@ class EpgSourceRepositoryImplTest {
val result = repository.refreshSource(10L)
assertThat(result is Result.Success).isTrue()
- verify(xmltvParser).parseStreamingWithChannels(any(), eq("America/New_York"), any(), any())
+ verify(xmltvParser).parseStreamingWithChannels(any(), eq("America/New_York"), any(), any(), any())
}
@Test
@@ -470,7 +552,7 @@ class EpgSourceRepositoryImplTest {
val result = repository.refreshSource(10L)
assertThat(result is Result.Success).isTrue()
- verify(xmltvParser).parseStreamingWithChannels(any(), isNull(), any(), any())
+ verify(xmltvParser).parseStreamingWithChannels(any(), isNull(), any(), any(), any())
}
@Test
diff --git a/data/src/test/java/com/streamvault/data/sync/JellyfinCatalogCheckpointTest.kt b/data/src/test/java/com/streamvault/data/sync/JellyfinCatalogCheckpointTest.kt
new file mode 100644
index 00000000..518a9cdb
--- /dev/null
+++ b/data/src/test/java/com/streamvault/data/sync/JellyfinCatalogCheckpointTest.kt
@@ -0,0 +1,110 @@
+package com.streamvault.data.sync
+
+import com.google.common.truth.Truth.assertThat
+import com.streamvault.data.remote.jellyfin.JellyfinCatalogLimitException
+import com.streamvault.data.remote.jellyfin.JellyfinPaginationException
+import org.junit.Test
+
+class JellyfinCatalogCheckpointTest {
+
+ @Test
+ fun `checkpoint round trip preserves process death continuation`() {
+ val beforeProcessDeath = JellyfinCatalogCheckpoint(
+ movieSessionId = 101,
+ seriesSessionId = 202,
+ phase = JellyfinCatalogPhase.SERIES,
+ movieStartIndex = 250,
+ movieTotal = 250,
+ seriesStartIndex = 100,
+ seriesTotal = 275
+ )
+
+ val restored = JellyfinCatalogCheckpoint.decode(beforeProcessDeath.encode())
+
+ assertThat(restored).isEqualTo(beforeProcessDeath)
+ assertThat(restored!!.isConsistent(movieStageCount = 250, seriesStageCount = 100)).isTrue()
+ assertThat(restored.seriesStartIndex).isEqualTo(100)
+ }
+
+ @Test
+ fun `torn staged page invalidates checkpoint instead of skipping data`() {
+ val checkpoint = JellyfinCatalogCheckpoint(
+ movieSessionId = 101,
+ seriesSessionId = 202,
+ phase = JellyfinCatalogPhase.MOVIES,
+ movieStartIndex = 200,
+ movieTotal = 350
+ )
+
+ assertThat(checkpoint.isConsistent(movieStageCount = 199, seriesStageCount = 0)).isFalse()
+ assertThat(checkpoint.isConsistent(movieStageCount = 201, seriesStageCount = 0)).isFalse()
+ }
+
+ @Test
+ fun `ready checkpoint requires both complete staged catalogs`() {
+ val ready = JellyfinCatalogCheckpoint(
+ movieSessionId = 101,
+ seriesSessionId = 202,
+ phase = JellyfinCatalogPhase.READY,
+ movieStartIndex = 2,
+ movieTotal = 2,
+ seriesStartIndex = 3,
+ seriesTotal = 3
+ )
+
+ assertThat(ready.isConsistent(2, 3)).isTrue()
+ assertThat(ready.copy(seriesStartIndex = 2).isConsistent(2, 2)).isFalse()
+ }
+
+ @Test
+ fun `malformed or unsafe checkpoint is rejected`() {
+ assertThat(JellyfinCatalogCheckpoint.decode(null)).isNull()
+ assertThat(JellyfinCatalogCheckpoint.decode("wrong|1|2|MOVIES|0|-1|0|-1")).isNull()
+ assertThat(JellyfinCatalogCheckpoint.decode("jellyfin-v1|-1|2|MOVIES|0|-1|0|-1")).isNull()
+ assertThat(JellyfinCatalogCheckpoint.decode("jellyfin-v1|1|2|MOVIES|-1|-1|0|-1")).isNull()
+ }
+
+ @Test
+ fun `server truncation advances through every explicit continuation`() {
+ var checkpoint = JellyfinCatalogCheckpoint(101, 202, JellyfinCatalogPhase.MOVIES)
+
+ checkpoint = checkpoint.afterMoviePage(reportedTotal = 150, pageItemCount = 40, stagedCount = 40)
+ checkpoint = checkpoint.afterMoviePage(reportedTotal = 150, pageItemCount = 40, stagedCount = 80)
+ checkpoint = checkpoint.afterMoviePage(reportedTotal = 150, pageItemCount = 40, stagedCount = 120)
+ checkpoint = checkpoint.afterMoviePage(reportedTotal = 150, pageItemCount = 30, stagedCount = 150)
+
+ assertThat(checkpoint.movieStartIndex).isEqualTo(150)
+ assertThat(checkpoint.movieTotal).isEqualTo(150)
+ }
+
+ @Test
+ fun `changing totals empty pages and repeated items are rejected`() {
+ val checkpoint = JellyfinCatalogCheckpoint(
+ movieSessionId = 101,
+ seriesSessionId = 202,
+ phase = JellyfinCatalogPhase.MOVIES,
+ movieStartIndex = 100,
+ movieTotal = 200
+ )
+
+ val changedTotal = runCatching { checkpoint.afterMoviePage(201, 100, 200) }.exceptionOrNull()
+ val emptyPage = runCatching { checkpoint.afterMoviePage(200, 0, 100) }.exceptionOrNull()
+ val repeatedItems = runCatching { checkpoint.afterMoviePage(200, 100, 150) }.exceptionOrNull()
+
+ assertThat(changedTotal).isInstanceOf(JellyfinPaginationException::class.java)
+ assertThat(emptyPage).isInstanceOf(JellyfinPaginationException::class.java)
+ assertThat(repeatedItems).isInstanceOf(JellyfinPaginationException::class.java)
+ }
+
+ @Test
+ fun `movie and series catalog ceilings are enforced before continuation`() {
+ val movies = JellyfinCatalogCheckpoint(101, 202, JellyfinCatalogPhase.MOVIES)
+ val series = movies.copy(phase = JellyfinCatalogPhase.SERIES, movieTotal = 0)
+
+ val movieFailure = runCatching { movies.afterMoviePage(200_001, 100, 100) }.exceptionOrNull()
+ val seriesFailure = runCatching { series.afterSeriesPage(100_001, 100, 100) }.exceptionOrNull()
+
+ assertThat(movieFailure).isInstanceOf(JellyfinCatalogLimitException::class.java)
+ assertThat(seriesFailure).isInstanceOf(JellyfinCatalogLimitException::class.java)
+ }
+}
diff --git a/data/src/test/java/com/streamvault/data/sync/ProviderWorkLockRegistryTest.kt b/data/src/test/java/com/streamvault/data/sync/ProviderWorkLockRegistryTest.kt
index 18e4420e..2272da0f 100644
--- a/data/src/test/java/com/streamvault/data/sync/ProviderWorkLockRegistryTest.kt
+++ b/data/src/test/java/com/streamvault/data/sync/ProviderWorkLockRegistryTest.kt
@@ -93,4 +93,36 @@ class ProviderWorkLockRegistryTest {
assertThat(registry.runWhenNoWorkActive { true }).isTrue()
}
+
+ @Test
+ fun `provider work waits until admitted maintenance completes`() = runTest {
+ val registry = ProviderWorkLockRegistry()
+ val maintenanceEntered = CompletableDeferred<Unit>()
+ val releaseMaintenance = CompletableDeferred<Unit>()
+ val providerEntered = CompletableDeferred<Unit>()
+
+ val maintenance = async {
+ registry.runWhenNoWorkActive {
+ maintenanceEntered.complete(Unit)
+ releaseMaintenance.await()
+ true
+ }
+ }
+ maintenanceEntered.await()
+
+ val providerWork = async {
+ registry.withProviderLock(7L) {
+ providerEntered.complete(Unit)
+ }
+ }
+ runCurrent()
+
+ assertThat(providerEntered.isCompleted).isFalse()
+
+ releaseMaintenance.complete(Unit)
+
+ assertThat(maintenance.await()).isTrue()
+ providerWork.await()
+ assertThat(providerEntered.isCompleted).isTrue()
+ }
}
diff --git a/data/src/test/java/com/streamvault/data/sync/ProviderWorkflowRunnerTest.kt b/data/src/test/java/com/streamvault/data/sync/ProviderWorkflowRunnerTest.kt
index 0dfb0abc..50f14a29 100644
--- a/data/src/test/java/com/streamvault/data/sync/ProviderWorkflowRunnerTest.kt
+++ b/data/src/test/java/com/streamvault/data/sync/ProviderWorkflowRunnerTest.kt
@@ -8,6 +8,10 @@ import com.streamvault.data.local.entity.ProviderWorkflowPhase
import com.streamvault.data.local.entity.ProviderWorkflowPhaseState
import com.streamvault.data.local.entity.ProviderWorkflowReason
import com.streamvault.data.local.entity.ProviderWorkflowState
+import com.streamvault.data.remote.jellyfin.JellyfinCatalogLimitException
+import com.streamvault.data.remote.jellyfin.JellyfinItemLimitException
+import com.streamvault.data.remote.jellyfin.JellyfinPaginationException
+import com.streamvault.data.remote.jellyfin.JellyfinResponseTooLargeException
import com.streamvault.domain.model.ProviderType
import java.io.IOException
import kotlinx.coroutines.CompletableDeferred
@@ -89,6 +93,15 @@ class ProviderWorkflowRunnerTest {
).isEqualTo(ProviderWorkflowPhaseState.FAILED_RETRYABLE)
}
+ @Test
+ fun `jellyfin bounded input violations are permanent while transport IO remains retryable`() {
+ assertThat(ProviderWorkFailureClassifier.isRetryable(IOException("offline"))).isTrue()
+ assertThat(ProviderWorkFailureClassifier.isRetryable(JellyfinPaginationException("unstable"))).isFalse()
+ assertThat(ProviderWorkFailureClassifier.isRetryable(JellyfinCatalogLimitException("too many"))).isFalse()
+ assertThat(ProviderWorkFailureClassifier.isRetryable(JellyfinResponseTooLargeException(5, 4))).isFalse()
+ assertThat(ProviderWorkFailureClassifier.isRetryable(JellyfinItemLimitException("item too large"))).isFalse()
+ }
+
@Test
fun `supersession during execution prevents old completion publication`() = runTest {
insertProvider()
diff --git a/data/src/test/java/com/streamvault/data/sync/SyncCatalogStoreTest.kt b/data/src/test/java/com/streamvault/data/sync/SyncCatalogStoreTest.kt
index cf6465df..d8e1e598 100644
--- a/data/src/test/java/com/streamvault/data/sync/SyncCatalogStoreTest.kt
+++ b/data/src/test/java/com/streamvault/data/sync/SyncCatalogStoreTest.kt
@@ -60,6 +60,10 @@ class SyncCatalogStoreTest {
runBlocking {
whenever(movieDao.getTmdbIdsByProvider(any())).thenReturn(emptyList())
whenever(seriesDao.getTmdbIdsByProvider(any())).thenReturn(emptyList())
+ whenever(catalogSyncDao.countChannelStages(any(), any())).thenReturn(0)
+ whenever(catalogSyncDao.countMovieStages(any(), any())).thenReturn(0)
+ whenever(catalogSyncDao.countSeriesStages(any(), any())).thenReturn(0)
+ whenever(catalogSyncDao.countCategoryStages(any(), any(), any())).thenReturn(0)
}
}
@@ -147,7 +151,8 @@ class SyncCatalogStoreTest {
store(transactionRunner = runner).applyStagedLiveCatalog(providerId = 7L, sessionId = 55L, categories = null)
- assertThat(runner.calls).isEqualTo(1)
+ // One transaction applies the catalog and one atomically clears its staging session.
+ assertThat(runner.calls).isEqualTo(2)
verify(catalogSyncDao).rebuildChannelFts()
}
@@ -377,6 +382,93 @@ class SyncCatalogStoreTest {
assertThat(stagedSeries.syncFingerprint).isNotEmpty()
}
+ @Test
+ fun `stageChannelBatch checks and inserts within one transaction`() = runTest {
+ val runner = TrackingTransactionRunner()
+ val providerId = 7L
+ val sessionId = 91L
+ whenever(catalogSyncDao.countChannelStages(providerId, sessionId)).thenAnswer {
+ assertThat(runner.isInTransaction).isTrue()
+ 0
+ }
+ doAnswer {
+ assertThat(runner.isInTransaction).isTrue()
+ Unit
+ }.whenever(catalogSyncDao).insertChannelStages(any())
+
+ store(
+ sizeLimits = CatalogSizeLimits(maxChannelsPerProvider = 1),
+ transactionRunner = runner
+ ).stageChannelBatch(
+ providerId,
+ sessionId,
+ listOf(
+ ChannelEntity(
+ streamId = 1L,
+ name = "News",
+ providerId = providerId,
+ streamUrl = "https://stream.example.com/news.ts"
+ )
+ )
+ )
+
+ assertThat(runner.calls).isEqualTo(1)
+ verify(catalogSyncDao).insertChannelStages(any())
+ }
+
+ @Test
+ fun `stageCategories rejects a provider type beyond the shared limit`() = runTest {
+ val providerId = 7L
+ val sessionId = 92L
+ whenever(catalogSyncDao.countCategoryStages(providerId, sessionId, ContentType.LIVE.name)).thenReturn(1)
+
+ val failure = runCatching {
+ store(CatalogSizeLimits(maxM3uCategoriesPerType = 1)).stageCategories(
+ providerId,
+ sessionId,
+ listOf(
+ CategoryEntity(
+ categoryId = 2L,
+ name = "Sports",
+ type = ContentType.LIVE,
+ providerId = providerId
+ )
+ )
+ )
+ }.exceptionOrNull()
+
+ assertThat(failure).isInstanceOf(CatalogAdmissionExceeded::class.java)
+ assertThat(failure).hasMessageThat().contains("live category limit exceeded")
+ verify(catalogSyncDao, never()).insertCategoryStages(any())
+ }
+
+ @Test
+ fun `discardStagedImport clears every stage table in one transaction`() = runTest {
+ val runner = TrackingTransactionRunner()
+ val providerId = 7L
+ val sessionId = 93L
+ doAnswer {
+ assertThat(runner.isInTransaction).isTrue()
+ Unit
+ }.whenever(catalogSyncDao).clearChannelStages(providerId, sessionId)
+ doAnswer {
+ assertThat(runner.isInTransaction).isTrue()
+ Unit
+ }.whenever(catalogSyncDao).clearMovieStages(providerId, sessionId)
+ doAnswer {
+ assertThat(runner.isInTransaction).isTrue()
+ Unit
+ }.whenever(catalogSyncDao).clearSeriesStages(providerId, sessionId)
+ doAnswer {
+ assertThat(runner.isInTransaction).isTrue()
+ Unit
+ }.whenever(catalogSyncDao).clearCategoryStages(providerId, sessionId)
+
+ store(transactionRunner = runner).discardStagedImport(providerId, sessionId)
+
+ assertThat(runner.calls).isEqualTo(1)
+ }
+
@Test
fun `replaceCategories preserves user protection when provider renames category`() = runTest {
val providerId = 7L
@@ -553,6 +645,36 @@ class SyncCatalogStoreTest {
verify(movieDao).restoreWatchProgress(7L)
}
+ @Test
+ fun `applyStagedJellyfinCatalog promotes movies and series in one transaction`() = runTest {
+ val runner = TrackingTransactionRunner()
+ whenever(categoryDao.getByProviderAndTypeSync(7L, ContentType.MOVIE.name)).thenReturn(emptyList())
+ whenever(categoryDao.getByProviderAndTypeSync(7L, ContentType.SERIES.name)).thenReturn(emptyList())
+ whenever(catalogSyncDao.getCategoryStages(7L, 101L, ContentType.MOVIE.name)).thenReturn(emptyList())
+ whenever(catalogSyncDao.getCategoryStages(7L, 202L, ContentType.SERIES.name)).thenReturn(emptyList())
+ doAnswer { assertThat(runner.isInTransaction).isTrue(); Unit }
+ .whenever(catalogSyncDao).rebuildMovieFts()
+ doAnswer { assertThat(runner.isInTransaction).isTrue(); Unit }
+ .whenever(catalogSyncDao).rebuildSeriesFts()
+ var callbackInTransaction = false
+
+ store(transactionRunner = runner).applyStagedJellyfinCatalog(
+ providerId = 7L,
+ movieSessionId = 101L,
+ seriesSessionId = 202L,
+ movieCategories = listOf(CategoryEntity(providerId = 7L, categoryId = 1L, name = "Movies", type = ContentType.MOVIE)),
+ seriesCategories = listOf(CategoryEntity(providerId = 7L, categoryId = 2L, name = "Series", type = ContentType.SERIES)),
+ afterCatalogApply = { callbackInTransaction = runner.isInTransaction }
+ )
+
+ assertThat(callbackInTransaction).isTrue()
+ verify(catalogSyncDao).rebuildMovieFts()
+ verify(catalogSyncDao).rebuildSeriesFts()
+ verify(movieDao).restoreWatchProgress(7L)
+ verify(catalogSyncDao).clearMovieStages(7L, 101L)
+ verify(catalogSyncDao).clearSeriesStages(7L, 202L)
+ }
+
private class TrackingTransactionRunner : DatabaseTransactionRunner {
var calls: Int = 0
private var depth: Int = 0
diff --git a/data/src/test/java/com/streamvault/data/sync/SyncManagerM3uImporterTest.kt b/data/src/test/java/com/streamvault/data/sync/SyncManagerM3uImporterTest.kt
new file mode 100644
index 00000000..d10e5949
--- /dev/null
+++ b/data/src/test/java/com/streamvault/data/sync/SyncManagerM3uImporterTest.kt
@@ -0,0 +1,446 @@
+package com.streamvault.data.sync
+
+import com.google.common.truth.Truth.assertThat
+import com.streamvault.data.parser.M3uParser
+import com.streamvault.domain.model.Provider
+import com.streamvault.domain.model.ProviderType
+import kotlinx.coroutines.CancellationException
+import kotlinx.coroutines.test.runTest
+import okhttp3.MediaType
+import okhttp3.MediaType.Companion.toMediaType
+import okhttp3.OkHttpClient
+import okhttp3.Protocol
+import okhttp3.Response
+import okhttp3.ResponseBody
+import okio.Buffer
+import okio.BufferedSource
+import org.junit.Test
+import org.mockito.kotlin.any
+import org.mockito.kotlin.anyOrNull
+import org.mockito.kotlin.doAnswer
+import org.mockito.kotlin.eq
+import org.mockito.kotlin.mock
+import org.mockito.kotlin.verify
+import org.mockito.kotlin.whenever
+import java.io.ByteArrayOutputStream
+import java.io.IOException
+import java.util.zip.GZIPOutputStream
+
+class SyncManagerM3uImporterTest {
+
+ @Test
+ fun `unknown length input accepts the exact decompressed byte limit`() = runTest {
+ val playlist = playlistEntry("News", "https://stream.example.com/news.ts").toByteArray()
+ val fixture = fixture(
+ body = playlist,
+ limits = CatalogSizeLimits(
+ maxM3uDecompressedBytes = playlist.size.toLong(),
+ maxM3uLineBytes = 1_024
+ )
+ )
+
+ val stats = fixture.importer.importPlaylist(provider(), onProgress = null)
+
+ assertThat(stats.liveCount).isEqualTo(1)
+ assertThat(fixture.finalized).isTrue()
+ verify(fixture.store).discardStagedImport(PROVIDER_ID, SESSION_ID)
+ }
+
+ @Test
+ fun `gzip expansion beyond the decompressed limit aborts before catalog apply`() = runTest {
+ val playlist = playlistEntry("News", "https://stream.example.com/news.ts").toByteArray()
+ val fixture = fixture(
+ body = gzip(playlist),
+ contentEncoding = "gzip",
+ limits = CatalogSizeLimits(
+ maxM3uDecompressedBytes = playlist.size.toLong() - 1,
+ maxM3uLineBytes = 1_024
+ )
+ )
+
+ val failure = runCatching {
+ fixture.importer.importPlaylist(provider(), onProgress = null)
+ }.exceptionOrNull()
+
+ assertThat(failure).isInstanceOf(CatalogAdmissionExceeded::class.java)
+ assertThat(failure).hasMessageThat().contains("decompressed byte limit")
+ assertThat(fixture.finalized).isFalse()
+ verify(fixture.store).discardStagedImport(PROVIDER_ID, SESSION_ID)
+ }
+
+ @Test
+ fun `declared response length beyond the limit is rejected before parsing`() = runTest {
+ val fixture = fixture(
+ body = playlistEntry("News", "https://stream.example.com/news.ts").toByteArray(),
+ reportedLength = 8_192,
+ limits = CatalogSizeLimits(maxM3uDecompressedBytes = 4_096, maxM3uLineBytes = 1_024)
+ )
+
+ val failure = runCatching {
+ fixture.importer.importPlaylist(provider(), onProgress = null)
+ }.exceptionOrNull()
+
+ assertThat(failure).isInstanceOf(CatalogAdmissionExceeded::class.java)
+ assertThat(failure).hasMessageThat().contains("response length limit")
+ assertThat(fixture.finalized).isFalse()
+ }
+
+ @Test
+ fun `overlong raw line is rejected before parser allocation grows further`() = runTest {
+ val playlist = playlistEntry("N".repeat(65), "https://stream.example.com/news.ts").toByteArray()
+ val fixture = fixture(
+ body = playlist,
+ limits = CatalogSizeLimits(maxM3uDecompressedBytes = 4_096, maxM3uLineBytes = 32)
+ )
+
+ val failure = runCatching {
+ fixture.importer.importPlaylist(provider(), onProgress = null)
+ }.exceptionOrNull()
+
+ assertThat(failure).isInstanceOf(CatalogAdmissionExceeded::class.java)
+ assertThat(failure).hasMessageThat().contains("line length limit")
+ assertThat(fixture.finalized).isFalse()
+ }
+
+ @Test
+ fun `persisted metadata fields are subject to the field limit`() = runTest {
+ val overlongTvgId = "t".repeat(65)
+ val playlist = """
+ #EXTM3U
+ #EXTINF:-1 tvg-id="$overlongTvgId" group-title="News",News
+ https://stream.example.com/news.ts
+ """.trimIndent().toByteArray()
+ val fixture = fixture(
+ body = playlist,
+ limits = CatalogSizeLimits(
+ maxM3uDecompressedBytes = 4_096,
+ maxM3uLineBytes = 1_024,
+ maxM3uFieldLength = 64
+ )
+ )
+
+ val failure = runCatching {
+ fixture.importer.importPlaylist(provider(), onProgress = null)
+ }.exceptionOrNull()
+
+ assertThat(failure).isInstanceOf(CatalogAdmissionExceeded::class.java)
+ assertThat(failure).hasMessageThat().contains("field length limit")
+ assertThat(fixture.finalized).isFalse()
+ }
+
+ @Test
+ fun `http content type charset is honored by the streaming importer`() = runTest {
+ val iso88591 = Charsets.ISO_8859_1
+ val playlist = "#EXTM3U\n#EXTINF:-1 tvg-id=\"café-tv\" group-title=\"Télévision\",Café\nhttps://stream.example.com/cafe.ts\n"
+ val fixture = fixture(
+ body = playlist.toByteArray(iso88591),
+ contentType = "application/x-mpegURL; charset=ISO-8859-1".toMediaType(),
+ limits = CatalogSizeLimits(maxM3uDecompressedBytes = 4_096, maxM3uLineBytes = 1_024)
+ )
+ var stagedChannels = emptyList<com.streamvault.data.local.entity.ChannelEntity>()
+ doAnswer { invocation ->
+ @Suppress("UNCHECKED_CAST")
+ stagedChannels = (invocation.arguments[2] as List<com.streamvault.data.local.entity.ChannelEntity>).toList()
+ Unit
+ }.whenever(fixture.store).stageChannelBatch(any(), any(), any())
+
+ fixture.importer.importPlaylist(provider(), onProgress = null)
+
+ val channel = stagedChannels.single()
+ assertThat(channel.name).isEqualTo("Café")
+ assertThat(channel.groupTitle).isEqualTo("Télévision")
+ assertThat(channel.epgChannelId).isEqualTo("café-tv")
+ }
+
+ @Test
+ fun `provider override disabled keeps vod-looking entries in live tv`() = runTest {
+ val fixture = fixture(
+ body = playlistEntry("Feature", "https://cdn.example.com/feature.mp4").toByteArray(),
+ limits = CatalogSizeLimits(maxM3uDecompressedBytes = 4_096, maxM3uLineBytes = 1_024)
+ )
+ var stagedChannels = emptyList<com.streamvault.data.local.entity.ChannelEntity>()
+ doAnswer { invocation ->
+ @Suppress("UNCHECKED_CAST")
+ stagedChannels = (invocation.arguments[2] as List<com.streamvault.data.local.entity.ChannelEntity>).toList()
+ Unit
+ }.whenever(fixture.store).stageChannelBatch(any(), any(), any())
+
+ val stats = fixture.importer.importPlaylist(
+ provider(vodClassificationEnabled = false),
+ onProgress = null
+ )
+
+ assertThat(stats.liveCount).isEqualTo(1)
+ assertThat(stats.movieCount).isEqualTo(0)
+ assertThat(stagedChannels.single().name).isEqualTo("Feature")
+ verify(fixture.store).finalizeStagedImport(
+ any(), any(), anyOrNull(), anyOrNull(), eq(true), eq(true), any()
+ )
+ }
+
+ @Test
+ fun `full refresh classifies vod and prunes the now-empty live section`() = runTest {
+ val fixture = fixture(
+ body = playlistEntry("Feature", "https://cdn.example.com/feature.mp4?token=abc").toByteArray(),
+ limits = CatalogSizeLimits(maxM3uDecompressedBytes = 4_096, maxM3uLineBytes = 1_024)
+ )
+ var stagedMovies = emptyList<com.streamvault.data.local.entity.MovieEntity>()
+ doAnswer { invocation ->
+ @Suppress("UNCHECKED_CAST")
+ stagedMovies = (invocation.arguments[2] as List<com.streamvault.data.local.entity.MovieEntity>).toList()
+ Unit
+ }.whenever(fixture.store).stageMovieBatch(any(), any(), any())
+
+ val stats = fixture.importer.importPlaylist(
+ provider(vodClassificationEnabled = true),
+ onProgress = null
+ )
+
+ assertThat(stats.liveCount).isEqualTo(0)
+ assertThat(stats.movieCount).isEqualTo(1)
+ assertThat(stagedMovies.single().name).isEqualTo("Feature")
+ verify(fixture.store).finalizeStagedImport(
+ any(), any(), anyOrNull(), anyOrNull(), eq(true), eq(true), any()
+ )
+ }
+
+ @Test
+ fun `invalid ratio is enforced when valid entries cross the sample threshold`() = runTest {
+ val playlist = buildString {
+ appendLine("#EXTM3U")
+ repeat(51) { index ->
+ appendLine("#EXTINF:-1,Invalid $index")
+ appendLine("not-a-stream-url")
+ }
+ repeat(49) { index ->
+ appendLine("#EXTINF:-1,Valid $index")
+ appendLine("https://stream.example.com/$index.ts")
+ }
+ }.toByteArray()
+ val fixture = fixture(
+ body = playlist,
+ limits = CatalogSizeLimits(
+ maxM3uDecompressedBytes = 64_000,
+ maxM3uLineBytes = 1_024,
+ maxM3uInvalidEntryRatioPercent = 50
+ )
+ )
+
+ val failure = runCatching {
+ fixture.importer.importPlaylist(provider(), onProgress = null)
+ }.exceptionOrNull()
+
+ assertThat(failure).isInstanceOf(CatalogAdmissionExceeded::class.java)
+ assertThat(failure).hasMessageThat().contains("invalid-entry ratio limit")
+ assertThat(fixture.finalized).isFalse()
+ verify(fixture.store).discardStagedImport(PROVIDER_ID, SESSION_ID)
+ }
+
+ @Test
+ fun `entry live movie and category maxima reject the first item beyond the boundary`() = runTest {
+ val liveEntries = buildString {
+ appendLine("#EXTM3U")
+ append(playlistRows("First", "News", "https://stream.example.com/1.ts"))
+ append(playlistRows("Second", "Sports", "https://stream.example.com/2.ts"))
+ }
+ val movieEntries = buildString {
+ appendLine("#EXTM3U")
+ append(playlistRows("First", "Movies", "https://stream.example.com/1.mp4"))
+ append(playlistRows("Second", "Movies", "https://stream.example.com/2.mp4"))
+ }
+ val cases = listOf(
+ Triple(
+ liveEntries,
+ CatalogSizeLimits(maxM3uEntries = 1, maxM3uDecompressedBytes = 8_192, maxM3uLineBytes = 1_024),
+ "entry limit"
+ ),
+ Triple(
+ liveEntries,
+ CatalogSizeLimits(maxChannelsPerProvider = 1, maxM3uDecompressedBytes = 8_192, maxM3uLineBytes = 1_024),
+ "live-channel limit"
+ ),
+ Triple(
+ movieEntries,
+ CatalogSizeLimits(maxMoviesPerProvider = 1, maxM3uDecompressedBytes = 8_192, maxM3uLineBytes = 1_024),
+ "movie limit"
+ ),
+ Triple(
+ liveEntries,
+ CatalogSizeLimits(maxM3uCategoriesPerType = 1, maxM3uDecompressedBytes = 8_192, maxM3uLineBytes = 1_024),
+ "category limit"
+ )
+ )
+
+ cases.forEach { (playlist, limits, expectedMessage) ->
+ val fixture = fixture(body = playlist.toByteArray(), limits = limits)
+ val failure = runCatching {
+ fixture.importer.importPlaylist(
+ provider = provider(vodClassificationEnabled = expectedMessage == "movie limit"),
+ onProgress = null
+ )
+ }.exceptionOrNull()
+
+ assertThat(failure).isInstanceOf(CatalogAdmissionExceeded::class.java)
+ assertThat(failure).hasMessageThat().contains(expectedMessage)
+ assertThat(fixture.finalized).isFalse()
+ }
+ }
+
+ @Test
+ fun `duplicate heavy input remains governed by the total entry limit`() = runTest {
+ val playlist = buildString {
+ appendLine("#EXTM3U")
+ repeat(3) { index ->
+ append(playlistRows("Duplicate $index", "News", "https://stream.example.com/same.ts"))
+ }
+ }
+ val fixture = fixture(
+ body = playlist.toByteArray(),
+ limits = CatalogSizeLimits(
+ maxM3uEntries = 2,
+ maxM3uDecompressedBytes = 8_192,
+ maxM3uLineBytes = 1_024
+ )
+ )
+
+ val failure = runCatching {
+ fixture.importer.importPlaylist(provider(), onProgress = null)
+ }.exceptionOrNull()
+
+ assertThat(failure).isInstanceOf(CatalogAdmissionExceeded::class.java)
+ assertThat(failure).hasMessageThat().contains("entry limit")
+ assertThat(fixture.finalized).isFalse()
+ }
+
+ @Test
+ fun `staging failure discards the session without applying the catalog`() = runTest {
+ val fixture = fixture(
+ body = playlistEntry("News", "https://stream.example.com/news.ts").toByteArray(),
+ limits = CatalogSizeLimits(maxM3uDecompressedBytes = 4_096, maxM3uLineBytes = 1_024)
+ )
+ doAnswer { throw IOException("database full") }
+ .whenever(fixture.store)
+ .stageChannelBatch(any(), any(), any())
+
+ val failure = runCatching {
+ fixture.importer.importPlaylist(provider(), onProgress = null, batchSize = 1)
+ }.exceptionOrNull()
+
+ assertThat(failure).isInstanceOf(IOException::class.java)
+ assertThat(fixture.finalized).isFalse()
+ verify(fixture.store).discardStagedImport(PROVIDER_ID, SESSION_ID)
+ }
+
+ @Test
+ fun `cancellation propagates after staging cleanup`() = runTest {
+ val fixture = fixture(
+ body = playlistEntry("News", "https://stream.example.com/news.ts").toByteArray(),
+ limits = CatalogSizeLimits(maxM3uDecompressedBytes = 4_096, maxM3uLineBytes = 1_024)
+ )
+ doAnswer { throw CancellationException("cancelled") }
+ .whenever(fixture.store)
+ .stageChannelBatch(any(), any(), any())
+
+ val failure = runCatching {
+ fixture.importer.importPlaylist(provider(), onProgress = null, batchSize = 1)
+ }.exceptionOrNull()
+
+ assertThat(failure).isInstanceOf(CancellationException::class.java)
+ assertThat(fixture.finalized).isFalse()
+ verify(fixture.store).discardStagedImport(PROVIDER_ID, SESSION_ID)
+ }
+
+ private suspend fun fixture(
+ body: ByteArray,
+ limits: CatalogSizeLimits,
+ contentEncoding: String? = null,
+ reportedLength: Long = -1L,
+ contentType: MediaType? = null
+ ): Fixture {
+ val store = mock<SyncCatalogStore>()
+ whenever(store.newSessionId()).thenReturn(SESSION_ID)
+ val fixture = Fixture(store)
+ doAnswer {
+ fixture.finalized = true
+ Unit
+ }.whenever(store).finalizeStagedImport(
+ any(),
+ any(),
+ anyOrNull(),
+ anyOrNull(),
+ any(),
+ any(),
+ any()
+ )
+ val client = OkHttpClient.Builder()
+ .addInterceptor { chain ->
+ Response.Builder()
+ .request(chain.request())
+ .protocol(Protocol.HTTP_1_1)
+ .code(200)
+ .message("OK")
+ .apply {
+ if (contentEncoding != null) header("Content-Encoding", contentEncoding)
+ }
+ .body(TestResponseBody(body, reportedLength, contentType))
+ .build()
+ }
+ .build()
+ fixture.importer = SyncManagerM3uImporter(
+ m3uParser = M3uParser(),
+ okHttpClient = client,
+ syncCatalogStore = store,
+ retryTransient = { operation -> operation() },
+ progress = { _, _, _ -> },
+ emitProgress = { _, _ -> },
+ sizeLimits = limits
+ )
+ return fixture
+ }
+
+ private fun provider(vodClassificationEnabled: Boolean = false) = Provider(
+ id = PROVIDER_ID,
+ name = "Test playlist",
+ type = ProviderType.M3U,
+ serverUrl = PLAYLIST_URL,
+ m3uUrl = PLAYLIST_URL,
+ m3uVodClassificationEnabled = vodClassificationEnabled
+ )
+
+ private fun playlistEntry(name: String, url: String): String = """
+ #EXTM3U
+ #EXTINF:-1 group-title="News",$name
+ $url
+ """.trimIndent()
+
+ private fun playlistRows(name: String, group: String, url: String): String =
+ "#EXTINF:-1 group-title=\"$group\",$name\n$url\n"
+
+ private fun gzip(bytes: ByteArray): ByteArray = ByteArrayOutputStream().use { output ->
+ GZIPOutputStream(output).use { it.write(bytes) }
+ output.toByteArray()
+ }
+
+ private class Fixture(val store: SyncCatalogStore) {
+ lateinit var importer: SyncManagerM3uImporter
+ var finalized: Boolean = false
+ }
+
+ private class TestResponseBody(
+ bytes: ByteArray,
+ private val reportedLength: Long,
+ private val mediaType: MediaType?
+ ) : ResponseBody() {
+ private val source = Buffer().write(bytes)
+
+ override fun contentType(): MediaType? = mediaType
+ override fun contentLength(): Long = reportedLength
+ override fun source(): BufferedSource = source
+ }
+
+ private companion object {
+ const val PROVIDER_ID = 41L
+ const val SESSION_ID = 73L
+ const val PLAYLIST_URL = "https://playlist.example.com/list.m3u"
+ }
+}
diff --git a/data/src/test/java/com/streamvault/data/sync/SyncManagerTest.kt b/data/src/test/java/com/streamvault/data/sync/SyncManagerTest.kt
index 26f84aaf..4b0a990c 100644
--- a/data/src/test/java/com/streamvault/data/sync/SyncManagerTest.kt
+++ b/data/src/test/java/com/streamvault/data/sync/SyncManagerTest.kt
@@ -53,7 +53,9 @@ import com.streamvault.data.remote.stalker.StalkerDeviceProfile
import com.streamvault.data.remote.jellyfin.JellyfinProvider
import com.streamvault.data.preferences.PreferencesRepository
import com.streamvault.domain.model.ContentType
+import com.streamvault.domain.model.EpgSource
import kotlinx.coroutines.CompletableDeferred
+import kotlinx.coroutines.CancellationException
import kotlinx.coroutines.ExperimentalCoroutinesApi
import kotlinx.coroutines.async
import kotlinx.coroutines.runBlocking
@@ -85,6 +87,7 @@ import org.mockito.kotlin.eq
import org.mockito.kotlin.mock
import org.mockito.kotlin.reset
import org.mockito.kotlin.verify
+import java.io.IOException
import java.util.Locale
import java.util.zip.GZIPOutputStream
@@ -310,6 +313,7 @@ class SyncManagerTest {
org.mockito.kotlin.whenever(catalogSyncDao.countChannelStages(any(), any())).thenReturn(0)
org.mockito.kotlin.whenever(catalogSyncDao.countMovieStages(any(), any())).thenReturn(0)
org.mockito.kotlin.whenever(catalogSyncDao.countSeriesStages(any(), any())).thenReturn(0)
+ org.mockito.kotlin.whenever(catalogSyncDao.countCategoryStages(any(), any(), any())).thenReturn(0)
org.mockito.kotlin.whenever(catalogSyncDao.getChannelStageCategorySummaries(any(), any())).thenReturn(emptyList())
org.mockito.kotlin.whenever(xtreamContentIndexDao.pruneStaleLocalContentRows(any(), any())).thenReturn(0)
// Default stubs for the streamed Stalker API methods. The tests in this file
@@ -331,6 +335,19 @@ class SyncManagerTest {
org.mockito.kotlin.whenever(epgSourceRepo.refreshAllForProvider(any())).thenReturn(Result.success(Unit))
org.mockito.kotlin.whenever(epgSourceRepo.resolveForProvider(any(), any()))
.thenReturn(com.streamvault.domain.model.EpgResolutionSummary())
+ org.mockito.kotlin.whenever(epgSourceRepo.getAllSources()).thenReturn(flowOf(emptyList()))
+ org.mockito.kotlin.whenever(epgSourceRepo.getAssignmentsForProvider(any())).thenReturn(flowOf(emptyList()))
+ org.mockito.kotlin.whenever(epgSourceRepo.addSource(any(), any(), any(), anyOrNull())).thenAnswer { invocation ->
+ Result.success(
+ EpgSource(
+ id = 1L,
+ name = invocation.getArgument(0),
+ url = invocation.getArgument(1)
+ )
+ )
+ }
+ org.mockito.kotlin.whenever(epgSourceRepo.assignSourceToProvider(any(), any(), any()))
+ .thenReturn(Result.success(Unit))
}
}
@@ -3358,7 +3375,7 @@ class SyncManagerTest {
advanceUntilIdle()
if (result is Result.Error) {
- error(result.message)
+ error("${result.message}: ${result.exception?.stackTraceToString()}")
}
assertThat(result.isSuccess).isTrue()
verify(catalogSyncDao, atLeast(3)).insertChannelStages(any())
@@ -3408,6 +3425,9 @@ class SyncManagerTest {
val result = mgr.sync(1L, force = true)
advanceUntilIdle()
+ if (result is Result.Error) {
+ error("${result.message}: ${result.exception?.stackTraceToString()}")
+ }
assertThat(result.isSuccess).isTrue()
val state = mgr.currentSyncState(1L)
assertThat(state).isInstanceOf(SyncState.Success::class.java)
@@ -3440,6 +3460,61 @@ class SyncManagerTest {
// ── M3U sync failure ────────────────────────────────────────────
+ @Test
+ fun `sync_m3u_isolates_each_epg_source_failure_and_preserves_source_priority`() = runTest {
+ val firstUrl = "https://epg.example.com/first.xml"
+ val secondUrl = "https://epg.example.com/second.xml"
+ val thirdUrl = "https://epg.example.com/third.xml"
+ val playlist = tempFolder.newFile("multi-epg-playlist.m3u")
+ playlist.writeText(
+ """
+ #EXTM3U x-tvg-url="$firstUrl, $secondUrl, $thirdUrl"
+ #EXTINF:-1 tvg-id="news",News
+ https://live.example.com/news.ts
+ """.trimIndent()
+ )
+ org.mockito.kotlin.whenever(epgSourceRepo.addSource(any(), any(), any(), anyOrNull())).thenAnswer { invocation ->
+ when (val url = invocation.getArgument<String>(1)) {
+ firstUrl -> throw IOException("first source unavailable")
+ secondUrl -> Result.success(EpgSource(id = 2L, name = "Second", url = url))
+ else -> Result.success(EpgSource(id = 3L, name = "Third", url = url))
+ }
+ }
+ org.mockito.kotlin.whenever(epgSourceRepo.assignSourceToProvider(any(), any(), any())).thenAnswer { invocation ->
+ if (invocation.getArgument<Long>(1) == 2L) throw IOException("second assignment unavailable")
+ Result.success(Unit)
+ }
+ val provider = sampleProvider(ProviderType.M3U).copy(
+ serverUrl = playlist.toURI().toString(),
+ m3uUrl = playlist.toURI().toString(),
+ epgUrl = ""
+ )
+ val manager = buildManager(providerType = ProviderType.M3U, providerEntity = provider)
+
+ val result = manager.sync(1L, force = true)
+ advanceUntilIdle()
+
+ assertThat(result.isSuccess).isTrue()
+ assertThat(manager.currentSyncState(1L)).isInstanceOf(SyncState.Partial::class.java)
+ verify(epgSourceRepo).addSource(eq("Playlist EPG 1"), eq(firstUrl), any(), anyOrNull())
+ verify(epgSourceRepo).addSource(eq("Playlist EPG 2"), eq(secondUrl), any(), anyOrNull())
+ verify(epgSourceRepo).addSource(eq("Playlist EPG 3"), eq(thirdUrl), any(), anyOrNull())
+ verify(epgSourceRepo).assignSourceToProvider(1L, 2L, 1)
+ verify(epgSourceRepo).assignSourceToProvider(1L, 3L, 2)
+ }
+
+ @Test
+ fun `jellyfin cancellation is propagated instead of reported as partial success`() = runTest {
+ org.mockito.kotlin.whenever(jellyfinProvider.fetchMoviesPage(any(), eq(0)))
+ .thenThrow(CancellationException("cancelled"))
+
+ val error = runCatching {
+ buildManager(providerType = ProviderType.JELLYFIN).sync(providerId = 1L)
+ }.exceptionOrNull()
+
+ assertThat(error).isInstanceOf(CancellationException::class.java)
+ }
+
@Test
fun `sync_m3u_networkError_transitionsToError`() = runTest {
val mgr = buildManager(providerType = ProviderType.M3U)
@@ -3535,17 +3610,17 @@ class SyncManagerTest {
@Test
fun `isVodEntry_movieGroupTitle_returnsTrue`() {
- assertThat(M3uParser.isVodEntry(entry(group = "Movies HD"))).isTrue()
+ assertThat(M3uParser.isVodEntry(entry(url = "http://vod.example.com/item/1", group = "Movies HD"))).isTrue()
}
@Test
fun `isVodEntry_vodGroupTitle_returnsTrue`() {
- assertThat(M3uParser.isVodEntry(entry(group = "VOD Library"))).isTrue()
+ assertThat(M3uParser.isVodEntry(entry(url = "http://vod.example.com/item/1", group = "VOD Library"))).isTrue()
}
@Test
fun `isVodEntry_filmGroupTitle_returnsTrue`() {
- assertThat(M3uParser.isVodEntry(entry(group = "Film Classics"))).isTrue()
+ assertThat(M3uParser.isVodEntry(entry(url = "http://vod.example.com/item/1", group = "Film Classics"))).isTrue()
}
@Test
diff --git a/data/src/test/java/com/streamvault/data/sync/SyncProgressBusTest.kt b/data/src/test/java/com/streamvault/data/sync/SyncProgressBusTest.kt
index 904b85fc..6ceb1a86 100644
--- a/data/src/test/java/com/streamvault/data/sync/SyncProgressBusTest.kt
+++ b/data/src/test/java/com/streamvault/data/sync/SyncProgressBusTest.kt
@@ -3,6 +3,11 @@ package com.streamvault.data.sync
import com.google.common.truth.Truth.assertThat
import com.streamvault.domain.sync.Section
import com.streamvault.domain.sync.SyncProgress
+import java.util.concurrent.Callable
+import java.util.concurrent.CountDownLatch
+import java.util.concurrent.Executors
+import java.util.concurrent.TimeUnit
+import kotlinx.coroutines.CancellationException
import org.junit.Test
class SyncProgressBusTest {
@@ -38,6 +43,7 @@ class SyncProgressBusTest {
assertThat(bus.progressByProvider.value[1L]?.session).isEqualTo(replacement)
assertThat(bus.progressByProvider.value[1L]?.progress).isEqualTo(replacementProgress)
+ assertThat(replacement.epoch).isGreaterThan(stale.epoch)
}
@Test
@@ -51,4 +57,63 @@ class SyncProgressBusTest {
assertThat(bus.aggregate.value?.activeProviderCount).isEqualTo(2)
assertThat(bus.aggregate.value?.representative?.session).isEqualTo(providerB)
}
+
+ @Test
+ fun cancellingOneProvider_finishesOnlyItsOwnSession() {
+ val bus = SyncProgressBus()
+ val cancelledProvider = bus.begin(1L)
+ val activeProvider = bus.begin(2L)
+ val activeProgress = progress(Section.SERIES)
+ bus.emit(cancelledProvider, progress())
+ bus.emit(activeProvider, activeProgress)
+
+ try {
+ throw CancellationException("manual sync cancelled")
+ } catch (_: CancellationException) {
+ // Mirrors a sync entry point's cancellation path.
+ } finally {
+ bus.finish(cancelledProvider)
+ }
+
+ assertThat(bus.progressByProvider.value.keys).containsExactly(2L)
+ assertThat(bus.progressByProvider.value[2L]?.progress).isEqualTo(activeProgress)
+ assertThat(bus.aggregate.value?.representative?.session).isEqualTo(activeProvider)
+ }
+
+ @Test
+ fun concurrentManualAndBackgroundSessions_remainProviderScoped() {
+ val bus = SyncProgressBus()
+ val executor = Executors.newFixedThreadPool(2)
+ val manualPublished = CountDownLatch(1)
+ val backgroundPublished = CountDownLatch(1)
+ val manualFinished = CountDownLatch(1)
+
+ try {
+ val manual = executor.submit {
+ val session = bus.begin(1L)
+ bus.emit(session, progress(Section.LIVE))
+ manualPublished.countDown()
+ check(backgroundPublished.await(5, TimeUnit.SECONDS))
+ bus.finish(session)
+ manualFinished.countDown()
+ }
+ val background = executor.submit(Callable {
+ check(manualPublished.await(5, TimeUnit.SECONDS))
+ val session = bus.begin(2L)
+ bus.emit(session, progress(Section.VOD))
+ backgroundPublished.countDown()
+ check(manualFinished.await(5, TimeUnit.SECONDS))
+ session
+ })
+
+ manual.get(5, TimeUnit.SECONDS)
+ val backgroundSession = background.get(5, TimeUnit.SECONDS)
+
+ assertThat(bus.progressByProvider.value.keys).containsExactly(2L)
+ assertThat(bus.progressByProvider.value[2L]?.session).isEqualTo(backgroundSession)
+ assertThat(bus.aggregate.value?.activeProviderCount).isEqualTo(1)
+ } finally {
+ executor.shutdownNow()
+ }
+ }
}
diff --git a/data/src/test/java/com/streamvault/data/sync/SyncWorkerPolicyTest.kt b/data/src/test/java/com/streamvault/data/sync/SyncWorkerPolicyTest.kt
new file mode 100644
index 00000000..5a98ab5a
--- /dev/null
+++ b/data/src/test/java/com/streamvault/data/sync/SyncWorkerPolicyTest.kt
@@ -0,0 +1,89 @@
+package com.streamvault.data.sync
+
+import android.database.sqlite.SQLiteException
+import android.database.sqlite.SQLiteFullException
+import com.google.common.truth.Truth.assertThat
+import com.streamvault.data.preferences.DatabaseMaintenanceSnapshot
+import kotlinx.coroutines.test.runTest
+import org.mockito.kotlin.mock
+import org.mockito.kotlin.whenever
+import org.junit.Test
+
+class SyncWorkerPolicyTest {
+
+ @Test
+ fun `completed maintenance persists one success snapshot`() = runTest {
+ val snapshots = mutableListOf<DatabaseMaintenanceSnapshot>()
+ val report = maintenanceReport()
+
+ val outcome = completeMaintenanceRun(
+ runMaintenance = { DatabaseMaintenanceManager.MaintenanceRunResult.Completed(report) },
+ persistSnapshot = { snapshot -> snapshots += snapshot },
+ nowMillis = { 1234L }
+ )
+
+ assertThat(outcome).isEqualTo(MaintenanceWorkerOutcome.Completed(report))
+ assertThat(snapshots).hasSize(1)
+ assertThat(snapshots.single().ranAt).isEqualTo(1234L)
+ assertThat(snapshots.single().deletedPrograms).isEqualTo(report.deletedPrograms)
+ assertThat(snapshots.single().reclaimableBytes).isEqualTo(report.statsAfterVacuum.reclaimableBytes)
+ }
+
+ @Test
+ fun `deferred maintenance does not persist a success snapshot`() = runTest {
+ val snapshots = mutableListOf<DatabaseMaintenanceSnapshot>()
+
+ val outcome = completeMaintenanceRun(
+ runMaintenance = { DatabaseMaintenanceManager.MaintenanceRunResult.DeferredForActiveSync },
+ persistSnapshot = { snapshot -> snapshots += snapshot }
+ )
+
+ assertThat(outcome).isEqualTo(MaintenanceWorkerOutcome.DeferredForActiveSync)
+ assertThat(snapshots).isEmpty()
+ }
+
+ @Test
+ fun `maintenance retries locked busy and full databases`() {
+ assertThat(shouldRetryMaintenance(sqliteException("database is locked"))).isTrue()
+ assertThat(shouldRetryMaintenance(sqliteException("database is busy"))).isTrue()
+ assertThat(shouldRetryMaintenance(SQLiteFullException("database or disk is full"))).isTrue()
+ assertThat(shouldRetryMaintenance(sqliteException("database or disk is full"))).isTrue()
+ assertThat(shouldRetryMaintenance(IllegalArgumentException("bad request"))).isFalse()
+ }
+
+ private fun sqliteException(message: String): SQLiteException = mock<SQLiteException>().also { error ->
+ whenever(error.message).thenReturn(message)
+ }
+
+ private fun maintenanceReport() = DatabaseMaintenanceManager.MaintenanceReport(
+ deletedPrograms = 3,
+ deletedOrphanEpisodes = 2,
+ deletedStaleFavorites = 1,
+ vacuumRan = true,
+ statsBeforeVacuum = DatabaseMaintenanceManager.DatabaseStorageStats(
+ pageSizeBytes = 4096,
+ pageCount = 100,
+ freelistCount = 10,
+ mainDbBytes = 409_600,
+ walBytes = 1024
+ ),
+ statsAfterVacuum = DatabaseMaintenanceManager.DatabaseStorageStats(
+ pageSizeBytes = 4096,
+ pageCount = 90,
+ freelistCount = 1,
+ mainDbBytes = 368_640,
+ walBytes = 0
+ ),
+ tableStats = DatabaseMaintenanceManager.TableRowStats(
+ channels = 10,
+ movies = 20,
+ series = 30,
+ episodes = 40,
+ programs = 50,
+ epgProgrammes = 60,
+ playbackHistory = 70,
+ favorites = 80,
+ programReminders = 90
+ )
+ )
+}
diff --git a/docs/IPTV_CODE_REVIEW.md b/docs/IPTV_CODE_REVIEW.md
index 3ff05374..e21d50d2 100644
--- a/docs/IPTV_CODE_REVIEW.md
+++ b/docs/IPTV_CODE_REVIEW.md
@@ -355,13 +355,13 @@ Phase 2 should review these files/modules first, in this order:
| INFRA-001 | Cancellation is broadly converted into normal failure/retry results | Implemented; adverse-case verification pending (2026-07-27) | High |
| NET-001 | Suspend call paths use blocking, non-cancellable OkHttp `execute()` | Implemented; MockWebServer race matrix pending (2026-07-27) | High |
| SYNC-001 | M3U staging has no effective response or catalog-size bound | Resolved (2026-07-26) | High |
-| SYNC-002 | A global progress bus corrupts concurrent per-provider sync progress | Resolved (2026-07-26) | High |
+| SYNC-002 | A global progress bus corrupts concurrent per-provider sync progress | Resolved (2026-08-01) | High |
| LIFE-001 | Every player engine permanently registers a timeshift component callback | Implemented; lifecycle stress verification pending (2026-07-27) | High |
| LIFE-002 | Player reset launches unowned cleanup that can stop a new timeshift session | Implemented; lifecycle stress verification pending (2026-07-27) | High |
| REC-001 | Recording stop/cancel neither cancels the HTTP call nor joins the writer | Implemented; capture matrix verification pending (2026-07-27) | High |
| PERF-001 | Playback snapshots synchronously rewrite a file every second on main | Resolved (2026-07-26) | High |
| PERF-002 | VOD progress triggers DB and Android TV rewrites every five seconds | Resolved (2026-07-26) | High |
-| WORK-001 | Local database maintenance is incorrectly gated on connectivity | Resolved (2026-07-26) | Medium |
+| WORK-001 | Local database maintenance is incorrectly gated on connectivity | Resolved (2026-08-01) | Medium |
| WORK-002 | Recording reconciliation retries every error without classification/cap | Resolved (2026-07-29) | Medium |
| UPDATE-001 | Failed automatic update checks suppress another attempt for 24 hours | Resolved (2026-07-29) | Medium |
| MEMORY-001 | Process-lifetime maps grow with provider/category/host/media identities | Architectural concern | Medium |
@@ -410,10 +410,12 @@ The urgent shared fixes are cancellation-safe HTTP, recording/timeshift lifecycl
- **Fix scope:** Architectural at catalog admission; parser counters are local.
- **Required tests:** Chunked/no length, compressed expansion, every catalog maximum, duplicate-heavy input, overlong lines, cancellation/space exhaustion, and preservation of the active catalog.
- **Resolution:** M3U imports now enforce declared response size, decompressed bytes, raw-line length, total entries, per-type entries, categories, field length, and invalid-entry ratio through `CatalogSizeLimits`. The importer raises `CatalogAdmissionExceeded` before finalization; its existing `finally` clears the staged session, preserving the active catalog. `SyncCatalogStore` independently rejects channel/movie batches that would exceed the durable stage limit, so non-M3U callers cannot bypass it. `M3uParser` reports malformed candidates to support ratio enforcement; its focused unit suite passes.
+- **2026-08-01 hardening:** Field admission now covers every persisted M3U/header string, and malformed candidates replaced by another `#EXTINF` or left pending at EOF contribute to the invalid ratio. The ratio is evaluated after both valid and invalid candidates, closing the malformed-prefix threshold bypass. Channel/movie count-and-insert admission and per-type category admission now run inside the shared Room transaction boundary. Provider/session cleanup is atomic, and session cleanup runs in a non-cancellable context so cancellation cannot skip staging disposal.
+- **Verification:** `SyncManagerM3uImporterTest` covers unknown-length exact-byte admission, declared length, gzip expansion, raw-line and metadata-field bounds, total/live/movie/category limits, duplicate-heavy input, invalid-ratio threshold ordering, storage failure, cancellation, no catalog apply on rejection, and staging disposal. Parser and catalog-store tests cover replaced/unterminated candidates, transactional admission, category limits, and atomic cleanup. The complete `:data:testDebugUnitTest` suite passed on 2026-08-01: 722 tests, zero failures, errors, or skips.
### SYNC-002 — A global progress bus corrupts concurrent per-provider sync progress
-- **Classification:** Resolved (2026-07-26)
+- **Classification:** Resolved (2026-08-01)
- **Severity:** High
- **Where:** `SyncProgressBus.kt:21-36`; `SyncManager.withProviderLock:296-306`; `SyncManager.sync:659-732` and `retrySection:845-898`; consumers such as `WelcomeViewModel`.
- **Current behavior:** Different providers have different mutexes and can sync concurrently. State is provider-scoped, but `SyncProgressBus` contains one unkeyed `StateFlow<SyncProgress?>`. Every sync emits into it and every sync's `finally` resets the same value.
@@ -422,7 +424,8 @@ The urgent shared fixes are cancellation-safe HTTP, recording/timeshift lifecycl
- **Recommended correction:** Key progress by provider ID and monotonic sync epoch, or return a per-session flow. Reset only the matching epoch; derive aggregate progress separately.
- **Fix scope:** Shared sync API and consumers.
- **Required tests:** Two interleaved providers; one completes/cancels without clearing the other; stale epoch after replacement; concurrent manual/background entry points.
-- **Resolution:** `SyncProgressBus` now owns an active session per provider, identified by a monotonic epoch. It exposes keyed provider progress and a separately derived aggregate; only the matching session can publish or finish. `SyncManager` opens and closes that session for full syncs, section retries, and Xtream index rebuilds, while M3U and Xtream emitters report through the provider-scoped bridge. Focused bus tests cover independent completion, stale-session replacement, and aggregation.
+- **Resolution:** `SyncProgressBus` now owns an active session per provider, identified by a monotonic epoch. It exposes keyed provider progress and a separately derived aggregate; only the matching session can publish or finish. `SyncManager` opens and closes that session for full syncs, section retries, and Xtream index rebuilds, while M3U and Xtream emitters report through the provider-scoped bridge. Focused bus tests cover two interleaved providers, independent completion and cancellation, stale-session replacement with increasing epochs, aggregate derivation, and genuinely concurrent manual/background lifecycles.
+- **Verification:** The complete data-module suite passed on 2026-08-01 with 778 tests and no failures, errors, or skips. App test compilation and `DataMaintenanceSchedulingTest` also passed after removing the ambiguous defaulted injectable constructor from `ProviderWorkflowCommitFence`.
### LIFE-001 — Every player engine permanently registers a timeshift component callback
@@ -498,7 +501,8 @@ The urgent shared fixes are cancellation-safe HTTP, recording/timeshift lifecycl
- **Recommended correction:** Remove network. Consider separate light pruning/checkpoint and expensive idle-only vacuum jobs; persist last success.
- **Fix scope:** Local scheduling.
- **Required tests:** Request constraints, offline execution, battery/idle variants, locked/full DB retry, and safe skip during active sync.
-- **Resolution:** Daily local maintenance no longer requires network connectivity while retaining the battery-not-low and device-idle constraints.
+- **Resolution (2026-08-01):** Daily local maintenance requires no network connection while retaining battery-not-low and device-idle constraints. The complete prune/statistics/checkpoint/VACUUM operation now holds the same no-provider-sync admission gate, so it cannot race catalog/EPG writes. If any provider work is active or queued, maintenance is deferred as retryable work without writing a false successful-maintenance snapshot; only a completed operation records the persisted success snapshot.
+- **Verification:** `DataMaintenanceSchedulingTest` asserts offline, battery, and idle constraints. `ProviderWorkLockRegistryTest` verifies both active-provider rejection and that provider work cannot enter while admitted maintenance is running. `SyncWorkerPolicyTest` verifies completed work persists one snapshot, deferred work persists none, and locked/busy/full database failures retry. The focused data tests and app-level `DataMaintenanceSchedulingTest` passed on 2026-08-01 after correcting the unrelated Hilt constructor ambiguity.
### WORK-002 — Recording reconciliation retries every error without classification or cap
@@ -606,7 +610,7 @@ Phase 3 should keep these shared defects separate from provider-specific behavio
| Xtream Codes | Cancellable OkHttp bridge, response byte budgets, streaming catalog decode, typed HTTP/parsing errors, compatibility fallbacks | Provider facade swallows cancellation; transient category failure is cached as authoritative adult classification |
| M3U/M3U8 | Streaming parser, defensive attribute parser, URL policy, stable 64-bit identity, staging swap | Fixed UTF-8 decode, only one header EPG URL, query-bearing VOD extensions misclassified; Phase 2 also found missing import bounds |
| XMLTV/manual EPG | XML encoding autodetection, offset-aware dates, staged atomic source swap, channel/program batching | Size limit is applied before decompression; no deterministic timezone when one source is shared across differing providers |
-| Jellyfin/local media | Stable SHA-256-derived remote IDs, bearer request profiles for playback, Quick Connect support | Entire libraries are unpaginated/buffered; image authorization cannot distinguish multiple accounts on one server |
+| Jellyfin/local media | Stable SHA-256-derived remote IDs, bounded verifiable catalog pagination with durable resume, bearer request profiles for playback, Quick Connect support | Image authorization device/migration verification remains pending for multiple accounts on one server |
| Plugin providers | Explicit Messenger request IDs, bind/unbind cleanup, capability contract, M3U provider integration | Name-based provider ownership, unscoped manifest IDs, orphan cleanup gap, sequential multi-minute playback chain |
## Phase 3 findings summary
@@ -620,12 +624,12 @@ Phase 3 should keep these shared defects separate from provider-specific behavio
| STALKER-005 | Portal-local dates are parsed as UTC | Confirmed defect | High |
| STALKER-006 | Non-numeric remote IDs collapse to a 31-bit Java hash | Confirmed defect | Medium |
| XTREAM-001 | Most Xtream provider operations swallow cancellation | Implemented; facade matrix verification pending (2026-07-27) | High |
-| XTREAM-002 | A transient category failure is cached as an empty adult-category set | Confirmed defect | Medium |
-| M3U-001 | Playlist decoding is fixed to UTF-8 and only the first header EPG URL is retained | Confirmed defect | Medium |
-| M3U-002 | Query-bearing VOD file URLs are classified as live streams | Confirmed defect | Medium |
-| XMLTV-001 | The 200 MB EPG limit measures compressed, not decompressed, bytes | Resolved (2026-07-26) | High |
+| XTREAM-002 | A transient category failure is cached as an empty adult-category set | Resolved; recommended-standard cache policy verified (2026-08-01) | Medium |
+| M3U-001 | Playlist decoding is fixed to UTF-8 and only the first header EPG URL is retained | Resolved; hardened and fully verified (2026-08-01) | Medium |
+| M3U-002 | Query-bearing VOD file URLs are classified as live streams | Resolved; hardened and fully verified (2026-08-01) | Medium |
+| XMLTV-001 | The 200 MB EPG limit measures compressed, not decompressed, bytes | Resolved; hardened and fully verified (2026-08-01) | High |
| XMLTV-002 | Shared no-offset XMLTV sources fall back to the device timezone | Implemented; migration/device verification pending (2026-07-30) | Medium |
-| JELLYFIN-001 | Jellyfin catalog APIs buffer unpaginated libraries without a byte bound | Confirmed defect | High |
+| JELLYFIN-001 | Jellyfin catalog APIs buffer unpaginated libraries without a byte bound | Resolved; hardened and fully verified (2026-08-01) | High |
| JELLYFIN-002 | Images from two accounts on one server use whichever provider matches first | Implemented; migration/device verification pending (2026-07-30) | High |
| PLUGIN-001 | Plugin provider fallback can overwrite/delete a user-created M3U provider | Implemented; ownership lifecycle matrix pending (2026-07-30) | High |
| PLUGIN-002 | Manifest IDs are not package-scoped and absent plugins are not reconciled | Implemented; package lifecycle matrix pending (2026-07-30) | High |
@@ -723,7 +727,7 @@ Phase 3 should keep these shared defects separate from provider-specific behavio
### XTREAM-002 — A transient category failure is cached as an empty adult-category set
-- **Classification:** Implemented; ownership lifecycle matrix pending (2026-07-30)
+- **Classification:** Resolved; recommended-standard cache policy verified (2026-08-01)
- **Severity:** Medium
- **Where:** `XtreamProvider.loadAdultCategoryIds:665-697`; long-lived provider caches in `MovieRepositoryImpl:138,1680-1713` and `SeriesRepositoryImpl:113,1660-1693`.
- **Current behavior:** Category fetch failure is caught, logged, converted to `emptyList`, then transformed to an empty set and cached by content type. Subsequent item mapping never retries category loading for that provider instance. Repository provider instances can live for the process.
@@ -732,13 +736,13 @@ Phase 3 should keep these shared defects separate from provider-specific behavio
- **Recommended correction:** Cache only successful category responses, or store success/error with TTL and retry backoff. Invalidate on sync/provider update and expose degraded classification state where policy requires it.
- **Fix scope:** Local provider cache policy plus repository invalidation.
- **Required tests:** First failure then recovery; cancellation does not cache; genuine successful empty list does cache; TTL/update invalidation; concurrent callers single-flight; explicit item adult flag still wins.
-- **Resolution:** Failed category prefetches now return an uncached empty set, while successful responses (including a genuinely empty response) remain cacheable. Cancellation is rethrown. Regression coverage verifies that a failed prefetch is retried and restored adult classification is applied.
+- **Resolution (2026-08-01):** Failed category prefetches return an uncached empty set, while successful responses (including a genuinely empty response) remain cacheable. Cancellation is rethrown. A mutex makes a successful prefetch single-flight, and a successful direct category refresh replaces the cached classification; provider configuration changes create a replacement provider through the repository signature cache. Focused coverage verifies recovery after failure, cancellation without caching, successful-empty caching, concurrent single-flight behavior, refresh invalidation, and explicit item-level adult flags.
## M3U/M3U8 review
### M3U-001 — Playlist decoding is fixed to UTF-8 and only the first header EPG URL is retained
-- **Classification:** Resolved (2026-07-26)
+- **Classification:** Resolved (hardened 2026-08-01)
- **Severity:** Medium
- **Where:** `M3uParser.parse/parseStreaming:56-134`, `M3uHeader:25-28`, `extractHeaderEpgUrl:220-230`; `SyncManagerM3uImporter` header handling at `93-99`.
- **Current behavior:** Both parser paths construct `InputStreamReader(..., UTF_8)` with no BOM/charset fallback. Header attributes such as `x-tvg-url` are split on commas and only the first non-empty URL survives in the singular `tvgUrl` field. XMLTV itself supports declaration-based encoding, so behavior differs across the paired formats.
@@ -747,11 +751,11 @@ Phase 3 should keep these shared defects separate from provider-specific behavio
- **Recommended correction:** Detect BOM, honor a validated HTTP charset when available, and use a documented UTF-8/fallback policy. Model header guide URLs as a bounded deduplicated list and assign/import each valid source with explicit priority.
- **Fix scope:** M3U parser/header model and importer assignment.
- **Required tests:** UTF-8 BOM, UTF-16 BOM if supported, Windows-1252/ISO-8859-1, invalid byte sequences, accented EPG matching, multiple/comma-spaced/duplicate guide URLs, and per-URL failure isolation.
-- **Resolution:** M3U parsing now detects UTF-8 and UTF-16 BOMs, falls back to UTF-8 otherwise, and retains up to eight distinct header EPG URLs in source order. The importer validates each URL independently.
+- **Resolution:** M3U parsing now gives UTF-8/UTF-16 BOMs precedence, honors allow-listed HTTP charsets, and otherwise decodes strict UTF-8 with a deterministic Windows-1252 fallback for legacy/invalid lines. Both buffered and streaming paths share this policy. Headers retain up to eight trimmed, distinct guide URLs in source order. The importer validates, imports, and assigns every accepted guide with stable priority; a returned error or exception from one guide is reported as a warning without preventing later guides from being processed. Regression coverage includes every encoding, matching, URL-list, and failure-isolation case listed above.
### M3U-002 — Query-bearing VOD file URLs are classified as live streams
-- **Classification:** Resolved (2026-07-26)
+- **Classification:** Resolved (hardened 2026-08-01)
- **Severity:** Medium
- **Where:** `M3uParser.isVodEntry:432-444`; classification branch in `SyncManagerM3uImporter:126-190`; tests cover bare `.mp4/.mkv` only.
- **Current behavior:** VOD classification checks `lowercaseUrl.endsWith(".mp4"/".mkv"/".avi")`, literal `/movie/`, or English group substrings. A tokenized URL such as `film.mp4?token=...` does not end with the extension and is imported as live unless its path/group happens to match another heuristic.
@@ -760,13 +764,13 @@ Phase 3 should keep these shared defects separate from provider-specific behavio
- **Recommended correction:** Parse the URI path and inspect its final extension independent of query/fragment. Make group/name heuristics configurable and localized, treat classification as confidence/evidence, and allow overrides.
- **Fix scope:** Local classifier plus tests/UI explanation.
- **Required tests:** Extensions with query/fragment and uppercase; encoded paths; extensionless VOD; localized groups; misleading live group names; explicit user override and stable reclassification on refresh.
-- **Resolution:** `M3uParser.isVodEntry` now inspects the parsed URI path, so query strings and fragments no longer hide supported VOD file extensions. Regression coverage includes query-bearing, fragment-bearing, and uppercase extensions.
+- **Resolution:** M3U classification now returns a deterministic media kind, confidence, and evidence. It decodes the URI path before matching a configurable extension/path rule set, recognizes localized group and title hints, gives strong live-stream evidence precedence over weak text labels, and supports an explicit override. The provider setting uses that override when classification is disabled. Successful full-playlist refreshes commit both Live and Movies even when one side becomes empty, so changing the setting reliably removes stale rows from the previous classification. The UI explains the evidence order. Regression coverage includes query/fragment/case variants, encoded and extensionless paths, localized groups, name hints, misleading live labels, custom rules, explicit override, deterministic results, importer routing, and both directions of empty-section pruning.
## XMLTV/manual EPG review
### XMLTV-001 — The 200 MB EPG limit measures compressed, not decompressed, bytes
-- **Classification:** Resolved (2026-07-26)
+- **Classification:** Resolved (hardened 2026-08-01)
- **Severity:** High
- **Where:** `EpgSourceRepositoryImpl.refreshSource:316-371`; `XmltvParser.maybeDecompressGzip:523-556`; `MAX_EPG_SIZE_BYTES` policy.
- **Current behavior:** A `FilterInputStream` counting up to 200 MB wraps the raw HTTP/file stream, then `maybeDecompressGzip` wraps that counted stream in `GZIPInputStream`. The parser consumes unlimited decompressed XML as long as the compressed input stays below 200 MB. Staged rows also grow with the expanded document.
@@ -775,7 +779,7 @@ Phase 3 should keep these shared defects separate from provider-specific behavio
- **Recommended correction:** Count decompressed bytes after decompression and retain a smaller raw/network bound before it. Add channel/program/field/depth limits and abort/clean staging with a typed oversize outcome.
- **Fix scope:** EPG source ingestion boundary.
- **Required tests:** Small gzip with expansion beyond 200 MB, chunked gzip, nested/very long text, maximum channel/program counts, cleanup after overflow, and active-source atomic preservation.
-- **Resolution:** The ingestion boundary now limits both raw bytes before decompression and decompressed XML bytes passed to the parser, preventing gzip expansion from bypassing the 200 MB ceiling.
+- **Resolution:** XMLTV downloads now request identity transfer encoding so the 64 MB raw-wire ceiling is measured before application-controlled gzip detection, while the parser-facing stream retains an independent 200 MB decompressed ceiling. Exact-limit inputs are accepted and the first excess byte raises a typed `XmltvLimitExceeded` identifying raw bytes, decompressed bytes, channels, programmes, field length, categories per programme, or XML depth. Streaming parsing additionally caps channels, programmes, persisted text/attributes, per-programme categories, and nesting. Any typed overflow removes the negative-ID staging rows and returns a typed repository error without deleting or moving active-source rows. Regression coverage includes gzip expansion, chunked gzip, the raw boundary, exact byte limits, long text, nesting, channel/programme/category maxima, cleanup, and active-source atomic preservation.
### XMLTV-002 — Shared no-offset XMLTV sources fall back to the device timezone
@@ -795,8 +799,8 @@ Phase 3 should keep these shared defects separate from provider-specific behavio
### JELLYFIN-001 — Jellyfin catalog APIs buffer unpaginated libraries without a byte bound
-- **Status:** Fixed (2026-07-26).
-- **Classification:** Resolved (2026-07-26)
+- **Status:** Fixed; hardened and fully verified (2026-08-01).
+- **Classification:** Resolved (2026-08-01)
- **Severity:** High
- **Where:** `JellyfinProvider.fetchMovies/fetchSeries/fetchEpisodes:82-119`, `fetchItems:186-198`, `fetchSeriesEpisodes:200-220`, `executeRequest:303-315`, DTOs at `363-369`.
- **Current behavior:** `/Items` and `/Shows/{id}/Episodes` requests omit `StartIndex`/`Limit`. Responses are read fully with `body.string()` and decoded into a complete list. There is a 60-second call timeout but no content-length/decompressed-byte/item limit. DTOs do not retain `TotalRecordCount`/`StartIndex`, so pagination cannot be verified or resumed.
@@ -806,9 +810,11 @@ Phase 3 should keep these shared defects separate from provider-specific behavio
- **Fix scope:** Jellyfin API/catalog sync architecture.
- **Required tests:** Multiple pages, server default truncation, changing totals, empty/repeated pages, huge single item/body, cancellation mid-page, resume/process death, and catalog limits.
-**Resolution:** Jellyfin movie, series, and episode endpoints now send explicit `StartIndex` and `Limit=100`. Catalog responses are decoded from a bounded stream rather than `body.string()`; each page has a 4 MiB read/decompressed-byte budget, rejects more than 100 records, and requires a valid `TotalRecordCount`. Movie and series pages stage incrementally and are promoted only after the reported total is reached without a changed total or early empty page. A failed, oversized, or unstable sync discards its staged snapshot and leaves the active catalog intact. Catalog ceilings are enforced before promotion (200,000 movies, 100,000 series, and 10,000 episodes per series). Episode hydration now pages through the same bounded API before replacing its cached episode set.
+**Resolution:** Jellyfin movie, series, and episode endpoints send explicit `StartIndex` and `Limit=100`, validate the echoed start index and stable `TotalRecordCount`, and reject oversized, empty-early, repeated, or non-advancing pages. Responses are decoded directly from a bounded stream with a 4 MiB decompressed/read budget per page; page item counts, individual string fields, nested collections, and overall catalog sizes are independently bounded. Episode hydration uses the same contract and detects repeated IDs and changing totals.
+
+Movie and series pages are staged incrementally under separate session IDs. A versioned workflow checkpoint records the phase, next indexes, totals, and session IDs after every committed page. Retryable failure and coroutine cancellation preserve that checkpoint and its matching staged rows across worker/process loss; resume verifies the staged counts before continuing, while force syncs and corrupt/torn checkpoints restart safely. Lease-token fencing prevents an obsolete worker from changing a checkpoint. Movies and series are published together in one database transaction only after both snapshots are complete, so a partial or invalid fetch cannot replace either active catalog. Cancellation is rethrown through both HTTP and sync boundaries.
-Focused JVM coverage verifies explicit continuation parameters, multi-page continuation metadata, and rejection of a server response that ignores the requested page size. Broader process-death and cancellation coverage remains part of the shared cancellation/recovery work package, but no longer permits an unbounded Jellyfin response to be loaded into memory.
+Focused JVM coverage now exercises multi-page/default-truncated responses, changing totals, mismatched starts, empty and repeated pages/items, oversized bodies and single items, mid-page cancellation, checkpoint serialization and process-loss reclaim, torn-stage rejection, stale lease fencing, atomic dual-catalog promotion, episode and catalog ceilings, and cancellation propagation. The full data-module unit suite also passes.
### JELLYFIN-002 — Images from two accounts on one server use whichever provider matches first