Skip to content

Open source, privacy-first web analytics. Lightweight, cookie-free Google Analytics alternative. Self-hosted or cloud.

Document reasoning behind disabling deduplication in import tables (#3439)

Adrian Gruntkowski127a9ef9baaf
diff --git a/lib/plausible/purge.ex b/lib/plausible/purge.ex
index 4288f74b53..b774798ba1 100644
--- a/lib/plausible/purge.ex
+++ b/lib/plausible/purge.ex
@@ -5,6 +5,30 @@ defmodule Plausible.Purge do
Stats are stored on Clickhouse, and unlike other databases data deletion is
done asynchronously.
+ All import tables have MergeTree's deduplication mechanism _disabled_ by setting
+ `replicated_deduplication_window` from default 100 to 0. When enabled, every insert
+ into a given table is compared against hashes of 100 previous inserts (as complete
+ parts, not concrete rows) and ignored when match is found. The prupose of that
+ mechanism is making inserts of exact same batches idempotent when retrying them
+ shortly after - for instance due to timeout, when the client can't easily tell if
+ previous insert succeeded or not. Deduplication, however, only considers inserts,
+ not mutations. Deletions do not affect stored hashes, so further inserts of parts
+ that were deleted will still be treated as duplicates. That's why this feature
+ is disabled for import tables.
+
+ Although deletions are asynchronous, the parts to delete are "remembered", so there's
+ no risk of overlapping deletion causing problems with import following right after it.
+
+ IMPORTANT: Deletion requires revision if/when import tables get moved to sharded CH
+ cluster setup. Mutation queries, which have to be run with `ON CLUSTER` in such setup,
+ dispatch independent queries across shards and those queries can start at different
+ times. This in turn means risk of deletions corrupting data of follow-up inserts
+ in some edge cases. Ideally, imported entries should be unique for a given import
+ - an extra `import_id` column can be introduced, holding identifier. Last processed
+ import identifier should be stored with other site data and should be used for scoping
+ imported stats queries. No longer used imports can then be safely removed fully
+ asynchronously.
+
- [Clickhouse `ALTER TABLE ... DELETE` Statement](https://clickhouse.com/docs/en/sql-reference/statements/alter/delete)
- [Synchronicity of `ALTER` Queries](https://clickhouse.com/docs/en/sql-reference/statements/alter/#synchronicity-of-alter-queries)
"""