Open source, privacy-first web analytics. Lightweight, cookie-free Google Analytics alternative. Self-hosted or cloud.
TypeScript analyticsanalytics-dashboardclickhouseelixirgoogle-analyticsmarketingopen-source-analyticsphoenixplausible-analyticspostgresqlprivacyprivacy-friendlyself-hostedsimple-analyticsstatisticstailwindcssweb-analyticswebsite-analyticswebsite-statswebsite-tracking
Document reasoning behind disabling deduplication in import tables (#3439)
Adrian Gruntkowski127a9ef9baaf
diff --git a/lib/plausible/purge.ex b/lib/plausible/purge.exindex 4288f74b53..b774798ba1 100644--- a/lib/plausible/purge.ex+++ b/lib/plausible/purge.ex@@ -5,6 +5,30 @@ defmodule Plausible.Purge doStats are stored on Clickhouse, and unlike other databases data deletion isdone asynchronously.+ All import tables have MergeTree's deduplication mechanism _disabled_ by setting+ `replicated_deduplication_window` from default 100 to 0. When enabled, every insert+ into a given table is compared against hashes of 100 previous inserts (as complete+ parts, not concrete rows) and ignored when match is found. The prupose of that+ mechanism is making inserts of exact same batches idempotent when retrying them+ shortly after - for instance due to timeout, when the client can't easily tell if+ previous insert succeeded or not. Deduplication, however, only considers inserts,+ not mutations. Deletions do not affect stored hashes, so further inserts of parts+ that were deleted will still be treated as duplicates. That's why this feature+ is disabled for import tables.++ Although deletions are asynchronous, the parts to delete are "remembered", so there's+ no risk of overlapping deletion causing problems with import following right after it.++ IMPORTANT: Deletion requires revision if/when import tables get moved to sharded CH+ cluster setup. Mutation queries, which have to be run with `ON CLUSTER` in such setup,+ dispatch independent queries across shards and those queries can start at different+ times. This in turn means risk of deletions corrupting data of follow-up inserts+ in some edge cases. Ideally, imported entries should be unique for a given import+ - an extra `import_id` column can be introduced, holding identifier. Last processed+ import identifier should be stored with other site data and should be used for scoping+ imported stats queries. No longer used imports can then be safely removed fully+ asynchronously.+- [Clickhouse `ALTER TABLE ... DELETE` Statement](https://clickhouse.com/docs/en/sql-reference/statements/alter/delete)- [Synchronicity of `ALTER` Queries](https://clickhouse.com/docs/en/sql-reference/statements/alter/#synchronicity-of-alter-queries)"""