:hedgehog: PostHog is the leading platform for building self-driving products. Our developer tools – AI observability, analytics, session replay, flags, experiments, error tracking, logs, and more – capture all the context agents need to diagnose problems, uncover opportunities, and ship fixes. Steer it all from Slack, web, desktop, or the MCP.
Python ab-testingai-analyticsanalyticscdpdata-warehouseexperimentsfeature-flagsjavascriptproduct-analyticspythonreactsession-replaysurveystypescriptweb-analytics
feat(ingestion-control-plane): back up and restore the personhog changelog config (#124313)
Zach Brownf0fa1ad3b6fc
diff --git a/rust/Cargo.lock b/rust/Cargo.lockindex 884b5643a337..82126aeade9c 100644--- a/rust/Cargo.lock+++ b/rust/Cargo.lock@@ -6517,6 +6517,9 @@ version = "0.1.0"dependencies = ["anyhow","assignment-coordination",+ "async-trait",+ "aws-config",+ "aws-sdk-s3","axum 0.7.9","chrono","common-alloc",diff --git a/rust/ingestion-control-plane/Cargo.toml b/rust/ingestion-control-plane/Cargo.tomlindex 6c3baa31d6f7..6c71a82c940b 100644--- a/rust/ingestion-control-plane/Cargo.toml+++ b/rust/ingestion-control-plane/Cargo.toml@@ -6,6 +6,9 @@ publish = false[dependencies]anyhow = { workspace = true }+async-trait = { workspace = true }+aws-config = { workspace = true }+aws-sdk-s3 = { workspace = true }assignment-coordination = { path = "../common/assignment-coordination" }axum = { workspace = true }chrono = { workspace = true }diff --git a/rust/ingestion-control-plane/README.md b/rust/ingestion-control-plane/README.mdindex c066eb96901f..ccc023e8370d 100644--- a/rust/ingestion-control-plane/README.md+++ b/rust/ingestion-control-plane/README.md@@ -23,7 +23,15 @@ Lists live ingestion-consumer pods and serves the consumer's routing debug UI pe### Personhog topology-Protocol-aware view of personhog's etcd coordination state (`PERSONHOG_ETCD_PREFIX`, default `/personhog/`): the whole prefix is read in one consistent snapshot and interpreted with the `personhog-coordination` types and the coordinator's own pure predicates, so the derived diagnostics cannot drift from the protocol. Shows the elected coordinator, registered pods and routers (with remaining lease TTLs), per-partition ownership, and in-flight handoffs with their phase age and what each is waiting on — distinguishing a stuck participant ("waiting on freeze acks from router-2") from a stuck coordinator ("quorum met; waiting on the coordinator to advance"). Derived issues cover stuck handoffs past their per-phase deadline, assignments owned by unregistered pods with no replanning handoff, lingering `Complete` records the coordinator hasn't cleaned up, dead new owners awaiting cancellation, unassigned partitions, stale acks from previous handoff attempts, and unparseable records. The deadline knobs (`PERSONHOG_HANDOFF_DEADLINE_SECS`, `PERSONHOG_WARMING_DEADLINE_SECS`) must match the coordinator's to agree with its cancellation decisions. The tool writes no coordination state except from the changelog panel (seeding `config/changelog`, adding or removing a consumer group that retire waits on, requesting or cancelling a migration, and retiring a drained generation), and a force-reassignment must go through a handoff record, never a raw assignment write.+Protocol-aware view of personhog's etcd coordination state (`PERSONHOG_ETCD_PREFIX`, default `/personhog/`): the whole prefix is read in one consistent snapshot and interpreted with the `personhog-coordination` types and the coordinator's own pure predicates, so the derived diagnostics cannot drift from the protocol. Shows the elected coordinator, registered pods and routers (with remaining lease TTLs), per-partition ownership, and in-flight handoffs with their phase age and what each is waiting on — distinguishing a stuck participant ("waiting on freeze acks from router-2") from a stuck coordinator ("quorum met; waiting on the coordinator to advance"). Derived issues cover stuck handoffs past their per-phase deadline, assignments owned by unregistered pods with no replanning handoff, lingering `Complete` records the coordinator hasn't cleaned up, dead new owners awaiting cancellation, unassigned partitions, stale acks from previous handoff attempts, and unparseable records. The deadline knobs (`PERSONHOG_HANDOFF_DEADLINE_SECS`, `PERSONHOG_WARMING_DEADLINE_SECS`) must match the coordinator's to agree with its cancellation decisions. The tool writes no coordination state except from the changelog panel (seeding `config/changelog`, adding or removing a consumer group that retire waits on, requesting or cancelling a migration, retiring a drained generation, and restoring `config/changelog` from a backup), and a force-reassignment must go through a handoff record, never a raw assignment write.++#### Changelog backups++With `CHANGELOG_BACKUP_BUCKET` set, the service keeps a copy of `config/changelog` in that versioned bucket at `<prefix>/config-changelog.json`, as `{"nonce", "config"}` with the stored bytes in `config`. It copies the config once at startup and again after every change, from whichever writer made it, and skips a write when the current copy already holds it. A deletion is never written, so a wiped etcd still has its last good config in the bucket. The bucket's versions are the history.++Every replica runs the copy. Each one reads the backup, then etcd, and uploads only if the backup it read is still current. When it is not, S3 refuses the upload and the replica tries again after a backoff, from a fresh read. So an upload never replaces a copy of a newer config. The nonce gives each upload its own ETag, because the config alone can return to bytes it held before. Each replica reports `ingestion_control_plane_changelog_backup_in_sync`: 1 after a sync succeeds, 0 from startup until then and after a failed attempt. The backup is at risk when no replica reports 1.++When etcd holds no config, the changelog panel offers a restore from the current backup or an earlier version. A restore writes the stored bytes back unchanged, only while the key is absent, and only after checking the topic's live partition count matches the backup. Running services ignore a config whose epoch is older than the one they adopted, so a rebuilt etcd is restored with personhog scaled to zero.### Personhog parity@@ -65,6 +73,7 @@ Both etcd tools are disabled (their APIs return 503) unless `ETCD_ENDPOINTS` is| `ETCD_ALLOWED_PREFIXES` | `/` | Comma-separated key prefixes the etcd explorer may read and write || `PERSONHOG_ETCD_PREFIX` | `/personhog/` | Prefix the personhog coordination state lives under; must match the personhog services' `ETCD_PREFIX` || `PERSONHOG_KAFKA_HOSTS` | `KAFKA_HOSTS` | Cluster the personhog changelog lives on, for the changelog panel |+| `CHANGELOG_BACKUP_BUCKET` | empty | Versioned S3 bucket for `config/changelog` backups; empty disables backups and restore. Credentials and region come from the standard AWS environment || `PERSONHOG_KAFKA_TLS` | `KAFKA_TLS` | TLS for `PERSONHOG_KAFKA_HOSTS` || `PERSONHOG_HANDOFF_DEADLINE_SECS` | `120` | Per-phase deadline for flagging stuck handoffs; must match `COORDINATOR_HANDOFF_DEADLINE_SECS` || `PERSONHOG_WARMING_DEADLINE_SECS` | `1800` | Warming-phase deadline; must match `COORDINATOR_WARMING_DEADLINE_SECS` |diff --git a/rust/ingestion-control-plane/src/api.rs b/rust/ingestion-control-plane/src/api.rsindex 0cb24a05d5fb..95dddaa08429 100644--- a/rust/ingestion-control-plane/src/api.rs+++ b/rust/ingestion-control-plane/src/api.rs@@ -15,7 +15,7 @@ use uuid::Uuid;use k8s_awareness::DiscoveredPod;-use crate::changelog;+use crate::changelog::{self, RestoreError};use crate::etcd;use crate::jobs::{AnalysisRequest, JobView};use crate::kafka::browse::{self, BrowseParams, BrowseStop, MessageFilter, MessageRecord};@@ -471,6 +471,58 @@ async fn personhog_seed_changelog(Ok(Json(seeded))}+fn backup_store(state: &AppState) -> Result<Arc<dyn crate::backup::BackupStore>, ApiError> {+ state.backups.clone().ok_or_else(|| {+ ApiError::unavailable("changelog backups are disabled (CHANGELOG_BACKUP_BUCKET not set)")+ })+}++async fn personhog_changelog_backups(+ State(state): State<AppState>,+) -> Result<Json<Vec<crate::backup::BackupVersion>>, ApiError> {+ let backups = backup_store(&state)?;+ let versions = changelog::backups(backups.as_ref(), &state.config.personhog_etcd_prefix)+ .await+ .map_err(|e| ApiError::upstream(format!("listing backups failed: {e:#}")))?;+ Ok(Json(versions))+}++#[derive(Deserialize)]+struct RestoreChangelogRequest {+ version_id: Option<String>,+}++async fn personhog_restore_changelog(+ State(state): State<AppState>,+ Json(request): Json<RestoreChangelogRequest>,+) -> Result<Json<ChangelogConfig>, ApiError> {+ let prefix = state.config.personhog_etcd_prefix.clone();+ validated_etcd_key(&state, &prefix)?;+ let backups = backup_store(&state)?;+ let client = etcd_client(&state).await?;+ let version_id = request+ .version_id+ .as_deref()+ .map(str::trim)+ .filter(|v| !v.is_empty());+ let restored = changelog::restore(+ &state.config,+ &client,+ &prefix,+ backups.as_ref(),+ version_id,+ )+ .await+ .map_err(|e| match e {+ RestoreError::NotFound(message) => ApiError::not_found(message),+ RestoreError::Refused(message) => {+ ApiError::bad_request(format!("restore refused: {message}"))+ }+ RestoreError::Upstream(e) => ApiError::upstream(format!("restore failed: {e:#}")),+ })?;+ Ok(Json(restored))+}+#[derive(Deserialize)]struct StartMigrationRequest {topic: String,@@ -604,6 +656,14 @@ pub fn router(state: AppState) -> Router {"/api/personhog/changelog/seed",post(personhog_seed_changelog),)+ .route(+ "/api/personhog/changelog/backups",+ get(personhog_changelog_backups),+ )+ .route(+ "/api/personhog/changelog/restore",+ post(personhog_restore_changelog),+ ).route("/api/personhog/migrations", post(personhog_start_migration)).route("/api/personhog/migrations/current",diff --git a/rust/ingestion-control-plane/src/backup.rs b/rust/ingestion-control-plane/src/backup.rsnew file mode 100644index 000000000000..c70bf47ca098--- /dev/null+++ b/rust/ingestion-control-plane/src/backup.rs@@ -0,0 +1,691 @@+use std::sync::Arc;+use std::time::{Duration, Instant};++use anyhow::Context;+use async_trait::async_trait;+use aws_sdk_s3::error::ProvideErrorMetadata;+use aws_sdk_s3::primitives::ByteStream;+use etcd_client::EventType;+use personhog_coordination::store::PersonhogStore;+use serde::{Deserialize, Serialize};+use uuid::Uuid;++use crate::changelog;+use crate::etcd::EtcdHandle;++const RETRY_INITIAL: Duration = Duration::from_secs(1);+const RETRY_MAX: Duration = Duration::from_secs(60);++#[derive(Debug, Clone, PartialEq, Serialize)]+pub struct BackupVersion {+ pub version_id: String,+ pub last_modified: Option<String>,+ pub is_latest: bool,+ pub size: i64,+}++#[derive(Debug, Clone, PartialEq)]+pub struct StoredObject {+ pub body: Vec<u8>,+ pub etag: String,+}++#[derive(Debug, Clone, PartialEq)]+pub enum Precondition {+ Absent,+ Matches(String),+}++#[derive(Debug, Clone, Copy, PartialEq)]+pub enum PutOutcome {+ Written,+ Conflict,+}++#[async_trait]+pub trait BackupStore: Send + Sync {+ /// The current version, or the named one; `None` when it does not exist.+ async fn get(+ &self,+ key: &str,+ version_id: Option<&str>,+ ) -> anyhow::Result<Option<StoredObject>>;+ async fn put(+ &self,+ key: &str,+ value: &[u8],+ precondition: Precondition,+ ) -> anyhow::Result<PutOutcome>;+ /// Newest first.+ async fn versions(&self, key: &str) -> anyhow::Result<Vec<BackupVersion>>;+}++pub struct S3BackupStore {+ client: aws_sdk_s3::Client,+ bucket: String,+}++impl S3BackupStore {+ pub fn new(client: aws_sdk_s3::Client, bucket: String) -> Self {+ Self { client, bucket }+ }++ pub async fn from_env(bucket: String) -> Self {+ let config = aws_config::load_defaults(aws_config::BehaviorVersion::latest()).await;+ // Local S3 (an `AWS_ENDPOINT_URL` such as the dev stack's SeaweedFS)+ // serves buckets by path, not by virtual host.+ let s3_config = aws_sdk_s3::config::Builder::from(&config)+ .force_path_style(config.endpoint_url().is_some())+ .build();+ Self::new(aws_sdk_s3::Client::from_conf(s3_config), bucket)+ }+}++#[async_trait]+impl BackupStore for S3BackupStore {+ async fn get(+ &self,+ key: &str,+ version_id: Option<&str>,+ ) -> anyhow::Result<Option<StoredObject>> {+ let result = self+ .client+ .get_object()+ .bucket(&self.bucket)+ .key(key)+ .set_version_id(version_id.map(str::to_string))+ .send()+ .await;+ let object = match result {+ Ok(object) => object,+ Err(e) if matches!(e.code(), Some("NoSuchKey" | "NoSuchVersion")) => return Ok(None),+ Err(e) => {+ return Err(anyhow::Error::from(e.into_service_error()))+ .with_context(|| format!("get s3://{}/{key}", self.bucket))+ }+ };+ let etag = object+ .e_tag()+ .with_context(|| format!("s3://{}/{key} has no ETag", self.bucket))?+ .to_string();+ let body = object.body.collect().await.context("read backup body")?;+ Ok(Some(StoredObject {+ body: body.into_bytes().to_vec(),+ etag,+ }))+ }++ async fn put(+ &self,+ key: &str,+ value: &[u8],+ precondition: Precondition,+ ) -> anyhow::Result<PutOutcome> {+ let request = self+ .client+ .put_object()+ .bucket(&self.bucket)+ .key(key)+ .content_type("application/json")+ .body(ByteStream::from(value.to_vec()));+ let request = match precondition {+ Precondition::Absent => request.if_none_match("*"),+ Precondition::Matches(etag) => request.if_match(etag),+ };+ match request.send().await {+ Ok(_) => Ok(PutOutcome::Written),+ // S3 answers 409 instead of 412 when another conditional upload+ // of the key is in flight.+ Err(e)+ if matches!(+ e.code(),+ Some("PreconditionFailed" | "ConditionalRequestConflict")+ ) =>+ {+ Ok(PutOutcome::Conflict)+ }+ Err(e) => Err(anyhow::Error::from(e.into_service_error()))+ .with_context(|| format!("put s3://{}/{key}", self.bucket)),+ }+ }++ async fn versions(&self, key: &str) -> anyhow::Result<Vec<BackupVersion>> {+ let resp = self+ .client+ .list_object_versions()+ .bucket(&self.bucket)+ .prefix(key)+ .send()+ .await+ .map_err(|e| anyhow::Error::from(e.into_service_error()))+ .with_context(|| format!("list versions of s3://{}/{key}", self.bucket))?;+ let mut versions: Vec<BackupVersion> = resp+ .versions()+ .iter()+ .filter(|v| v.key() == Some(key))+ .filter_map(|v| {+ Some(BackupVersion {+ version_id: v.version_id()?.to_string(),+ last_modified: v.last_modified().map(|t| t.to_string()),+ is_latest: v.is_latest().unwrap_or(false),+ size: v.size().unwrap_or(0),+ })+ })+ .collect();+ // Two writes can share a timestamp, so the current version leads explicitly.+ versions.sort_by(|a, b| {+ b.is_latest+ .cmp(&a.is_latest)+ .then_with(|| b.last_modified.cmp(&a.last_modified))+ });+ Ok(versions)+ }+}++/// The stored form of a config. The nonce gives every upload distinct bytes,+/// and so a distinct ETag: the config alone can return to earlier bytes, and+/// an ETag condition would then pass for an object written since it was read.+#[derive(Serialize, Deserialize)]+struct Envelope {+ nonce: String,+ config: String,+}++fn wrap_config(config: &[u8]) -> anyhow::Result<Vec<u8>> {+ let envelope = Envelope {+ nonce: Uuid::new_v4().to_string(),+ config: String::from_utf8(config.to_vec()).context("config/changelog is not UTF-8")?,+ };+ Ok(serde_json::to_vec(&envelope)?)+}++pub fn unwrap_config(body: &[u8]) -> anyhow::Result<Vec<u8>> {+ let envelope: Envelope =+ serde_json::from_slice(body).context("the backup is not a backup envelope")?;+ Ok(envelope.config.into_bytes())+}++fn record(outcome: &str) {+ common_metrics::inc(+ "ingestion_control_plane_changelog_backups_total",+ &[("outcome".to_string(), outcome.to_string())],+ 1,+ );+}++fn set_in_sync(in_sync: bool) {+ common_metrics::gauge(+ "ingestion_control_plane_changelog_backup_in_sync",+ &[],+ if in_sync { 1.0 } else { 0.0 },+ );+}++pub fn object_key(prefix: &str) -> String {+ format!("{}/config-changelog.json", prefix.trim_matches('/'))+}++/// The backup is read before etcd, and the upload requires that backup to+/// still be current, so an upload never replaces a newer config.+async fn sync(store: &PersonhogStore, backups: &dyn BackupStore, key: &str) -> anyhow::Result<()> {+ let backup = backups.get(key, None).await?;+ let (config, _) = store.get_changelog_config_raw_with_revision().await?;+ let Some(config) = config else {+ set_in_sync(true);+ return Ok(());+ };+ let precondition = match backup {+ Some(object) if unwrap_config(&object.body).ok().as_deref() == Some(config.as_slice()) => {+ record("unchanged");+ set_in_sync(true);+ return Ok(());+ }+ Some(object) => Precondition::Matches(object.etag),+ None => Precondition::Absent,+ };+ match backups+ .put(key, &wrap_config(&config)?, precondition)+ .await?+ {+ PutOutcome::Written => {+ record("uploaded");+ set_in_sync(true);+ tracing::info!(key, "backed up config/changelog");+ Ok(())+ }+ PutOutcome::Conflict => {+ record("conflict");+ anyhow::bail!("the backup changed after it was read")+ }+ }+}++/// A deletion is never backed up, because the last good copy is what a+/// rebuild restores.+async fn watch_once(+ store: &PersonhogStore,+ backups: &dyn BackupStore,+ key: &str,+) -> anyhow::Result<()> {+ let (_, revision) = store.get_changelog_config_raw_with_revision().await?;+ sync(store, backups, key).await?;+ let mut stream = store.watch_changelog_config_from(revision + 1).await?;+ while let Some(response) = stream.message().await? {+ if response.canceled() {+ anyhow::bail!(+ "config/changelog watch cancelled by etcd: {}",+ response.cancel_reason()+ );+ }+ for event in response.events() {+ match event.event_type() {+ EventType::Put => sync(store, backups, key).await?,+ EventType::Delete => {+ record("deleted_skipped");+ tracing::warn!(key, "config/changelog was deleted; keeping the last backup");+ }+ }+ }+ }+ anyhow::bail!("config/changelog watch ended")+}++pub async fn run(etcd: Arc<EtcdHandle>, prefix: String, backups: Arc<dyn BackupStore>) {+ let key = object_key(&prefix);+ let mut delay = RETRY_INITIAL;+ set_in_sync(false);+ loop {+ let started = Instant::now();+ let attempt = async {+ let client = etcd.client().await?;+ watch_once(&changelog::store(&client, &prefix), backups.as_ref(), &key).await+ };+ if let Err(e) = attempt.await {+ record("error");+ set_in_sync(false);+ tracing::warn!(key, error = %format!("{e:#}"), "config/changelog backup failed; retrying");+ }+ if started.elapsed() > RETRY_MAX {+ delay = RETRY_INITIAL;+ }+ tokio::time::sleep(delay).await;+ delay = (delay * 2).min(RETRY_MAX);+ }+}++#[cfg(test)]+pub(crate) mod tests {+ use std::collections::HashMap;+ use std::hash::{DefaultHasher, Hash, Hasher};+ use std::sync::atomic::{AtomicBool, Ordering};+ use std::sync::Mutex;++ use envconfig::Envconfig;+ use tokio::sync::Notify;++ use super::*;+ use crate::config::Config;++ /// Versions in insertion order, the last one current. Like S3, the ETag+ /// is a hash of the body, so identical bodies share an ETag.+ #[derive(Default)]+ pub(crate) struct MemoryBackups {+ pub(crate) versions: Mutex<Vec<Vec<u8>>>,+ }++ fn etag(body: &[u8]) -> String {+ let mut hasher = DefaultHasher::new();+ body.hash(&mut hasher);+ format!("{:x}", hasher.finish())+ }++ impl MemoryBackups {+ fn configs(&self) -> Vec<Vec<u8>> {+ let versions = self.versions.lock().unwrap();+ versions.iter().map(|v| unwrap_config(v).unwrap()).collect()+ }+ }++ #[async_trait]+ impl BackupStore for MemoryBackups {+ async fn get(+ &self,+ _key: &str,+ version_id: Option<&str>,+ ) -> anyhow::Result<Option<StoredObject>> {+ let versions = self.versions.lock().unwrap();+ let body = match version_id {+ None => versions.last(),+ Some(id) => id.parse::<usize>().ok().and_then(|i| versions.get(i)),+ };+ Ok(body.map(|body| StoredObject {+ body: body.clone(),+ etag: etag(body),+ }))+ }++ async fn put(+ &self,+ _key: &str,+ value: &[u8],+ precondition: Precondition,+ ) -> anyhow::Result<PutOutcome> {+ let mut versions = self.versions.lock().unwrap();+ let current = versions.last().map(|body| etag(body));+ let holds = match precondition {+ Precondition::Absent => current.is_none(),+ Precondition::Matches(expected) => current == Some(expected),+ };+ if !holds {+ return Ok(PutOutcome::Conflict);+ }+ versions.push(value.to_vec());+ Ok(PutOutcome::Written)+ }++ async fn versions(&self, _key: &str) -> anyhow::Result<Vec<BackupVersion>> {+ let versions = self.versions.lock().unwrap();+ Ok((0..versions.len())+ .rev()+ .map(|i| BackupVersion {+ version_id: i.to_string(),+ last_modified: None,+ is_latest: i + 1 == versions.len(),+ size: versions[i].len() as i64,+ })+ .collect())+ }+ }++ #[derive(Clone, Copy, Debug, PartialEq)]+ enum Stall {+ BeforeRead,+ BeforeUpload,+ }++ struct Stalled {+ inner: Arc<MemoryBackups>,+ at: Stall,+ stalled: AtomicBool,+ arrived: Notify,+ release: Notify,+ }++ impl Stalled {+ fn new(inner: Arc<MemoryBackups>, at: Stall) -> Arc<Self> {+ Arc::new(Self {+ inner,+ at,+ stalled: AtomicBool::new(false),+ arrived: Notify::new(),+ release: Notify::new(),+ })+ }++ async fn stall(&self, at: Stall) {+ if at == self.at && !self.stalled.swap(true, Ordering::SeqCst) {+ self.arrived.notify_one();+ self.release.notified().await;+ }+ }+ }++ #[async_trait]+ impl BackupStore for Stalled {+ async fn get(+ &self,+ key: &str,+ version_id: Option<&str>,+ ) -> anyhow::Result<Option<StoredObject>> {+ self.stall(Stall::BeforeRead).await;+ self.inner.get(key, version_id).await+ }++ async fn put(+ &self,+ key: &str,+ value: &[u8],+ precondition: Precondition,+ ) -> anyhow::Result<PutOutcome> {+ self.stall(Stall::BeforeUpload).await;+ self.inner.put(key, value, precondition).await+ }++ async fn versions(&self, key: &str) -> anyhow::Result<Vec<BackupVersion>> {+ self.inner.versions(key).await+ }+ }++ pub(crate) async fn test_etcd(name: &str) -> (Arc<EtcdHandle>, String) {+ let nanos = std::time::SystemTime::now()+ .duration_since(std::time::UNIX_EPOCH)+ .unwrap()+ .as_nanos();+ let env = HashMap::from([(+ "ETCD_ENDPOINTS".to_string(),+ "http://localhost:2379".to_string(),+ )]);+ let config = Config::init_from_hashmap(&env).unwrap();+ let etcd = Arc::new(EtcdHandle::from_config(&config).unwrap());+ (etcd, format!("/test-control-plane-{name}-{nanos}/"))+ }++ async fn local_s3() -> S3BackupStore {+ use aws_sdk_s3::config::{Credentials, Region};+ use aws_sdk_s3::types::{BucketVersioningStatus, VersioningConfiguration};++ const BUCKET: &str = "control-plane-backup-test";+ let config = aws_sdk_s3::Config::builder()+ .behavior_version(aws_config::BehaviorVersion::latest())+ .region(Region::new("us-east-1"))+ .endpoint_url("http://localhost:19000")+ .credentials_provider(Credentials::new(+ "object_storage_root_user",+ "object_storage_root_password",+ None,+ None,+ "test",+ ))+ .force_path_style(true)+ .build();+ let client = aws_sdk_s3::Client::from_conf(config);+ // Creating an existing bucket fails; the versioning call below is what must succeed.+ drop(client.create_bucket().bucket(BUCKET).send().await);+ client+ .put_bucket_versioning()+ .bucket(BUCKET)+ .versioning_configuration(+ VersioningConfiguration::builder()+ .status(BucketVersioningStatus::Enabled)+ .build(),+ )+ .send()+ .await+ .expect("local S3 at localhost:19000");+ S3BackupStore::new(client, BUCKET.to_string())+ }++ #[tokio::test]+ async fn s3_store_reads_versions_and_enforces_upload_conditions() {+ let store = local_s3().await;+ let nanos = std::time::SystemTime::now()+ .duration_since(std::time::UNIX_EPOCH)+ .unwrap()+ .as_nanos();+ let key = format!("test-{nanos}/config-changelog.json");+ assert_eq!(store.get(&key, None).await.unwrap(), None);++ let put = |value: &'static [u8], precondition| store.put(&key, value, precondition);+ assert_eq!(+ put(b"first", Precondition::Absent).await.unwrap(),+ PutOutcome::Written+ );+ assert_eq!(+ put(b"again", Precondition::Absent).await.unwrap(),+ PutOutcome::Conflict+ );+ let first = store.get(&key, None).await.unwrap().unwrap();+ assert_eq!(+ put(b"second", Precondition::Matches(first.etag.clone()))+ .await+ .unwrap(),+ PutOutcome::Written+ );+ assert_eq!(+ put(b"stale", Precondition::Matches(first.etag))+ .await+ .unwrap(),+ PutOutcome::Conflict+ );++ assert_eq!(+ store.get(&key, None).await.unwrap().unwrap().body,+ b"second"+ );+ let versions = store.versions(&key).await.unwrap();+ assert_eq!(versions.len(), 2);+ assert!(versions[0].is_latest);+ let earlier = store+ .get(&key, Some(&versions[1].version_id))+ .await+ .unwrap();+ assert_eq!(earlier.unwrap().body, b"first");+ }++ async fn wait_for_versions(backups: &MemoryBackups, count: usize) {+ let deadline = Instant::now() + Duration::from_secs(10);+ while backups.versions.lock().unwrap().len() < count {+ assert!(Instant::now() < deadline, "expected {count} backups");+ tokio::time::sleep(Duration::from_millis(20)).await;+ }+ }++ #[tokio::test]+ async fn every_change_is_backed_up_and_a_deletion_never_is() {+ let (etcd, prefix) = test_etcd("backup-watch").await;+ let store = changelog::store(&etcd.client().await.unwrap(), &prefix);+ assert!(store+ .seed_changelog_config("personhog_updates_64", 64, "personhog-writer")+ .await+ .unwrap());+ let (seeded, _) = store+ .get_changelog_config_raw_with_revision()+ .await+ .unwrap();+ let seeded = seeded.unwrap();+ let backups = Arc::new(MemoryBackups::default());+ let watcher = tokio::spawn(run(etcd.clone(), prefix.clone(), backups.clone()));+ wait_for_versions(&backups, 1).await;++ let inner = store.inner();+ let key = format!("{}config/changelog", inner.prefix());+ inner.put_raw(&key, seeded.clone()).await.unwrap();+ let moved = br#"{"topic":"personhog_updates_128","partitions":128,"epoch":2}"#.to_vec();+ inner.put_raw(&key, moved.clone()).await.unwrap();+ wait_for_versions(&backups, 2).await;+ inner.delete(&key).await.unwrap();+ // Events arrive in order, so this one being backed up proves the+ // deletion before it was handled.+ let reseeded = br#"{"topic":"personhog_updates_128","partitions":128,"epoch":3}"#.to_vec();+ inner.put_raw(&key, reseeded.clone()).await.unwrap();+ wait_for_versions(&backups, 3).await;+ watcher.abort();++ assert_eq!(backups.configs(), vec![seeded, moved, reseeded]);+ }++ fn spawn_sync(+ client: &etcd_client::Client,+ prefix: &str,+ backups: Arc<Stalled>,+ ) -> tokio::task::JoinHandle<anyhow::Result<()>> {+ let store = changelog::store(client, prefix);+ let key = object_key(prefix);+ tokio::spawn(async move { sync(&store, backups.as_ref(), &key).await })+ }++ #[tokio::test]+ async fn a_stalled_pod_never_replaces_a_newer_backup() {+ for at in [Stall::BeforeRead, Stall::BeforeUpload] {+ let (etcd, prefix) = test_etcd("backup-race").await;+ let client = etcd.client().await.unwrap();+ let store = changelog::store(&client, &prefix);+ let inner = store.inner();+ let etcd_key = format!("{}config/changelog", inner.prefix());+ let key = object_key(&prefix);+ let without_group =+ br#"{"topic":"personhog_updates_64","partitions":64,"epoch":1,"writer_group":"personhog-writer","draining":[]}"#.to_vec();+ let with_group =+ br#"{"topic":"personhog_updates_64","partitions":64,"epoch":1,"writer_group":"personhog-writer","retire_groups":["relay"],"draining":[]}"#.to_vec();+ let backups = Arc::new(MemoryBackups::default());+ inner+ .put_raw(&etcd_key, without_group.clone())+ .await+ .unwrap();+ sync(&store, backups.as_ref(), &key).await.unwrap();++ inner.put_raw(&etcd_key, with_group.clone()).await.unwrap();+ let stalled = Stalled::new(backups.clone(), at);+ let pod_a = spawn_sync(&client, &prefix, stalled.clone());+ stalled.arrived.notified().await;++ // Pod B backs up the added group; the group is removed again,+ // which restores the original bytes, and pod C backs that up.+ sync(&store, backups.as_ref(), &key).await.unwrap();+ inner+ .put_raw(&etcd_key, without_group.clone())+ .await+ .unwrap();+ sync(&store, backups.as_ref(), &key).await.unwrap();+ stalled.release.notify_one();+ // Pod A either finds the backup current or has its upload refused.+ let _refused = pod_a.await.unwrap();++ assert_eq!(+ backups.configs().last(),+ Some(&without_group),+ "stalled {at:?}"+ );+ }+ }++ #[tokio::test]+ async fn a_refused_upload_fails_and_the_next_attempt_backs_up_the_latest_config() {+ let (etcd, prefix) = test_etcd("backup-retry").await;+ let client = etcd.client().await.unwrap();+ let store = changelog::store(&client, &prefix);+ let inner = store.inner();+ let etcd_key = format!("{}config/changelog", inner.prefix());+ let config = |epoch: u64| {+ format!(r#"{{"topic":"personhog_updates_64","partitions":64,"epoch":{epoch},"writer_group":"personhog-writer","draining":[]}}"#)+ .into_bytes()+ };+ let backups = Arc::new(MemoryBackups::default());+ inner.put_raw(&etcd_key, config(1)).await.unwrap();+ sync(&store, backups.as_ref(), &object_key(&prefix))+ .await+ .unwrap();++ inner.put_raw(&etcd_key, config(2)).await.unwrap();+ let w = Stalled::new(backups.clone(), Stall::BeforeUpload);+ let pod_w = spawn_sync(&client, &prefix, w.clone());+ w.arrived.notified().await;+ inner.put_raw(&etcd_key, config(3)).await.unwrap();+ let a = Stalled::new(backups.clone(), Stall::BeforeUpload);+ let pod_a = spawn_sync(&client, &prefix, a.clone());+ a.arrived.notified().await;++ // W's older upload lands first, so A's is refused.+ w.release.notify_one();+ pod_w.await.unwrap().unwrap();+ a.release.notify_one();+ assert!(pod_a.await.unwrap().is_err(), "the refusal is reported");+ assert_eq!(backups.configs().last(), Some(&config(2)));++ sync(&store, backups.as_ref(), &object_key(&prefix))+ .await+ .unwrap();++ assert_eq!(backups.configs().last(), Some(&config(3)));+ }+}diff --git a/rust/ingestion-control-plane/src/changelog.rs b/rust/ingestion-control-plane/src/changelog.rsindex 71aa08104bd9..fc95a30efb65 100644--- a/rust/ingestion-control-plane/src/changelog.rs+++ b/rust/ingestion-control-plane/src/changelog.rs@@ -10,11 +10,12 @@ use personhog_coordination::types::{use rdkafka::consumer::Consumer;use serde::Serialize;+use crate::backup::{self, BackupStore, BackupVersion};use crate::config::Config;use crate::kafka::client::{base_client_config, metadata_consumer};use crate::kafka::lag::{scan_group_lag, ConsumerTarget};-fn store(client: &Client, prefix: &str) -> PersonhogStore {+pub(crate) fn store(client: &Client, prefix: &str) -> PersonhogStore {PersonhogStore::new(EtcdStore::from_client(client.clone(),StoreConfig {@@ -72,6 +73,72 @@ pub async fn seed(})}+pub async fn backups(+ backups: &dyn BackupStore,+ prefix: &str,+) -> anyhow::Result<Vec<BackupVersion>> {+ backups.versions(&backup::object_key(prefix)).await+}++#[derive(Debug)]+pub enum RestoreError {+ NotFound(String),+ Refused(String),+ Upstream(anyhow::Error),+}++impl From<anyhow::Error> for RestoreError {+ fn from(error: anyhow::Error) -> Self {+ Self::Upstream(error)+ }+}++pub async fn restore(+ config: &Config,+ client: &Client,+ prefix: &str,+ backups: &dyn BackupStore,+ version_id: Option<&str>,+) -> Result<ChangelogConfig, RestoreError> {+ let key = backup::object_key(prefix);+ let object = backups.get(&key, version_id).await?.ok_or_else(|| {+ RestoreError::NotFound(match version_id {+ Some(version) => format!("no backup version {version} of {key}"),+ None => format!("no backup at {key}"),+ })+ })?;+ let value =+ backup::unwrap_config(&object.body).map_err(|e| RestoreError::Refused(format!("{e:#}")))?;+ let restored: ChangelogConfig = serde_json::from_slice(&value)+ .map_err(|e| RestoreError::Refused(format!("the backup is not a changelog config: {e}")))?;+ let live_partitions = topic_partition_count(&config.personhog_kafka(), &restored.topic).await?;+ apply_restore(&store(client, prefix), &value, restored, live_partitions).await+}++async fn apply_restore(+ store: &PersonhogStore,+ value: &[u8],+ restored: ChangelogConfig,+ live_partitions: u32,+) -> Result<ChangelogConfig, RestoreError> {+ if live_partitions != restored.partitions {+ return Err(RestoreError::Refused(format!(+ "topic {} has {live_partitions} partitions but the backup records {}; the backup may be from another environment",+ restored.topic, restored.partitions+ )));+ }+ let created = store+ .restore_changelog_config_raw(value)+ .await+ .map_err(|e| RestoreError::Upstream(e.into()))?;+ if !created {+ return Err(RestoreError::Refused(+ "config/changelog already exists; restore only rebuilds an empty etcd".to_string(),+ ));+ }+ Ok(restored)+}+#[derive(Serialize)]pub struct GroupLag {pub group: String,@@ -405,3 +472,76 @@ async fn group_lag(config: &Config, group: &str, topic: &str) -> GroupLag {},}}++#[cfg(test)]+mod tests {+ use super::*;+ use crate::backup::tests::test_etcd;++ async fn empty_store(name: &str) -> PersonhogStore {+ let (etcd, prefix) = test_etcd(name).await;+ store(&etcd.client().await.unwrap(), &prefix)+ }++ fn backed_up(partitions: u32) -> (Vec<u8>, ChangelogConfig) {+ // A field this build does not know must survive the round trip.+ let value = format!(+ r#"{{"topic":"personhog_updates_64","partitions":{partitions},"epoch":4,"writer_group":"personhog-writer","draining":[],"future_field":"kept"}}"#+ )+ .into_bytes();+ let parsed = serde_json::from_slice(&value).unwrap();+ (value, parsed)+ }++ #[tokio::test]+ async fn restore_writes_the_backed_up_bytes_into_an_empty_etcd() {+ let store = empty_store("restore-empty").await;+ let (value, parsed) = backed_up(64);++ let restored = apply_restore(&store, &value, parsed, 64).await.unwrap();++ assert_eq!(restored.epoch, 4);+ let (stored, _) = store+ .get_changelog_config_raw_with_revision()+ .await+ .unwrap();+ assert_eq!(stored.unwrap(), value);+ }++ #[tokio::test]+ async fn restore_refuses_a_seeded_etcd() {+ let store = empty_store("restore-seeded").await;+ assert!(store+ .seed_changelog_config("personhog_updates_64", 64, "personhog-writer")+ .await+ .unwrap());+ let (value, parsed) = backed_up(64);++ let refused = apply_restore(&store, &value, parsed, 64).await;++ assert!(+ matches!(refused, Err(RestoreError::Refused(ref m)) if m.contains("already exists")),+ "{refused:?}"+ );+ let current = store.get_changelog_config().await.unwrap().unwrap();+ assert_eq!(current.epoch, 1, "the seeded config is untouched");+ }++ #[tokio::test]+ async fn restore_refuses_a_backup_whose_partitions_do_not_match_the_topic() {+ let store = empty_store("restore-mismatch").await;+ let (value, parsed) = backed_up(128);++ let refused = apply_restore(&store, &value, parsed, 64).await;++ assert!(+ matches!(refused, Err(RestoreError::Refused(ref m)) if m.contains("another environment")),+ "{refused:?}"+ );+ let (stored, _) = store+ .get_changelog_config_raw_with_revision()+ .await+ .unwrap();+ assert!(stored.is_none(), "nothing is written");+ }+}diff --git a/rust/ingestion-control-plane/src/config.rs b/rust/ingestion-control-plane/src/config.rsindex a1d89ec7527c..2ce5e7da4438 100644--- a/rust/ingestion-control-plane/src/config.rs+++ b/rust/ingestion-control-plane/src/config.rs@@ -165,6 +165,11 @@ pub struct Config {#[envconfig(from = "PERSONHOG_ETCD_PREFIX", default = "/personhog/")]pub personhog_etcd_prefix: String,+ /// Versioned S3 bucket that keeps a copy of `config/changelog` after+ /// every change; empty disables backups and restore.+ #[envconfig(from = "CHANGELOG_BACKUP_BUCKET", default = "")]+ pub changelog_backup_bucket: String,+/// The personhog changelog's Kafka cluster; unset means `KAFKA_HOSTS`.#[envconfig(from = "PERSONHOG_KAFKA_HOSTS")]pub personhog_kafka_hosts: Option<String>,diff --git a/rust/ingestion-control-plane/src/lib.rs b/rust/ingestion-control-plane/src/lib.rsindex a1816528f251..e631c077468d 100644--- a/rust/ingestion-control-plane/src/lib.rs+++ b/rust/ingestion-control-plane/src/lib.rs@@ -1,4 +1,5 @@pub mod api;+pub mod backup;pub mod cache;pub mod changelog;pub mod config;diff --git a/rust/ingestion-control-plane/src/main.rs b/rust/ingestion-control-plane/src/main.rsindex 343476be5e42..437a4567c455 100644--- a/rust/ingestion-control-plane/src/main.rs+++ b/rust/ingestion-control-plane/src/main.rs@@ -1,15 +1,17 @@use std::future::ready;+use std::sync::Arc;use std::time::Duration;use axum::routing::get;use common_metrics::setup_metrics_routes;use health::{readiness_handler, HealthRegistry};use ingestion_control_plane::api;+use ingestion_control_plane::backup::{self, BackupStore, S3BackupStore};use ingestion_control_plane::config::Config;use ingestion_control_plane::state::AppState;use tokio::signal;-use tracing::info;use tracing::level_filters::LevelFilter;+use tracing::{info, warn};use tracing_subscriber::{layer::SubscriberExt, util::SubscriberInitExt, EnvFilter, Layer};common_alloc::used!();@@ -63,7 +65,8 @@ async fn main() {}});- let state = AppState::new(config).expect("failed to initialize app state");+ let mut state = AppState::new(config).expect("failed to initialize app state");+ start_changelog_backups(&mut state).await;let router = api::router(state).route("/_readiness", get(readiness_handler))@@ -83,3 +86,22 @@ async fn main() {.await.expect("server failed");}++async fn start_changelog_backups(state: &mut AppState) {+ let bucket = state.config.changelog_backup_bucket.clone();+ if bucket.is_empty() {+ info!("CHANGELOG_BACKUP_BUCKET not set; changelog backups disabled");+ return;+ }+ let Some(etcd) = state.etcd.clone() else {+ warn!("CHANGELOG_BACKUP_BUCKET is set without ETCD_ENDPOINTS; changelog backups disabled");+ return;+ };+ let backups: Arc<dyn BackupStore> = Arc::new(S3BackupStore::from_env(bucket).await);+ state.backups = Some(backups.clone());+ tokio::spawn(backup::run(+ etcd,+ state.config.personhog_etcd_prefix.clone(),+ backups,+ ));+}diff --git a/rust/ingestion-control-plane/src/state.rs b/rust/ingestion-control-plane/src/state.rsindex 4b0133efb24b..c419b3e15119 100644--- a/rust/ingestion-control-plane/src/state.rs+++ b/rust/ingestion-control-plane/src/state.rs@@ -4,6 +4,7 @@ use std::time::Duration;use common_database::PoolConfig;use tokio::sync::Semaphore;+use crate::backup::BackupStore;use crate::cache::TtlCache;use crate::config::Config;use crate::etcd::EtcdHandle;@@ -28,6 +29,7 @@ pub struct AppState {pub browse_permits: Arc<Semaphore>,/// `None` when `ETCD_ENDPOINTS` is unset; the etcd tools then 503.pub etcd: Option<Arc<EtcdHandle>>,+ pub backups: Option<Arc<dyn BackupStore>>,}impl AppState {@@ -77,6 +79,7 @@ impl AppState {))),browse_permits: Arc::new(Semaphore::new(config.browse_max_concurrent.max(1))),etcd: etcd.map(Arc::new),+ backups: None,config: Arc::new(config),})}diff --git a/rust/ingestion-control-plane/src/ui/index.html b/rust/ingestion-control-plane/src/ui/index.htmlindex 89862f1dc3a2..132966c0042a 100644--- a/rust/ingestion-control-plane/src/ui/index.html+++ b/rust/ingestion-control-plane/src/ui/index.html@@ -1418,7 +1418,7 @@this.ui.changelogError.replaceChildren();this.ui.changelogForm.replaceChildren(...(view.config? [this.migrationForm(view), this.retireGroupForm()]- : [this.seedForm()]));+ : [this.seedForm(), this.restoreForm()]));}if (!view.config) {this.ui.changelogStatus.replaceChildren(@@ -1526,6 +1526,48 @@error);},+ restoreForm() {+ const versions = el("select", { disabled: true }, el("option", {}, "loading backups…"));+ const error = el("div");+ const button = el("button", { disabled: true }, "Restore from backup");+ fetchJSON("api/personhog/changelog/backups")+ .then((list) => {+ if (!list.length) {+ versions.replaceChildren(el("option", {}, "no backups yet"));+ return;+ }+ versions.replaceChildren(...list.map((v) => el("option", { value: v.version_id },+ `${v.is_latest ? "current" : "earlier"}: ${v.last_modified || v.version_id}`)));+ versions.disabled = false;+ button.disabled = false;+ })+ .catch((e) => {+ versions.replaceChildren(el("option", {}, "backups unavailable"));+ error.replaceChildren(el("div", { class: "muted" }, String(e)));+ });+ button.onclick = async () => {+ const version_id = versions.value;+ const label = versions.selectedOptions[0] ? versions.selectedOptions[0].textContent : version_id;+ if (!window.confirm(`Restore config/changelog from the backup ${label}?\n\nThe config is written exactly as it was backed up, epoch and draining generations included. Restore only works while etcd holds no config. Running services ignore a config with an older epoch than they adopted, so scale personhog to zero first when rebuilding etcd.`)) return;+ button.disabled = true;+ try {+ await fetchJSON("api/personhog/changelog/restore", {+ method: "POST",+ headers: { "content-type": "application/json" },+ body: JSON.stringify({ version_id }),+ });+ this.refresh();+ } catch (e) {+ error.replaceChildren(el("div", { class: "error-box" }, String(e)));+ } finally {+ button.disabled = false;+ }+ };+ return el("div", { style: "padding: 0 14px 6px" },+ el("div", { class: "row" }, versions, button),+ error);+ },+migrationForm(view) {const m = view.migration;const r = view.request;diff --git a/rust/personhog-coordination/src/store.rs b/rust/personhog-coordination/src/store.rsindex fd4a88bf0f81..7ac123d76789 100644--- a/rust/personhog-coordination/src/store.rs+++ b/rust/personhog-coordination/src/store.rs@@ -1472,22 +1472,49 @@ impl PersonhogStore {writer_group: &str,) -> Result<bool> {count_call("seed_changelog_config");- let key = self.key(StoreKey::ChangelogConfig);let config = ChangelogConfig {writer_group: writer_group.to_string(),..ChangelogConfig::new(topic, partitions)};+ self.create_changelog_config(serde_json::to_vec(&config)?)+ .await+ }++ /// Create-if-absent with bytes a backup recorded, written back as they+ /// were stored so fields this build does not know survive the round trip.+ pub async fn restore_changelog_config_raw(&self, value: &[u8]) -> Result<bool> {+ count_call("restore_changelog_config_raw");+ self.create_changelog_config(value.to_vec()).await+ }++ async fn create_changelog_config(&self, value: Vec<u8>) -> Result<bool> {+ let key = self.key(StoreKey::ChangelogConfig);let txn = Txn::new().when(vec![Compare::create_revision(key.clone(),CompareOp::Equal,0,)])- .and_then(vec![TxnOp::put(key, serde_json::to_vec(&config)?, None)]);+ .and_then(vec![TxnOp::put(key, value, None)]);let resp = self.inner.txn(txn).await?;Ok(resp.succeeded())}+ /// The stored config bytes, unparsed, and the revision of the read.+ pub async fn get_changelog_config_raw_with_revision(&self) -> Result<(Option<Vec<u8>>, i64)> {+ count_call("get_changelog_config_raw_with_revision");+ let key = self.key(StoreKey::ChangelogConfig);+ let resp = self+ .inner+ .client()+ .clone()+ .get(key, None)+ .await+ .map_err(assignment_coordination::error::Error::from)?;+ let revision = resp.header().map(|h| h.revision()).unwrap_or(0);+ Ok((resp.kvs().first().map(|kv| kv.value().to_vec()), revision))+ }+/// Bypasses the protocol: see the crate's `test-support` feature.#[cfg(any(test, feature = "test-support"))]pub async fn put_changelog_config(&self, config: &ChangelogConfig) -> Result<()> {