Skip to content

armgit / macro

Public Imported from GitHub
0

Macro is a unified workspace for teams: email, chat, docs, tasks, agents, calls, and CRM — @-linked together with shared AI memory.

fix(search): prevent accidental agent index creation (#6495)

Eric Hayese768fde12edf
diff --git a/crates/opensearch_client/src/upsert/agent_session.rs b/crates/opensearch_client/src/upsert/agent_session.rs
index a1f53bd669..06bd805879 100644
--- a/crates/opensearch_client/src/upsert/agent_session.rs
+++ b/crates/opensearch_client/src/upsert/agent_session.rs
@@ -174,6 +174,9 @@ pub(crate) async fn reconcile_agent_session(
&args.agent_session_id,
))
.routing(routing)
+ // Normal writes must never auto-create an index under the alias name.
+ // Backfills may explicitly target a pre-provisioned physical index.
+ .require_alias(index_override.is_none())
.body(parent_document(args))
.refresh(opensearch::params::Refresh::WaitFor)
.send()
@@ -201,6 +204,7 @@ pub(crate) async fn reconcile_agent_session(
let response = client
.bulk(BulkParts::Index(index))
+ .require_alias(index_override.is_none())
.body(body)
.refresh(opensearch::params::Refresh::WaitFor)
.send()
diff --git a/crates/opensearch_client/src/upsert/agent_session/test.rs b/crates/opensearch_client/src/upsert/agent_session/test.rs
index 4ebf4579f5..6ee43fdc44 100644
--- a/crates/opensearch_client/src/upsert/agent_session/test.rs
+++ b/crates/opensearch_client/src/upsert/agent_session/test.rs
@@ -71,6 +71,80 @@ fn index_override_selects_a_physical_backfill_index() {
);
}
+/// Capture actual HTTP requests so both index and bulk serialization are covered.
+async fn capture_reconcile_requests(index_override: Option<&str>) -> Vec<String> {
+ use std::io::{BufRead, Read, Write};
+ use std::net::TcpListener;
+ use std::time::Duration;
+
+ let listener = TcpListener::bind("127.0.0.1:0").unwrap();
+ let address = listener.local_addr().unwrap();
+ let server = std::thread::spawn(move || {
+ let mut requests = Vec::new();
+ for body in [
+ r#"{}"#,
+ r#"{"errors":false}"#,
+ r#"{"failures":[],"timed_out":false}"#,
+ ] {
+ let (mut stream, _) = listener.accept().unwrap();
+ stream
+ .set_read_timeout(Some(Duration::from_secs(5)))
+ .unwrap();
+ let mut reader = std::io::BufReader::new(&mut stream);
+ let mut request = String::new();
+ reader.read_line(&mut request).unwrap();
+ requests.push(request);
+ let mut content_length = 0;
+ loop {
+ let mut line = String::new();
+ reader.read_line(&mut line).unwrap();
+ if line == "\r\n" {
+ break;
+ }
+ if let Some(value) = line.to_ascii_lowercase().strip_prefix("content-length:") {
+ content_length = value.trim().parse::<usize>().unwrap();
+ }
+ }
+ reader.read_exact(&mut vec![0; content_length]).unwrap();
+ write!(stream, "HTTP/1.1 200 OK\r\nContent-Type: application/json\r\nContent-Length: {}\r\nConnection: close\r\n\r\n{body}", body.len()).unwrap();
+ }
+ requests
+ });
+ let client =
+ crate::OpensearchClient::new(format!("http://{address}"), "".into(), "".into()).unwrap();
+ let result = client
+ .reconcile_agent_session(&args(), index_override)
+ .await;
+ result.unwrap();
+ server.join().unwrap()
+}
+
+#[tokio::test]
+async fn normal_parent_and_bulk_writes_require_an_alias() {
+ let requests = capture_reconcile_requests(None).await;
+ assert!(
+ requests[0].starts_with("POST /agent_sessions/_doc/session-1?"),
+ "{requests:?}"
+ );
+ assert!(requests[1].starts_with("POST /agent_sessions/_bulk?"));
+ for request in &requests[..2] {
+ assert!(request.contains("require_alias=true"), "{request}");
+ }
+}
+
+#[tokio::test]
+async fn explicit_backfill_overrides_allow_a_physical_index() {
+ let requests = capture_reconcile_requests(Some("scratch-index")).await;
+ assert!(
+ requests[0].starts_with("POST /scratch-index/_doc/session-1?"),
+ "{requests:?}"
+ );
+ assert!(requests[1].starts_with("POST /scratch-index/_bulk?"));
+ for request in &requests[..2] {
+ assert!(request.contains("require_alias=false"), "{request}");
+ }
+}
+
/// Uses only a uniquely named scratch index; never changes application indices.
#[tokio::test]
#[ignore = "requires local OpenSearch at http://localhost:9200"]
diff --git a/infra/stacks/opensearch/helpers/README.md b/infra/stacks/opensearch/helpers/README.md
index c6ab7e3782..ee5a7daa1a 100644
--- a/infra/stacks/opensearch/helpers/README.md
+++ b/infra/stacks/opensearch/helpers/README.md
@@ -42,11 +42,79 @@ that can be swapped without a code deploy.
All migration scripts default to `DRY_RUN=true`; pass `DRY_RUN=false` to apply.
+## Manual provisioning
+
+Provision indices and aliases manually using the canonical declarations in
+`create_indices.ts`, and verify them before deploying their consumers. Deployment
+does not create or reconcile indices.
+
+Normal agent-session parent and bulk writes set `require_alias=true`. A missing
+alias fails the write instead of automatically creating a bare index with an
+inferred mapping. Explicit backfill `index_override` requests may still target
+a physical index. This guard does not provision a schema or recover events
+already dropped by the consumer.
+
+## One-time repair: incorrectly auto-created agent-session index
+
+This is an operator procedure, separate from deployment. Confirm that
+`agent_sessions` is a bare physical index with the incorrect mapping, not an
+existing alias. If it is already an alias, inspect its target before proceeding.
+Configure the intended cluster's usual `OPENSEARCH_*` credentials and use the
+production VPC connection/tunnel when appropriate.
+
+Create the versioned destination from the canonical declaration and verify it:
+
+```sh
+ENVIRONMENT=prod INDEX=agent_sessions bun scripts/create_indices.ts
+ENVIRONMENT=prod INDEX=agent_sessions DRY_RUN=false bun scripts/create_indices.ts
+ENVIRONMENT=prod INDEX=agent_sessions bun scripts/verify_mappings.ts
+```
+
+Inspect `GET /agent_sessions_v1/_mapping` against `AGENT_SESSIONS_V1_BODY` in
+`create_indices.ts`: the existing verifier checks field presence and types,
+not all mapping parameters. Confirm `dynamic:false`, the
+`agent_session_relation` join from `agent_session` to `message`, field-alias
+paths, and date formats before proceeding.
+
+A bare index blocks alias creation, so creation deliberately leaves the alias
+for this manual cutover. If the destination already contains data, inspect it
+before deciding to use it. Wait for `agent_sessions_v1` health to be at least
+yellow (all primary shards available), and recheck the source/alias state.
+
+For a future-writes-only repair, submit this **single** request through the
+OpenSearch API after explicitly accepting deletion of the old search projection:
+
+```http
+POST /_aliases
+{
+ "actions": [
+ { "remove_index": { "index": "agent_sessions" } },
+ { "add": { "index": "agent_sessions_v1", "alias": "agent_sessions", "is_write_index": true } }
+ ]
+}
+```
+
+Do not issue a separate DELETE followed by alias creation: a writer could
+recreate the bare index between those operations. The atomic request deletes
+the old index and its search data; Postgres session data is untouched. It does
+not reindex or backfill. The generic `reindex_with_alias_swap.ts` helper copies
+old documents, so it is not a substitute for this no-copy operation.
+
+Require an acknowledged response, then run `verify_mappings.ts` with
+`INDEX=agent_sessions`. Inspect `GET /_alias/agent_sessions`: its sole target
+must be `agent_sessions_v1` and `is_write_index` must be true or omitted, never
+false. Create a new session, complete a turn, and verify its title and unique
+phrases from both authors are searchable.
+Check indexing errors. In-flight reconciles can straddle the cutover; backfill
+those sessions and any desired historical sessions using the existing
+`POST /internal/backfill/agent-sessions` endpoint. Record the repair outcome in
+the incident runbook.
+
## Adding a field to an existing index
The index bodies in `create_indices.ts` are the source of truth for mappings.
Every body is `dynamic: 'false'`, so a field the service writes but the live
-mapping lacks is dropped silently — and a search over it matches nothing
+mapping lacks remains in `_source` but is not indexed — and a search over it matches nothing
rather than erroring. That is how `call_records_v2` shipped without
`properties`/`name` and left tag filters on calls returning empty (macro-2731).
diff --git a/services/search_processing_service/README.md b/services/search_processing_service/README.md
index 8293b1045d..487ae1a2f8 100644
--- a/services/search_processing_service/README.md
+++ b/services/search_processing_service/README.md
@@ -42,6 +42,12 @@ the parent and both authors' folded messages, refreshes them, then removes
documents from older projection generations. Missing sessions delete their
entire projection, even when triggered by an older event.
+Provision and verify the `agent_sessions` index and alias manually before
+deploying this consumer; deployment does not create indices. Normal parent and
+bulk writes require an alias, so a missing alias fails instead of automatically
+creating an index. An explicit backfill `index_override` permits physical
+indices. Use the OpenSearch helper runbook for an explicit repair.
+
For rollout:
1. Verify the `agent_sessions` alias exists with the join mapping defined in