Skip to content

armgit / macro

Public Imported from GitHub
0

Macro is a unified workspace for teams: email, chat, docs, tasks, agents, calls, and CRM — @-linked together with shared AI memory.

fix(search): index unreadable PDFs by name instead of retrying them (#8419)

teo5bfe2259b472
diff --git a/services/search_processing_service/README.md b/services/search_processing_service/README.md
index 487ae1a2f8..334c2055e2 100644
--- a/services/search_processing_service/README.md
+++ b/services/search_processing_service/README.md
@@ -105,7 +105,7 @@ Upserts read the current call state from Postgres and overwrite the indexed repr
### Kafka Delivery Contract
- The durable consumer group is `search-processing-service`. Its subscriptions are declared in `src/inbound/kafka_consumer.rs`.
-- The poll loop shards decoded events by entity ordering key across ten sequential workers, each with a bounded 16-message channel. Sending waits when the selected channel is full, stopping polling and committing until that worker catches up. Events for the same entity stay ordered.
+- The poll loop shards decoded events by entity ordering key across ten sequential workers, each with a bounded 16-message channel. Sending waits when the selected channel is full, stopping polling and committing until that worker catches up. Events for the same entity stay ordered within one consumer, but not end to end: producers can publish them out of order, and a rebalance does not wait for the previous owner's workers, which can still be processing older events while the new owner processes later ones. Handlers must not assume an entity's newest event arrives last.
- An offset is committed asynchronously immediately after a successful in-memory handoff, not after OpenSearch processing. This commit-after-handoff design creates a loss window: a process or host crash can lose committed events still in those buffers. If the worker channel closes before handoff, the offset is left uncommitted for redelivery.
- Malformed, keyless, and unsupported-schema records are logged and committed without handoff so a poison record cannot wedge a partition. Commit failures are logged and may cause duplicate delivery, which is safe because full upserts and per-call deletes are idempotent.
- Processing gets three total attempts, with retries after 1 and 2 seconds. Exhausted events are logged and dropped so later events can continue.
@@ -164,7 +164,7 @@ cargo run --features disable_processing
1. Configure `KAFKA_BROKERS` and the required MSK read permissions, then deploy the Kafka consumer while the live SQS call producer is still enabled.
2. On the first deployment, the new `search-processing-service` group has no committed offsets, so `auto.offset.reset=earliest` replays the retained `macro.calls` history. This replay and the temporary dual delivery from Kafka and SQS are safe: call upserts are full idempotent overwrites, and deletes are idempotent delete-by-query operations.
3. Before disabling live SQS call publication, confirm the consumer-group lag on `macro.calls` has caught up, worker errors are healthy, and the `call_records` index is fresh. Continue monitoring lag after cutover.
-4. Watch service logs for Kafka receive/decode/commit errors, processing retry warnings, events dropped after five attempts, and consumer supervisor restarts. A commit error can result in a safe duplicate; a dropped processing event or crash in the commit-after-handoff window can require recovery.
+4. Watch service logs for Kafka receive/decode/commit errors, processing retry warnings, events dropped after three attempts, and consumer supervisor restarts. A commit error can result in a safe duplicate; a dropped processing event or crash in the commit-after-handoff window can require recovery.
For the local broker, inspect group lag with:
diff --git a/services/search_processing_service/src/parsers/pdf.rs b/services/search_processing_service/src/parsers/pdf.rs
index 2806ccb691..84849c3216 100644
--- a/services/search_processing_service/src/parsers/pdf.rs
+++ b/services/search_processing_service/src/parsers/pdf.rs
@@ -1,7 +1,9 @@
-use pdfium_render::prelude::Pdfium;
+use pdfium_render::prelude::{Pdfium, PdfiumError, PdfiumInternalError};
-/// Parses the pdf into individual pages of content
-pub fn parse_pdf_pages(content: Vec<u8>) -> anyhow::Result<Vec<String>> {
+/// Parses the pdf into individual pages of content, or `None` when pdfium
+/// cannot open it: a password-protected, corrupt, or unsupported file fails
+/// the same way on every attempt.
+pub fn parse_pdf_pages(content: Vec<u8>) -> anyhow::Result<Option<Vec<String>>> {
tracing::trace!("parsing pdf");
let pdfium = Pdfium::new(
@@ -15,7 +17,18 @@ pub fn parse_pdf_pages(content: Vec<u8>) -> anyhow::Result<Vec<String>> {
);
tracing::trace!("initialized pdfium");
- let document = pdfium.load_pdf_from_byte_vec(content, None)?;
+ let document = match pdfium.load_pdf_from_byte_vec(content, None) {
+ Ok(document) => document,
+ Err(PdfiumError::PdfiumLibraryInternalError(
+ error @ (PdfiumInternalError::PasswordError
+ | PdfiumInternalError::FormatError
+ | PdfiumInternalError::SecurityError),
+ )) => {
+ tracing::warn!(error = ?error, "pdfium cannot open the pdf");
+ return Ok(None);
+ }
+ Err(error) => return Err(error.into()),
+ };
tracing::trace!("pdf loaded into pdfium");
@@ -31,5 +44,8 @@ pub fn parse_pdf_pages(content: Vec<u8>) -> anyhow::Result<Vec<String>> {
})
.collect::<Vec<_>>();
- Ok(content)
+ Ok(Some(content))
}
+
+#[cfg(test)]
+mod test;
diff --git a/services/search_processing_service/src/parsers/pdf/test.rs b/services/search_processing_service/src/parsers/pdf/test.rs
new file mode 100644
index 0000000000..c402c4667b
--- /dev/null
+++ b/services/search_processing_service/src/parsers/pdf/test.rs
@@ -0,0 +1,63 @@
+use super::*;
+
+/// A one-page PDF encrypted with the standard security handler under a user
+/// password, which pdfium cannot open without it.
+fn password_protected_pdf() -> Vec<u8> {
+ let objects = [
+ "<< /Type /Catalog /Pages 2 0 R >>",
+ "<< /Type /Pages /Kids [3 0 R] /Count 1 >>",
+ "<< /Type /Page /Parent 2 0 R /MediaBox [0 0 612 792] >>",
+ concat!(
+ "<< /Filter /Standard /V 1 /R 2 /P -4 ",
+ "/O <0000000000000000000000000000000000000000000000000000000000000000> ",
+ "/U <0000000000000000000000000000000000000000000000000000000000000000> >>",
+ ),
+ ];
+
+ let mut pdf = b"%PDF-1.4\n".to_vec();
+ let mut offsets = Vec::new();
+ for (index, object) in objects.iter().enumerate() {
+ offsets.push(pdf.len());
+ pdf.extend(format!("{} 0 obj\n{object}\nendobj\n", index + 1).bytes());
+ }
+ let xref = pdf.len();
+ pdf.extend(format!("xref\n0 {}\n0000000000 65535 f \n", objects.len() + 1).bytes());
+ for offset in offsets {
+ pdf.extend(format!("{offset:010} 00000 n \n").bytes());
+ }
+ pdf.extend(
+ format!(
+ concat!(
+ "trailer\n<< /Size {} /Root 1 0 R /Encrypt 4 0 R ",
+ "/ID [<00000000000000000000000000000000> <00000000000000000000000000000000>] >>\n",
+ "startxref\n{}\n%%EOF\n",
+ ),
+ objects.len() + 1,
+ xref,
+ )
+ .bytes(),
+ );
+ pdf
+}
+
+#[test]
+fn test_pdfs_pdfium_cannot_open_are_unreadable() {
+ for (name, content) in [
+ ("password protected", password_protected_pdf()),
+ ("not a pdf", b"not a pdf".to_vec()),
+ ] {
+ let pages = parse_pdf_pages(content).expect("an unreadable pdf is not an error");
+ assert!(pages.is_none(), "{name}");
+ }
+}
+
+#[test]
+fn test_readable_pdfs_parse_into_pages() {
+ let content = std::fs::read("fixtures/example.pdf").expect("the fixture exists");
+
+ let pages = parse_pdf_pages(content)
+ .expect("the fixture parses")
+ .expect("the fixture is readable");
+
+ assert!(!pages.is_empty());
+}
diff --git a/services/search_processing_service/src/process/document/raw_document.rs b/services/search_processing_service/src/process/document/raw_document.rs
index 6535bee2a0..ee5500c504 100644
--- a/services/search_processing_service/src/process/document/raw_document.rs
+++ b/services/search_processing_service/src/process/document/raw_document.rs
@@ -4,6 +4,7 @@ use anyhow::Context;
use chrono::Utc;
use document_sub_type::DocumentSubType;
use lexical_client::types::MarkdownParseResult;
+use macro_db_client::document::get_document_version_id;
use model::document::{DocumentMetadata, FileType};
use model_owner::Owner;
use models_properties::EntityType;
@@ -218,11 +219,12 @@ async fn read_design_text(file_type: FileType, content: Vec<u8>) -> anyhow::Resu
.unwrap_or_else(|e| Err(anyhow::anyhow!("the design engine failed: {e}")))
}
-/// Indexes a design the engines cannot read (an old or damaged file, or one
-/// too large to decode here) by name only, dropping the text an earlier
-/// version left, so it stays findable. Reading it again would fail the same
-/// way, so this is not an error to retry.
-async fn update_search_with_unreadable_design(
+/// Indexes a file its parser cannot read (a password-protected or damaged
+/// PDF, or a design that is old, damaged, or too large to decode here) by
+/// name only, dropping the text an earlier version left, so it stays
+/// findable. Reading it again would fail the same way, so this is not an
+/// error to retry.
+async fn update_search_with_unreadable_file(
opensearch_client: &OpensearchClient,
db: &sqlx::Pool<sqlx::Postgres>,
search_extractor_message: &SearchExtractorMessage,
@@ -237,9 +239,39 @@ async fn update_search_with_unreadable_design(
update_search_with_parent_only_document(opensearch_client, db, search_extractor_message).await
}
+/// The version whose stored content search indexes. A PDF is served from its
+/// first version, and a save adds a version without rewriting that file, so
+/// text read from any other version (a message delivered out of order, or the
+/// bytes of an API save) is not what the document shows. A DOCX is indexed
+/// from its converted PDF, which each conversion overwrites in place. Every
+/// other file is indexed at its latest version.
+async fn indexed_version_id(
+ db: &sqlx::Pool<sqlx::Postgres>,
+ search_extractor_message: &SearchExtractorMessage,
+ latest_version_id: i64,
+) -> anyhow::Result<String> {
+ if !matches!(
+ search_extractor_message.file_type,
+ FileType::Pdf | FileType::Docx
+ ) {
+ return Ok(latest_version_id.to_string());
+ }
+ let message_version_id = search_extractor_message
+ .document_version_id
+ .as_deref()
+ .context("expected document version id to be provided for pdf/docx")?;
+ if message_version_id == CONVERTED_DOCUMENT_FILE_NAME {
+ return Ok(CONVERTED_DOCUMENT_FILE_NAME.to_string());
+ }
+ let (served_version_id, _) = get_document_version_id(db, &search_extractor_message.document_id)
+ .await
+ .context("failed to get the served document version")?;
+ Ok(served_version_id.to_string())
+}
+
/// The S3 key of the content to index, or `None` when the message names a
-/// superseded version. Content is stored under the document's owner; sync
-/// messages carry no `user_id`.
+/// version other than the indexed one. Content is stored under the
+/// document's owner; sync messages carry no `user_id`.
fn raw_document_key(
owner: &Owner,
search_extractor_message: &SearchExtractorMessage,
@@ -307,17 +339,12 @@ pub async fn update_search_with_raw_document(
// TODO: this is hacky, update the search event message to use the correctly serialized
// document key parts from the s3_key crate
- let document_version_id = match search_extractor_message.file_type {
- // For static/converted files, we want to use the version from the search extractor message since
- // that is what is in s3 and document saves don't change the actual file in s3.
- FileType::Pdf | FileType::Docx => search_extractor_message
- .document_version_id
- .clone()
- .context("expected document version id to be provided for pdf/docx")?,
- // For all other files we want to ensure we are only updating search if this message
- // contains the latest document version id
- _ => document_info.document_version_id.to_string(),
- };
+ let document_version_id = indexed_version_id(
+ db,
+ search_extractor_message,
+ document_info.document_version_id,
+ )
+ .await?;
if document_info.file_type.is_none() {
tracing::debug!("file type is none");
@@ -339,7 +366,7 @@ pub async fn update_search_with_raw_document(
tracing::debug!(
msg_version_id = ?search_extractor_message.document_version_id,
document_version_id,
- "document version is not latest, skipping"
+ "message names a version search does not index, skipping"
);
return Ok(());
};
@@ -364,7 +391,16 @@ pub async fn update_search_with_raw_document(
FileType::Pdf | FileType::Docx => {
#[cfg(feature = "pdf")]
{
- let pages_content = parse_pdf_pages(content).context("unable to parse pdf")?;
+ let Some(pages_content) =
+ parse_pdf_pages(content).context("unable to parse pdf")?
+ else {
+ return update_search_with_unreadable_file(
+ opensearch_client,
+ db,
+ search_extractor_message,
+ )
+ .await;
+ };
pages_content
.iter()
.enumerate()
@@ -430,7 +466,7 @@ pub async fn update_search_with_raw_document(
if let Err(error) = &read {
tracing::warn!(error = ?error, "indexing an unreadable design by name only");
}
- return update_search_with_unreadable_design(
+ return update_search_with_unreadable_file(
opensearch_client,
db,
search_extractor_message,
diff --git a/services/search_processing_service/src/process/document/raw_document/test.rs b/services/search_processing_service/src/process/document/raw_document/test.rs
index 708593b0e6..4122f01dfb 100644
--- a/services/search_processing_service/src/process/document/raw_document/test.rs
+++ b/services/search_processing_service/src/process/document/raw_document/test.rs
@@ -211,6 +211,185 @@ fn raw_document_key_is_under_the_stored_owner() {
);
}
+/// The document in `basic_versioned_document`: a PDF uploaded as version 1
+/// and saved again as version 2, so it is served from version 1.
+const SAVED_PDF: &str = "document-one";
+const BUCKET: &str = "document-storage";
+
+fn saved_pdf_message(file_type: FileType, version: &str) -> SearchExtractorMessage {
+ SearchExtractorMessage {
+ user_id: "macro|[email protected]".to_string(),
+ document_id: SAVED_PDF.to_string(),
+ file_type,
+ document_version_id: Some(version.to_string()),
+ index_override: None,
+ }
+}
+
+#[sqlx::test(
+ migrations = "../../crates/macro_db_client/migrations",
+ fixtures(
+ path = "../../../../../../crates/macro_db_client/fixtures",
+ scripts("basic_versioned_document")
+ )
+)]
+async fn a_pdf_is_indexed_from_the_version_it_is_served_from(pool: sqlx::PgPool) {
+ let owner: Owner = "macro|[email protected]".to_string().try_into().unwrap();
+ let served = build_cloud_storage_bucket_document_key(&owner, SAVED_PDF, 1);
+ let latest = build_cloud_storage_bucket_document_key(&owner, SAVED_PDF, 2);
+ let converted = build_docx_to_pdf_converted_document_key(&owner, SAVED_PDF);
+
+ for (file_type, version, expected) in [
+ (FileType::Pdf, "1", Some(&served)),
+ (FileType::Pdf, "2", None),
+ (
+ FileType::Pdf,
+ CONVERTED_DOCUMENT_FILE_NAME,
+ Some(&converted),
+ ),
+ (
+ FileType::Docx,
+ CONVERTED_DOCUMENT_FILE_NAME,
+ Some(&converted),
+ ),
+ (FileType::Md, "1", None),
+ (FileType::Md, "2", Some(&latest)),
+ ] {
+ let message = saved_pdf_message(file_type, version);
+ let indexed = indexed_version_id(&pool, &message, 2).await.unwrap();
+ assert_eq!(
+ raw_document_key(&owner, &message, &indexed).as_ref(),
+ expected,
+ "{file_type} at version {version}"
+ );
+ }
+}
+
+type Requests = std::sync::Arc<std::sync::Mutex<Vec<(String, String)>>>;
+
+/// Stands in for both S3 and OpenSearch: every object read returns `object`,
+/// and each request is recorded as `METHOD path` with its body.
+async fn fake_storage(object: &'static [u8]) -> (String, Requests) {
+ use axum::{
+ Json,
+ body::Bytes,
+ http::{Method, StatusCode, Uri},
+ response::IntoResponse,
+ };
+
+ let requests = Requests::default();
+ let recorded = requests.clone();
+ let app = axum::Router::new().fallback(move |method: Method, uri: Uri, body: Bytes| {
+ let recorded = recorded.clone();
+ async move {
+ let path = uri.path().to_string();
+ recorded.lock().unwrap().push((
+ format!("{method} {path}"),
+ String::from_utf8_lossy(&body).into_owned(),
+ ));
+ if method == Method::GET {
+ object.into_response()
+ } else if path.ends_with("/_delete_by_query") {
+ Json(serde_json::json!({ "deleted": 1 })).into_response()
+ } else if path.ends_with("/_bulk") {
+ Json(serde_json::json!({
+ "errors": false,
+ "items": [{ "index": { "status": 201 } }],
+ }))
+ .into_response()
+ } else {
+ StatusCode::NOT_FOUND.into_response()
+ }
+ }
+ });
+ let listener = tokio::net::TcpListener::bind("127.0.0.1:0").await.unwrap();
+ let endpoint = format!("http://{}", listener.local_addr().unwrap());
+ tokio::spawn(async move { axum::serve(listener, app).await.unwrap() });
+ (endpoint, requests)
+}
+
+fn fake_s3(endpoint: &str) -> s3_client::S3 {
+ use aws_sdk_s3::config::{BehaviorVersion, Credentials, Region, retry::RetryConfig};
+
+ s3_client::S3::new(aws_sdk_s3::Client::from_conf(
+ aws_sdk_s3::Config::builder()
+ .behavior_version(BehaviorVersion::latest())
+ .region(Region::new("us-east-1"))
+ .credentials_provider(Credentials::new("test", "test", None, None, "test"))
+ .endpoint_url(endpoint)
+ .force_path_style(true)
+ .retry_config(RetryConfig::disabled())
+ .build(),
+ ))
+}
+
+fn fake_opensearch(endpoint: &str) -> OpensearchClient {
+ OpensearchClient::new(endpoint.to_string(), String::new(), String::new()).unwrap()
+}
+
+#[sqlx::test(
+ migrations = "../../crates/macro_db_client/migrations",
+ fixtures(
+ path = "../../../../../../crates/macro_db_client/fixtures",
+ scripts("basic_versioned_document")
+ )
+)]
+async fn a_message_for_a_version_the_pdf_is_not_served_from_changes_nothing(pool: sqlx::PgPool) {
+ let (endpoint, requests) = fake_storage(b"not a pdf").await;
+
+ update_search_with_raw_document(
+ &fake_opensearch(&endpoint),
+ &pool,
+ &fake_s3(&endpoint),
+ BUCKET,
+ &saved_pdf_message(FileType::Pdf, "2"),
+ )
+ .await
+ .unwrap();
+
+ assert_eq!(*requests.lock().unwrap(), Vec::new());
+}
+
+#[cfg(feature = "pdf")]
+#[sqlx::test(
+ migrations = "../../crates/macro_db_client/migrations",
+ fixtures(
+ path = "../../../../../../crates/macro_db_client/fixtures",
+ scripts("basic_versioned_document")
+ )
+)]
+async fn an_unreadable_pdf_is_indexed_by_name_only(pool: sqlx::PgPool) {
+ let (endpoint, requests) = fake_storage(b"not a pdf").await;
+
+ update_search_with_raw_document(
+ &fake_opensearch(&endpoint),
+ &pool,
+ &fake_s3(&endpoint),
+ BUCKET,
+ &saved_pdf_message(FileType::Pdf, "1"),
+ )
+ .await
+ .expect("an unreadable pdf is not an error to retry");
+
+ let requests = requests.lock().unwrap().clone();
+ let lines: Vec<&str> = requests.iter().map(|(line, _)| line.as_str()).collect();
+ assert_eq!(lines.len(), 3, "{lines:?}");
+ assert!(
+ lines[0].starts_with(&format!("GET /{BUCKET}/")) && lines[0].ends_with("/document-one/1"),
+ "{lines:?}"
+ );
+ assert_eq!(lines[1], "POST /documents/_delete_by_query");
+ assert_eq!(lines[2], "POST /documents/_bulk");
+ let bulk: Vec<serde_json::Value> = requests[2]
+ .1
+ .lines()
+ .map(|line| serde_json::from_str(line).unwrap())
+ .collect();
+ assert_eq!(bulk.len(), 2, "only the parent doc is written: {bulk:?}");
+ assert_eq!(bulk[1]["document_name"], "test_document_name");
+ assert_eq!(bulk[1]["document_relation"], "document");
+}
+
#[tokio::test]
async fn test_unreadable_designs_are_read_errors() {
for file_type in [FileType::Psd, FileType::Psb, FileType::Ai, FileType::Md] {