Supercharge your AI agents with data from the web and beyond. Building the library for superintelligence. 🔥
TypeScript aiai-agentsai-crawlerai-scrapingai-searchcrawlerdata-extractionhtml-to-markdownllmmarkdownscraperscrapingweb-crawlerweb-dataweb-data-extractionweb-scraperweb-scrapingweb-searchwebscraping
feat(html-transformer): exclude
Gergő Móricz1f21ec32ba75
diff --git a/apps/api/sharedLibs/html-transformer/src/lib.rs b/apps/api/sharedLibs/html-transformer/src/lib.rsindex 65630866c..ff65a4978 100644--- a/apps/api/sharedLibs/html-transformer/src/lib.rs+++ b/apps/api/sharedLibs/html-transformer/src/lib.rs@@ -252,7 +252,7 @@ pub unsafe extern "C" fn extract_metadata(html: *const libc::c_char) -> *mut libCString::new(metadata_out).unwrap().into_raw()}-const EXCLUDE_NON_MAIN_TAGS: [&str; 41] = [+const EXCLUDE_NON_MAIN_TAGS: [&str; 42] = ["header","footer","nav",@@ -294,6 +294,7 @@ const EXCLUDE_NON_MAIN_TAGS: [&str; 41] = ["#widget",".cookie","#cookie",+ ".fc-decoration",];const FORCE_INCLUDE_MAIN_TAGS: [&str; 13] = [