web search done right
decrease robotstxt char limit to guard against adversarial input
Mikkel Denkerf5cea25a9a91
diff --git a/crates/robotstxt/src/lib.rs b/crates/robotstxt/src/lib.rsindex d0a98aea..1f785cb6 100644--- a/crates/robotstxt/src/lib.rs+++ b/crates/robotstxt/src/lib.rs@@ -17,7 +17,7 @@//! A robots.txt parser and matcher compliant with [RFC 9309](https://www.rfc-editor.org/rfc/rfc9309.html)//! and [Google's Robots.txt parser](https://github.com/google/robotstxt) (with some extensions).-const MAX_CHAR_LIMIT_DEFAULT: usize = 4_000_000; // 4 MB+const MAX_CHAR_LIMIT_DEFAULT: usize = 512_000; // 512 KBmod parser;mod pattern;