Skip to content

web search done right

Remove complete trust with canonical urls. If siteA has a canonical url for siteB, then siteB might show up for queries where it shouldn't actually match but where siteA matches. A bad actor might use this to have weird canonical urls to some sites that they don't like. This could of course be fixed by only respecting canonical urls within the same domain, but even in that case I don't see how the canonical site will benefit the user. If siteA and siteB actually has the same content, then one of them will already be downranked due to duplication detection.

Mikkel Denker403a740aaf51
diff --git a/core/src/webpage/mod.rs b/core/src/webpage/mod.rs
index 8340c87b..11c7b2f9 100644
--- a/core/src/webpage/mod.rs
+++ b/core/src/webpage/mod.rs
@@ -392,10 +392,6 @@ impl Html {
robots: None,
};
- if let Some(url) = res.canonical_url() {
- res.url = url;
- }
-
let queries: Vec<_> = res
.url
.query_pairs()
@@ -420,7 +416,7 @@ impl Html {
Ok(res)
}
- fn canonical_url(&self) -> Option<Url> {
+ pub fn canonical_url(&self) -> Option<Url> {
let mut canonical_url = None;
for node in self.root.select("link").unwrap() {
@@ -2236,11 +2232,6 @@ mod tests {
Some(Url::parse("https://example.com/canonical.html").unwrap())
);
- assert_eq!(
- html.url(),
- &Url::parse("https://example.com/canonical.html").unwrap()
- );
-
let html = Html::parse(
r#"
<html>