Skip to content

High-performance browser automation bridge and multi-instance orchestrator with advanced stealth injection and real-time dashboard.

test(extract): add CC BY / CC BY-SA attribution for the corpus page copies and correct the Wikinews licence to CC BY 4.0 (PIN-395)

Luigi Agostib992caea0e26
diff --git a/internal/extract/testdata/corpus/README.md b/internal/extract/testdata/corpus/README.md
index 25f27a5c..3db3278e 100644
--- a/internal/extract/testdata/corpus/README.md
+++ b/internal/extract/testdata/corpus/README.md
@@ -92,6 +92,26 @@ floors in step with this directory, and the
page through the live pipeline, reporting every field and failing an entry
whose hits fall below its floor.
-The toscrape pages are Zyte's public scraping sandbox. The Wikinews article is
-CC BY 2.5 and the Wikipedia pages CC BY-SA 4.0, both credited to their
-contributors at the URLs above.
+## Attribution and licences
+
+The snapshots here and the HTML copies under `tests/e2e/fixtures/corpus/` carry
+third-party page content, captured on 2026-09-13 from the URLs in the table
+above:
+
+- `article`: "Grateful Dead rhythm guitarist Bob Weir dies, aged 78", Wikinews
+ contributors, licensed CC BY 4.0
+ (https://creativecommons.org/licenses/by/4.0/; the page's own footer puts
+ text created after 2024-12-16 under 4.0). Attributed to Wikinews.
+- `infobox`: "Mount Twynam", Wikipedia contributors, and `search`: the
+ Wikipedia search results page for "headless browser", whose snippets are
+ excerpts of Wikipedia articles by Wikipedia contributors; both licensed
+ CC BY-SA 4.0 (https://creativecommons.org/licenses/by-sa/4.0/). The copies
+ are adaptations and are shared under the same licence.
+- `product`, `listing`, `login`: Zyte's public scraping sandboxes
+ (books.toscrape.com, quotes.toscrape.com), published for scraping practice.
+- `pricing`, `stories`: snapshot only, no HTML copy; see the e2e manifest.
+
+Changes from the originals: the snapshots are reformatted to one node per line;
+the HTML copies have scripts, iframes and external resources removed,
+stylesheets reduced and inlined, and images replaced by a 1x1 pixel. No text
+was edited.
diff --git a/tests/e2e/fixtures/corpus/README.md b/tests/e2e/fixtures/corpus/README.md
new file mode 100644
index 00000000..88837cf5
--- /dev/null
+++ b/tests/e2e/fixtures/corpus/README.md
@@ -0,0 +1,23 @@
+# e2e extract corpus
+
+Static copies of the pages in `internal/extract/testdata/corpus/`, used by
+`tests/e2e/scenarios/api/extract-corpus-extended.sh`. That README describes how
+they were made and how `manifest.json` gates them.
+
+## Attribution and licences
+
+All pages captured on 2026-09-13.
+
+| file | source | author / licence |
+| -------------- | ---------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------- |
+| `article.html` | https://en.wikinews.org/wiki/Grateful_Dead_rhythm_guitarist_Bob_Weir_dies,_aged_78 | Wikinews contributors, CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/) |
+| `infobox.html` | https://en.wikipedia.org/wiki/Mount_Twynam | Wikipedia contributors, CC BY-SA 4.0 (https://creativecommons.org/licenses/by-sa/4.0/) |
+| `search.html` | https://en.wikipedia.org/w/index.php?search=headless+browser&title=Special:Search&profile=advanced&fulltext=1&ns0=1 | Wikipedia contributors (article excerpts), CC BY-SA 4.0 |
+| `product.html` | https://books.toscrape.com/catalogue/a-light-in-the-attic_1000/index.html | Zyte's public scraping sandbox |
+| `listing.html` | https://books.toscrape.com/catalogue/category/books/travel_2/index.html | Zyte's public scraping sandbox |
+| `login.html` | https://quotes.toscrape.com/login | Zyte's public scraping sandbox |
+
+Changes from the originals: scripts, iframes and external resources removed,
+stylesheets reduced to the matching rules and inlined, images replaced by a
+1x1 pixel. No text was edited. The Wikipedia-derived copies are adaptations
+shared under CC BY-SA 4.0.