Skip to content

https://arxiv.org/abs/2606.28344. The end of web parsing. The beginning of scalable pixel-native search. link: https://pixelrag.ai/

docs(train): link synthetic data pipeline from Data Curation section (#60)

Yichuan Wang6d9cdda76f64
diff --git a/README.md b/README.md
index f35813c..12a28b9 100644
--- a/README.md
+++ b/README.md
@@ -238,6 +238,8 @@ You don't need to retrain to use the model — the trained adapters are publishe
We also release the full training set
([`Chrisyichuan/screenshot-training-natural-filtered-v2`](https://huggingface.co/datasets/Chrisyichuan/screenshot-training-natural-filtered-v2)),
so you can adapt other backbones yourself — a larger Qwen, or any other embedding model.
+The data curation pipeline (LLM-augmented query generation, filtering, hard-negative mining)
+is documented in [`train/docs/synthetic_data_pipeline.md`](train/docs/synthetic_data_pipeline.md).
## Acknowledgments
diff --git a/train/README.md b/train/README.md
index 257ff51..25820a9 100644
--- a/train/README.md
+++ b/train/README.md
@@ -408,4 +408,6 @@ packaging, and tests, see [`docs/training_dev_notes.md`](./docs/training_dev_not
Visualization of some very early version of the training data:
[early training data viewer](https://yichuan-w.github.io/share/blog-review-first100-light/)
-Reproduce: TBD
+For details on the synthetic data generation pipeline (LLM-augmented query generation,
+filtering, and hard-negative mining), see
+[`docs/synthetic_data_pipeline.md`](./docs/synthetic_data_pipeline.md).