Apache Lucene open-source search software
Recommend multi-stage retrieval pipelines in oal.search javadocs. (#14310)
Adrien Grandfcffbe1ebb5e
diff --git a/lucene/core/src/java/org/apache/lucene/search/package-info.java b/lucene/core/src/java/org/apache/lucene/search/package-info.javaindex 664e8e61693..b01de702cc1 100644--- a/lucene/core/src/java/org/apache/lucene/search/package-info.java+++ b/lucene/core/src/java/org/apache/lucene/search/package-info.java@@ -350,6 +350,41 @@** <a id="customQueriesExpert"></a>*+ * <h3>Multi-stage retrieval pipelines</h3>+ *+ * <p>The above explains how to influence the score when evaluating all matches of the query. This+ * is expensive by design since it applies to all matches of the query, which could be millions. In+ * order to apply more sophisticated ranking logic, a good approach consists of having a retrieval+ * pipeline that runs a simple candidate retrieval stage that retrieves e.g. 1,000 hits, followed by+ * a more sophisticated reranking stage that reranks these 1,000 hits to select the best 100 hits+ * among them. Since the number of hits that this retrieval stage needs to operate on is bounded, it+ * allows it to be more sophisticated.+ *+ * <p>Lucene exposes reranking via the {@link org.apache.lucene.search.Rescorer} abstract class,+ * which has two main sub-classes:+ *+ * <ul>+ * <li>{@link org.apache.lucene.search.QueryRescorer}, to rescore using a query. For instance, the+ * query string could be parsed as phrase query using {@link+ * org.apache.lucene.util.QueryBuilder#createPhraseQuery} instead of a boolean query in order+ * to help boost hits which also match the query string as a phrase.+ * <li>{@link org.apache.lucene.search.SortRescorer}, to rescore using a {@link+ * org.apache.lucene.search.Sort}. For instance, the best 1,000 hits by BM25 score may be+ * sorted by descending popularity in order to compute the final top-100 hits.+ * </ul>+ *+ * <h3>Top hits fusion</h3>+ *+ * <p>Sometimes, multiple retrieval pipelines may make sense, having their own pros and cons. A+ * typical example would be a lexical retrieval pipeline, matching exactly what the user requested,+ * and a semantic retrieval pipeline, matching documents that are closest to the user's query from a+ * semantic perspective. Combining scores is hazardous as different retrieval pipelines often+ * produce scores that not only have different ranges, but also different distributions within this+ * range. A robust way of combining multiple retrieval pipelines consists of combining the top hits+ * that they produce through their ranks rather than through their scores using reciprocal rank+ * fusion. This is exposed via {@link org.apache.lucene.search.TopDocs#rrf(int topN, int k,+ * TopDocs[] hits)}.+ ** <h2>Custom Queries — Expert Level</h2>** <p>Custom queries are an expert level task, so tread carefully and be prepared to share your code