web search done right
update quickstart.optic
Mikkel Denker465a49bf101b
diff --git a/optics/testcases/quickstart.optic b/optics/testcases/quickstart.opticindex fcd2257d..ca7eeb5d 100644--- a/optics/testcases/quickstart.optic+++ b/optics/testcases/quickstart.optic@@ -1,14 +1,17 @@-// You can get syntax highlighting for this file with the following vscode extension: https://marketplace.visualstudio.com/items?itemName=Cuely.Optics+// You can get syntax highlighting for this file with the following vscode+// extension: https://marketplace.visualstudio.com/items?itemName=Cuely.Optics// Optics is a domain-specific language that lets you write small programs to-// influence the search results that gets returned to you from Cuely or other supported search engines.-// The goal is to give you very fine grained control over your own search experience.-// Let's see how this is done.+// influence the search results that gets returned to you from Cuely or other+// search engines which support Optics. The goal of Optics is to give you very+// fine grained control over your own search experience. Let's see how this is+// done.-// As you might have guessed, comments can either be line-comments with "//" or block-comments "/* ... */".-// Most optics contains a sequence of rules that defines how a particular search result should be altered-// given it matches the specific rule.+// As you might have guessed, comments can either be line-comments with "//" or+// block-comments "/* ... */". Optics constists of a sequence of rules where+// each rule defines how a particular search result should be altered given that+// they match the specific rule.Rule {Matches {Title("Top * of 2022")@@ -22,23 +25,33 @@ Rule {Action(Boost(3))};-// Let's unwrap what happens in the first two rules.-// The first rule matches all search results where some pattern occurs in the title of the search result.-// The pattern "Top * of 2022" indicates that a matching search result must contain the term "top" (non-case-sensitive) followed-// by any number of wildcard terms, which is then followed by the terms "of" and "2022" in sequence. This will e.g. match the titles-// "Top movies of 2022" and "See the top tourist attractions of 2022 in Paris". Note that the wildcard can match more than one term-// and that the pattern does not have to be at the beggining or end of the title.-// You can use the special token "|" in a pattern to indicate the beggining or end of the title, so the pattern "|Top * of 2022" would only match-// "Top movies of 2022" but not "See the top tourist attractions of 2022 in Paris". Patterns also only match entire terms-// so the pattern "an*how" would not match the term "anyhow" but instead "an <wildcard> how".--// The second rule matches search results where the pattern matches the url of the result. Both rules specify an action that will be applied to-// the matching results: the first rule downranks these types of listicles whereas the second rule boosts reddit comments.-// The action can either be `Action(Boost(<int>))`, `Action(Downrank(<int>))` or `Action(Discard)` to completely discard the result.-// If no action is specified in a rule it is equivalent to `Action(Boost(1))` which will slightly boost a result.--// You can specify any number of patterns in a `Matches` block. A result will only match if it matches all the patterns.-// As an example, this rule will only discard listicles from "esquire.com" but other results from the same domain will not be discarded.+// Let's unwrap what happens in the first two rules. The first rule matches all+// search results where some pattern occurs in the title of the search result.+// The pattern "Top * of 2022" indicates that a matching search result must+// contain the term "top" (non-case-sensitive) followed by any number of+// wildcard terms, which is then followed by the terms "of" and "2022" in+// sequence. This will e.g. match the titles "Top movies of 2022" and "See the+// top tourist attractions of 2022 in Paris". Note that the wildcard can match+// more than one term and that the pattern does not have to be at the beginning+// or end of the title. You can use the special token "|" in a pattern to+// indicate the beginning or end of the title, so the pattern "|Top * of 2022"+// would only match "Top movies of 2022" but not "See the top tourist+// attractions of 2022 in Paris". Patterns also only match entire terms so the+// pattern "an*how" would not match the term "anyhow" but instead "an <wildcard>+// how".++// The second rule matches search results where the pattern matches the url of+// the result. Both rules specify an action that will be applied to the matching+// results: the first rule downranks these types of listicles whereas the second+// rule boosts reddit comments. The action can either be `Action(Boost(<int>))`,+// `Action(Downrank(<int>))` or `Action(Discard)` to completely discard the+// result. If no action is specified in a rule it is equivalent to+// `Action(Boost(1))` which will slightly boost a result.++// You can specify any number of patterns in a `Matches` block. A result will+// only match if it matches all the patterns in the block. As an example, this+// rule will only discard listicles from "esquire.com" but other results from+// the same domain will not be discarded.Rule {Matches {Title("Top * of 2022"),@@ -56,9 +69,12 @@ Rule {// - Content// - Schema-// The most special of these is probably the `Schema` location. This allows you to match results that contains specific https://schema.org entities.-// This is also the only match-location where you cannot use the special pattern characters "*" and "|". `Schema` only supports simple patterns.-// The following rule boosts all pages that contains the https://schema.org/DiscussionForumPosting entity+// The most special of these is probably the `Schema` location. This allows you+// to match results that contains specific https://schema.org entities. This is+// also the only match-location where you cannot use the special pattern+// characters "*" and "|". `Schema` only supports simple patterns. The following+// rule boosts all pages that contains the+// https://schema.org/DiscussionForumPosting entityRule {Matches {Schema("DiscussionForumPosting")@@ -66,47 +82,63 @@ Rule {Action(Boost(5))};-// By default, any search result that does not match the optic would simply not have any special action applied to it-// and the search result's score would be unaltered. This behaviour can be changed by specifying `DiscardNonMatching`+// By default, any search result that does not match the optic would simply not+// have any special action applied to it and the search result's score would be+// unaltered. This behaviour can be changed by specifying `DiscardNonMatching`DiscardNonMatching;-// Now all search resuts that does not match any of the specified rules will be discarded.---// The `Rule` block we have looked at so far only alters search results that matches the specific rules and leaves other-// results intact. To explain the next part, it will be helpful to know a little bit about how most web search engines-// rank their results.--// Back in the days when Google launched, it completely disrupted the existing search engines by having a lot of-// technical advantages. One of the most ground breaking things they did was the introduction of their core ranking-// algorithm called PageRank. The idea is, that websites that have links from other trustworthy websites must themselves be-// trustworthy. The idea to analyze website links and use this for ranking gave Google way better search results than their-// competitors. However, websites quickly realized this and started actively building links to their own website, which skews-// the PageRank metric. In essense, websites with a high PageRank are no longer guaranteed to be trustworthy but might-// just be the best websites at link building.--// At Cuely we use a similar link based metric called Harmonic Centrality. It is calculated by taking the sum of the inverse distances-// from all other websites to a particular website. Imagine site A links to B that then links to C. This gives the following normalized centralities-// A ---> B ---> C-// 0 0.33 0.5--// While harmonic centrality is believed to be more resistant to link building than page rank, it is still susceptible to it.-// Let's now get back from our detour and see what this means for our optics.--// In optics you have the ability to like and dislike sites. We then calculate a harmonic centrality score from all your liked sites,-// and subtract the harmonic centrality from your disliked sites. What this means is, that sites that has a short distance from your liked sites-// and a long distance from your disliked sites will be heavily prioritized in your search results.-// This is the same thing that happens whenever you like a search result directly in Cuely.+// Now all search resuts that does not match any of the specified rules will be+// discarded.+++// The `Rule` block we have looked at so far only alters search results that+// matches the specific rules and leaves other results intact. To explain the+// next part, it will be helpful to know a little bit about how most web search+// engines rank their results.++// Back in the days when Google launched, it completely disrupted the existing+// search engines by having a lot of technical advantages. One of the most+// ground breaking things they did was the introduction of their core ranking+// algorithm called PageRank. The idea is, that websites that have links from+// other trustworthy websites must themselves be trustworthy. The idea to+// analyze website links and use this for ranking gave Google way better search+// results than their competitors. However, websites quickly realized this and+// started actively building links to their own website, which skews the+// PageRank metric. In essense, websites with a high PageRank are no longer+// guaranteed to be trustworthy but might just be the best websites at link+// building.++// At Cuely we use a similar link based metric called Harmonic Centrality. It is+// calculated by taking the sum of the inverse distances from all other websites+// to a particular website. Imagine site A links to B that then links to C. This+// gives the following normalized centralities+// A ---> B ---> C+// 0 0.33 0.5++// While harmonic centrality is believed to be more resistant to link building+// than PageRank, it is still susceptible to it. Let's now get back from our+// detour and see what this means for our optics.++// In optics you have the ability to like and dislike sites. We then calculate a+// harmonic centrality score from all your liked sites, and subtract the+// harmonic centrality from your disliked sites. What this means is, that sites+// that has a short distance from your liked sites and a long distance from your+// disliked sites will be heavily prioritized in your search results. This is+// the same thing that happens whenever you like a search result directly in+// Cuely.Like(Site("news.ycombinator.com"));Dislike(Site("w3schools.com"));-// One thing to note here is that only simple patterns can be used inside `Like(Site("..."))` and `Dislike(Site("..."))`.-// The special tokens "*" and "|" will not work here.+// One thing to note here is that only simple patterns can be used inside+// `Like(Site("..."))` and `Dislike(Site("..."))`. The special tokens "*" and+// "|" will not work here.-// We have now covered how you can alter the scores of specific search results and indicate which sites you like more than others.-// While harmonic centrality plays a large role in the final score of a search result, in reality it is actually a weighted-// sum of a number of signals where harmonic centrality is simply one of them.-// You can change the coefficient for each of these signals as follows+// We have now covered how you can alter the scores of specific search results+// and indicate which sites you like more than others. While harmonic centrality+// plays a large role in the final score of a search result, in reality it is+// actually a weighted sum of a number of signals where harmonic centrality is+// simply one of them. You can change the coefficient for each of these signals+// as followsRanking {Signal("bm25"),3@@ -116,10 +148,11 @@ Ranking {10000};-// Note that different signals might be scaled differently based on the nature of how they are calculated, so a higher coefficient does-// not necesarrily result in a larger influence on the final score.-// Currently we support the following signals, but since this list is highly dynamic we encourage you to consult the source code for the-// most up-to-date list:+// Note that different signals might be scaled differently based on the nature+// of how they are calculated, so a higher coefficient does not necesarrily+// result in a larger influence on the final score. Currently we support the+// following signals, but since this list is highly dynamic we encourage you to+// consult the source code for the most up-to-date list:// - bm25// - host_centrality// - page_centrality@@ -133,13 +166,16 @@ Ranking {// - query_centrality-// You can also boost text-matches at different parts of a page+// You can also boost text-matches at different parts of a page. As an example,+// this sets the coefficient for pages where the query matches in the title to+// 10:Ranking {Field("title"),10}-// The current list of supported fields is listed here, but again we encourage you to consult the code for the most up-to-date list:+// The current list of supported fields is listed here, but again we encourage+// you to consult the code for the most up-to-date list:// - title// - body// - url@@ -158,6 +194,10 @@ Ranking {// - dmoz_description-// Last but not least, when you have developed your optic it can be installed in Cuely by hosting it somewhere-// that returns a simple plain-text HTTP response and is publicly available. The optic url can then be added-// to https://cuely.io/optics and you should be able to use it during search.\ No newline at end of file+// Last but not least, when you have developed your optic it can be installed in+// Cuely by uploading the Optic to github, and copy the "raw" url into+// https://cuely.io/settings. E.g. the url for this Optic is+// "https://raw.githubusercontent.com/Cuely/sample-optics/main/quickstart.optic"++// Alternatively you can host it anywhere that returns a simple plain-text HTTP+// response and is publicly available.\ No newline at end of file