Skip to content

web search done right

update quickstart.optic

Mikkel Denker465a49bf101b
diff --git a/optics/testcases/quickstart.optic b/optics/testcases/quickstart.optic
index fcd2257d..ca7eeb5d 100644
--- a/optics/testcases/quickstart.optic
+++ b/optics/testcases/quickstart.optic
@@ -1,14 +1,17 @@
-// You can get syntax highlighting for this file with the following vscode extension: https://marketplace.visualstudio.com/items?itemName=Cuely.Optics
+// You can get syntax highlighting for this file with the following vscode
+// extension: https://marketplace.visualstudio.com/items?itemName=Cuely.Optics
// Optics is a domain-specific language that lets you write small programs to
-// influence the search results that gets returned to you from Cuely or other supported search engines.
-// The goal is to give you very fine grained control over your own search experience.
-// Let's see how this is done.
+// influence the search results that gets returned to you from Cuely or other
+// search engines which support Optics. The goal of Optics is to give you very
+// fine grained control over your own search experience. Let's see how this is
+// done.
-// As you might have guessed, comments can either be line-comments with "//" or block-comments "/* ... */".
-// Most optics contains a sequence of rules that defines how a particular search result should be altered
-// given it matches the specific rule.
+// As you might have guessed, comments can either be line-comments with "//" or
+// block-comments "/* ... */". Optics constists of a sequence of rules where
+// each rule defines how a particular search result should be altered given that
+// they match the specific rule.
Rule {
Matches {
Title("Top * of 2022")
@@ -22,23 +25,33 @@ Rule {
Action(Boost(3))
};
-// Let's unwrap what happens in the first two rules.
-// The first rule matches all search results where some pattern occurs in the title of the search result.
-// The pattern "Top * of 2022" indicates that a matching search result must contain the term "top" (non-case-sensitive) followed
-// by any number of wildcard terms, which is then followed by the terms "of" and "2022" in sequence. This will e.g. match the titles
-// "Top movies of 2022" and "See the top tourist attractions of 2022 in Paris". Note that the wildcard can match more than one term
-// and that the pattern does not have to be at the beggining or end of the title.
-// You can use the special token "|" in a pattern to indicate the beggining or end of the title, so the pattern "|Top * of 2022" would only match
-// "Top movies of 2022" but not "See the top tourist attractions of 2022 in Paris". Patterns also only match entire terms
-// so the pattern "an*how" would not match the term "anyhow" but instead "an <wildcard> how".
-
-// The second rule matches search results where the pattern matches the url of the result. Both rules specify an action that will be applied to
-// the matching results: the first rule downranks these types of listicles whereas the second rule boosts reddit comments.
-// The action can either be `Action(Boost(<int>))`, `Action(Downrank(<int>))` or `Action(Discard)` to completely discard the result.
-// If no action is specified in a rule it is equivalent to `Action(Boost(1))` which will slightly boost a result.
-
-// You can specify any number of patterns in a `Matches` block. A result will only match if it matches all the patterns.
-// As an example, this rule will only discard listicles from "esquire.com" but other results from the same domain will not be discarded.
+// Let's unwrap what happens in the first two rules. The first rule matches all
+// search results where some pattern occurs in the title of the search result.
+// The pattern "Top * of 2022" indicates that a matching search result must
+// contain the term "top" (non-case-sensitive) followed by any number of
+// wildcard terms, which is then followed by the terms "of" and "2022" in
+// sequence. This will e.g. match the titles "Top movies of 2022" and "See the
+// top tourist attractions of 2022 in Paris". Note that the wildcard can match
+// more than one term and that the pattern does not have to be at the beginning
+// or end of the title. You can use the special token "|" in a pattern to
+// indicate the beginning or end of the title, so the pattern "|Top * of 2022"
+// would only match "Top movies of 2022" but not "See the top tourist
+// attractions of 2022 in Paris". Patterns also only match entire terms so the
+// pattern "an*how" would not match the term "anyhow" but instead "an <wildcard>
+// how".
+
+// The second rule matches search results where the pattern matches the url of
+// the result. Both rules specify an action that will be applied to the matching
+// results: the first rule downranks these types of listicles whereas the second
+// rule boosts reddit comments. The action can either be `Action(Boost(<int>))`,
+// `Action(Downrank(<int>))` or `Action(Discard)` to completely discard the
+// result. If no action is specified in a rule it is equivalent to
+// `Action(Boost(1))` which will slightly boost a result.
+
+// You can specify any number of patterns in a `Matches` block. A result will
+// only match if it matches all the patterns in the block. As an example, this
+// rule will only discard listicles from "esquire.com" but other results from
+// the same domain will not be discarded.
Rule {
Matches {
Title("Top * of 2022"),
@@ -56,9 +69,12 @@ Rule {
// - Content
// - Schema
-// The most special of these is probably the `Schema` location. This allows you to match results that contains specific https://schema.org entities.
-// This is also the only match-location where you cannot use the special pattern characters "*" and "|". `Schema` only supports simple patterns.
-// The following rule boosts all pages that contains the https://schema.org/DiscussionForumPosting entity
+// The most special of these is probably the `Schema` location. This allows you
+// to match results that contains specific https://schema.org entities. This is
+// also the only match-location where you cannot use the special pattern
+// characters "*" and "|". `Schema` only supports simple patterns. The following
+// rule boosts all pages that contains the
+// https://schema.org/DiscussionForumPosting entity
Rule {
Matches {
Schema("DiscussionForumPosting")
@@ -66,47 +82,63 @@ Rule {
Action(Boost(5))
};
-// By default, any search result that does not match the optic would simply not have any special action applied to it
-// and the search result's score would be unaltered. This behaviour can be changed by specifying `DiscardNonMatching`
+// By default, any search result that does not match the optic would simply not
+// have any special action applied to it and the search result's score would be
+// unaltered. This behaviour can be changed by specifying `DiscardNonMatching`
DiscardNonMatching;
-// Now all search resuts that does not match any of the specified rules will be discarded.
-
-
-// The `Rule` block we have looked at so far only alters search results that matches the specific rules and leaves other
-// results intact. To explain the next part, it will be helpful to know a little bit about how most web search engines
-// rank their results.
-
-// Back in the days when Google launched, it completely disrupted the existing search engines by having a lot of
-// technical advantages. One of the most ground breaking things they did was the introduction of their core ranking
-// algorithm called PageRank. The idea is, that websites that have links from other trustworthy websites must themselves be
-// trustworthy. The idea to analyze website links and use this for ranking gave Google way better search results than their
-// competitors. However, websites quickly realized this and started actively building links to their own website, which skews
-// the PageRank metric. In essense, websites with a high PageRank are no longer guaranteed to be trustworthy but might
-// just be the best websites at link building.
-
-// At Cuely we use a similar link based metric called Harmonic Centrality. It is calculated by taking the sum of the inverse distances
-// from all other websites to a particular website. Imagine site A links to B that then links to C. This gives the following normalized centralities
-// A ---> B ---> C
-// 0 0.33 0.5
-
-// While harmonic centrality is believed to be more resistant to link building than page rank, it is still susceptible to it.
-// Let's now get back from our detour and see what this means for our optics.
-
-// In optics you have the ability to like and dislike sites. We then calculate a harmonic centrality score from all your liked sites,
-// and subtract the harmonic centrality from your disliked sites. What this means is, that sites that has a short distance from your liked sites
-// and a long distance from your disliked sites will be heavily prioritized in your search results.
-// This is the same thing that happens whenever you like a search result directly in Cuely.
+// Now all search resuts that does not match any of the specified rules will be
+// discarded.
+
+
+// The `Rule` block we have looked at so far only alters search results that
+// matches the specific rules and leaves other results intact. To explain the
+// next part, it will be helpful to know a little bit about how most web search
+// engines rank their results.
+
+// Back in the days when Google launched, it completely disrupted the existing
+// search engines by having a lot of technical advantages. One of the most
+// ground breaking things they did was the introduction of their core ranking
+// algorithm called PageRank. The idea is, that websites that have links from
+// other trustworthy websites must themselves be trustworthy. The idea to
+// analyze website links and use this for ranking gave Google way better search
+// results than their competitors. However, websites quickly realized this and
+// started actively building links to their own website, which skews the
+// PageRank metric. In essense, websites with a high PageRank are no longer
+// guaranteed to be trustworthy but might just be the best websites at link
+// building.
+
+// At Cuely we use a similar link based metric called Harmonic Centrality. It is
+// calculated by taking the sum of the inverse distances from all other websites
+// to a particular website. Imagine site A links to B that then links to C. This
+// gives the following normalized centralities
+// A ---> B ---> C
+// 0 0.33 0.5
+
+// While harmonic centrality is believed to be more resistant to link building
+// than PageRank, it is still susceptible to it. Let's now get back from our
+// detour and see what this means for our optics.
+
+// In optics you have the ability to like and dislike sites. We then calculate a
+// harmonic centrality score from all your liked sites, and subtract the
+// harmonic centrality from your disliked sites. What this means is, that sites
+// that has a short distance from your liked sites and a long distance from your
+// disliked sites will be heavily prioritized in your search results. This is
+// the same thing that happens whenever you like a search result directly in
+// Cuely.
Like(Site("news.ycombinator.com"));
Dislike(Site("w3schools.com"));
-// One thing to note here is that only simple patterns can be used inside `Like(Site("..."))` and `Dislike(Site("..."))`.
-// The special tokens "*" and "|" will not work here.
+// One thing to note here is that only simple patterns can be used inside
+// `Like(Site("..."))` and `Dislike(Site("..."))`. The special tokens "*" and
+// "|" will not work here.
-// We have now covered how you can alter the scores of specific search results and indicate which sites you like more than others.
-// While harmonic centrality plays a large role in the final score of a search result, in reality it is actually a weighted
-// sum of a number of signals where harmonic centrality is simply one of them.
-// You can change the coefficient for each of these signals as follows
+// We have now covered how you can alter the scores of specific search results
+// and indicate which sites you like more than others. While harmonic centrality
+// plays a large role in the final score of a search result, in reality it is
+// actually a weighted sum of a number of signals where harmonic centrality is
+// simply one of them. You can change the coefficient for each of these signals
+// as follows
Ranking {
Signal("bm25"),
3
@@ -116,10 +148,11 @@ Ranking {
10000
};
-// Note that different signals might be scaled differently based on the nature of how they are calculated, so a higher coefficient does
-// not necesarrily result in a larger influence on the final score.
-// Currently we support the following signals, but since this list is highly dynamic we encourage you to consult the source code for the
-// most up-to-date list:
+// Note that different signals might be scaled differently based on the nature
+// of how they are calculated, so a higher coefficient does not necesarrily
+// result in a larger influence on the final score. Currently we support the
+// following signals, but since this list is highly dynamic we encourage you to
+// consult the source code for the most up-to-date list:
// - bm25
// - host_centrality
// - page_centrality
@@ -133,13 +166,16 @@ Ranking {
// - query_centrality
-// You can also boost text-matches at different parts of a page
+// You can also boost text-matches at different parts of a page. As an example,
+// this sets the coefficient for pages where the query matches in the title to
+// 10:
Ranking {
Field("title"),
10
}
-// The current list of supported fields is listed here, but again we encourage you to consult the code for the most up-to-date list:
+// The current list of supported fields is listed here, but again we encourage
+// you to consult the code for the most up-to-date list:
// - title
// - body
// - url
@@ -158,6 +194,10 @@ Ranking {
// - dmoz_description
-// Last but not least, when you have developed your optic it can be installed in Cuely by hosting it somewhere
-// that returns a simple plain-text HTTP response and is publicly available. The optic url can then be added
-// to https://cuely.io/optics and you should be able to use it during search.
\ No newline at end of file
+// Last but not least, when you have developed your optic it can be installed in
+// Cuely by uploading the Optic to github, and copy the "raw" url into
+// https://cuely.io/settings. E.g. the url for this Optic is
+// "https://raw.githubusercontent.com/Cuely/sample-optics/main/quickstart.optic"
+
+// Alternatively you can host it anywhere that returns a simple plain-text HTTP
+// response and is publicly available.
\ No newline at end of file