Apache Solr open-source search software
SOLR-14444: Sentences start on newlines (#199)
Cassandra Targett97efeadd02a4
diff --git a/solr/solr-ref-guide/src/about-this-guide.adoc b/solr/solr-ref-guide/src/about-this-guide.adocindex 4f6a71af83a..00d243b7702 100644--- a/solr/solr-ref-guide/src/about-this-guide.adoc+++ b/solr/solr-ref-guide/src/about-this-guide.adoc@@ -20,15 +20,20 @@ This guide describes all of the important features and functions of Apache Solr.Solr is free to download from http://solr.apache.org/.-Designed to provide high-level documentation, this guide is intended to be more encyclopedic and less of a cookbook. It is structured to address a broad spectrum of needs, ranging from new developers getting started to well-experienced developers extending their application or troubleshooting. It will be of use at any point in the application life cycle, for whenever you need authoritative information about Solr.+Designed to provide high-level documentation, this guide is intended to be more encyclopedic and less of a cookbook.+It is structured to address a broad spectrum of needs, ranging from new developers getting started to well-experienced developers extending their application or troubleshooting.+It will be of use at any point in the application life cycle, for whenever you need authoritative information about Solr.-The material as presented assumes that you are familiar with some basic search concepts and that you can read XML. It does not assume that you are a Java programmer, although knowledge of Java is helpful when working directly with Lucene or when developing custom extensions to a Lucene/Solr installation.+The material as presented assumes that you are familiar with some basic search concepts and that you can read XML.+It does not assume that you are a Java programmer, although knowledge of Java is helpful when working directly with Lucene or when developing custom extensions to a Lucene/Solr installation.== Hosts and Port Examples-The default port when running Solr is 8983. The samples, URLs and screenshots in this guide may show different ports, because the port number that Solr uses is configurable.+The default port when running Solr is 8983.+The samples, URLs and screenshots in this guide may show different ports, because the port number that Solr uses is configurable.-If you have not customized your installation of Solr, please make sure that you use port 8983 when following the examples, or configure your own installation to use the port numbers shown in the examples. For information about configuring port numbers, see the section <<monitoring-solr.adoc#,Monitoring Solr>>.+If you have not customized your installation of Solr, please make sure that you use port 8983 when following the examples, or configure your own installation to use the port numbers shown in the examples.+For information about configuring port numbers, see the section <<monitoring-solr.adoc#,Monitoring Solr>>.Similarly, URL examples use `localhost` throughout; if you are accessing Solr from a location remote to the server hosting Solr, replace `localhost` with the proper domain or IP where Solr is running.@@ -36,7 +41,11 @@ For example, we might provide a sample query like:`\http://localhost:8983/solr/gettingstarted/select?q=brown+cow`-There are several items in this URL you might need to change locally. First, if your server is running at "www.example.com", you'll replace "localhost" with the proper domain. If you aren't using port 8983, you'll replace that also. Finally, you'll want to replace "gettingstarted" (the collection or core name) with the proper one in use in your implementation. The URL would then become:+There are several items in this URL you might need to change locally.+First, if your server is running at "www.example.com", you'll replace "localhost" with the proper domain.+If you aren't using port 8983, you'll replace that also.+Finally, you'll want to replace "gettingstarted" (the collection or core name) with the proper one in use in your implementation.+The URL would then become:`\http://www.example.com/solr/mycollection/select?q=brown+cow`@@ -44,27 +53,34 @@ There are several items in this URL you might need to change locally. First, ifPath information is given relative to `solr.home`, which is the location under the main Solr installation where Solr's collections and their `conf` and `data` directories are stored.-In many cases, this is is in the `server/solr` directory of your installation. However, there can be exceptions, particularly if your installation has customized this.+In many cases, this is is in the `server/solr` directory of your installation.+However, there can be exceptions, particularly if your installation has customized this.-In several cases of this Guide, our examples are built from the the "techproducts" example (i.e., you have started Solr with the command `bin/solr -e techproducts`). In this case, `solr.home` will be a sub-directory of the `example/` directory created for you automatically.+In several cases of this Guide, our examples are built from the the "techproducts" example (i.e., you have started Solr with the command `bin/solr -e techproducts`).+In this case, `solr.home` will be a sub-directory of the `example/` directory created for you automatically.See also the section <<configuration-files.adoc#solr-home,Solr Home>> for further details on what is contained in this directory.== API Examples-Solr has two styles of APIs that currently co-exist. The first has grown somewhat organically as Solr has developed over time, but the second, referred to as the "V2 API", redesigns many of the original APIs with a modernized and self-documenting API interface.+Solr has two styles of APIs that currently co-exist.+The first has grown somewhat organically as Solr has developed over time, but the second, referred to as the "V2 API", redesigns many of the original APIs with a modernized and self-documenting API interface.-In many cases, but not all, the parameters and outputs of API calls are the same between the two styles. In all cases the paths and endpoints used are different.+In many cases, but not all, the parameters and outputs of API calls are the same between the two styles.+In all cases the paths and endpoints used are different.Throughout this Guide, we have added examples of both styles with sections labeled "V1 API" and "V2 API". As of the 7.2 version of this Guide, these examples are not yet complete - more coverage will be added as future versions of the Guide are released.The section <<v2-api.adoc#,V2 API>> provides more information about how to work with the new API structure, including how to disable it if you choose to do so.-All APIs return a response header that includes the status of the request and the time to process it. Some APIs will also include the parameters used for the request. Many of the examples in this Guide omit this header information, which you can do locally by adding the parameter `omitHeader=true` to any request.+All APIs return a response header that includes the status of the request and the time to process it.+Some APIs will also include the parameters used for the request.+Many of the examples in this Guide omit this header information, which you can do locally by adding the parameter `omitHeader=true` to any request.== Special Inline Notes-Special notes are included throughout these pages. There are several types of notes:+Special notes are included throughout these pages.+There are several types of notes:NOTE: Information blocks provide additional information that's useful for you to know.diff --git a/solr/solr-ref-guide/src/aliases.adoc b/solr/solr-ref-guide/src/aliases.adocindex e30d712a866..7a2c5128b41 100644--- a/solr/solr-ref-guide/src/aliases.adoc+++ b/solr/solr-ref-guide/src/aliases.adoc@@ -149,8 +149,8 @@ First you create a category routed alias using the <<alias-management.adoc#creatMost of the settings are editable at a later time using the <<alias-management.adoc#aliasprop,ALIASPROP>> command.The alias will be created with a special place-holder collection which will always be named `myAlias\__CRA__NEW_CATEGORY_ROUTED_ALIAS_WAITING_FOR_DATA\__TEMP`.-The first document indexed into the CRA will create a second collection named `myAlias__CRA__foo` (for a routed field value of `foo`). The second document- indexed will cause the temporary place holder collection to be deleted.+The first document indexed into the CRA will create a second collection named `myAlias__CRA__foo` (for a routed field value of `foo`).+The second document indexed will cause the temporary place holder collection to be deleted.Thereafter collections will be created whenever a new value for the field is encountered.CAUTION: To guard against runaway collection creation options for limiting the total number of categories, and for rejecting values that don't match, a regular expression parameter is provided (see <<alias-management.adoc#category-routed-alias-parameters,Category Routed Alias Parameters>> for details).diff --git a/solr/solr-ref-guide/src/analytics-expression-sources.adoc b/solr/solr-ref-guide/src/analytics-expression-sources.adocindex 4096afa1704..8e823446652 100644--- a/solr/solr-ref-guide/src/analytics-expression-sources.adoc+++ b/solr/solr-ref-guide/src/analytics-expression-sources.adoc@@ -60,7 +60,8 @@ In order to save duplicates, you must use PointField types.== Constants-Constants can be included in expressions to use along side fields and functions. The available constants are shown below.+Constants can be included in expressions to use along side fields and functions.+The available constants are shown below.Constants do not need to be surrounded by any function to define them, they can be used exactly like fields in an expression.=== Strings@@ -76,14 +77,16 @@ There are two possible ways of specifying constant strings, as shown below.=== Dates-Dates can be specified in the same way as they are in Solr queries. Just use ISO-8601 format.+Dates can be specified in the same way as they are in Solr queries.+Just use ISO-8601 format.For more information, refer to the <<date-formatting-math.adoc#,Working with Dates>> section.* `2017-07-17T19:35:08Z`=== Numeric-Any non-decimal number will be read as an integer, or as a long if it is too large for an integer. All decimal numbers will be read as doubles.+Any non-decimal number will be read as an integer, or as a long if it is too large for an integer.+All decimal numbers will be read as doubles.* `-123421`: Integer* `800000000000`: Longdiff --git a/solr/solr-ref-guide/src/analytics-mapping-functions.adoc b/solr/solr-ref-guide/src/analytics-mapping-functions.adocindex debd9a5a515..ab4367c1276 100644--- a/solr/solr-ref-guide/src/analytics-mapping-functions.adoc+++ b/solr/solr-ref-guide/src/analytics-mapping-functions.adoc@@ -181,7 +181,8 @@ Checks whether any value(s) exist for the expression.== Comparison=== Equality-Checks whether two expressions' values are equal. The parameters must be the same type, after implicit casting.+Checks whether two expressions' values are equal.+The parameters must be the same type, after implicit casting.`equal(< _Single_ T >, < _Single_ T >)` => `< _Single Bool_ >`::* `equal(F, F)` => `T`@@ -316,7 +317,8 @@ Explicitly converts the values of a `String` or `Long` expression into `Dates`.[[analytics-date-math]]=== Date Math-Compute the given date math strings for the values of a `Date` expression. The date math strings *must* be <<analytics-expression-sources.adoc#strings, constant>>.+Compute the given date math strings for the values of a `Date` expression.+The date math strings *must* be <<analytics-expression-sources.adoc#strings, constant>>.`date_math(< _Date_ >, < _Constant String_ >...)` => `< _Date_ >`::* `date_math(1800-04-15, '+1DAY', '-1MONTH')` => `1800-03-16`diff --git a/solr/solr-ref-guide/src/analytics-reduction-functions.adoc b/solr/solr-ref-guide/src/analytics-reduction-functions.adocindex 83c4093a985..b6f57d8093c 100644--- a/solr/solr-ref-guide/src/analytics-reduction-functions.adoc+++ b/solr/solr-ref-guide/src/analytics-reduction-functions.adoc@@ -27,7 +27,8 @@ These can be combined using mapping functions to implement more complex function== Counting Reductions=== Count-The number of existing values for an expression. For single-valued expressions, this is equivalent to `docCount`.+The number of existing values for an expression.+For single-valued expressions, this is equivalent to `docCount`.If no expression is given, the number of matching documents is returned.`count()` => `< _Single Long_ >`@@ -35,7 +36,8 @@ If no expression is given, the number of matching documents is returned.`count(< T >)` => `< _Single Long_ >`=== Doc Count-The number of documents for which an expression has existing values. For single-valued expressions, this is equivalent to `count`.+The number of documents for which an expression has existing values.+For single-valued expressions, this is equivalent to `count`.If no expression is given, the number of matching documents is returned.`doc_count()` => `< _Single Long_ >`@@ -49,7 +51,8 @@ The number of documents for which an expression has no existing value.[[analytics-unique]]=== Unique-The number of unique values for an expression. This function accepts `Numeric`, `Date` and `String` expressions.+The number of unique values for an expression.+This function accepts `Numeric`, `Date` and `String` expressions.`unique(< T >)` => `< _Single Long_ >`@@ -85,17 +88,20 @@ NOTE: The expressions must satisfy the rules for `mult` function parameters.== Ordering Reductions=== Minimum-Returns the minimum value for the expression. This function accepts `Numeric`, `Date` and `String` expressions.+Returns the minimum value for the expression.+This function accepts `Numeric`, `Date` and `String` expressions.`min(< T >)` => `< _Single_ T >`=== Maximum-Returns the maximum value for the expression. This function accepts `Numeric`, `Date` and `String` expressions.+Returns the maximum value for the expression.+This function accepts `Numeric`, `Date` and `String` expressions.`max(< T >)` => `< _Single_ T >`=== Median-Returns the median of all values for the expression. This function accepts `Numeric` and `Date` expressions.+Returns the median of all values for the expression.+This function accepts `Numeric` and `Date` expressions.`median(< T >)` => `< _Single_ T >`diff --git a/solr/solr-ref-guide/src/analytics.adoc b/solr/solr-ref-guide/src/analytics.adocindex 6bcbe9602d4..bfae49df68f 100644--- a/solr/solr-ref-guide/src/analytics.adoc+++ b/solr/solr-ref-guide/src/analytics.adoc@@ -32,9 +32,12 @@ Since the Analytics framework is a _search component_, it must be declared as suFor distributed analytics requests over cloud collections, the component uses the `AnalyticsHandler` strictly for inter-shard communication.The Analytics Handler should not be used by users to submit analytics requests.-To use the Analytics Component, the first step is to install this contrib module's plugins into Solr -- see the <<solr-plugins.adoc#installing-plugins,Solr Plugins>> section on how to do this. Note: Method with `<lib/>` directive doesn't work. Instead copy `${solr.install.dir}/dist/solr-analytics-x.x.x.jar` to `${solr.install.dir}/server/solr-webapp/webapp/WEB-INF/lib/`, as described in the <<libs.adoc#lib-directories,lib directories documentation>>.+To use the Analytics Component, the first step is to install this contrib module's plugins into Solr -- see the <<solr-plugins.adoc#installing-plugins,Solr Plugins>> section on how to do this.+Note: Method with `<lib/>` directive doesn't work.+Instead copy `${solr.install.dir}/dist/solr-analytics-x.x.x.jar` to `${solr.install.dir}/server/solr-webapp/webapp/WEB-INF/lib/`, as described in the <<libs.adoc#lib-directories,lib directories documentation>>.-Next you need to register the request handler and search component. Add the following lines to `solrconfig.xml`, near the defintions for other request handlers:+Next you need to register the request handler and search component.+Add the following lines to `solrconfig.xml`, near the defintions for other request handlers:[source,xml].solrconfig.xml@@ -75,11 +78,14 @@ curl --data-binary 'analytics={There are 3 main parts of any analytics request:Expressions::-A list of calculations to perform over the entire result set. Expressions aggregate the search results into a single value to return.-This list is entirely independent of the expressions defined in each of the groupings. Find out more about them in the section <<Expressions>>.+A list of calculations to perform over the entire result set.+Expressions aggregate the search results into a single value to return.+This list is entirely independent of the expressions defined in each of the groupings.+Find out more about them in the section <<Expressions>>.Functions::-One or more <<variable-functions, Variable Functions>> to be used throughout the rest of the request. These are essentially lambda functions and can be combined in a number of ways.+One or more <<variable-functions, Variable Functions>> to be used throughout the rest of the request.+These are essentially lambda functions and can be combined in a number of ways.These functions for the expressions defined in `expressions` as well as `groupings`.Groupings::@@ -144,13 +150,15 @@ The `functions` parameter is always optional.== Expressions-Expressions are the way to request pieces of information from the analytics component. These are the statistical expressions that you want computed and returned in your response.+Expressions are the way to request pieces of information from the analytics component.+These are the statistical expressions that you want computed and returned in your response.=== Constructing an Expression==== Expression Components-An expression is built using fields, constants, mapping functions and reduction functions. The ways that these can be defined are described below.+An expression is built using fields, constants, mapping functions and reduction functions.+The ways that these can be defined are described below.Sources::* Constants: The values defined in the expression.@@ -203,7 +211,8 @@ With the above definitions and ordering, an example expression can be broken up[source,bash]div(sum(a,fill_missing(b,0)),add(10.5,count(mult(a,c)))))-As a whole, this is a reduced mapping function. The `div` function is a reduced mapping function since it is a <<analytics-mapping-functions.adoc#division,provided mapping function>> and has reduced arguments.+As a whole, this is a reduced mapping function.+The `div` function is a reduced mapping function since it is a <<analytics-mapping-functions.adoc#division,provided mapping function>> and has reduced arguments.If we break down the expression further:@@ -227,10 +236,12 @@ If we break down the expression further:=== Expression Cardinality (Multi-Valued and Single-Valued)-The root of all multi-valued expressions are multi-valued fields. Single-valued expressions can be started with constants or single-valued fields.+The root of all multi-valued expressions are multi-valued fields.+Single-valued expressions can be started with constants or single-valued fields.All single-valued expressions can be treated as multi-valued expressions that contain one value.-Single-valued expressions and multi-valued expressions can be used together in many mapping functions, as well as multi-valued expressions being used alone, and many single-valued expressions being used together. For example:+Single-valued expressions and multi-valued expressions can be used together in many mapping functions, as well as multi-valued expressions being used alone, and many single-valued expressions being used together.+For example:`add(<single-valued double>, <single-valued double>, ...)`::Returns a single-valued double expression where the value of the values of each expression are added.@@ -265,7 +276,9 @@ An implicit cast means that if a function requires a certain type of value as aFor example, `concat()` only accepts string parameters and since all types can be implicitly cast to strings, any type is accepted as an argument.-This also goes for dynamically typed functions. `fill_missing()` requires two arguments of the same type. However, two types that implicitly cast to the same type can also be used.+This also goes for dynamically typed functions.+`fill_missing()` requires two arguments of the same type.+However, two types that implicitly cast to the same type can also be used.For example, `fill_missing(<long>,<float>)` will be cast to `fill_missing(<double>,<double>)` since long cannot be cast to float and float cannot be cast to long implicitly.@@ -285,7 +298,8 @@ However `round()`, `floor()` and `cell()` can return either int or long, dependi== Variable Functions-Variable functions are a way to shorten your expressions and make writing analytics queries easier. They are essentially lambda functions defined in a request.+Variable functions are a way to shorten your expressions and make writing analytics queries easier.+They are essentially lambda functions defined in a request.[source,json].Example Basic Function@@ -301,7 +315,8 @@ Variable functions are a way to shorten your expressions and make writing analyt}-----In the above request, instead of writing `mult(price,quantity)` twice, a function `sale()` was defined to abstract this idea. Then that function was used in the multiple expressions.+In the above request, instead of writing `mult(price,quantity)` twice, a function `sale()` was defined to abstract this idea.+Then that function was used in the multiple expressions.Suppose that we want to look at the sales of specific categories:@@ -486,7 +501,8 @@ The list of criteria to sort the facet by.+It takes the following parameters:-`type`::: The type of sort. There are two possible values:+`type`::: The type of sort.+There are two possible values:* `expression`: Sort by the value of an expression defined in the same grouping.* `facetvalue`: Sort by the string-representation of the facet value.@@ -594,11 +610,14 @@ The second pivot given will be treated like one value facet for each value of thEach of these second-level value facets will be limited to the documents in their first-level facet bucket.This continues for however many pivots are provided.-Sorting is enabled on a per-pivot basis. This means that if your top pivot has a sort with `limit:1`, then only that first value of the facet will be drilled down into. Sorting in each pivot is independent of the other pivots.+Sorting is enabled on a per-pivot basis.+This means that if your top pivot has a sort with `limit:1`, then only that first value of the facet will be drilled down into.+Sorting in each pivot is independent of the other pivots.==== Parameters-`pivots`:: The list of pivots to calculate a drill-down facet for. The list is ordered by top-most to bottom-most level.+`pivots`:: The list of pivots to calculate a drill-down facet for.+The list is ordered by top-most to bottom-most level.`name`::: The name of the pivot.`expression`::: The expression to choose a facet bucket for each document.`sort`::: A <<Facet Sorting,sort>> for the results of the pivot.@@ -684,16 +703,20 @@ Refer to the <<faceting.adoc#range-faceting,Range Facet documentation>> for addi`field`:: Field to be faceted over`start`:: The bottom end of the range`end`:: The top end of the range-`gap`:: A list of range gaps to generate facet buckets. If the buckets do not add up to fit the `start` to `end` range,+`gap`:: A list of range gaps to generate facet buckets.+If the buckets do not add up to fit the `start` to `end` range,then the last `gap` value will repeated as many times as needed to fill any unused range.-`hardend`:: Whether to cutoff the last facet bucket range at the `end` value if it spills over. Defaults to `false`.-`include`:: The boundaries to include in the facet buckets. Defaults to `lower`.+`hardend`:: Whether to cutoff the last facet bucket range at the `end` value if it spills over.+Defaults to `false`.+`include`:: The boundaries to include in the facet buckets.+Defaults to `lower`.* `lower` - All gap-based ranges include their lower bound.* `upper` - All gap-based ranges include their upper bound.* `edge` - The first and last gap ranges include their edge bounds (lower for the first one, upper for the last one) even if the corresponding upper/lower option is not specified.* `outer` - The `before` and `after` ranges will be inclusive of their bounds, even if the first or last ranges already include those boundaries.* `all` - Includes all options: `lower`, `upper`, `edge`, and `outer`-`others`:: Additional ranges to include in the facet. Defaults to `none`.+`others`:: Additional ranges to include in the facet.+Defaults to `none`.* `before` - All records with field values lower then lower bound of the first range.* `after` - All records with field values greater then the upper bound of the last range.* `between` - All records with field values between the lower bound of the first range and the upper bound of the last range.diff --git a/solr/solr-ref-guide/src/analyzers.adoc b/solr/solr-ref-guide/src/analyzers.adocindex ac5b26e34da..535926a8dfe 100644--- a/solr/solr-ref-guide/src/analyzers.adoc+++ b/solr/solr-ref-guide/src/analyzers.adoc@@ -58,7 +58,8 @@ For example:</analyzer></fieldType>-----Tokenizer and filter factory classes are referred by their symbolic names (SPI names). Here, name="standard" refers `org.apache.lucene.analysis.standard.StandardTokenizerFactory`.+Tokenizer and filter factory classes are referred by their symbolic names (SPI names).+Here, name="standard" refers `org.apache.lucene.analysis.standard.StandardTokenizerFactory`.====[example.tab-pane#byclass]====diff --git a/solr/solr-ref-guide/src/audit-logging.adoc b/solr/solr-ref-guide/src/audit-logging.adocindex 86b3baf1404..44efcd7f4c5 100644--- a/solr/solr-ref-guide/src/audit-logging.adoc+++ b/solr/solr-ref-guide/src/audit-logging.adoc@@ -19,8 +19,8 @@Solr has the ability to log an audit trail of all HTTP requests entering the system.Audit loggers are pluggable to suit any possible format or log destination.-[quote]-An audit trail (also called audit log) is a security-relevant chronological record, set of records, and/or destination and source of records that provide documentary evidence of the sequence of activities that have affected at any time a specific operation, procedure, event, or device. (https://en.wikipedia.org/wiki/Audit_trail[Wikipedia])+[quote, Wikipedia, https://en.wikipedia.org/wiki/Audit_trail]+An audit trail (also called audit log) is a security-relevant chronological record, set of records, and/or destination and source of records that provide documentary evidence of the sequence of activities that have affected at any time a specific operation, procedure, event, or device.== Configuring Audit LoggingAudit logging is configured in `security.json` under the `auditlogging` key.@@ -239,7 +239,7 @@ The individual metrics are:* `lost`: (_meter_) Records number and rate of events lost if the queue is full and `blockAsync=false`.* `requestTimes`: (_timer_) Records latency and percentiles for audit logging performance.* `totalTime`: (_counter_) Records total time spent logging.-* `queueCapacity`: (_gauge_). Records the maximum size of the async logging queue.+* `queueCapacity`: (_gauge_) Records the maximum size of the async logging queue.* `queueSize`: (_gauge_) Records the number of events currently waiting in the queue.* `queuedTime`: (_timer_) Records the amount of time events waited in queue.Adding this with the `requestTimes` metric will show the total time from event to logging complete.diff --git a/solr/solr-ref-guide/src/authentication-and-authorization-plugins.adoc b/solr/solr-ref-guide/src/authentication-and-authorization-plugins.adocindex 62a38ac30f1..8cd2002b4dc 100644--- a/solr/solr-ref-guide/src/authentication-and-authorization-plugins.adoc+++ b/solr/solr-ref-guide/src/authentication-and-authorization-plugins.adoc@@ -188,7 +188,8 @@ include::securing-solr.adoc[tag=list-of-authorization-plugins]Whenever an authentication plugin is enabled, authentication is also required for all or some operations in the Admin UI.The Admin UI is an AngularJS application running inside your browser, and is treated as any other external client by Solr.-When authentication is required the Admin UI will presented you with a login dialogue. The authentication plugins currently supported by the Admin UI are:+When authentication is required the Admin UI will presented you with a login dialogue.+The authentication plugins currently supported by the Admin UI are:* <<basic-authentication-plugin.adoc#,Basic Authentication Plugin>>* <<jwt-authentication-plugin.adoc#,JWT Authentication Plugin>>diff --git a/solr/solr-ref-guide/src/backup-restore.adoc b/solr/solr-ref-guide/src/backup-restore.adocindex 5c05cfddb04..3a25bf67c5d 100644--- a/solr/solr-ref-guide/src/backup-restore.adoc+++ b/solr/solr-ref-guide/src/backup-restore.adoc@@ -201,7 +201,8 @@ If no repository is specified then the local filesystem repository will be usedThe `restore` command is an asynchronous call.Once the restore is complete the data reflected will be of the backed up index which was restored.-Only one `restore` call can can be made against a core at one point in time. While an ongoing restore operation is happening subsequent calls for restoring will throw an exception.+Only one `restore` call can can be made against a core at one point in time.+While an ongoing restore operation is happening subsequent calls for restoring will throw an exception.=== Restore Status API@@ -376,7 +377,8 @@ A LocalFileSystemRepository instance is used as a default by any backup and restLocalFileSystemRepository accepts the following configuration options:`location`::-A valid file path (accessible to Solr locally) to use for backup storage and retrieval. Used as a fallback when user's don't provide a `location` parameter in their Backup or Restore API commands+A valid file path (accessible to Solr locally) to use for backup storage and retrieval.+Used as a fallback when user's don't provide a `location` parameter in their Backup or Restore API commandsAn example configuration using this property can be found below.@@ -411,7 +413,8 @@ A HDFS URI in the format `hdfs://<host>:<port>/<hdfsBaseFilePath>` that points SA permission umask used when creating files in HDFS.`location`::-A valid directory path on the HDFS cluster to use for backup storage and retrieval. Used as a fallback when users don't provide a `location` parameter in their Backup or Restore API commands+A valid directory path on the HDFS cluster to use for backup storage and retrieval.+Used as a fallback when users don't provide a `location` parameter in their Backup or Restore API commands.An example configuration using these properties can be found below:diff --git a/solr/solr-ref-guide/src/block-join-query-parser.adoc b/solr/solr-ref-guide/src/block-join-query-parser.adocindex 47ab4bf0ea5..5f3055d1b3b 100644--- a/solr/solr-ref-guide/src/block-join-query-parser.adoc+++ b/solr/solr-ref-guide/src/block-join-query-parser.adoc@@ -166,7 +166,8 @@ Overall the idea is similar to <<faceting.adoc#tagging-and-excluding-filters, ex=== Scoring with the Block Join Parent Query ParserYou can optionally use the `score` local parameter to return scores of the subordinate query.-The values to use for this parameter define the type of aggregation, which are `avg` (average), `max` (maximum), `min` (minimum), `total (sum)`. Implicit default is `none` which returns `0.0`.+The values to use for this parameter define the type of aggregation, which are `avg` (average), `max` (maximum), `min` (minimum), `total (sum)`.+Implicit default is `none` which returns `0.0`.=== All Parents Syntaxdiff --git a/solr/solr-ref-guide/src/circuit-breakers.adoc b/solr/solr-ref-guide/src/circuit-breakers.adocindex b28bda68631..83fa418cdee 100644--- a/solr/solr-ref-guide/src/circuit-breakers.adoc+++ b/solr/solr-ref-guide/src/circuit-breakers.adoc@@ -16,18 +16,18 @@// specific language governing permissions and limitations// under the License.-Solr's circuit breaker infrastructure allows prevention of actions that can cause a node to go beyond its capacity or to go down. The-premise of circuit breakers is to ensure a higher quality of service and only accept request loads that are serviceable in the current+Solr's circuit breaker infrastructure allows prevention of actions that can cause a node to go beyond its capacity or to go down.+The premise of circuit breakers is to ensure a higher quality of service and only accept request loads that are serviceable in the currentresource configuration.== When To Use Circuit Breakers-Circuit breakers should be used when the user wishes to trade request throughput for a higher Solr stability. If circuit breakers-are enabled, requests may be rejected under the condition of high node duress with an appropriate HTTP error code (typically 503).+Circuit breakers should be used when the user wishes to trade request throughput for a higher Solr stability.+If circuit breakers are enabled, requests may be rejected under the condition of high node duress with an appropriate HTTP error code (typically 503).It is up to the client to handle this error and potentially build a retrial logic as this should ideally be a transient situation.== Circuit Breaker Configurations-All circuit breaker configurations are listed in the circuitBreaker tags in solrconfig.xml as shown below:+All circuit breaker configurations are listed in the `<circuitBreaker>` tags in `solrconfig.xml` as shown below:[source,xml]----@@ -36,21 +36,21 @@ All circuit breaker configurations are listed in the circuitBreaker tags in solr</circuitBreaker>-----The "enabled" attribute controls the global activation/deactivation of circuit breakers. If this flag is disabled, all circuit breakers-will be disabled globally. Per circuit breaker configurations are specified in their respective sections later.+The `enabled` attribute controls the global activation/deactivation of circuit breakers.+If this flag is disabled, all circuit breakers will be disabled globally.+Per circuit breaker configurations are specified in their respective sections later.-This attribute acts as the highest authority and global controller of circuit breakers. For using specific circuit breakers, each one-needs to be individually enabled in addition to this flag being enabled.+This attribute acts as the highest authority and global controller of circuit breakers.+For using specific circuit breakers, each one needs to be individually enabled in addition to this flag being enabled.-CircuitBreakerManager is the default manager for all circuit breakers that should be defined in the tag unless the user wishes to use-a custom implementation.+`CircuitBreakerManager` is the default manager for all circuit breakers that should be defined in the tag unless the user wishes to use a custom implementation.== Currently Supported Circuit Breakers-=== JVM Heap Usage Based Circuit Breaker-This circuit breaker tracks JVM heap memory usage and rejects incoming search requests with a 503 error code if the heap usage-exceeds a configured percentage of maximum heap allocated to the JVM (-Xmx). The main configuration for this circuit breaker is-controlling the threshold percentage at which the breaker will trip.+=== JVM Heap Usage++This circuit breaker tracks JVM heap memory usage and rejects incoming search requests with a 503 error code if the heap usage exceeds a configured percentage of maximum heap allocated to the JVM (-Xmx).+The main configuration for this circuit breaker is controlling the threshold percentage at which the breaker will trip.Configuration for JVM heap usage based circuit breaker:@@ -59,30 +59,30 @@ Configuration for JVM heap usage based circuit breaker:<str name="memEnabled">true</str>-----Note that this configuration will be overridden by the global circuit breaker flag -- if circuit breakers are disabled, this flag-will not help you.+Note that this configuration will be overridden by the global circuit breaker flag -- if circuit breakers are disabled, this flag will not help you.-The triggering threshold is defined as a percentage of the max heap allocated to the JVM. It is controlled by the below configuration:+The triggering threshold is defined as a percentage of the max heap allocated to the JVM.+It is controlled by the below configuration:[source,xml]----<str name="memThreshold">75</str>-----It does not logically make sense to have a threshold below 50% and above 95% of the max heap allocated to the JVM. Hence, the range-of valid values for this parameter is [50, 95], both inclusive.+It does not logically make sense to have a threshold below 50% and above 95% of the max heap allocated to the JVM.+Hence, the range of valid values for this parameter is [50, 95], both inclusive.Consider the following example:-JVM has been allocated a maximum heap of 5GB (-Xmx) and memoryCircuitBreakerThresholdPct is set to 75. In this scenario, the heap usage-at which the circuit breaker will trip is 3.75GB.+JVM has been allocated a maximum heap of 5GB (-Xmx) and `memoryCircuitBreakerThresholdPct` is set to `75`.+In this scenario, the heap usage at which the circuit breaker will trip is 3.75GB.++=== CPU Utilization-=== CPU Utilization Based Circuit Breaker-This circuit breaker tracks CPU utilization and triggers if the average CPU utilization over the last one minute-exceeds a configurable threshold. Note that the value used in computation is over the last one minute -- so a sudden-spike in traffic that goes down might still cause the circuit breaker to trigger for a short while before it resolves-and updates the value. For more details of the calculation, please see https://en.wikipedia.org/wiki/Load_(computing)+This circuit breaker tracks CPU utilization and triggers if the average CPU utilization over the last one minute exceeds a configurable threshold.+Note that the value used in computation is over the last one minute -- so a sudden spike in traffic that goes down might still cause the circuit breaker to trigger for a short while before it resolves and updates the value.+For more details of the calculation, please see https://en.wikipedia.org/wiki/Load_(computing)Configuration for CPU utilization based circuit breaker:@@ -91,10 +91,10 @@ Configuration for CPU utilization based circuit breaker:<str name="cpuEnabled">true</str>-----Note that this configuration will be overridden by the global circuit breaker flag -- if circuit breakers are disabled, this flag-will not help you.+Note that this configuration will be overridden by the global circuit breaker flag -- if circuit breakers are disabled, this flag will not help you.-The triggering threshold is defined in units of CPU utilization. The configuration to control this is as below:+The triggering threshold is defined in units of CPU utilization.+The configuration to control this is as below:[source,xml]----@@ -102,8 +102,7 @@ The triggering threshold is defined in units of CPU utilization. The configurati----== Performance Considerations-It is worth noting that while JVM or CPU circuit breakers do not add any noticeable overhead per query, having too many-circuit breakers checked for a single request can cause a performance overhead.-In addition, it is a good practice to exponentially back off while retrying requests on a busy node.+It is worth noting that while JVM or CPU circuit breakers do not add any noticeable overhead per query, having too many circuit breakers checked for a single request can cause a performance overhead.+In addition, it is a good practice to exponentially back off while retrying requests on a busy node.diff --git a/solr/solr-ref-guide/src/cluster-node-management.adoc b/solr/solr-ref-guide/src/cluster-node-management.adocindex 2f37b5ac57b..3e69179bc6d 100644--- a/solr/solr-ref-guide/src/cluster-node-management.adoc+++ b/solr/solr-ref-guide/src/cluster-node-management.adoc@@ -314,7 +314,8 @@ Support for the "collectionDefaults" key will be removed in Solr 9.=== Default Shard PreferencesUsing the `defaultShardPreferences` parameter, you can implement rack or availability zone awareness.-First, make sure to "label" your nodes using a <<property-substitution.adoc#jvm-system-properties,system property>> (e.g., `-Drack=rack1`). Then, set the value of `defaultShardPreferences` to `node.sysprop:sysprop.YOUR_PROPERTY_NAME` like this:+First, make sure to "label" your nodes using a <<property-substitution.adoc#jvm-system-properties,system property>> (e.g., `-Drack=rack1`).+Then, set the value of `defaultShardPreferences` to `node.sysprop:sysprop.YOUR_PROPERTY_NAME` like this:[source,bash]----@@ -422,7 +423,8 @@ Examining the clusterstate after issuing this call should show exactly one repli[[replacenode]]== REPLACENODE: Move All Replicas in a Node to Another-This command recreates replicas in one node (the source) on another node(s) (the target). After each replica is copied, the replicas in the source node are deleted.+This command recreates replicas in one node (the source) on another node(s) (the target).+After each replica is copied, the replicas in the source node are deleted.For source replicas that are also shard leaders the operation will wait for the number of seconds set with the `timeout` parameter to make sure there's an active replica that can become a leader, either an existing replica becoming a leader or the new replica completing recovery and becoming a leader).diff --git a/solr/solr-ref-guide/src/cluster-plugins.adoc b/solr/solr-ref-guide/src/cluster-plugins.adocindex 8649ad63e1c..ad43ebaf5a4 100644--- a/solr/solr-ref-guide/src/cluster-plugins.adoc+++ b/solr/solr-ref-guide/src/cluster-plugins.adoc@@ -19,46 +19,43 @@// under the License.== Cluster (CoreContainer-level) Plugins Subsystem-Cluster plugins are pluggable components that are defined and instantiated at the-`CoreContainer` (node) level. These components usually provide admin-level functionality-and APIs for additional functionality at the Solr node level.+Cluster plugins are pluggable components that are defined and instantiated at the `CoreContainer` (node) level.+These components usually provide admin-level functionality and APIs for additional functionality at the Solr node level.=== Plugin ConfigurationsPlugin configurations are maintained using `/cluster/plugin` API.This API endpoint allows adding, removing and updating plugin configurations.-Each plugin MUST have a unique name under which it's registered. Attempting to-add a plugin with a duplicate name is an error. Some types of plugins use-pre-defined names, and they MUST be registered under these names in order to-properly function.+Each plugin MUST have a unique name under which it's registered.+Attempting to add a plugin with a duplicate name is an error.+Some types of plugins use pre-defined names, and they MUST be registered under these names in order to properly function.-Internally, as of Solr 9.0 plugin configurations are maintained in ZooKeeper in the-`/clusterprops.json` file, under the `plugin` entry. The configuration is a JSON map-where keys are the unique plugin names, and values are serialized-`org.apache.solr.client.solrj.request.beans.PluginMeta` beans.+Internally, as of Solr 9.0 plugin configurations are maintained in ZooKeeper in the `/clusterprops.json` file, under the `plugin` entry.+The configuration is a JSON map where keys are the unique plugin names, and values are serialized `org.apache.solr.client.solrj.request.beans.PluginMeta` beans.The following common plugin properties are supported:`name`::-(required) unique plugin name. Some plugin types require using one of the-pre-defined names to properly function. By convention such predefined names use-a leading-dot prefix (e.g., `.placement-plugin`)+(required) unique plugin name.+Some plugin types require using one of the pre-defined names to properly function.+By convention such predefined names use a leading-dot prefix (e.g., `.placement-plugin`)`class`::-(required) implementation class. This can be specified as a fully-qualified-class name if the class is available as a part of Solr, or it can be also-specified using the `<package>:<className>` syntax to refer to a class inside-one of the Solr packages.+(required) implementation class.+This can be specified as a fully-qualified class name if the class is available as a part of Solr, or it can be also specified using the `<package>:<className>` syntax to refer to a class inside one of the Solr packages.`version`::-(required when class is loaded from a package). Solr package version.+(required when class is loaded from a package).+Solr package version.`path-prefix`::-(optional, default is none). Path prefix to be added to the REST API endpoints defined in the plugin.+(optional, default is `none`).+Path prefix to be added to the REST API endpoints defined in the plugin.`config`::-(optional, default is none). A JSON map of additional plugin configuration parameters.+(optional, default is `none`).+A JSON map of additional plugin configuration parameters.Plugins that implement `ConfigurablePlugin` interface will be initialized with aplugin-specific configuration object deserialized from this map.@@ -85,42 +82,35 @@ curl -X POST -H 'Content-type: application/json' -d '{=== Types of Cluster PluginsClasses loaded from plugins in general support two types of functionality (not mutually exclusive):-* request handler plugins that expose REST API endpoints (the implementing class is annotated with-`@EndPoint` and optionally `@Command` annotations). The APIs of these plugins are automatically-registered as REST endpoints under the paths defined in the `@EndPoint` annotations.+* request handler plugins that expose REST API endpoints (the implementing class is annotated with `@EndPoint` and optionally `@Command` annotations).+The APIs of these plugins are automatically registered as REST endpoints under the paths defined in the `@EndPoint` annotations.-* plugins that implement a specific interface, for use as an internal component. Upon loading they are-automatically discovered and registered with sub-systems that use this type of plugin. Examples here-include the `ClusterSingleton`, ClusterEventProducer`, `ClusterEventListener`-and `PlacementPluginFactory`.+* plugins that implement a specific interface, for use as an internal component.+Upon loading they are automatically discovered and registered with sub-systems that use this type of plugin.+Examples here include the `ClusterSingleton`, `ClusterEventProducer`, `ClusterEventListener` and `PlacementPluginFactory`.=== Plugin Lifecycle-Plugin instances are loaded and initialized when Solr's `CoreContainer` is first created during-Solr node start-up.+Plugin instances are loaded and initialized when Solr's `CoreContainer` is first created during Solr node start-up.-Then on each update of the configurations each node is notified about the change,-and then the existing plugins are compared with the new configs, and plugin instances-present on the node are respectively created, removed, or-replaced (i.e., removed and added using the new configuration).+Then on each update of the configurations each node is notified about the change, and then the existing plugins are compared with the new configs, and plugin instances present on the node are respectively created, removed, or replaced (i.e., removed and added using the new configuration).In practice this means that cluster-level plugins managed by this API can be-dynamically changed and reconfigured without restarting the Solr nodes, and the changes-apply to all nodes nearly simultaneously.+dynamically changed and reconfigured without restarting the Solr nodes, and the changes apply to all nodes nearly simultaneously.== Plugin Types=== Predefined Plugin Names-Plugins with these names are used in specific parts of Solr. Their names are reserved-and cannot be used for other plugin types:+Plugins with these names are used in specific parts of Solr.+Their names are reserved and cannot be used for other plugin types:`.placement-plugin`::-plugin that implements `PlacementPluginFactory` interface. This type of plugin-determines the replica placement strategy in the cluster.+A plugin that implements `PlacementPluginFactory` interface.+This type of plugin determines the replica placement strategy in the cluster.`.cluster-event-producer`::-plugin that implements `ClusterEventProducer` interface. This type of plugin-is used for generating cluster-level events.+A plugin that implements `ClusterEventProducer` interface.+This type of plugin is used for generating cluster-level events.=== PlacementPluginFactory PluginsThis type of plugin supports configurable placement strategies for collection@@ -128,9 +118,8 @@ replicas.=== ClusterSingleton PluginsPlugins that implement `ClusterSingleton` interface are instantiated on each-Solr node. However, their start/stop life-cycle, as defined in the interface,-is controlled in such a way that only a single running instance of the plugin-is present in the cluster at any time.+Solr node.+However, their start/stop life-cycle, as defined in the interface, is controlled in such a way that only a single running instance of the plugin is present in the cluster at any time.(Currently this is implemented by re-using the Overseer leader election, so all`ClusterSingleton`-s that are in the RUNNING state execute on the Overseer leader node).@@ -140,18 +129,15 @@ it requires this cluster singleton behavior.=== ClusterEventProducer PluginsIn order to support the generation of cluster-level events an implementation of-`ClusterEventProducer` is created on each Solr node. This component is also a-`ClusterSingleton`, which means that only one active instance is present in the+`ClusterEventProducer` is created on each Solr node.+This component is also a `ClusterSingleton`, which means that only one active instance is present in thecluster at any time.If no plugin configuration is specified then the default implementation-`org.apache.solr.cluster.events.impl.NoOpProducer` is used, which doesn't generate-any events - this means that by default event generation is turned off. An implementation-that supports node and collection event generation is also available in-`org.apache.solr.cluster.events.impl.DefaultClusterEventProducer`.+`org.apache.solr.cluster.events.impl.NoOpProducer` is used, which doesn't generate any events - this means that by default event generation is turned off.+An implementation that supports node and collection event generation is also available in `org.apache.solr.cluster.events.impl.DefaultClusterEventProducer`.-Event producer configuration can be changed dynamically by changing the predefined-plugin configuration, for example:+Event producer configuration can be changed dynamically by changing the predefined plugin configuration, for example:[source,bash]----@@ -175,21 +161,17 @@ curl -X POST -H 'Content-type: application/json' -d '{=== ClusterEventListener Plugins-Plugins that implement the `ClusterEventListener` interface will be automatically-registered with the instance of `ClusterEventProducer`.+Plugins that implement the `ClusterEventListener` interface will be automatically registered with the instance of `ClusterEventProducer`.// XXX edit this once SOLR-14977 is doneImplementations will be notified of all events that are generated by the`ClusterEventProducer` and need to select only events that they are interested in.==== org.apache.solr.cluster.events.impl.CollectionsRepairEventListener-An implementation of listener that reacts to NODE_LOST events and checks what replicas-need to be re-added to other nodes to keep the replication counts the same as before.+An implementation of listener that reacts to NODE_LOST events and checks what replicas need to be re-added to other nodes to keep the replication counts the same as before.-This implementation waits for a certain period (default is 30s) to make sure the node-is really down, and for the replicas located on nodes that were down sufficiently long-it generates appropriate ADDREPLICA commands to counter-balance the lost replicas on-these nodes.+This implementation waits for a certain period (default is 30s) to make sure the node is really down.+For the replicas located on nodes that were down sufficiently long it generates appropriate ADDREPLICA commands to counter-balance the lost replicas on these nodes.Example plugin configuration:@@ -214,8 +196,8 @@ curl http://localhost:8983/api/cluster/plugin----=== Add Plugin-This command uses HTTP POST to add a new plugin configuration. If a plugin with the-same name already exists this results in an error.+This command uses HTTP POST to add a new plugin configuration.+If a plugin with the same name already exists this results in an error.Example command, which adds a plugin contained in a Solr package:[source,bash]@@ -230,12 +212,12 @@ curl -X POST -H 'Content-type: application/json' -d '{----=== Update Plugin-This command uses HTTP POST to update an existing plugin configuration. If a plugin-with this name doesn't exist this results in an error.+This command uses HTTP POST to update an existing plugin configuration.+If a plugin with this name doesn't exist this results in an error.++This example updates an existing plugin, possibly changing its configuration parameters.+The old instance of the plugin is removed and a new instance is created using the supplied configuration.-This example updates an existing plugin, possibly changing its configuration paramers.-The old instance of the plugin is removed and a new instance is created using the supplied-configuration.[source,bash]----curl -X POST -H 'Content-type: application/json' -d '{@@ -249,11 +231,10 @@ curl -X POST -H 'Content-type: application/json' -d '{----=== Remove Plugin-This command uses HTTP POST to delete an existing plugin configuration. If a plugin-with this name doesn't exist this results in an error.+This command uses HTTP POST to delete an existing plugin configuration.+If a plugin with this name doesn't exist this results in an error.-Unlike other commands the command payload here consists just of-the name of the plugin to remove, as a string.+Unlike other commands the command payload here consists just of the name of the plugin to remove, as a string.[source,bash]----diff --git a/solr/solr-ref-guide/src/codec-factory.adoc b/solr/solr-ref-guide/src/codec-factory.adocindex 7f77028a7a3..590f73e1996 100644--- a/solr/solr-ref-guide/src/codec-factory.adoc+++ b/solr/solr-ref-guide/src/codec-factory.adoc@@ -45,7 +45,10 @@ Example:This factory for Lucene's {solr-javadocs}/core/org/apache/solr/core/SimpleTextCodecFactory.html[`SimpleTextCodecFactory`] produces a plain text human-readable index format.-CAUTION: *FOR RECREATIONAL USE ONLY*. This codec should never be used in production. `SimpleTextCodec` is relatively slow and takes up a large amount of disk space. Its use should be limited to educational and debugging purposes.+CAUTION: *FOR RECREATIONAL USE ONLY*.+This codec should never be used in production.+`SimpleTextCodec` is relatively slow and takes up a large amount of disk space.+Its use should be limited to educational and debugging purposes.Example:diff --git a/solr/solr-ref-guide/src/collapse-and-expand-results.adoc b/solr/solr-ref-guide/src/collapse-and-expand-results.adocindex ce39c1cd968..4f7d87a9d1d 100644--- a/solr/solr-ref-guide/src/collapse-and-expand-results.adoc+++ b/solr/solr-ref-guide/src/collapse-and-expand-results.adoc@@ -18,16 +18,23 @@The Collapsing query parser and the Expand component combine to form an approach to grouping documents for field collapsing in search results.-The Collapsing query parser groups documents (collapsing the result set) according to your parameters, while the Expand component provides access to documents in the collapsed group for use in results display or other processing by a client application. Collapse & Expand can together do what the older <<result-grouping.adoc#,Result Grouping>> (`group=true`) does for _most_ use-cases but not all. Collapse and Expand are not supported when Result Grouping is enabled. Generally, you should prefer Collapse & Expand.+The Collapsing query parser groups documents (collapsing the result set) according to your parameters, while the Expand component provides access to documents in the collapsed group for use in results display or other processing by a client application.+Collapse & Expand can together do what the older <<result-grouping.adoc#,Result Grouping>> (`group=true`) does for _most_ use-cases but not all.+Collapse and Expand are not supported when Result Grouping is enabled.+Generally, you should prefer Collapse & Expand.[IMPORTANT]====-In order to use these features with SolrCloud, the documents must be located on the same shard. To ensure document co-location, you can define the `router.name` parameter as `compositeId` when creating the collection. For more information on this option, see the section <<solrcloud-shards-indexing.adoc#document-routing,Document Routing>>.+In order to use these features with SolrCloud, the documents must be located on the same shard.+To ensure document co-location, you can define the `router.name` parameter as `compositeId` when creating the collection.+For more information on this option, see the section <<solrcloud-shards-indexing.adoc#document-routing,Document Routing>>.====== Collapsing Query Parser-The `CollapsingQParser` is really a _post filter_ that provides more performant field collapsing than Solr's standard approach when the number of distinct groups in the result set is high. This parser collapses the result set to a single document per group before it forwards the result set to the rest of the search components. So all downstream components (faceting, highlighting, etc.) will work with the collapsed result set.+The `CollapsingQParser` is really a _post filter_ that provides more performant field collapsing than Solr's standard approach when the number of distinct groups in the result set is high.+This parser collapses the result set to a single document per group before it forwards the result set to the rest of the search components.+So all downstream components (faceting, highlighting, etc.) will work with the collapsed result set.The CollapsingQParserPlugin fully supports the QueryElevationComponent.@@ -36,26 +43,30 @@ The CollapsingQParserPlugin fully supports the QueryElevationComponent.The CollapsingQParser accepts the following local params:`field`::-The field that is being collapsed on. The field must be a single valued String, Int or Float-type of field.+The field that is being collapsed on.+The field must be a single valued String, Int or Float-type of field.`min` or `max`::Selects the group head document for each group based on which document has the min or max value of the specified numeric field or <<function-queries.adoc#,function query>>.+At most only one of the `min`, `max`, or `sort` (see below) parameters may be specified.+-If none are specified, the group head document of each group will be selected based on the highest scoring document in that group. The default is none.+If none are specified, the group head document of each group will be selected based on the highest scoring document in that group.+The default is none.`sort`::Selects the group head document for each group based on which document comes first according to the specified <<common-query-parameters.adoc#sort-parameter,sort string>>.+At most only one of the `min`, `max`, (see above) or `sort` parameters may be specified.+-If none are specified, the group head document of each group will be selected based on the highest scoring document in that group. The default is none.+If none are specified, the group head document of each group will be selected based on the highest scoring document in that group.+The default is none.`nullPolicy`::There are three available null policies:+-* `ignore`: removes documents with a null value in the collapse field. This is the default.+* `ignore`: removes documents with a null value in the collapse field.+This is the default.* `expand`: treats each document with a null value in the collapse field as a separate group.* `collapse`: collapses all documents with a null value into a single group using either highest score, or minimum/maximum.+@@ -67,7 +78,10 @@ There are two hint options available:+`top_fc`::: This stands for top level FieldCache.+-The `hint=top_fc` hint is only available when collapsing on String fields. `top_fc` usually provides the best query time speed but takes the longest to warm on startup or following a commit. `top_fc` will also result in having the collapsed field cached in memory twice if it's used for faceting or sorting. For very high cardinality (high distinct count) fields, `top_fc` may not fare so well.+The `hint=top_fc` hint is only available when collapsing on String fields.+`top_fc` usually provides the best query time speed but takes the longest to warm on startup or following a commit.+`top_fc` will also result in having the collapsed field cached in memory twice if it's used for faceting or sorting.+For very high cardinality (high distinct count) fields, `top_fc` may not fare so well.+`hint=block`::: This indicates that the field being collapsed on is suitable for the optimzed <<#block-collapsing,Block Collapse>> logic described below.+@@ -76,7 +90,8 @@ The default is none.`size`::Sets the initial size of the collapse data structures when collapsing on a *numeric field only*.+-The data structures used for collapsing grow dynamically when collapsing on numeric fields. Setting the size above the number of results expected in the result set will eliminate the resizing cost.+The data structures used for collapsing grow dynamically when collapsing on numeric fields.+Setting the size above the number of results expected in the result set will eliminate the resizing cost.+The default is 100,000.@@ -108,14 +123,16 @@ Collapse on `group_field` selecting the document in each group with the maximumfq={!collapse field=group_field max=numeric_field}-----Collapse on `group_field` selecting the document in each group with the maximum value of a function. Note that the *cscore()* function can be used with the min/max options to use the score of the current document being collapsed.+Collapse on `group_field` selecting the document in each group with the maximum value of a function.+Note that the *cscore()* function can be used with the min/max options to use the score of the current document being collapsed.[source,text]----fq={!collapse field=group_field max=sum(cscore(),numeric_field)}-----Collapse on `group_field` with a null policy so that all docs that do not have a value in the `group_field` will be treated as a single group. For each group, the selected document will be based first on a `numeric_field`, but ties will be broken by score:+Collapse on `group_field` with a null policy so that all docs that do not have a value in the `group_field` will be treated as a single group.+For each group, the selected document will be based first on a `numeric_field`, but ties will be broken by score:[source,text]----@@ -143,7 +160,9 @@ The default collapsing logic must keep track of all group head documents -- forWhen collapsing on the `\_root_` field however, the logic knows that as it scans over the index, it will never encounter any new documents in a group that it has previously processed.-This more efficient logic can also be used with other `collapseField` values, via the `hint=block` local param. This can be useful when you have deeply nested documents and you'd like to collapse on a field that does not contain identical values for all documents with a common `\_root_` but is a unique and identical value for sets of contiguous documents under a common `\_root_`. For example: searching for "grand child" documents and collapsing on a field that is unique per "child document"+This more efficient logic can also be used with other `collapseField` values, via the `hint=block` local param.+This can be useful when you have deeply nested documents and you'd like to collapse on a field that does not contain identical values for all documents with a common `\_root_` but is a unique and identical value for sets of contiguous documents under a common `\_root_`.+For example: searching for "grand child" documents and collapsing on a field that is unique per "child document"[CAUTION]====@@ -163,9 +182,11 @@ Example usage with the CollapsingQParserPlugin:q=foo&fq={!collapse field=ISBN}-----In the query above, the CollapsingQParserPlugin will collapse the search results on the _ISBN_ field. The main search results will contain the highest ranking document from each book.+In the query above, the CollapsingQParserPlugin will collapse the search results on the _ISBN_ field.+The main search results will contain the highest ranking document from each book.-The ExpandComponent can now be used to expand the results so you can see the documents grouped by ISBN. For example:+The ExpandComponent can now be used to expand the results so you can see the documents grouped by ISBN.+For example:[source,text]----@@ -180,7 +201,8 @@ If there are multiple collapse groups with same cost then the first specified onWhen enabled, the ExpandComponent adds a new section to the search output labeled `expanded`.-Inside the `expanded` section there is a _map_ with each group head pointing to the expanded documents that are within the group. As applications iterate the main collapsed result set, they can access the _expanded_ map to retrieve the expanded groups.+Inside the `expanded` section there is a _map_ with each group head pointing to the expanded documents that are within the group.+As applications iterate the main collapsed result set, they can access the _expanded_ map to retrieve the expanded groups.The ExpandComponent has the following parameters:@@ -188,25 +210,31 @@ The ExpandComponent has the following parameters:When `true`, the ExpandComponent is enabled.`expand.field`::-Field on which expand documents need to be populated. When `expand=true`, either this parameter needs to be specified or should be used with CollapsingQParserPlugin.+Field on which expand documents need to be populated.+When `expand=true`, either this parameter needs to be specified or should be used with CollapsingQParserPlugin.When both are specified, this parameter is given higher priority.`expand.sort`::-Orders the documents within the expanded groups. The default is `score desc`.+Orders the documents within the expanded groups.+The default is `score desc`.`expand.rows`::-The number of rows to display in each group. The default is 5 rows.+The number of rows to display in each group.+The default is 5 rows.+[IMPORTANT]====-When `expand.rows=0`, only the number of documents found for each expanded value is returned. Hence, scores won't be computed even if requested and `maxScore` is set to 0.+When `expand.rows=0`, only the number of documents found for each expanded value is returned.+Hence, scores won't be computed even if requested and `maxScore` is set to 0.====`expand.q`::-Overrides the main query (`q`), determines which documents to include in the main group. The default is to use the main query.+Overrides the main query (`q`), determines which documents to include in the main group.+The default is to use the main query.`expand.fq`::-Overrides main filter queries (`fq`), determines which documents to include in the main group. The default is to use the main filter queries.+Overrides main filter queries (`fq`), determines which documents to include in the main group.+The default is to use the main filter queries.`expand.nullGroup`::Indicates if an expanded group can be returned containing documents with no value in the expanded field.diff --git a/solr/solr-ref-guide/src/collection-management.adoc b/solr/solr-ref-guide/src/collection-management.adocindex 788e0d5e129..fa658bfae94 100644--- a/solr/solr-ref-guide/src/collection-management.adoc+++ b/solr/solr-ref-guide/src/collection-management.adoc@@ -149,7 +149,8 @@ If the field specified is null in the document, the document will be rejected.Please note that <<realtime-get.adoc#,RealTime Get>> or retrieval by document ID would also require the parameter `\_route_` (or `shard.keys`) to avoid a distributed search.`perReplicaState`::-If `true` the states of individual replicas will be maintained as individual child of the `state.json`. The default is `false`.+If `true` the states of individual replicas will be maintained as individual child of the `state.json`.+The default is `false`.`property._name_=_value_`::Set core property _name_ to _value_. See the section <<core-discovery.adoc#,Core Discovery>> for details on supported properties and values.@@ -663,7 +664,8 @@ This parameter is required.`split.key` (v1), `splitKey` (v2)::The routing key prefix.-For example, if the uniqueKey of a document is "a!123", then you would use `split.key=a!`. This parameter is required.+For example, if the uniqueKey of a document is "a!123", then you would use `split.key=a!`.+This parameter is required.`forward.timeout` (v1), `forwardTimeout` (v2)::The timeout, in seconds, until which write requests made to the source collection for the given `split.key` will be forwarded to the target shard.@@ -851,7 +853,8 @@ Such incompatibilities may result from incompatible schema changes or after migr=== COLSTATUS Parameters`collection`::-Collection name (optional). If missing then it means all collections.+Collection name (optional).+If missing then it means all collections.`coreInfo`::Optional boolean.@@ -1704,7 +1707,8 @@ Values \<=0 are use the default value Integer.MAX_VALUE.When this number is reached, the process waits for one or more leaders to be successfully assigned before adding more to the queue.`maxWaitSeconds`::-Defaults to `60`. This is the timeout value when waiting for leaders to be reassigned.+Defaults to `60`.+This is the timeout value when waiting for leaders to be reassigned.If `maxAtOnce` is less than the number of reassignments that will take place, this is the maximum interval that any _single_ wait for at least one reassignment.+For example, if 10 reassignments are to take place and `maxAtOnce` is `1` and `maxWaitSeconds` is `60`, the upper bound on the time that the command may wait is 10 minutes.diff --git a/solr/solr-ref-guide/src/collections-api.adoc b/solr/solr-ref-guide/src/collections-api.adocindex 03c778c50fe..a2404a5fba3 100644--- a/solr/solr-ref-guide/src/collections-api.adoc+++ b/solr/solr-ref-guide/src/collections-api.adoc@@ -17,7 +17,8 @@// specific language governing permissions and limitations// under the License.-A SolrCloud cluster includes a number of components. The Collections API is provided to allow you to control your cluster, including the collections, shards, replicas, backups, leader election, and other operations needs.+A SolrCloud cluster includes a number of components.+The Collections API is provided to allow you to control your cluster, including the collections, shards, replicas, backups, leader election, and other operations needs.Because this API has a large number of commands and options, we've grouped the commands into the following sub-sections:@@ -33,9 +34,12 @@ Because this API has a large number of commands and options, we've grouped the c== Asynchronous Calls-Since some collection API calls can be long running tasks (such as SPLITSHARD), you can optionally have the calls run asynchronously. Specifying `async=<request-id>` enables you to make an asynchronous call, the status of which can be requested using the <<requeststatus,REQUESTSTATUS>> call at any time.+Since some collection API calls can be long running tasks (such as SPLITSHARD), you can optionally have the calls run asynchronously.+Specifying `async=<request-id>` enables you to make an asynchronous call, the status of which can be requested using the <<requeststatus,REQUESTSTATUS>> call at any time.-As of now, REQUESTSTATUS does not automatically clean up the tracking data structures, meaning the status of completed or failed tasks stays stored in ZooKeeper unless cleared manually. DELETESTATUS can be used to clear the stored statuses. However, there is a limit of 10,000 on the number of async call responses stored in a cluster.+As of now, REQUESTSTATUS does not automatically clean up the tracking data structures, meaning the status of completed or failed tasks stays stored in ZooKeeper unless cleared manually.+DELETESTATUS can be used to clear the stored statuses.+However, there is a limit of 10,000 on the number of async call responses stored in a cluster.=== Examples of Async Requests@@ -93,7 +97,8 @@ curl -X POST http://localhost:8983/api/collections/collection1/shards -H 'Conten[[requeststatus]]== REQUESTSTATUS: Request Status of an Async Call-Request the status and response of an already submitted <<Asynchronous Calls,Asynchronous Collection API>> (below) call. This call is also used to clear up the stored statuses.+Request the status and response of an already submitted <<Asynchronous Calls,Asynchronous Collection API>> (below) call.+This call is also used to clear up the stored statuses.[.dynamic-tabs]--@@ -122,7 +127,9 @@ curl -X GET http://localhost:8983/api/cluster/command-status/1000=== REQUESTSTATUS Parameters`requestid`::-The user defined request ID for the request. This can be used to track the status of the submitted asynchronous task. This parameter is required.+The user defined request ID for the request.+This can be used to track the status of the submitted asynchronous task.+This parameter is required.=== Examples using REQUESTSTATUSdiff --git a/solr/solr-ref-guide/src/commits-transaction-logs.adoc b/solr/solr-ref-guide/src/commits-transaction-logs.adocindex cf5903cb5dd..1d5c26279f4 100644--- a/solr/solr-ref-guide/src/commits-transaction-logs.adoc+++ b/solr/solr-ref-guide/src/commits-transaction-logs.adoc@@ -216,7 +216,8 @@ One point of confusion is how much data is contained in a transaction log.A tlog does not contain all documents, only the ones since the last hard commit.Older transaction log files are deleted when no longer needed.-WARNING: Implicit in the above is that transaction logs will grow forever if hard commits are disabled. Therefore it is important that hard commits be enabled when indexing.+WARNING: Implicit in the above is that transaction logs will grow forever if hard commits are disabled.+Therefore it is important that hard commits be enabled when indexing.=== Transaction Log Configuration@@ -265,7 +266,7 @@ The number of update records to keep per log.|Optional |Default: `10`|===+-The maximum number of logs keep. The default is `10`.+The maximum number of logs keep.`numVersionBuckets`::+diff --git a/solr/solr-ref-guide/src/common-query-parameters.adoc b/solr/solr-ref-guide/src/common-query-parameters.adocindex 62627a4db73..314fc2a43c8 100644--- a/solr/solr-ref-guide/src/common-query-parameters.adoc+++ b/solr/solr-ref-guide/src/common-query-parameters.adoc@@ -79,7 +79,8 @@ Users looking to avoid this behavior can add an additional sort criteria on a unWhen specified, the `start` parameter specifies an offset into a query's result set and instructs Solr to begin displaying results from this offset.-The default value is `0`. In other words, by default, Solr returns results without an offset, beginning where the results themselves begin.+The default value is `0`.+In other words, by default, Solr returns results without an offset, beginning where the results themselves begin.Setting the `start` parameter to some other number, such as `3`, causes Solr to skip over the preceding records and start at the document identified by the offset.@@ -91,7 +92,8 @@ For example, if the `rows` parameter is set to 10, you could display three succeYou can use the `rows` parameter to paginate results from a query.The parameter specifies the maximum number of documents from the complete result set that Solr should return to the client at one time.-The default value is `10`. That is, by default, Solr returns 10 documents at a time in response to a query.+The default value is `10`.+That is, by default, Solr returns 10 documents at a time in response to a query.== canCancel Parameter@@ -453,7 +455,8 @@ q=quick brown fox&minExactCount=100&rows=10"docs": [{"doc1"}]}-----Since `numFoundExact=false`, we know the number of documents matching the query is greater or equal to 153. If we specify a higher value for `minExactCount`:+Since `numFoundExact=false`, we know the number of documents matching the query is greater or equal to 153.+If we specify a higher value for `minExactCount`:[source,text]q=quick brown fox&minExactCount=200&rows=10diff --git a/solr/solr-ref-guide/src/computational-geometry.adoc b/solr/solr-ref-guide/src/computational-geometry.adocindex 50d7ac63a71..3be20a023f1 100644--- a/solr/solr-ref-guide/src/computational-geometry.adoc+++ b/solr/solr-ref-guide/src/computational-geometry.adoc@@ -21,29 +21,26 @@ This section of the math expressions user guide covers computational geometry fu== Convex Hull-A convex hull is the smallest convex set of points that encloses a data set. Math expressions has support for computing-the convex hull of a 2D data set. Once a convex hull has been calculated, a set of math expression functions+A convex hull is the smallest convex set of points that encloses a data set.+Math expressions has support for computing the convex hull of a 2D data set.+Once a convex hull has been calculated, a set of math expression functionscan be applied to geometrically describe and visualize the convex hull.=== Visualization-The `convexHull` function can be used to visualize a border around a-set of 2D points. Border visualizations can be useful for understanding where data points are-in relation to the border.+The `convexHull` function can be used to visualize a border around a set of 2D points.+Border visualizations can be useful for understanding where data points are in relation to the border.-In the examples below the `convexHull` function is used-to visualize a border for a set of latitude and longitude points of rat sightings in the NYC311-complaints database. An investigation of the border around the rat sightings can be done-to better understand how rats may be entering or exiting the specific region.+In the examples below the `convexHull` function is used to visualize a border for a set of latitude and longitude points of rat sightings in the NYC311+complaints database.+An investigation of the border around the rat sightings can be done to better understand how rats may be entering or exiting the specific region.==== Scatter PlotBefore visualizing the convex hull its often useful to visualize the 2D points as a scatter plot.-In this example the `random` function draws a sample of records from the NYC311 (complaints database) collection where-the complaint description matches "rat sighting" and the zip code is 11238. The latitude and longitude fields-are then vectorized and plotted as a scatter plot with longitude on x-axis and latitude on the-y-axis.+In this example the `random` function draws a sample of records from the NYC311 (complaints database) collection where the complaint description matches "rat sighting" and the zip code is 11238.+The latitude and longitude fields are then vectorized and plotted as a scatter plot with longitude on x-axis and latitude on the y-axis.image::images/math-expressions/convex0.png[]@@ -51,10 +48,10 @@ Notice from the scatter plot that many of the points appear to lie near the bord==== Convex Hull Plot-The `convexHull` function can be used to visualize the border. The example uses the same points-drawn from the NYC311 database. But instead of plotting the points directly the latitude and-longitude points are added as rows to a matrix. The matrix is then transposed with `transpose`-function so that each row of the matrix contains a single latitude and longitude point.+The `convexHull` function can be used to visualize the border.+The example uses the same points drawn from the NYC311 database.+But instead of plotting the points directly the latitude and longitude points are added as rows to a matrix.+The matrix is then transposed with `transpose` function so that each row of the matrix contains a single latitude and longitude point.The `convexHull` function is then used calculate the convex hull for the matrix of points.The convex hull is set a variable called `hull`.@@ -62,49 +59,44 @@ The convex hull is set a variable called `hull`.Once the convex hull has been created the `getVertices` function can be used toretrieve the matrix of points in the scatter plot that comprise the convex border around the scatter plot.The `colAt` function can then be used to retrieve the latitude and longitude vectors from the matrix-so they can visualized by the `zplot` function. In the example below the convex hull points are-visualized as a scatter plot.+so they can visualized by the `zplot` function.+In the example below the convex hull points are visualized as a scatter plot.image::images/math-expressions/hullplot.png[]-Notice that the 15 points in the scatter plot describe that latitude and longitude points of the-convex hull.+Notice that the 15 points in the scatter plot describe that latitude and longitude points of the convex hull.==== Projecting and Clustering-The once a convex hull as been calculated the `projectToBorder` can then be used to project-points to the nearest point on the border. In the example below the `projectToBorder` function-is used to project the original scatter scatter plot points to the nearest border.+The once a convex hull as been calculated the `projectToBorder` can then be used to project points to the nearest point on the border.+In the example below the `projectToBorder` function is used to project the original scatter scatter plot points to the nearest border.-The `projectToBorder` function returns a matrix of lat/lon points for the border projections. In-the example the matrix of border points is then clustered into 7 clusters using kmeans clustering.+The `projectToBorder` function returns a matrix of lat/lon points for the border projections.+In the example the matrix of border points is then clustered into 7 clusters using kmeans clustering.The `zplot` function is then used to plot the clustered border points.image::images/math-expressions/convex1.png[]-Notice in the visualization its easy to see which spots along the border have the highest-density of points. In the case or the rat sightings this information is useful in understanding-which border points are closest for the rats to enter or exit from.+Notice in the visualization its easy to see which spots along the border have the highest density of points.+In the case of the rat sightings this information is useful in understanding which border points are closest for the rats to enter or exit from.==== Plotting the CentroidsOnce the border points have been clustered its very easy to extract the centroids of the clusters-and plot them on a map. The example below extracts the centroids from the clusters using the-`getCentroids` function. `getCentroids` returns the matrix of lat/lon points which represent-the centroids of border clusters. The `colAt` function can then be used to extract the lat/lon-vectors so they can be plotted on a map using `zplot`.+and plot them on a map.+The example below extracts the centroids from the clusters using the `getCentroids` function.+`getCentroids` returns the matrix of lat/lon points which represent the centroids of border clusters.+The `colAt` function can then be used to extract the lat/lon vectors so they can be plotted on a map using `zplot`.image::images/math-expressions/convex2.png[]-The map above shows the centroids of the border clusters. The centroids from the highest-density clusters can now be zoomed and investigated geo-spatially to determine what might be-the best places to begin an investigation of the border.+The map above shows the centroids of the border clusters.+The centroids from the highest density clusters can now be zoomed and investigated geo-spatially to determine what might be the best places to begin an investigation of the border.== Enclosing DiskThe `enclosingDisk` function finds the smallest enclosing circle the encloses a 2D data set.-Once an enclosing disk has been calculated, a set of math expression functions-can be applied to geometrically describe the enclosing disk.+Once an enclosing disk has been calculated, a set of math expression functions can be applied to geometrically describe the enclosing disk.In the example below an enclosing disk is calculated for a randomly generated set of 1000 2D observations.diff --git a/solr/solr-ref-guide/src/config-api.adoc b/solr/solr-ref-guide/src/config-api.adocindex 258f3084afc..d2410a51ab7 100644--- a/solr/solr-ref-guide/src/config-api.adoc+++ b/solr/solr-ref-guide/src/config-api.adoc@@ -893,7 +893,8 @@ If `params.json` is modified, the params object is just updated without a core r=== Empty Command-If an empty command is sent to the `/config` endpoint, the watch is triggered on all cores using this configset. For example:+If an empty command is sent to the `/config` endpoint, the watch is triggered on all cores using this configset.+For example:[.dynamic-tabs]--diff --git a/solr/solr-ref-guide/src/configsets-api.adoc b/solr/solr-ref-guide/src/configsets-api.adocindex c0d9a9ccc8b..8b85228950c 100644--- a/solr/solr-ref-guide/src/configsets-api.adoc+++ b/solr/solr-ref-guide/src/configsets-api.adoc@@ -19,7 +19,8 @@The Configsets API enables you to upload new configsets to ZooKeeper, create, and delete configsets when Solr is running SolrCloud mode.-Configsets are a collection of configuration files such as `solrconfig.xml`, `synonyms.txt`, the schema, language-specific files, and other collection-level configuration files (everything that normally lives in the `conf` directory). Solr ships with two example configsets (`_default` and `sample_techproducts_configs`) which can be used when creating collections.+Configsets are a collection of configuration files such as `solrconfig.xml`, `synonyms.txt`, the schema, language-specific files, and other collection-level configuration files (everything that normally lives in the `conf` directory).+Solr ships with two example configsets (`_default` and `sample_techproducts_configs`) which can be used when creating collections.Using the same concept, you can create your own configsets and make them available when creating collections.This API provides a way to upload configuration files to ZooKeeper and share the same set of configuration files between two or more collections.diff --git a/solr/solr-ref-guide/src/configuring-solr-xml.adoc b/solr/solr-ref-guide/src/configuring-solr-xml.adocindex 0e0f74a2972..68eea5fbda0 100644--- a/solr/solr-ref-guide/src/configuring-solr-xml.adoc+++ b/solr/solr-ref-guide/src/configuring-solr-xml.adoc@@ -308,7 +308,8 @@ If you would like to customize the metrics for your installation, see the sectio== Substituting JVM System Properties in solr.xmlSolr supports variable substitution of JVM system property values in `solr.xml`, which allows runtime specification of various configuration options.-The syntax is `${propertyname[:option default value]}`. This allows defining a default that can be overridden when Solr is launched.+The syntax is `${propertyname[:option default value]}`.+This allows defining a default that can be overridden when Solr is launched.If a default value is not specified, then the property must be specified at runtime or the `solr.xml` file will generate an error when parsed.Any JVM system properties usually specified using the `-D` flag when starting the JVM, can be used as variables in the `solr.xml` file.diff --git a/solr/solr-ref-guide/src/copy-fields.adoc b/solr/solr-ref-guide/src/copy-fields.adocindex 2404b702b8e..c51f6c44937 100644--- a/solr/solr-ref-guide/src/copy-fields.adoc+++ b/solr/solr-ref-guide/src/copy-fields.adoc@@ -33,7 +33,8 @@ In the example above, if the `text` destination field has data of its own in theRemember to configure your fields as `multivalued="true"` if they will ultimately get multiple values (either from a multivalued source or from multiple `copyField` directives).A common usage for this functionality is to create a single "search" field that will serve as the default query field when users or clients do not specify a field to query.-For example, `title`, `author`, `keywords`, and `body` may all be fields that should be searched by default, with copy field rules for each field to copy to a `catchall` field (for example, it could be named anything). Later you can set a rule in `solrconfig.xml` to search the `catchall` field by default.+For example, `title`, `author`, `keywords`, and `body` may all be fields that should be searched by default, with copy field rules for each field to copy to a `catchall` field (for example, it could be named anything).+Later you can set a rule in `solrconfig.xml` to search the `catchall` field by default.One caveat to this is your index will grow when using copy fields.However, whether this becomes problematic for you and the final size will depend on the number of fields being copied, the number of destination fields being copied to, the analysis in use, and the available disk space.diff --git a/solr/solr-ref-guide/src/coreadmin-api.adoc b/solr/solr-ref-guide/src/coreadmin-api.adocindex 5900191f4fb..5f34f15094e 100644--- a/solr/solr-ref-guide/src/coreadmin-api.adoc+++ b/solr/solr-ref-guide/src/coreadmin-api.adoc@@ -85,7 +85,8 @@ curl -X GET http://localhost:8983/api/cores?indexInfo=false=== STATUS Parameters`core`::-The name of a core, as listed in the "name" attribute of a `<core>` element in `solr.xml`. This parameter is required in v1, and part of the url in the v2 API.+The name of a core, as listed in the "name" attribute of a `<core>` element in `solr.xml`.+This parameter is required in v1, and part of the url in the v2 API.`indexInfo`::If `false`, information about the index will not be returned with a core STATUS request.diff --git a/solr/solr-ref-guide/src/curve-fitting.adoc b/solr/solr-ref-guide/src/curve-fitting.adocindex 966e8882b93..d5a79ede503 100644--- a/solr/solr-ref-guide/src/curve-fitting.adoc+++ b/solr/solr-ref-guide/src/curve-fitting.adoc@@ -20,84 +20,68 @@ These functions support constructing a curve through bivariate non-linear data.== Polynomial Curve Fitting-The `polyfit` function is a general purpose curve fitter used to model-the non-linear relationship between two random variables.+The `polyfit` function is a general purpose curve fitter used to model the non-linear relationship between two random variables.The `polyfit` function is passed x- and y-axes and fits a smooth curve to the data.-If only a single array is provided it is treated as the y-axis and a sequence is generated-for the x-axis. A third parameter can be added that specifies the degree of the polynomial. If the degree is-not provided a 3 degree polynomial is used by default. The higher-the degree the more curves that can be modeled.+If only a single array is provided it is treated as the y-axis and a sequence is generated for the x-axis.+A third parameter can be added that specifies the degree of the polynomial.+If the degree is not provided a 3 degree polynomial is used by default.+The higher the degree the more curves that can be modeled.-The `polyfit` function can be visualized in a similar manner to linear regression with-Zeppelin-Solr.+The `polyfit` function can be visualized in a similar manner to linear regression with Zeppelin-Solr.-The example below uses the `polyfit` function to fit a non-linear curve to a scatter-plot of a random sample. The blue points are the scatter plot of the original observations and the red points-are the predicted curve.+The example below uses the `polyfit` function to fit a non-linear curve to a scatter plot of a random sample.+The blue points are the scatter plot of the original observations and the red points are the predicted curve.image::images/math-expressions/polyfit.png[]-In the example above a random sample containing two fields, `filesize_d`-and `response_d`, is drawn from the `logs` collection.+In the example above a random sample containing two fields, `filesize_d` and `response_d`, is drawn from the `logs` collection.The two fields are vectorized and set to the variables `x` and `y`.-Then the `polyfit` function is used to fit a non-linear model to the data using a 5 degree-polynomial. The `polyfit` function returns a model that is then directly plotted-by `zplot` along with the original observations.+Then the `polyfit` function is used to fit a non-linear model to the data using a 5 degree polynomial.+The `polyfit` function returns a model that is then directly plotted by `zplot` along with the original observations.-The fitted model can also be used-by the `predict` function in the same manner as linear regression. The example below-uses the fitted model to predict a response time for a file size of 42000.+The fitted model can also be used by the `predict` function in the same manner as linear regression.+The example below uses the fitted model to predict a response time for a file size of 42000.image::images/math-expressions/polyfit-predict.png[]If an array of predictor values is provided an array of predictions will be returned.-The `polyfit` model performs both *interpolation* and *extrapolation*,-which means that it can predict results both within the bounds of the data set-and beyond the bounds.+The `polyfit` model performs both *interpolation* and *extrapolation*, which means that it can predict results both within the bounds of the data set and beyond the bounds.=== Residuals-The residuals can be calculated and visualized in the same manner as linear-regression as well. In the example below the `ebeSubtract` function is used-to subtract the fitted model from the observed values, to-calculate a vector of residuals. The residuals are then plotted in a *residual plot*-with the predictions along the x-axis and the model error on the y-axis.+The residuals can be calculated and visualized in the same manner as linear regression as well.+In the example below the `ebeSubtract` function is used to subtract the fitted model from the observed values, to calculate a vector of residuals.+The residuals are then plotted in a *residual plot* with the predictions along the x-axis and the model error on the y-axis.image::images/math-expressions/polyfit-resid.png[]== Gaussian Curve Fitting-The `gaussfit` function fits a smooth curve through a Gaussian peak. The `gaussfit`-function takes an x- and y-axis and fits a smooth gaussian curve to the data. If-only one vector of numbers is passed, `gaussfit` will treat it as the y-axis-and will generate a sequence for the x-axis.+The `gaussfit` function fits a smooth curve through a Gaussian peak.+The `gaussfit` function takes an x- and y-axis and fits a smooth gaussian curve to the data.+If only one vector of numbers is passed, `gaussfit` will treat it as the y-axis and will generate a sequence for the x-axis.-One of the interesting use cases for `gaussfit` is to visualize how well a regression-model's residuals fit a normal distribution.+One of the interesting use cases for `gaussfit` is to visualize how well a regression model's residuals fit a normal distribution.One of the characteristics of a well-fit regression model is that its residuals will ideally fit a normal distribution.-We can-test this by building a histogram of the residuals and then fitting a gaussian curve to the curve of the histogram.+We can test this by building a histogram of the residuals and then fitting a gaussian curve to the curve of the histogram.-In the example below the residuals from a `polyfit` regression are modeled with the-`hist` function to return a histogram with 32 bins. The `hist` function returns-a list of tuples with statistics about each bin. In the example the `col` function is-used to return a vector with the `N` column for each bin, which is the count of-observations in the-bin. If the residuals are normally distributed we would expect the bin counts-to roughly follow a gaussian curve.+In the example below the residuals from a `polyfit` regression are modeled with the `hist` function to return a histogram with 32 bins.+The `hist` function returns a list of tuples with statistics about each bin.+In the example the `col` function is used to return a vector with the `N` column for each bin, which is the count of observations in the+bin.+If the residuals are normally distributed we would expect the bin counts to roughly follow a gaussian curve.-The bin count vector is then passed to `gaussfit` as the y-axis. `gaussfit` generates-a sequence for the x-axis and then fits the gaussian curve to data.+The bin count vector is then passed to `gaussfit` as the y-axis.+`gaussfit` generates a sequence for the x-axis and then fits the gaussian curve to data.-`zplot` is then used to plot the original bin counts and the fitted curve. In the-example below, the blue line is the bin counts, and the smooth yellow line is the-fitted curve. We can see that the binned residuals fit fairly well to a normal-distribution.+`zplot` is then used to plot the original bin counts and the fitted curve.+In the example below, the blue line is the bin counts, and the smooth yellow line is the fitted curve.+We can see that the binned residuals fit fairly well to a normal distribution.image::images/math-expressions/gaussfit.png[]@@ -110,26 +94,23 @@ image::images/math-expressions/gaussfit2.png[]The `harmonicFit` function (or `harmfit`, for short) fits a smooth line through control points of a sine wave.The `harmfit` function is passed x- and y-axes and fits a smooth curve to the data.-If a single array is provided it is treated as the y-axis and a sequence is generated-for the x-axis.+If a single array is provided it is treated as the y-axis and a sequence is generated for the x-axis.-The example below shows `harmfit` fitting a single oscillation of a sine wave. The `harmfit` function-returns the smoothed values at each control point. The return value is also a model which can be used by-the `predict`, `derivative` and `integrate` functions.--NOTE: The `harmfit` function works best when run on a single oscillation rather than a long sequence of-oscillations. This is particularly true if the sine wave has noise. After the curve has been fit it can be-extrapolated to any point in time in the past or future.+The example below shows `harmfit` fitting a single oscillation of a sine wave.+The `harmfit` function returns the smoothed values at each control point.+The return value is also a model which can be used by the `predict`, `derivative` and `integrate` functions.+NOTE: The `harmfit` function works best when run on a single oscillation rather than a long sequence of oscillations.+This is particularly true if the sine wave has noise.+After the curve has been fit it can be extrapolated to any point in time in the past or future.In the example below the original control points are shown in blue and the fitted curve is shown in yellow.image::images/math-expressions/harmfit.png[]--The output of `harmfit` is a model that can be used by the `predict` function to interpolate and extrapolate-the sine wave. In the example below the `natural` function creates an x-axis from 0 to 127-used to predict results for the model. This extrapolates the sine wave out to 128 points, when-the original model curve had only 19 control points.+The output of `harmfit` is a model that can be used by the `predict` function to interpolate and extrapolate the sine wave.+In the example below the `natural` function creates an x-axis from 0 to 127+used to predict results for the model.+This extrapolates the sine wave out to 128 points, when the original model curve had only 19 control points.image::images/math-expressions/harmfit2.png[]diff --git a/solr/solr-ref-guide/src/date-formatting-math.adoc b/solr/solr-ref-guide/src/date-formatting-math.adocindex 8bd66c25a3f..1753e276730 100644--- a/solr/solr-ref-guide/src/date-formatting-math.adoc+++ b/solr/solr-ref-guide/src/date-formatting-math.adoc@@ -31,7 +31,8 @@ The format used is a restricted form of the canonical representation of dateTime* `ss` is seconds.* `Z` is a literal 'Z' character indicating that this string representation of the date is in UTC-Note that no time zone can be specified; the String representations of dates is always expressed in Coordinated Universal Time (UTC). Here is an example value:+Note that no time zone can be specified; the String representations of dates is always expressed in Coordinated Universal Time (UTC).+Here is an example value:`1972-05-20T17:33:18Z`@@ -42,7 +43,8 @@ Here are example values with sub-seconds:* `1972-05-20T17:33:18.77Z`* `1972-05-20T17:33:18.7Z`-There must be a leading `'-'` for dates prior to year 0000, and Solr will format dates with a leading `'+'` for years after 9999. Year 0000 is considered year 1 BC; there is no such thing as year 0 AD or BC.+There must be a leading `'-'` for dates prior to year 0000, and Solr will format dates with a leading `'+'` for years after 9999.+Year 0000 is considered year 1 BC; there is no such thing as year 0 AD or BC..Query escaping may be required[WARNING]@@ -62,7 +64,8 @@ These are valid queries: +Solr's `DateRangeField` supports the same point in time date syntax described above (with _date math_ described below) and more to express date ranges.One class of examples is truncated dates, which represent the entire date span to the precision indicated.-The other class uses the range syntax (`[ TO ]`). Here are some examples:+The other class uses the range syntax (`[ TO ]`).+Here are some examples:* `2000-11` – The entire month of November, 2000.* `1605-11-05` – The Fifth of November.@@ -185,6 +188,7 @@ fq={!field f=dateRange op=Contains}[2013 TO 2018]----Unlike most local params, `op` is actually _not_ defined by any query parser (`field`), it is defined by the field type, in this case `DateRangeField`.-In the above example, it would find documents with indexed ranges that _contain_ (or equals) the range 2013 thru 2018. Multi-valued overlapping indexed ranges in a document are effectively coalesced.+In the above example, it would find documents with indexed ranges that _contain_ (or equals) the range 2013 thru 2018.+Multi-valued overlapping indexed ranges in a document are effectively coalesced.For a DateRangeField example use-case, see https://cwiki.apache.org/confluence/display/solr/DateRangeField[see Solr's community wiki].diff --git a/solr/solr-ref-guide/src/de-duplication.adoc b/solr/solr-ref-guide/src/de-duplication.adocindex 893c0c712a9..9d24cff5fe6 100644--- a/solr/solr-ref-guide/src/de-duplication.adoc+++ b/solr/solr-ref-guide/src/de-duplication.adoc@@ -78,7 +78,8 @@ By default, all fields on the document will be used.`signatureField`::The name of the field used to hold the fingerprint/signature.-The field should be defined in `schema.xml`. The default is `signatureField`.+The field should be defined in your schema.+The default is `signatureField`.`enabled`::Set to *false* to disable de-duplication processing.diff --git a/solr/solr-ref-guide/src/dismax-query-parser.adoc b/solr/solr-ref-guide/src/dismax-query-parser.adocindex 46c0c89e485..c16489889c2 100644--- a/solr/solr-ref-guide/src/dismax-query-parser.adoc+++ b/solr/solr-ref-guide/src/dismax-query-parser.adoc@@ -16,13 +16,22 @@// specific language governing permissions and limitations// under the License.-The DisMax query parser is designed to process simple phrases (without complex syntax) entered by users and to search for individual terms across several fields using different weighting (boosts) based on the significance of each field. Additional options enable users to influence the score based on rules specific to each use case (independent of user input).+The DisMax query parser is designed to process simple phrases (without complex syntax) entered by users and to search for individual terms across several fields using different weighting (boosts) based on the significance of each field.+Additional options enable users to influence the score based on rules specific to each use case (independent of user input).-In general, the DisMax query parser's interface is more like that of Google than the interface of the 'lucene' Solr query parser. This similarity makes DisMax the appropriate query parser for many consumer applications. It accepts a simple syntax, and it rarely produces error messages.+In general, the DisMax query parser's interface is more like that of Google than the interface of the 'lucene' Solr query parser.+This similarity makes DisMax the appropriate query parser for many consumer applications.+It accepts a simple syntax, and it rarely produces error messages.-The DisMax query parser supports an extremely simplified subset of the Lucene QueryParser syntax. As in Lucene, quotes can be used to group phrases, and +/- can be used to denote mandatory and optional clauses. All other Lucene query parser special characters (except AND and OR) are escaped to simplify the user experience. The DisMax query parser takes responsibility for building a good query from the user's input using Boolean clauses containing DisMax queries across fields and boosts specified by the user. It also lets the Solr administrator provide additional boosting queries, boosting functions, and filtering queries to artificially affect the outcome of all searches. These options can all be specified as default parameters for the request handler in the `solrconfig.xml` file or overridden in the Solr query URL.+The DisMax query parser supports an extremely simplified subset of the Lucene QueryParser syntax.+As in Lucene, quotes can be used to group phrases, and +/- can be used to denote mandatory and optional clauses.+All other Lucene query parser special characters (except AND and OR) are escaped to simplify the user experience.+The DisMax query parser takes responsibility for building a good query from the user's input using Boolean clauses containing DisMax queries across fields and boosts specified by the user.+It also lets the Solr administrator provide additional boosting queries, boosting functions, and filtering queries to artificially affect the outcome of all searches.+These options can all be specified as default parameters for the request handler in the `solrconfig.xml` file or overridden in the Solr query URL.-Interested in the technical concept behind the DisMax name? DisMax stands for Maximum Disjunction. Here's a definition of a Maximum Disjunction or "DisMax" query:+Interested in the technical concept behind the DisMax name? DisMax stands for Maximum Disjunction.+Here's a definition of a Maximum Disjunction or "DisMax" query:[quote]____@@ -33,29 +42,37 @@ Whether or not you remember this explanation, do remember that the DisMax Query== DisMax Query Parser Parameters-In addition to the common request parameters, highlighting parameters, and simple facet parameters, the DisMax query parser supports the parameters described below. Like the standard query parser, the DisMax query parser allows default parameter values to be specified in `solrconfig.xml`, or overridden by query-time values in the request.+In addition to the common request parameters, highlighting parameters, and simple facet parameters, the DisMax query parser supports the parameters described below.+Like the standard query parser, the DisMax query parser allows default parameter values to be specified in `solrconfig.xml`, or overridden by query-time values in the request.The sections below explain these parameters in detail.=== q Parameter-The `q` parameter defines the main "query" constituting the essence of the search. The parameter supports raw input strings provided by users with no special escaping. The + and - characters are treated as "mandatory" and "prohibited" modifiers for terms. Text wrapped in balanced quote characters (for example, "San Jose") is treated as a phrase. Any query containing an odd number of quote characters is evaluated as if there were no quote characters at all.+The `q` parameter defines the main "query" constituting the essence of the search.+The parameter supports raw input strings provided by users with no special escaping.+The + and - characters are treated as "mandatory" and "prohibited" modifiers for terms.+Text wrapped in balanced quote characters (for example, "San Jose") is treated as a phrase.+Any query containing an odd number of quote characters is evaluated as if there were no quote characters at all.IMPORTANT: The `q` parameter does not support wildcard characters such as *.=== q.alt Parameter-If specified, the `q.alt` parameter defines a query (which by default will be parsed using standard query parsing syntax) when the main q parameter is not specified or is blank. The `q.alt` parameter comes in handy when you need something like a query to match all documents (don't forget `&rows=0` for that one!) in order to get collection-wide faceting counts.+If specified, the `q.alt` parameter defines a query (which by default will be parsed using standard query parsing syntax) when the main q parameter is not specified or is blank.+The `q.alt` parameter comes in handy when you need something like a query to match all documents (don't forget `&rows=0` for that one!) in order to get collection-wide faceting counts.=== qf (Query Fields) Parameter-The `qf` parameter introduces a list of fields, each of which is assigned a boost factor to increase or decrease that particular field's importance in the query. For example, the query below:+The `qf` parameter introduces a list of fields, each of which is assigned a boost factor to increase or decrease that particular field's importance in the query.+For example, the query below:`qf="fieldOne^2.3 fieldTwo fieldThree^0.4"`-assigns `fieldOne` a boost of 2.3, leaves `fieldTwo` with the default boost (because no boost factor is specified), and `fieldThree` a boost of 0.4. These boost factors make matches in `fieldOne` much more significant than matches in `fieldTwo`, which in turn are much more significant than matches in `fieldThree`.+assigns `fieldOne` a boost of 2.3, leaves `fieldTwo` with the default boost (because no boost factor is specified), and `fieldThree` a boost of 0.4.+These boost factors make matches in `fieldOne` much more significant than matches in `fieldTwo`, which in turn are much more significant than matches in `fieldThree`.=== mm (Minimum Should Match) Parameter@@ -80,10 +97,16 @@ The table below explains the various ways that mm values can be specified.When specifying `mm` values, keep in mind the following:-* When dealing with percentages, negative values can be used to get different behavior in edge cases. 75% and -25% mean the same thing when dealing with 4 clauses, but when dealing with 5 clauses 75% means 3 are required, but -25% means 4 are required.-* If the calculations based on the parameter arguments determine that no optional clauses are needed, the usual rules about Boolean queries still apply at search time. (That is, a Boolean query containing no required clauses must still match at least one optional clause).-* No matter what number the calculation arrives at, Solr will never use a value greater than the number of optional clauses, or a value less than 1. In other words, no matter how low or how high the calculated result, the minimum number of required matches will never be less than 1 or greater than the number of clauses.-* When searching across multiple fields that are configured with different query analyzers, the number of optional clauses may differ between the fields. In such a case, the value specified by mm applies to the maximum number of optional clauses. For example, if a query clause is treated as stopword for one of the fields, the number of optional clauses for that field will be smaller than for the other fields. A query with such a stopword clause would not return a match in that field if mm is set to 100% because the removed clause does not count as matched.+* When dealing with percentages, negative values can be used to get different behavior in edge cases.+75% and -25% mean the same thing when dealing with 4 clauses, but when dealing with 5 clauses 75% means 3 are required, but -25% means 4 are required.+* If the calculations based on the parameter arguments determine that no optional clauses are needed, the usual rules about Boolean queries still apply at search time.+(That is, a Boolean query containing no required clauses must still match at least one optional clause).+* No matter what number the calculation arrives at, Solr will never use a value greater than the number of optional clauses, or a value less than 1.+In other words, no matter how low or how high the calculated result, the minimum number of required matches will never be less than 1 or greater than the number of clauses.+* When searching across multiple fields that are configured with different query analyzers, the number of optional clauses may differ between the fields.+In such a case, the value specified by mm applies to the maximum number of optional clauses.+For example, if a query clause is treated as stopword for one of the fields, the number of optional clauses for that field will be smaller than for the other fields.+A query with such a stopword clause would not return a match in that field if mm is set to 100% because the removed clause does not count as matched.The default value of `mm` is 0% (all clauses optional), unless `q.op` is specified as "AND", in which case `mm` defaults to 100% (all clauses required).@@ -97,26 +120,33 @@ The format is the same as that used by the `qf` parameter: a list of fields and=== ps (Phrase Slop) Parameter-The `ps` parameter specifies the amount of "phrase slop" to apply to queries specified with the pf parameter. Phrase slop is the number of positions one token needs to be moved in relation to another token in order to match a phrase specified in a query.+The `ps` parameter specifies the amount of "phrase slop" to apply to queries specified with the pf parameter.+Phrase slop is the number of positions one token needs to be moved in relation to another token in order to match a phrase specified in a query.=== qs (Query Phrase Slop) Parameter-The `qs` parameter specifies the amount of slop permitted on phrase queries explicitly included in the user's query string with the `qf` parameter. As explained above, slop refers to the number of positions one token needs to be moved in relation to another token in order to match a phrase specified in a query.+The `qs` parameter specifies the amount of slop permitted on phrase queries explicitly included in the user's query string with the `qf` parameter.+As explained above, slop refers to the number of positions one token needs to be moved in relation to another token in order to match a phrase specified in a query.=== The tie (Tie Breaker) ParameterThe `tie` parameter specifies a float value (which should be something much less than 1) to use as tiebreaker in DisMax queries.-When a term from the user's input is tested against multiple fields, more than one field may match. If so, each field will generate a different score based on how common that word is in that field (for each document relative to all other documents). The `tie` parameter lets you control how much the final score of the query will be influenced by the scores of the lower scoring fields compared to the highest scoring field.+When a term from the user's input is tested against multiple fields, more than one field may match.+If so, each field will generate a different score based on how common that word is in that field (for each document relative to all other documents).+The `tie` parameter lets you control how much the final score of the query will be influenced by the scores of the lower scoring fields compared to the highest scoring field.-A value of "0.0" - the default - makes the query a pure "disjunction max query": that is, only the maximum scoring subquery contributes to the final score. A value of "1.0" makes the query a pure "disjunction sum query" where it doesn't matter what the maximum scoring sub query is, because the final score will be the sum of the subquery scores. Typically a low value, such as 0.1, is useful.+A value of "0.0" - the default - makes the query a pure "disjunction max query": that is, only the maximum scoring subquery contributes to the final score.+A value of "1.0" makes the query a pure "disjunction sum query" where it doesn't matter what the maximum scoring sub query is, because the final score will be the sum of the subquery scores.+Typically a low value, such as 0.1, is useful.=== bq (Boost Query) Parameter-The `bq` parameter specifies an additional, optional, query clause that will be _added_ to the user's main query as optional clauses that will influence the score. For example, if you wanted to add a boost for documents that are in a particular category you could use:+The `bq` parameter specifies an additional, optional, query clause that will be _added_ to the user's main query as optional clauses that will influence the score.+For example, if you wanted to add a boost for documents that are in a particular category you could use:[source,text]----@@ -147,7 +177,8 @@ The only difference between the above examples, is that using the `bq` parameter[[bq-bf-shortcomings]].Additive Boosts vs Multiplicative Boosts====-Generally speaking, using `bq` (or `bf`, below) is considered a poor way to "boost" documents by a secondary query because it has an "Additive" effect on the final score. The overall impact a particular `bq` parameter will have on a given document can vary a lot depending on the _absolute_ values of the scores from the original query as well as the `bq` query, which in turn depends on the complexity of the original query, and various scoring factors (TF, IDF, average field length, etc.)+Generally speaking, using `bq` (or `bf`, below) is considered a poor way to "boost" documents by a secondary query because it has an "Additive" effect on the final score.+The overall impact a particular `bq` parameter will have on a given document can vary a lot depending on the _absolute_ values of the scores from the original query as well as the `bq` query, which in turn depends on the complexity of the original query, and various scoring factors (TF, IDF, average field length, etc.)"Multiplicative Boosting" is generally considered to be a more predictable method of influencing document score, because it acts as a "scaling factor" -- increasing (or decreasing) the scores of each document by a _relative_ amount.@@ -157,7 +188,9 @@ The <<other-parsers.adoc#boost-query-parser,`{!boost}` QParser>> provides a conv=== bf (Boost Functions) Parameter-The `bf` parameter specifies functions (with optional <<standard-query-parser.adoc#boosting-a-term-with,query boost>>) that will be used to construct FunctionQueries which will be _added_ to the user's main query as optional clauses that will influence the score. Any <<function-queries.adoc#available-functions,function supported natively by Solr>> can be used, along with a boost value. For example:+The `bf` parameter specifies functions (with optional <<standard-query-parser.adoc#boosting-a-term-with,query boost>>) that will be used to construct FunctionQueries which will be _added_ to the user's main query as optional clauses that will influence the score.+Any <<function-queries.adoc#available-functions,function supported natively by Solr>> can be used, along with a boost value.+For example:[source,text]----@@ -215,7 +248,9 @@ Another request handler is registered at "/instock" and has slightly different c`\http://localhost:8983/solr/techproducts/instock?defType=dismax&q=video&fl=name,score,inStock`-One of the other really cool features in this parser is robust support for specifying the "BooleanQuery.minimumNumberShouldMatch" you want to be used based on how many terms are in your user's query. These allows flexibility for typos and partial matches. For the dismax parser, one and two word queries require that all of the optional clauses match, but for three to five word queries one missing word is allowed.+One of the other really cool features in this parser is robust support for specifying the "BooleanQuery.minimumNumberShouldMatch" you want to be used based on how many terms are in your user's query.+These allows flexibility for typos and partial matches.+For the dismax parser, one and two word queries require that all of the optional clauses match, but for three to five word queries one missing word is allowed.`\http://localhost:8983/solr/techproducts/select?defType=dismax&q=belkin+ipod`diff --git a/solr/solr-ref-guide/src/docker-faq.adoc b/solr/solr-ref-guide/src/docker-faq.adocindex df90235a6fc..bbd0731bc68 100644--- a/solr/solr-ref-guide/src/docker-faq.adoc+++ b/solr/solr-ref-guide/src/docker-faq.adoc@@ -19,9 +19,8 @@== How do I persist Solr data and config?Your data is persisted already, in your container's filesystem.-If you `docker run`, add data to Solr, then `docker stop` and later-`docker start`, then your data is still there. The same is true for-changes to configuration files.+If you `docker run`, add data to Solr, then `docker stop` and later `docker start`, then your data is still there.+The same is true for changes to configuration files.Equally, if you `docker commit` your container, you can later create a newcontainer from that image, and that will have your data in it.@@ -49,13 +48,10 @@ This is useful if you want to inspect or modify the data in the Docker hostwhen the container is not running, and later easily run new containers against that data.This is indeed possible, but there are a few gotchas.-Solr stores its core data in the `server/solr` directory, in sub-directories-for each core. The `server/solr` directory also contains configuration files-that are part of the Solr distribution.-Now, if we mounted volumes for each core individually, then that would-interfere with Solr trying to create those directories. If instead we make-the whole directory a volume, then we need to provide those configuration files-in our volume, which we can do by copying them from a temporary container.+Solr stores its core data in the `server/solr` directory, in sub-directories for each core.+The `server/solr` directory also contains configuration files that are part of the Solr distribution.+Now, if we mounted volumes for each core individually, then that would interfere with Solr trying to create those directories.+If instead we make the whole directory a volume, then we need to provide those configuration files in our volume, which we can do by copying them from a temporary container.For example:[source,bash]@@ -181,12 +177,14 @@ This is especially a problem for ZooKeeper 3.4.6; future versions are better atDocker 1.10 has a new `--ip` configuration option that allows you to specify an IP address for a container.It also has a `--ip-range` option that allows you to specify the range that other containers get addresses from.-Used together, you can implement static addresses. See the <<docker-networking.adoc#,Solr Docker networking guide>> for more information.+Used together, you can implement static addresses.+See the <<docker-networking.adoc#,Solr Docker networking guide>> for more information.== Can I run ZooKeeper and Solr with Docker Links?Docker's https://docs.docker.com/engine/userguide/networking/default_network/dockerlinks/[Legacy container links] provide a way to-pass connection configuration between containers. It only works on a single machine, on the default bridge.+pass connection configuration between containers.+It only works on a single machine, on the default bridge.It provides no facilities for static IPs.Note: this feature is expected to be deprecated and removed in a future release.So really, see the "Can I run ZooKeeper and Solr clusters under Docker?" option above instead.@@ -249,9 +247,8 @@ set by the `GC_TUNE` environment variable and further down the list the overridi== I'm confused about the different invocations of Solr -- help?-The different invocations of the Solr docker image can look confusing, because the name of the-image is "solr" and the Solr command is also "solr", and the image interprets various arguments in-special ways. I'll illustrate the various invocations:+The different invocations of the Solr docker image can look confusing, because the name of the image is "solr" and the Solr command is also "solr", and the image interprets various arguments in special ways.+I'll illustrate the various invocations:To run an arbitrary command in the image:@@ -270,8 +267,7 @@ To run the Solr server:docker run -it solr-----Here "solr" is the name of the image, and there is no specific command,-so the image defaults to run the "solr" command with "-f" to run it in the foreground.+Here "solr" is the name of the image, and there is no specific command, so the image defaults to run the "solr" command with "-f" to run it in the foreground.To run the Solr server with extra arguments:@@ -290,32 +286,31 @@ To run solr as an arbitrary command:docker run -it solr solr zk --help-----Here the first "solr" is the image name, and the second "solr"-is the "solr" command. The image runs the command exactly as specified;-no "-f" is implicitly added. The container will print help text, and exit.+Here the first "solr" is the image name, and the second "solr" is the "solr" command.+The image runs the command exactly as specified; no "-f" is implicitly added.+The container will print help text, and exit.-If you find this visually confusing, it might be helpful to use more specific image tags,-and specific command paths. For example:+If you find this visually confusing, it might be helpful to use more specific image tags, and specific command paths.+For example:[source,bash]----docker run -it solr bin/solr -f -h myhostname-----Finally, the Solr docker image offers several commands that do some work before-then invoking the Solr server, like "solr-precreate" and "solr-demo".+Finally, the Solr docker image offers several commands that do some work before then invoking the Solr server, like "solr-precreate" and "solr-demo".See the README.md for usage.-These are implemented by the `docker-entrypoint.sh` script, and must be passed-as the first argument to the image. For example:+These are implemented by the `docker-entrypoint.sh` script, and must be passed as the first argument to the image.+For example:[source,bash]----docker run -it solr solr-demo-----It's important to understand an implementation detail here. The Dockerfile uses-`solr-foreground` as the `CMD`, and the `docker-entrypoint.sh` implements-that by by running "solr -f". So these two are equivalent:+It's important to understand an implementation detail here.+The Dockerfile uses `solr-foreground` as the `CMD`, and the `docker-entrypoint.sh` implements that by by running "solr -f".+So these two are equivalent:[source,bash]----@@ -330,13 +325,11 @@ whereas:docker run -it solr solr -f-----is slightly different: the "solr" there is a generic command, not treated in any-special way by `docker-entrypoint.sh`. In particular, this means that the-`docker-entrypoint-initdb.d` mechanism is not applied.-So, if you want to use `docker-entrypoint-initdb.d`, then you must use one-of the other two invocations.-You also need to keep that in mind when you want to invoke solr from the bash-command. For example, this does NOT run `docker-entrypoint-initdb.d` scripts:+is slightly different: the "solr" there is a generic command, not treated in any special way by `docker-entrypoint.sh`.+In particular, this means that the `docker-entrypoint-initdb.d` mechanism is not applied.+So, if you want to use `docker-entrypoint-initdb.d`, then you must use one of the other two invocations.+You also need to keep that in mind when you want to invoke solr from the bash command.+For example, this does NOT run `docker-entrypoint-initdb.d` scripts:[source,bash]----diff --git a/solr/solr-ref-guide/src/docker-networking.adoc b/solr/solr-ref-guide/src/docker-networking.adocindex 608cfc69c56..f0639df5632 100644--- a/solr/solr-ref-guide/src/docker-networking.adoc+++ b/solr/solr-ref-guide/src/docker-networking.adoc@@ -30,10 +30,9 @@ NOTE: this example requires Docker 1.10.I'll run these commands from the first machine, trinity10.-Create a network named "netzksolr" for this cluster. The `--ip-range` specifies the range of-addresses to use for containers, whereas the `--subnet` specifies all possible addresses in this-network. So effectively, addresses in the subnet but outside the range are reserved for containers-that specifically use the `--ip` option.+Create a network named "netzksolr" for this cluster.+The `--ip-range` specifies the range of addresses to use for containers, whereas the `--subnet` specifies all possible addresses in this network.+So effectively, addresses in the subnet but outside the range are reserved for containers that specifically use the `--ip` option.[source,bash]----@@ -233,7 +232,8 @@ docker port zksolrproxy 8002Or use a suitably configured HAProxy to round-robin between all Solr nodes. Or, instead of the overlay network, use http://www.projectcalico.org[Project Calico] and configure L3 routing so you do not need to mess with proxies.-Now I can get to Solr on `http://trinity10:32774/solr/#/`. In the Cloud -> Tree -> /live_nodes view I see the Solr nodes.+Now I can get to Solr on `http://trinity10:32774/solr/#/`.+In the Cloud -> Tree -> /live_nodes view I see the Solr nodes.From the Solr UI select the collection1 core, and click on Cloud -> Graph to see how it has createdtwo shards across our Solr nodes.diff --git a/solr/solr-ref-guide/src/document-analysis.adoc b/solr/solr-ref-guide/src/document-analysis.adocindex 2f60f12860d..e24ccaaa507 100644--- a/solr/solr-ref-guide/src/document-analysis.adoc+++ b/solr/solr-ref-guide/src/document-analysis.adoc@@ -26,9 +26,13 @@The following sections describe how Solr breaks down and works with textual data.There are three main concepts to understand: analyzers, tokenizers, and filters.-* <<analyzers.adoc#,Field analyzers>> are used both during ingestion, when a document is indexed, and at query time. An analyzer examines the text of fields and generates a token stream. Analyzers may be a single class or they may be composed of a series of tokenizer and filter classes.+* <<analyzers.adoc#,Field analyzers>> are used both during ingestion, when a document is indexed, and at query time.+An analyzer examines the text of fields and generates a token stream.+Analyzers may be a single class or they may be composed of a series of tokenizer and filter classes.* <<tokenizers.adoc#,Tokenizers>> break field data into lexical units, or _tokens_.-* <<filters.adoc#,Filters>> examine a stream of tokens and keep them, transform or discard them, or create new ones. Tokenizers and filters may be combined to form pipelines, or _chains_, where the output of one is input to the next. Such a sequence of tokenizers and filters is called an _analyzer_ and the resulting output of an analyzer is used to match query results or build indices.+* <<filters.adoc#,Filters>> examine a stream of tokens and keep them, transform or discard them, or create new ones.+Tokenizers and filters may be combined to form pipelines, or _chains_, where the output of one is input to the next.+Such a sequence of tokenizers and filters is called an _analyzer_ and the resulting output of an analyzer is used to match query results or build indices.== Using Analyzers, Tokenizers, and Filtersdiff --git a/solr/solr-ref-guide/src/document-transformers.adoc b/solr/solr-ref-guide/src/document-transformers.adocindex 689fdf74dc0..c6d3a8e8354 100644--- a/solr/solr-ref-guide/src/document-transformers.adoc+++ b/solr/solr-ref-guide/src/document-transformers.adoc@@ -98,7 +98,8 @@ Augments each document with an inline explanation of its score exactly like theq=features:cache&fl=id,[explain style=nl]-----Supported values for `style` are `text`, `html`, and `nl` which returns the information as structured data. Here is the output of the above request using `style=nl`:+Supported values for `style` are `text`, `html`, and `nl` which returns the information as structured data.+Here is the output of the above request using `style=nl`:[source,json]----@@ -125,7 +126,8 @@ A default style can be configured by specifying an `args` parameter in your `sol=== [child] - ChildDocTransformerFactory-This transformer returns all <<indexing-nested-documents.adoc#,descendant documents>> of each parent document matching your query. This is useful when you have indexed nested child documents and want to retrieve the child documents for the relevant parent documents for any type of search query.+This transformer returns all <<indexing-nested-documents.adoc#,descendant documents>> of each parent document matching your query.+This is useful when you have indexed nested child documents and want to retrieve the child documents for the relevant parent documents for any type of search query.Note that this transformer can be used even when the query used to match the result documents is not a <<block-join-query-parser.adoc#,Block Join query>>.@@ -138,20 +140,25 @@ q=book_title:Solr&fl=id,[child childFilter=doc_type:chapter limit=100]If the documents involved include a `\_nest_path_` field, then it is used to re-create the hierarchical structure of the descendent documents using the original pseudo-field names the documents were indexed with, otherwise the descendent documents are returned as a flat list of <<indexing-nested-documents#indexing-anonymous-children,anonymous children>>.`childFilter`::-A query to filter which child documents should be included. This can be particularly useful when you have multiple levels of hierarchical documents. The default is all children.+A query to filter which child documents should be included.+This can be particularly useful when you have multiple levels of hierarchical documents.+The default is all children.`limit`::-The maximum number of child documents to be returned per parent document. The default is `10`.+The maximum number of child documents to be returned per parent document.+The default is `10`.`fl`::-The field list which the transformer is to return. The default is the top level `fl`).+The field list which the transformer is to return.+The default is the top level `fl`).+There is a further limitation in which the fields here should be a subset of those specified by the top level `fl` parameter.`parentFilter`::Serves the same purpose as the `of`/`which` params in `{!child}`/`{!parent}` query parsers: toidentify the set of "all parents" for the purpose of identifying the beginning & end of each-nested document block. This recently became fully optional and appears to be obsolete.+nested document block.+This recently became fully optional and appears to be obsolete.It is likely to be removed in a future Solr release, so _if you find it has some use, let theproject know!_@@ -229,7 +236,8 @@ fl=id,[elevated],[excluded]&excludeIds=GB18030TEST&elevateIds=6H500F0&markExclud=== [json] / [xml]-These transformers replace a field value containing a string representation of a valid XML or JSON structure with the actual raw XML or JSON structure instead of just the string value. Each applies only to the specific writer, such that `[json]` only applies to `wt=json` and `[xml]` only applies to `wt=xml`.+These transformers replace a field value containing a string representation of a valid XML or JSON structure with the actual raw XML or JSON structure instead of just the string value.+Each applies only to the specific writer, such that `[json]` only applies to `wt=json` and `[xml]` only applies to `wt=xml`.[source,plain]----@@ -239,12 +247,14 @@ fl=id,source_s:[json]&wt=json=== [subquery]-This transformer executes a separate query per transforming document passing document fields as an input for subquery parameters. It's usually used with `{!join}` and `{!parent}` query parsers, and is intended to be an improvement for `[child]`.+This transformer executes a separate query per transforming document passing document fields as an input for subquery parameters.+It's usually used with `{!join}` and `{!parent}` query parsers, and is intended to be an improvement for `[child]`.* It must be given an unique name: `fl=*,children:[subquery]`* There might be a few of them, e.g., `fl=*,sons:[subquery],daughters:[subquery]`.* Every `[subquery]` occurrence adds a field into a result document with the given name, the value of this field is a document list, which is a result of executing subquery using document fields as an input.-* Subquery will use the `/select` search handler by default, and will return an error if `/select` is not configured. This can be changed by supplying `foo.qt` parameter.+* Subquery will use the `/select` search handler by default, and will return an error if `/select` is not configured.+This can be changed by supplying `foo.qt` parameter.Here is how it looks like using various formats:@@ -299,7 +309,8 @@ Here is how it looks like using various formats:To appear in subquery document list, a field should be specified in both `fl` parameters: in the main `fl` (despite the main result documents have no this field), and in subquery's `fl` (e.g., `foo.fl`).-Wildcards can be used in one or both of these parameters. For example, if field `title` should appear in categories subquery, it can be done via one of these ways:+Wildcards can be used in one or both of these parameters.+For example, if field `title` should appear in categories subquery, it can be done via one of these ways:[source,plain]----@@ -311,21 +322,25 @@ fl=...*,categories:[subquery]&categories.fl=*&categories.q=...==== Subquery Parameters Shift-If a subquery is declared as `fl=*,foo:[subquery]`, subquery parameters are prefixed with the given name and period. For example:+If a subquery is declared as `fl=*,foo:[subquery]`, subquery parameters are prefixed with the given name and period.+For example:[source,plain]q=*:*&fl=*,**foo**:[subquery]&**foo.**q=to be continued&**foo.**rows=10&**foo.**sort=id desc==== Document Field as an Input for Subquery Parameters-It's necessary to pass some document field values as a parameter for subquery. It's supported via an implicit *`row.__fieldname__`* parameter, and can be (but might not only) referred via local params syntax:+It's necessary to pass some document field values as a parameter for subquery.+It's supported via an implicit *`row.__fieldname__`* parameter, and can be (but might not only) referred via local params syntax:[source,plain,subs="quotes"]q=name:john&fl=name,id,depts:[subquery]&depts.q={!terms f=id **v=$row.dept_id**}&depts.rows=10-Here departments are retrieved per every employee in search result. We can say that it's like SQL `join ON emp.dept_id=dept.id`.+Here departments are retrieved per every employee in search result.+We can say that it's like SQL `join ON emp.dept_id=dept.id`.-Note, when a document field has multiple values they are concatenated with a comma by default. This can be changed with the local parameter `foo:[subquery separator=' ']`, this mimics *`{!terms}`* to work smoothly with it.+Note, when a document field has multiple values they are concatenated with a comma by default.+This can be changed with the local parameter `foo:[subquery separator=' ']`, this mimics *`{!terms}`* to work smoothly with it.To log substituted subquery request parameters, add the corresponding parameter names, as in: `depts.logParamsList=q,fl,rows,**row.dept_id**`@@ -351,11 +366,18 @@ Otherwise you'll get `NullPointerException` from `QueryComponent.mergeIds`.=== [geo] - Geospatial formatter-Formats spatial data from a spatial field using a designated format type name. Two inner parameters are required: `f` for the field name, and `w` for the format name. Example: `geojson:[geo f=mySpatialField w=GeoJSON]`.+Formats spatial data from a spatial field using a designated format type name.+Two inner parameters are required: `f` for the field name, and `w` for the format name.+Example: `geojson:[geo f=mySpatialField w=GeoJSON]`.-Normally you'll simply be consistent in choosing the format type you want by setting the `format` attribute on the spatial field type to `WKT` or `GeoJSON` – see the section <<spatial-search.adoc#,Spatial Search>> for more information. If you are consistent, it'll come out the way you stored it. This transformer offers a convenience to transform the spatial format to something different on retrieval.+Normally you'll simply be consistent in choosing the format type you want by setting the `format` attribute on the spatial field type to `WKT` or `GeoJSON` – see the section <<spatial-search.adoc#,Spatial Search>> for more information.+If you are consistent, it'll come out the way you stored it.+This transformer offers a convenience to transform the spatial format to something different on retrieval.-In addition, this feature is very useful with the `RptWithGeometrySpatialField` to avoid double-storage of the potentially large vector geometry. This transformer will detect that field type and fetch the geometry from an internal compact binary representation on disk (in docValues), and then format it as desired. As such, you needn't mark the field as stored, which would be redundant. In a sense this double-storage between docValues and stored-value storage isn't unique to spatial but with polygonal geometry it can be a lot of data, and furthermore you'd like to avoid storing it in a verbose format (like GeoJSON or WKT).+In addition, this feature is very useful with the `RptWithGeometrySpatialField` to avoid double-storage of the potentially large vector geometry.+This transformer will detect that field type and fetch the geometry from an internal compact binary representation on disk (in docValues), and then format it as desired.+As such, you needn't mark the field as stored, which would be redundant.+In a sense this double-storage between docValues and stored-value storage isn't unique to spatial but with polygonal geometry it can be a lot of data, and furthermore you'd like to avoid storing it in a verbose format (like GeoJSON or WKT).=== [features] - LTRFeatureLoggerTransformerFactorydiff --git a/solr/solr-ref-guide/src/documents-fields-schema-design.adoc b/solr/solr-ref-guide/src/documents-fields-schema-design.adocindex 33f5ac56dde..b105683ea0d 100644--- a/solr/solr-ref-guide/src/documents-fields-schema-design.adoc+++ b/solr/solr-ref-guide/src/documents-fields-schema-design.adoc@@ -44,7 +44,8 @@ A recipe document would contain the ingredients, the instructions, the preparatiA document about a person, for example, might contain the person's name, biography, favorite color, and shoe size.A document about a book could contain the title, author, year of publication, number of pages, and so on.-In the Solr universe, documents are composed of _fields_, which are more specific pieces of information. Shoe size could be a field.+In the Solr universe, documents are composed of _fields_, which are more specific pieces of information.+Shoe size could be a field.First name and last name could be fields.Fields can contain different kinds of data.@@ -94,6 +95,6 @@ How will fields from documents be displayed to users?If you aren't sure yet, plan on some test indexing runs to see how the data in your documents is indexed with default settings.Build into your implementation plan some time for iteration and start small.-The more you're able to define your schema before indexing all of your documents, the higher your chances for a successful search application for your users.+The more you're able to define your schema before indexing all of your documents, the higher your chances for a successful search application for your users.More information about the schema is in the section <<schema-elements.adoc#,Schema Elements>>.diff --git a/solr/solr-ref-guide/src/documents-screen.adoc b/solr/solr-ref-guide/src/documents-screen.adocindex 15fe607131d..fa527e14bc7 100644--- a/solr/solr-ref-guide/src/documents-screen.adoc+++ b/solr/solr-ref-guide/src/documents-screen.adoc@@ -36,11 +36,18 @@ There are other ways to load data, see also these sections:====== Common Fields-* Request-Handler: The first step is to define the RequestHandler. By default `/update` will be defined. Change the request handler to `/update/extract` to use Solr Cell.-* Document Type: Select the Document Type to define the format of document to load. The remaining parameters may change depending on the document type selected.-* Document(s): Enter a properly-formatted Solr document corresponding to the `Document Type` selected. XML and JSON documents must be formatted in a Solr-specific format, a small illustrative document will be shown. CSV files should have headers corresponding to fields defined in the schema. More details can be found at: <<indexing-with-update-handlers.adoc#,Indexing with Update Handlers>>.+* Request-Handler: The first step is to define the RequestHandler.+By default `/update` will be defined.+Change the request handler to `/update/extract` to use Solr Cell.+* Document Type: Select the Document Type to define the format of document to load.+The remaining parameters may change depending on the document type selected.+* Document(s): Enter a properly-formatted Solr document corresponding to the `Document Type` selected.+XML and JSON documents must be formatted in a Solr-specific format, a small illustrative document will be shown.+CSV files should have headers corresponding to fields defined in the schema.+More details can be found at: <<indexing-with-update-handlers.adoc#,Indexing with Update Handlers>>.* Commit Within: Specify the number of milliseconds between the time the document is submitted and when it is available for searching.-* Overwrite: If `true` the new document will replace an existing document with the same value in the `id` field. If `false` multiple documents with the same id can be added.+* Overwrite: If `true` the new document will replace an existing document with the same value in the `id` field.+If `false` multiple documents with the same id can be added.[TIP]====@@ -49,7 +56,9 @@ Setting `Overwrite` to `false` is very rare in production situations, the defaul== CSV, JSON and XML Documents-When using these document types the functionality is similar to submitting documents via `curl` or similar. The document structure must be in a Solr-specific format appropriate for the document type. Examples are illustrated in the Document(s) text box when you select the various types.+When using these document types the functionality is similar to submitting documents via `curl` or similar.+The document structure must be in a Solr-specific format appropriate for the document type.+Examples are illustrated in the Document(s) text box when you select the various types.These options will only add or overwrite documents; for other update tasks, see the <<Solr Command>> option.@@ -59,14 +68,18 @@ The Document Builder provides a wizard-like interface to enter fields of a docum== File Upload-The File Upload option allows choosing a prepared file and uploading it. If using `/update` for the Request-Handler option, you will be limited to XML, CSV, and JSON.+The File Upload option allows choosing a prepared file and uploading it.+If using `/update` for the Request-Handler option, you will be limited to XML, CSV, and JSON.-Other document types (e.g., Word, PDF, etc.) can be indexed using the ExtractingRequestHandler (aka, Solr Cell). You must modify the RequestHandler to `/update/extract`, which must be defined in your `solrconfig.xml` file with your desired defaults. You should also add `&literal.id` shown in the "Extracting Request Handler Params" field so the file chosen is given a unique id.+Other document types (e.g., Word, PDF, etc.) can be indexed using the ExtractingRequestHandler (aka, Solr Cell).+You must modify the RequestHandler to `/update/extract`, which must be defined in your `solrconfig.xml` file with your desired defaults.+You should also add `&literal.id` shown in the "Extracting Request Handler Params" field so the file chosen is given a unique id.More information can be found at: <<indexing-with-tika.adoc#,Indexing with Solr Cell and Apache Tika>>.== Solr Command-The Solr Command option allows you use the `/update` request handler with XML or JSON formatted commands to perform specific actions. A few examples are:+The Solr Command option allows you use the `/update` request handler with XML or JSON formatted commands to perform specific actions.+A few examples are:* Deleting documents* Updating only certain fields of documentsdiff --git a/solr/solr-ref-guide/src/docvalues.adoc b/solr/solr-ref-guide/src/docvalues.adocindex ecf4a6fe9e6..e46d1c1a574 100644--- a/solr/solr-ref-guide/src/docvalues.adoc+++ b/solr/solr-ref-guide/src/docvalues.adoc@@ -20,15 +20,22 @@ DocValues are a way of recording field values internally that is more efficient== Why DocValues?-The standard way that Solr builds the index is with an _inverted index_. This style builds a list of terms found in all the documents in the index and next to each term is a list of documents that the term appears in (as well as how many times the term appears in that document). This makes search very fast - since users search by terms, having a ready list of term-to-document values makes the query process faster.+The standard way that Solr builds the index is with an _inverted index_. This style builds a list of terms found in all the documents in the index and next to each term is a list of documents that the term appears in (as well as how many times the term appears in that document).+This makes search very fast - since users search by terms, having a ready list of term-to-document values makes the query process faster.-For other features that we now commonly associate with search, such as sorting, faceting, and highlighting, this approach is not very efficient. The faceting engine, for example, must look up each term that appears in each document that will make up the result set and pull the document IDs in order to build the facet list. In Solr, this is maintained in memory, and can be slow to load (depending on the number of documents, terms, etc.).+For other features that we now commonly associate with search, such as sorting, faceting, and highlighting, this approach is not very efficient.+The faceting engine, for example, must look up each term that appears in each document that will make up the result set and pull the document IDs in order to build the facet list.+In Solr, this is maintained in memory, and can be slow to load (depending on the number of documents, terms, etc.).-In Lucene 4.0, a new approach was introduced. DocValue fields are now column-oriented fields with a document-to-value mapping built at index time. This approach promises to relieve some of the memory requirements of the fieldCache and make lookups for faceting, sorting, and grouping much faster.+In Lucene 4.0, a new approach was introduced.+DocValue fields are now column-oriented fields with a document-to-value mapping built at index time.+This approach promises to relieve some of the memory requirements of the fieldCache and make lookups for faceting, sorting, and grouping much faster.== Enabling DocValues-To use docValues, you only need to enable it for a field that you will use it with. As with all schema design, you need to define a field type and then define fields of that type with docValues enabled. All of these actions are done in `schema.xml`.+To use docValues, you only need to enable it for a field that you will use it with.+As with all schema design, you need to define a field type and then define fields of that type with docValues enabled.+All of these actions are done in `schema.xml`.Enabling a field for docValues only requires adding `docValues="true"` to the field (or field type) definition, as in this example from the `schema.xml` of Solr's `sample_techproducts_configs` <<config-sets.adoc#,configset>>:@@ -40,24 +47,31 @@ Enabling a field for docValues only requires adding `docValues="true"` to the fi[IMPORTANT]If you have already indexed data into your Solr index, you will need to completely reindex your content after changing your field definitions in `schema.xml` in order to successfully use docValues.-DocValues are only available for specific field types. The types chosen determine the underlying Lucene docValue type that will be used. The available Solr field types are:+DocValues are only available for specific field types.+The types chosen determine the underlying Lucene docValue type that will be used.+The available Solr field types are:* `StrField`, and `UUIDField`:** If the field is single-valued (i.e., multi-valued is false), Lucene will use the `SORTED` type.-** If the field is multi-valued, Lucene will use the `SORTED_SET` type. Entries are kept in sorted order and duplicates are removed.+** If the field is multi-valued, Lucene will use the `SORTED_SET` type.+Entries are kept in sorted order and duplicates are removed.* `BoolField`:** If the field is single-valued (i.e., multi-valued is false), Lucene will use the `SORTED` type.-** If the field is multi-valued, Lucene will use the `SORTED_SET` type. Entries are kept in sorted order and duplicates are removed.+** If the field is multi-valued, Lucene will use the `SORTED_SET` type.+Entries are kept in sorted order and duplicates are removed.* Any `*PointField` Numeric or Date fields, `EnumFieldType`, and `CurrencyFieldType`:** If the field is single-valued (i.e., multi-valued is false), Lucene will use the `NUMERIC` type.-** If the field is multi-valued, Lucene will use the `SORTED_NUMERIC` type. Entries are kept in sorted order and duplicates are kept.+** If the field is multi-valued, Lucene will use the `SORTED_NUMERIC` type.+Entries are kept in sorted order and duplicates are kept.* Any of the deprecated `Trie*` Numeric or Date fields, `EnumField` and `CurrencyField`:** If the field is single-valued (i.e., multi-valued is false), Lucene will use the `NUMERIC` type.-** If the field is multi-valued, Lucene will use the `SORTED_SET` type. Entries are kept in sorted order and duplicates are removed.+** If the field is multi-valued, Lucene will use the `SORTED_SET` type.+Entries are kept in sorted order and duplicates are removed.These Lucene types are related to how the {lucene-javadocs}/core/org/apache/lucene/index/DocValuesType.html[values are sorted and stored].-There is an additional configuration option available, which is to modify the `docValuesFormat` <<field-type-definitions-and-properties.adoc#docvaluesformat,used by the field type>>. The default implementation employs a mixture of loading some things into memory and keeping some on disk. In some cases, however, you may choose to specify an alternative {lucene-javadocs}/core/org/apache/lucene/codecs/DocValuesFormat.html[DocValuesFormat implementation]. For example, you could choose to keep everything in memory by specifying `docValuesFormat="Direct"` on a field type:+There is an additional configuration option available, which is to modify the `docValuesFormat` <<field-type-definitions-and-properties.adoc#docvaluesformat,used by the field type>>. The default implementation employs a mixture of loading some things into memory and keeping some on disk.+In some cases, however, you may choose to specify an alternative {lucene-javadocs}/core/org/apache/lucene/codecs/DocValuesFormat.html[DocValuesFormat implementation]. For example, you could choose to keep everything in memory by specifying `docValuesFormat="Direct"` on a field type:[source,xml]----@@ -67,7 +81,8 @@ There is an additional configuration option available, which is to modify the `dPlease note that the `docValuesFormat` option may change in future releases.[NOTE]-Lucene index back-compatibility is only supported for the default codec. If you choose to customize the `docValuesFormat` in your `schema.xml`, upgrading to a future version of Solr may require you to either switch back to the default codec and optimize your index to rewrite it into the default codec before upgrading, or re-build your entire index from scratch after upgrading.+Lucene index back-compatibility is only supported for the default codec.+If you choose to customize the `docValuesFormat` in your `schema.xml`, upgrading to a future version of Solr may require you to either switch back to the default codec and optimize your index to rewrite it into the default codec before upgrading, or re-build your entire index from scratch after upgrading.== Using DocValues@@ -78,7 +93,8 @@ If `docValues="true"` for a field, then DocValues will automatically be used any=== Retrieving DocValues During SearchField values retrieved during search queries are typically returned from stored values.-However, non-stored docValues fields will be also returned along with other stored fields when all fields (or pattern matching globs) are specified to be returned (e.g., "`fl=*`") for search queries depending on the effective value of the `useDocValuesAsStored` parameter for each field. For schema versions >= 1.6, the implicit default is `useDocValuesAsStored="true"`.+However, non-stored docValues fields will be also returned along with other stored fields when all fields (or pattern matching globs) are specified to be returned (e.g., "`fl=*`") for search queries depending on the effective value of the `useDocValuesAsStored` parameter for each field.+For schema versions >= 1.6, the implicit default is `useDocValuesAsStored="true"`.See <<field-type-definitions-and-properties.adoc#,Field Type Definitions and Properties>> & <<fields.adoc#,Fields>> for more details.When `useDocValuesAsStored="false"`, non-stored DocValues fields can still be explicitly requested by name in the <<common-query-parameters.adoc#fl-field-list-parameter,`fl` parameter>>, but will not match glob patterns (`"*"`).@@ -92,5 +108,8 @@ In cases where the query is returning _only_ docValues fields performance may imWhen retrieving fields from their docValues form (such as when using the <<exporting-result-sets.adoc#,/export handler>>, <<streaming-expressions.adoc#,streaming expressions>> or if the field is requested in the `fl` parameter), two important differences between regular stored fields and docValues fields must be understood:-1. Order is _not_ preserved. When retrieving stored fields, the insertion order is the return order. For docValues, it is the _sorted_ order.-2. For field types using `SORTED_SET` (see above), multiple identical entries are collapsed into a single value. Thus if values 4, 5, 2, 4, 1 are inserted, the values returned will be 1, 2, 4, 5.+. Order is _not_ preserved.+When retrieving stored fields, the insertion order is the return order.+For docValues, it is the _sorted_ order.+. For field types using `SORTED_SET` (see above), multiple identical entries are collapsed into a single value.+Thus if values 4, 5, 2, 4, 1 are inserted, the values returned will be 1, 2, 4, 5.diff --git a/solr/solr-ref-guide/src/dsp.adoc b/solr/solr-ref-guide/src/dsp.adocindex 50923e84aad..54775c0da29 100644--- a/solr/solr-ref-guide/src/dsp.adoc+++ b/solr/solr-ref-guide/src/dsp.adoc@@ -16,107 +16,101 @@// specific language governing permissions and limitations// under the License.-This section of the user guide explores functions that are commonly used in the field of-Digital Signal Processing (DSP).+This section of the user guide explores functions that are commonly used in the field of Digital Signal Processing (DSP).== Convolution-The `conv` function calculates the convolution of two vectors. The convolution is calculated by *reversing*-the second vector and sliding it across the first vector. The dot product of the two vectors-is calculated at each point as the second vector is slid across the first vector.+The `conv` function calculates the convolution of two vectors.+The convolution is calculated by *reversing* the second vector and sliding it across the first vector.+The dot product of the two vectors is calculated at each point as the second vector is slid across the first vector.The dot products are collected in a third vector which is the convolution of the two vectors.=== Moving Average Function-Before looking at an example of convolution it's useful to review the `movingAvg` function. The moving average-function computes a moving average by sliding a window across a vector and computing-the average of the window at each shift. If that sounds similar to convolution, that's because the `movingAvg`-function involves a sliding window approach similar to convolution.+Before looking at an example of convolution it's useful to review the `movingAvg` function.+The moving average function computes a moving average by sliding a window across a vector and computing the average of the window at each shift.+If that sounds similar to convolution, that's because the `movingAvg` function involves a sliding window approach similar to convolution.-Below is an example of a moving average with a window size of 5. Notice that the original vector has 13 elements-but the result of the moving average has only 9 elements. This is because the `movingAvg` function-only begins generating results when it has a full window. The `ltrim` function is used to trim the-first four elements from the original `y` array to line up with the moving average.+Below is an example of a moving average with a window size of 5.+Notice that the original vector has 13 elements but the result of the moving average has only 9 elements.+This is because the `movingAvg` function only begins generating results when it has a full window.+The `ltrim` function is used to trim the first four elements from the original `y` array to line up with the moving average.image::images/math-expressions/conv1.png[]=== Convolutional Smoothing-The moving average can also be computed using convolution. In the example-below the `conv` function is used to compute the moving average of the first array-by applying the second array as a filter.+The moving average can also be computed using convolution.+In the example below the `conv` function is used to compute the moving average of the first array by applying the second array as a filter.-Looking at the result, we see that the convolution produced an array with 17 values instead of the 9 values created by the-moving average. That is because the `conv` function pads zeros-to the front and back of the first vector so that the window size is always full.+Looking at the result, we see that the convolution produced an array with 17 values instead of the 9 values created by the moving average.+That is because the `conv` function pads zeros to the front and back of the first vector so that the window size is always full.image::images/math-expressions/conv2.png[]-We achieve the same result as the `movingAvg` function by trimming the first and last 4 values of-the convolution result using the `ltrim` and `rtrim` functions.+We achieve the same result as the `movingAvg` function by trimming the first and last 4 values of the convolution result using the `ltrim` and `rtrim` functions.-The example below plots both the trimmed convolution and the moving average on the same plot. Notice that-they perfectly overlap.+The example below plots both the trimmed convolution and the moving average on the same plot.+Notice that they perfectly overlap.image::images/math-expressions/conv3.png[]-This demonstrates how convolution can be used to smooth a signal by sliding a filter across the signal and-computing the dot product at each point. The smoothing effect is caused by the design of the filter.-In the example, the filter length is 5 and each value in the filter is .2. This filter calculates a-simple moving average with a window size of 5.+This demonstrates how convolution can be used to smooth a signal by sliding a filter across the signal and computing the dot product at each point.+The smoothing effect is caused by the design of the filter.+In the example, the filter length is 5 and each value in the filter is .2.+This filter calculates a simple moving average with a window size of 5.-The formula for computing a simple moving average using convolution is to make the filter length the window-size and make the values of the filter all the same and sum to 1. A moving average with a window size of 4-can be computed by changing the filter to a length of 4 with each value being .25.+The formula for computing a simple moving average using convolution is to make the filter length the window size and make the values of the filter all the same and sum to 1.+A moving average with a window size of 4 can be computed by changing the filter to a length of 4 with each value being .25.==== Changing the Weights-The filter, which is sometimes called the *kernel*, can be viewed as a vector of weights. In the initial-example all values in the filter have the same weight (.2). The weights in the filter can be changed to-produce different smoothing effects. This is demonstrated in the example below.+The filter, which is sometimes called the *kernel*, can be viewed as a vector of weights.+In the initial example all values in the filter have the same weight (.2).+The weights in the filter can be changed to produce different smoothing effects.+This is demonstrated in the example below.-In this example the filter increases in weight from .1 to .3. This places more weight towards the front-of the filter. Notice that the filter is reversed with the `rev` function before the `conv` function applies it.-This is done because convolution will reverse-the filter. In this case we reverse it ahead of time and when convolution reverses it back, it is the same-as the original filter.+In this example the filter increases in weight from .1 to .3.+This places more weight towards the front of the filter.+Notice that the filter is reversed with the `rev` function before the `conv` function applies it.+This is done because convolution will reverse the filter.+In this case we reverse it ahead of time and when convolution reverses it back, it is the same as the original filter.-The plot shows the effect of the different weights in the filter. The dark blue line is the initial array.-The light blue line is the convolution and the orange line is the moving average. Notice that the convolution-responds quicker to the movements in the underlying array. This is because more weight has been placed-at the front of the filter.+The plot shows the effect of the different weights in the filter.+The dark blue line is the initial array.+The light blue line is the convolution and the orange line is the moving average.+Notice that the convolution responds quicker to the movements in the underlying array.+This is because more weight has been placed at the front of the filter.image::images/math-expressions/conv4.png[]--== Cross-Correlation-Cross-correlation is used to determine the delay between two signals. This is accomplished by sliding one signal across another-and calculating the dot product at each shift. The dot products are collected into a vector which represents the correlation-at each shift. The highest dot product in the cross-correlation vector is the point where the two signals are most closely correlated.+Cross-correlation is used to determine the delay between two signals.+This is accomplished by sliding one signal across another and calculating the dot product at each shift.+The dot products are collected into a vector which represents the correlation at each shift.+The highest dot product in the cross-correlation vector is the point where the two signals are most closely correlated.-The sliding dot product used in convolution can also be used to represent cross-correlation between two vectors. The only-difference in the formula when representing correlation is that the second vector is *not reversed*.+The sliding dot product used in convolution can also be used to represent cross-correlation between two vectors.+The only difference in the formula when representing correlation is that the second vector is *not reversed*.Notice in the example below that the second vector is reversed by the `rev` function before it is operated on by the `conv` function.-The `conv` function reverses the second vector so it will be flipped back to its original order to perform the correlation calculation-rather than the convolution calculation.+The `conv` function reverses the second vector so it will be flipped back to its original order to perform the correlation calculation rather than the convolution calculation.-Notice in the result the highest value is 217. This is the point where the two vectors have the highest correlation.+Notice in the result the highest value is 217.+This is the point where the two vectors have the highest correlation.image::images/math-expressions/crosscorr.png[]== Find Delay-It is fairly simple to compute the delay from the cross-correlation result, but a convenience function called `finddelay` can-be used to find the delay directly. Under the covers `finddelay` uses convolutional math to compute the cross-correlation vector-and then computes the delay between the two signals.+It is fairly simple to compute the delay from the cross-correlation result, but a convenience function called `finddelay` can be used to find the delay directly.+Under the covers `finddelay` uses convolutional math to compute the cross-correlation vector and then computes the delay between the two signals.-Below is an example of the `finddelay` function. Notice that the `finddelay` function reports a 3 period delay between the first-and second signal.+Below is an example of the `finddelay` function.+Notice that the `finddelay` function reports a 3 period delay between the first and second signal.image::images/math-expressions/delay.png[]@@ -146,132 +140,125 @@ image::images/math-expressions/sinewave256.png[]== Autocorrelation-Autocorrelation measures the degree to which a signal is correlated with itself. Autocorrelation is used to determine-if a vector contains a signal or is purely random.+Autocorrelation measures the degree to which a signal is correlated with itself.+Autocorrelation is used to determine if a vector contains a signal or is purely random.A few examples, with plots, will help to understand the concepts.-The first example simply revisits the example above of an extrapolated sine wave. The result of this-is plotted in the image below. Notice that there is a structure to the plot that is clearly not random.+The first example simply revisits the example above of an extrapolated sine wave.+The result of this is plotted in the image below.+Notice that there is a structure to the plot that is clearly not random.image::images/math-expressions/sinewave256.png[]-In the next example the `sample` function is used to draw 256 samples from a `uniformDistribution` to create a-vector of random data. The result of this is plotted in the image below. Notice that there is no clear structure to the-data and the data appears to be random.+In the next example the `sample` function is used to draw 256 samples from a `uniformDistribution` to create a vector of random data.+The result of this is plotted in the image below.+Notice that there is no clear structure to the data and the data appears to be random.image::images/math-expressions/noise.png[]-In the next example the random noise is added to the sine wave using the `ebeAdd` function.-The result of this is plotted in the image below. Notice that the sine wave has been hidden-somewhat within the noise. Its difficult to say for sure if there is structure. As plots-becomes more dense it can become harder to see a pattern hidden within noise.-+The result of this is plotted in the image below.+Notice that the sine wave has been hidden somewhat within the noise.+Its difficult to say for sure if there is structure.+As plots becomes more dense it can become harder to see a pattern hidden within noise.image::images/math-expressions/hidden-signal.png[]--In the next examples autocorrelation is performed with each of the vectors shown above to see what the-autocorrelation plots look like.+In the next examples autocorrelation is performed with each of the vectors shown above to see what the autocorrelation plots look like.In the example below the `conv` function is used to autocorrelate the first vector which is the sine wave.Notice that the `conv` function is simply correlating the sine wave with itself.-The plot has a very distinct structure to it. As the sine wave is slid across a copy of itself the correlation-moves up and down in increasing intensity until it reaches a peak. This peak is directly in the center and is the-the point where the sine waves are directly lined up. Following the peak the correlation moves up and down in decreasing-intensity as the sine wave slides farther away from being directly lined up.+The plot has a very distinct structure to it.+As the sine wave is slid across a copy of itself the correlation moves up and down in increasing intensity until it reaches a peak.+This peak is directly in the center and is the the point where the sine waves are directly lined up.+Following the peak the correlation moves up and down in decreasing intensity as the sine wave slides farther away from being directly lined up.This is the autocorrelation plot of a pure signal.-image::images/math-expressions/signal-autocorrelation.png[]--In the example below autocorrelation is performed with the vector of pure noise. Notice that the autocorrelation-plot has a very different plot then the sine wave. In this plot there is long period of low intensity correlation that appears-to be random. Then in the center a peak of high intensity correlation where the vectors are directly lined up.+In the example below autocorrelation is performed with the vector of pure noise.+Notice that the autocorrelation plot has a very different plot then the sine wave.+In this plot there is long period of low intensity correlation that appears to be random.+Then in the center a peak of high intensity correlation where the vectors are directly lined up.This is followed by another long period of low intensity correlation.This is the autocorrelation plot of pure noise.-image::images/math-expressions/noise-autocorrelation.png[]-In the example below autocorrelation is performed on the vector with the sine wave hidden within the noise.-Notice that this plot shows very clear signs of structure which is similar to autocorrelation plot of the-pure signal. The correlation is less intense due to noise but the shape of the correlation plot suggests-strongly that there is an underlying signal hidden within the noise.-+Notice that this plot shows very clear signs of structure which is similar to autocorrelation plot of the pure signal.+The correlation is less intense due to noise but the shape of the correlation plot suggests strongly that there is an underlying signal hidden within the noise.image::images/math-expressions/hidden-signal-autocorrelation.png[]-== Discrete Fourier Transform-The convolution-based functions described above are operating on signals in the time domain. In the time-domain the x-axis is time and the y-axis is the quantity of some value at a specific point in time.+The convolution-based functions described above are operating on signals in the time domain.+In the time domain the x-axis is time and the y-axis is the quantity of some value at a specific point in time.The discrete Fourier Transform translates a time domain signal into the frequency domain.In the frequency domain the x-axis is frequency, and y-axis is the accumulated power at a specific frequency.-The basic principle is that every time domain signal is composed of one or more signals (sine waves)-at different frequencies. The discrete Fourier transform decomposes a time domain signal into its component-frequencies and measures the power at each frequency.+The basic principle is that every time domain signal is composed of one or more signals (sine waves) at different frequencies.+The discrete Fourier transform decomposes a time domain signal into its component frequencies and measures the power at each frequency.-The discrete Fourier transform has many important uses. In the example below, the discrete Fourier transform is used-to determine if a signal has structure or if it is purely random.+The discrete Fourier transform has many important uses.+In the example below, the discrete Fourier transform is used to determine if a signal has structure or if it is purely random.=== Complex Result-The `fft` function performs the discrete Fourier Transform on a vector of *real* data. The result-of the `fft` function is returned as *complex* numbers. A complex number has two parts, *real* and *imaginary*.+The `fft` function performs the discrete Fourier Transform on a vector of *real* data.+The result of the `fft` function is returned as *complex* numbers.+A complex number has two parts, *real* and *imaginary*.The *real* part of the result describes the magnitude of the signal at different frequencies.-The *imaginary* part of the result describes the *phase*. The examples below deal only with the *real*-part of the result.+The *imaginary* part of the result describes the *phase*.+The examples below deal only with the *real* part of the result.-The `fft` function returns a `matrix` with two rows. The first row in the matrix is the *real*-part of the complex result. The second row in the matrix is the *imaginary* part of the complex result.+The `fft` function returns a `matrix` with two rows.+The first row in the matrix is the *real* part of the complex result.+The second row in the matrix is the *imaginary* part of the complex result.The `rowAt` function can be used to access the rows so they can be processed as vectors.-=== Fast Fourier Transform ExamplesIn the first example the `fft` function is called on the sine wave used in the autocorrelation example.-The results of the `fft` function is a matrix. The `rowAt` function is used to return the first row of-the matrix which is a vector containing the real values of the `fft` response.--The plot of the real values of the `fft` response is shown below. Notice there are two-peaks on opposite sides of the plot. The plot is actually showing a mirrored response. The right side-of the plot is an exact mirror of the left side. This is expected when the `fft` is run on real rather than-complex data.+The results of the `fft` function is a matrix.+The `rowAt` function is used to return the first row of the matrix which is a vector containing the real values of the `fft` response.-Also notice that the `fft` has accumulated significant power in a single peak. This is the power associated with-the specific frequency of the sine wave. The vast majority of frequencies in the plot have close to 0 power-associated with them. This `fft` shows a clear signal with very low levels of noise.+The plot of the real values of the `fft` response is shown below.+Notice there are two peaks on opposite sides of the plot.+The plot is actually showing a mirrored response.+The right side of the plot is an exact mirror of the left side.+This is expected when the `fft` is run on real rather than complex data.+Also notice that the `fft` has accumulated significant power in a single peak.+This is the power associated with the specific frequency of the sine wave.+The vast majority of frequencies in the plot have close to 0 power associated with them.+This `fft` shows a clear signal with very low levels of noise.image::images/math-expressions/signal-fft.png[]-In the second example the `fft` function is called on a vector of random data similar to one used in the-autocorrelation example. The plot of the real values of the `fft` response is shown below.+In the second example the `fft` function is called on a vector of random data similar to one used in the autocorrelation example.+The plot of the real values of the `fft` response is shown below.-Notice that in is this response there is no clear peak. Instead all frequencies have accumulated a random level of-power. This `fft` shows no clear sign of signal and appears to be noise.+Notice that in is this response there is no clear peak.+Instead all frequencies have accumulated a random level of power.+This `fft` shows no clear sign of signal and appears to be noise.image::images/math-expressions/noise-fft.png[]-In the third example the `fft` function is called on the same signal hidden within noise that was used for-the autocorrelation example. The plot of the real values of the `fft` response is shown below.--Notice that there are two clear mirrored peaks, at the same locations as the `fft` of the pure signal. But-there is also now considerable noise on the frequencies. The `fft` has found the signal and but also-shows that there is considerable noise along with the signal.+In the third example the `fft` function is called on the same signal hidden within noise that was used for the autocorrelation example.+The plot of the real values of the `fft` response is shown below.+Notice that there are two clear mirrored peaks, at the same locations as the `fft` of the pure signal.+But there is also now considerable noise on the frequencies.+The `fft` has found the signal and but also shows that there is considerable noise along with the signal.image::images/math-expressions/hidden-signal-fft.png[]diff --git a/solr/solr-ref-guide/src/edismax-query-parser.adoc b/solr/solr-ref-guide/src/edismax-query-parser.adocindex c015ac73206..9aeff4a5fb6 100644--- a/solr/solr-ref-guide/src/edismax-query-parser.adoc+++ b/solr/solr-ref-guide/src/edismax-query-parser.adoc@@ -79,7 +79,8 @@ Defaults to `false`.`ps`::Phrase Slop.-The default amount of slop - distance between terms - on phrase queries built with `pf`, `pf2` and/or `pf3` fields (affects boosting). See also the section <<Using 'Slop'>> below.+The default amount of slop - distance between terms - on phrase queries built with `pf`, `pf2` and/or `pf3` fields (affects boosting).+See also the section <<Using 'Slop'>> below.`pf2`::@@ -87,14 +88,16 @@ A multivalued list of fields with optional weights.Similar to `pf`, but based on word _pair_ shingles.`ps2`::-This is similar to `ps` but overrides the slop factor used for `pf2`. If not specified, `ps` is used.+This is similar to `ps` but overrides the slop factor used for `pf2.+If not specified, `ps` is used.`pf3`::A multivalued list of fields with optional weights, based on triplets of word shingles.Similar to `pf`, except that instead of building a phrase per field out of all the words in the input, it builds a set of phrases for each field out of word _triplet_ shingles.`ps3`::-This is similar to `ps` but overrides the slop factor used for `pf3`. If not specified, `ps` is used.+This is similar to `ps` but overrides the slop factor used for `pf3`.+If not specified, `ps` is used.`stopwords`::A Boolean parameter indicating if the `StopFilterFactory` configured in the query analyzer should be respected when parsing the query.diff --git a/solr/solr-ref-guide/src/external-files-processes.adoc b/solr/solr-ref-guide/src/external-files-processes.adocindex 7f44ee4f4d3..29251804be1 100644--- a/solr/solr-ref-guide/src/external-files-processes.adoc+++ b/solr/solr-ref-guide/src/external-files-processes.adoc@@ -52,7 +52,8 @@ A `defVal` defines a default value that will be used if there is no entry in the=== Format of the External FileThe file itself is located in Solr's index directory, which by default is `$SOLR_HOME/data`.-The name of the file should be `external_fieldname_` or `external_fieldname_.*`. For the example above, then, the file could be named `external_entryRankFile` or `external_entryRankFile.txt`.+The name of the file should be `external_fieldname_` or `external_fieldname_.*`.+For the example above, then, the file could be named `external_entryRankFile` or `external_entryRankFile.txt`.[TIP]====@@ -186,7 +187,8 @@ name ::= textvalue ::= text-----Special characters in "text" values can be escaped using the escape character `\`. The following escape sequences are recognized:+Special characters in "text" values can be escaped using the escape character `\`.+The following escape sequences are recognized:[width="60%",options="header",]|===diff --git a/solr/solr-ref-guide/src/faceting.adoc b/solr/solr-ref-guide/src/faceting.adocindex 81c78942e73..c75db251206 100644--- a/solr/solr-ref-guide/src/faceting.adoc+++ b/solr/solr-ref-guide/src/faceting.adoc@@ -18,7 +18,8 @@Faceting is the arrangement of search results into categories based on indexed terms.-Searchers are presented with the indexed terms, along with numerical counts of how many matching documents were found for each term. Faceting makes it easy for users to explore search results, narrowing in on exactly the results they are looking for.+Searchers are presented with the indexed terms, along with numerical counts of how many matching documents were found for each term.+Faceting makes it easy for users to explore search results, narrowing in on exactly the results they are looking for.See also <<json-facet-api.adoc#, JSON Facet API>> for an alternative approach to this.@@ -27,16 +28,23 @@ See also <<json-facet-api.adoc#, JSON Facet API>> for an alternative approach toThere are two general parameters for controlling faceting.`facet`::-If set to `true`, this parameter enables facet counts in the query response. If set to `false`, a blank or missing value, this parameter disables faceting. None of the other parameters listed below will have any effect unless this parameter is set to `true`. The default value is blank (false).+If set to `true`, this parameter enables facet counts in the query response.+If set to `false`, a blank or missing value, this parameter disables faceting.+None of the other parameters listed below will have any effect unless this parameter is set to `true`.+The default value is blank (false).`facet.query`::This parameter allows you to specify an arbitrary query in the Lucene default syntax to generate a facet count.+-By default, Solr's faceting feature automatically determines the unique terms for a field and returns a count for each of those terms. Using `facet.query`, you can override this default behavior and select exactly which terms or expressions you would like to see counted. In a typical implementation of faceting, you will specify a number of `facet.query` parameters. This parameter can be particularly useful for numeric-range-based facets or prefix-based facets.+By default, Solr's faceting feature automatically determines the unique terms for a field and returns a count for each of those terms.+Using `facet.query`, you can override this default behavior and select exactly which terms or expressions you would like to see counted.+In a typical implementation of faceting, you will specify a number of `facet.query` parameters.+This parameter can be particularly useful for numeric-range-based facets or prefix-based facets.+You can set the `facet.query` parameter multiple times to indicate that multiple queries should be used as separate facet constraints.+-To use facet queries in a syntax other than the default syntax, prefix the facet query with the name of the query notation. For example, to use the hypothetical `myfunc` query parser, you could set the `facet.query` parameter like so:+To use facet queries in a syntax other than the default syntax, prefix the facet query with the name of the query notation.+For example, to use the hypothetical `myfunc` query parser, you could set the `facet.query` parameter like so:+`facet.query={!myfunc}name~fred`@@ -44,24 +52,30 @@ To use facet queries in a syntax other than the default syntax, prefix the facetSeveral parameters can be used to trigger faceting based on the indexed terms in a field.-When using these parameters, it is important to remember that "term" is a very specific concept in Lucene: it relates to the literal field/value pairs that are indexed after any analysis occurs. For text fields that include stemming, lowercasing, or word splitting, the resulting terms may not be what you expect.+When using these parameters, it is important to remember that "term" is a very specific concept in Lucene: it relates to the literal field/value pairs that are indexed after any analysis occurs.+For text fields that include stemming, lowercasing, or word splitting, the resulting terms may not be what you expect.-If you want Solr to perform both analysis (for searching) and faceting on the full literal strings, use the `copyField` directive in your Schema to create two versions of the field: one Text and one String. The Text field should have `indexed="true" docValues=“false"` if used for searching but not faceting and the String field should have `indexed="false" docValues="true"` if used for faceting but not searching.+If you want Solr to perform both analysis (for searching) and faceting on the full literal strings, use the `copyField` directive in your Schema to create two versions of the field: one Text and one String.+The Text field should have `indexed="true" docValues=“false"` if used for searching but not faceting and the String field should have `indexed="false" docValues="true"` if used for faceting but not searching.(For more information about the `copyField` directive, see <<fields-and-schema-design.adoc#,Fields and Schema Design>>.)Unless otherwise specified, all of the parameters below can be specified on a per-field basis with the syntax of `f.<fieldname>.facet.<parameter>``facet.field`::-The `facet.field` parameter identifies a field that should be treated as a facet. It iterates over each Term in the field and generate a facet count using that Term as the constraint. This parameter can be specified multiple times in a query to select multiple facet fields.+The `facet.field` parameter identifies a field that should be treated as a facet.+It iterates over each Term in the field and generate a facet count using that Term as the constraint.+This parameter can be specified multiple times in a query to select multiple facet fields.+IMPORTANT: If you do not set this parameter to at least one field in the schema, none of the other parameters described in this section will have any effect.`facet.prefix`::-The `facet.prefix` parameter limits the terms on which to facet to those starting with the given string prefix. This does not limit the query in any way, only the facets that would be returned in response to the query.+The `facet.prefix` parameter limits the terms on which to facet to those starting with the given string prefix.+This does not limit the query in any way, only the facets that would be returned in response to the query.+`facet.contains`::-The `facet.contains` parameter limits the terms on which to facet to those containing the given substring. This does not limit the query in any way, only the facets that would be returned in response to the query.+The `facet.contains` parameter limits the terms on which to facet to those containing the given substring.+This does not limit the query in any way, only the facets that would be returned in response to the query.`facet.contains.ignoreCase`::@@ -78,13 +92,16 @@ There are two options for this parameter.+--`count`::: Sort the constraints by count (highest count first).-`index`::: Return the constraints sorted in their index order (lexicographic by indexed term). For terms in the ASCII range, this will be alphabetically sorted.+`index`::: Return the constraints sorted in their index order (lexicographic by indexed term).+For terms in the ASCII range, this will be alphabetically sorted.--+-The default is `count` if `facet.limit` is greater than 0, otherwise, the default is `index`. Note that the default logic is changed when <<#limiting-facet-with-certain-terms>>+The default is `count` if `facet.limit` is greater than 0, otherwise, the default is `index`.+Note that the default logic is changed when <<#limiting-facet-with-certain-terms>>`facet.limit`::-This parameter specifies the maximum number of constraint counts (essentially, the number of facets for a field that are returned) that should be returned for the facet fields. A negative value means that Solr will return unlimited number of constraint counts.+This parameter specifies the maximum number of constraint counts (essentially, the number of facets for a field that are returned) that should be returned for the facet fields.+A negative value means that Solr will return unlimited number of constraint counts.+The default value is `100`.@@ -96,7 +113,8 @@ The default value is `0`.`facet.mincount`::-The `facet.mincount` parameter specifies the minimum counts required for a facet field to be included in the response. If a field's counts are below the minimum, the field's facet is not returned.+The `facet.mincount` parameter specifies the minimum counts required for a facet field to be included in the response.+If a field's counts are below the minimum, the field's facet is not returned.+The default value is `0`.@@ -113,72 +131,98 @@ The following methods are available.--`enum`::: Enumerates all terms in a field, calculating the set intersection of documents that match the term with documents that match the query.+-This method is recommended for faceting multi-valued fields that have only a few distinct values. The average number of values per document does not matter.+This method is recommended for faceting multi-valued fields that have only a few distinct values.+The average number of values per document does not matter.+-For example, faceting on a field with U.S. States such as `Alabama, Alaska, ... Wyoming` would lead to fifty cached filters which would be used over and over again. The `filterCache` should be large enough to hold all the cached filters.+For example, faceting on a field with U.S. States such as `Alabama, Alaska, ... Wyoming` would lead to fifty cached filters which would be used over and over again.+The `filterCache` should be large enough to hold all the cached filters.`fc`::: Calculates facet counts by iterating over documents that match the query and summing the terms that appear in each document.+-This is currently implemented using an `UnInvertedField` cache if the field either is multi-valued or is tokenized (according to `FieldType.isTokened()`). Each document is looked up in the cache to see what terms/values it contains, and a tally is incremented for each value.+This is currently implemented using an `UnInvertedField` cache if the field either is multi-valued or is tokenized (according to `FieldType.isTokened()`).+Each document is looked up in the cache to see what terms/values it contains, and a tally is incremented for each value.+-This method is excellent for situations where the number of indexed values for the field is high, but the number of values per document is low. For multi-valued fields, a hybrid approach is used that uses term filters from the `filterCache` for terms that match many documents. The letters `fc` stand for field cache.+This method is excellent for situations where the number of indexed values for the field is high, but the number of values per document is low.+For multi-valued fields, a hybrid approach is used that uses term filters from the `filterCache` for terms that match many documents.+The letters `fc` stand for field cache.-`fcs`::: Per-segment field faceting for single-valued string fields. Enable with `facet.method=fcs` and control the number of threads used with the `threads` local parameter. This parameter allows faceting to be faster in the presence of rapid index changes.+`fcs`::: Per-segment field faceting for single-valued string fields.+Enable with `facet.method=fcs` and control the number of threads used with the `threads` local parameter.+This parameter allows faceting to be faster in the presence of rapid index changes.--+The default value is `fc` (except for fields using the `BoolField` field type and when `facet.exists=true` is requested) since it tends to use less memory and is faster when a field has many unique terms in the index.`facet.enum.cache.minDf`::-This parameter indicates the minimum document frequency (the number of documents matching a term) for which the filterCache should be used when determining the constraint count for that term. This is only used with the `facet.method=enum` method of faceting.+This parameter indicates the minimum document frequency (the number of documents matching a term) for which the filterCache should be used when determining the constraint count for that term.+This is only used with the `facet.method=enum` method of faceting.+-A value greater than zero decreases the filterCache's memory usage, but increases the time required for the query to be processed. If you are faceting on a field with a very large number of terms, and you wish to decrease memory usage, try setting this parameter to a value between `25` and `50`, and run a few tests. Then, optimize the parameter setting as necessary.+A value greater than zero decreases the filterCache's memory usage, but increases the time required for the query to be processed.+If you are faceting on a field with a very large number of terms, and you wish to decrease memory usage, try setting this parameter to a value between `25` and `50`, and run a few tests.+Then, optimize the parameter setting as necessary.+The default value is `0`, causing the filterCache to be used for all terms in the field.`facet.exists`::-To cap facet counts by 1, specify `facet.exists=true`. This parameter can be used with `facet.method=enum` or when it's omitted. It can be used only on non-trie fields (such as strings). It may speed up facet counting on large indices and/or high-cardinality facet values.+To cap facet counts by 1, specify `facet.exists=true`.+This parameter can be used with `facet.method=enum` or when it's omitted.+It can be used only on non-trie fields (such as strings).+It may speed up facet counting on large indices and/or high-cardinality facet values.`facet.excludeTerms`::If you want to remove terms from facet counts but keep them in the index, the `facet.excludeTerms` parameter allows you to do that.`facet.overrequest.count` and `facet.overrequest.ratio`::-In some situations, the accuracy in selecting the "top" constraints returned for a facet in a distributed Solr query can be improved by "over requesting" the number of desired constraints (i.e., `facet.limit`) from each of the individual shards. In these situations, each shard is by default asked for the top `10 + (1.5 * facet.limit)` constraints.+In some situations, the accuracy in selecting the "top" constraints returned for a facet in a distributed Solr query can be improved by "over requesting" the number of desired constraints (i.e., `facet.limit`) from each of the individual shards.+In these situations, each shard is by default asked for the top `10 + (1.5 * facet.limit)` constraints.+-In some situations, depending on how your docs are partitioned across your shards and what `facet.limit` value you used, you may find it advantageous to increase or decrease the amount of over-requesting Solr does. This can be achieved by setting the `facet.overrequest.count` (defaults to `10`) and `facet.overrequest.ratio` (defaults to `1.5`) parameters.+In some situations, depending on how your docs are partitioned across your shards and what `facet.limit` value you used, you may find it advantageous to increase or decrease the amount of over-requesting Solr does.+This can be achieved by setting the `facet.overrequest.count` (defaults to `10`) and `facet.overrequest.ratio` (defaults to `1.5`) parameters.`facet.threads`::-This parameter will cause loading the underlying fields used in faceting to be executed in parallel with the number of threads specified. Specify as `facet.threads=N` where `N` is the maximum number of threads used.+This parameter will cause loading the underlying fields used in faceting to be executed in parallel with the number of threads specified.+Specify as `facet.threads=N` where `N` is the maximum number of threads used.+-Omitting this parameter or specifying the thread count as `0` will not spawn any threads, and only the main request thread will be used. Specifying a negative number of threads will create up to `Integer.MAX_VALUE` threads.+Omitting this parameter or specifying the thread count as `0` will not spawn any threads, and only the main request thread will be used.+Specifying a negative number of threads will create up to `Integer.MAX_VALUE` threads.== Range Faceting-You can use Range Faceting on any date field or any numeric field that supports range queries. This is particularly useful for stitching together a series of range queries (as facet by query) for things like prices.+You can use Range Faceting on any date field or any numeric field that supports range queries.+This is particularly useful for stitching together a series of range queries (as facet by query) for things like prices.`facet.range`::-The `facet.range` parameter defines the field for which Solr should create range facets. For example:+The `facet.range` parameter defines the field for which Solr should create range facets.+For example:+`facet.range=price&facet.range=age`+`facet.range=lastModified_dt``facet.range.start`::-The `facet.range.start` parameter specifies the lower bound of the ranges. You can specify this parameter on a per field basis with the syntax of `f.<fieldname>.facet.range.start`. For example:+The `facet.range.start` parameter specifies the lower bound of the ranges.+You can specify this parameter on a per field basis with the syntax of `f.<fieldname>.facet.range.start`.+For example:+`f.price.facet.range.start=0.0&f.age.facet.range.start=10`+`f.lastModified_dt.facet.range.start=NOW/DAY-30DAYS``facet.range.end`::-The `facet.range.end` specifies the upper bound of the ranges. You can specify this parameter on a per field basis with the syntax of `f.<fieldname>.facet.range.end`. For example:+The `facet.range.end` specifies the upper bound of the ranges.+You can specify this parameter on a per field basis with the syntax of `f.<fieldname>.facet.range.end`.+For example:+`f.price.facet.range.end=1000.0&f.age.facet.range.start=99`+`f.lastModified_dt.facet.range.end=NOW/DAY+30DAYS``facet.range.gap`::-The span of each range expressed as a value to be added to the lower bound. For date fields, this should be expressed using the {solr-javadocs}/core/org/apache/solr/util/DateMathParser.html[`DateMathParser` syntax] (such as, `facet.range.gap=%2B1DAY ... '+1DAY'`). You can specify this parameter on a per-field basis with the syntax of `f.<fieldname>.facet.range.gap`. For example:+The span of each range expressed as a value to be added to the lower bound.+For date fields, this should be expressed using the {solr-javadocs}/core/org/apache/solr/util/DateMathParser.html[`DateMathParser` syntax] (such as, `facet.range.gap=%2B1DAY ... '+1DAY'`).+You can specify this parameter on a per-field basis with the syntax of `f.<fieldname>.facet.range.gap`.+For example:+`f.price.facet.range.gap=100&f.age.facet.range.gap=10`+@@ -187,12 +231,17 @@ The span of each range expressed as a value to be added to the lower bound. For`facet.range.hardend`::The `facet.range.hardend` parameter is a Boolean parameter that specifies how Solr should handle cases where the `facet.range.gap` does not divide evenly between `facet.range.start` and `facet.range.end`.+-If `true`, the last range constraint will have the `facet.range.end` value as an upper bound. If `false`, the last range will have the smallest possible upper bound greater then `facet.range.end` such that the range is the exact width of the specified range gap. The default value for this parameter is false.+If `true`, the last range constraint will have the `facet.range.end` value as an upper bound.+If `false`, the last range will have the smallest possible upper bound greater then `facet.range.end` such that the range is the exact width of the specified range gap.+The default value for this parameter is false.+This parameter can be specified on a per field basis with the syntax `f.<fieldname>.facet.range.hardend`.`facet.range.include`::-By default, the ranges used to compute range faceting between `facet.range.start` and `facet.range.end` are inclusive of their lower bounds and exclusive of the upper bounds. The "before" range defined with the `facet.range.other` parameter is exclusive and the "after" range is inclusive. This default, equivalent to "lower" below, will not result in double counting at the boundaries. You can use the `facet.range.include` parameter to modify this behavior using the following options:+By default, the ranges used to compute range faceting between `facet.range.start` and `facet.range.end` are inclusive of their lower bounds and exclusive of the upper bounds.+The "before" range defined with the `facet.range.other` parameter is exclusive and the "after" range is inclusive.+This default, equivalent to "lower" below, will not result in double counting at the boundaries.+You can use the `facet.range.include` parameter to modify this behavior using the following options:+--* `lower`: All gap-based ranges include their lower bound.@@ -217,15 +266,20 @@ The `facet.range.other` parameter specifies that in addition to the counts for e* `all`: Compute counts for before, between, and after.--+-This parameter can be specified on a per field basis with the syntax of `f.<fieldname>.facet.range.other`. In addition to the `all` option, this parameter can be specified multiple times to indicate multiple choices, but `none` will override all other options.+This parameter can be specified on a per field basis with the syntax of `f.<fieldname>.facet.range.other`.+In addition to the `all` option, this parameter can be specified multiple times to indicate multiple choices, but `none` will override all other options.`facet.range.method`::-The `facet.range.method` parameter selects the type of algorithm or method Solr should use for range faceting. Both methods produce the same results, but performance may vary.+The `facet.range.method` parameter selects the type of algorithm or method Solr should use for range faceting.+Both methods produce the same results, but performance may vary.+---filter::: This method generates the ranges based on other facet.range parameters, and for each of them executes a filter that later intersects with the main query resultset to get the count. It will make use of the filterCache, so it will benefit of a cache large enough to contain all ranges.+filter::: This method generates the ranges based on other facet.range parameters, and for each of them executes a filter that later intersects with the main query resultset to get the count.+It will make use of the filterCache, so it will benefit of a cache large enough to contain all ranges.+-dv::: This method iterates the documents that match the main query, and for each of them finds the correct range for the value. This method will make use of <<docvalues.adoc#,docValues>> (if enabled for the field) or fieldCache. The `dv` method is not supported for field type DateRangeField or when using <<result-grouping.adoc#,group.facets>>.+dv::: This method iterates the documents that match the main query, and for each of them finds the correct range for the value.+This method will make use of <<docvalues.adoc#,docValues>> (if enabled for the field) or fieldCache.+The `dv` method is not supported for field type DateRangeField or when using <<result-grouping.adoc#,group.facets>>.--+The default value for this parameter is `filter`.@@ -242,19 +296,26 @@ For more information, see the examples in the <<date-formatting-math.adoc#,Worki=== facet.mincount in Range Faceting-The `facet.mincount` parameter, the same one as used in field faceting is also applied to range faceting. When used, no ranges with a count below the minimum will be included in the response.+The `facet.mincount` parameter, the same one as used in field faceting is also applied to range faceting.+When used, no ranges with a count below the minimum will be included in the response.== Pivot (Decision Tree) Faceting-Pivoting is a summarization tool that lets you automatically sort, count, total or average data stored in a table. The results are typically displayed in a second table showing the summarized data. Pivot faceting lets you create a summary table of the results from a faceting documents by multiple fields.+Pivoting is a summarization tool that lets you automatically sort, count, total or average data stored in a table.+The results are typically displayed in a second table showing the summarized data.+Pivot faceting lets you create a summary table of the results from a faceting documents by multiple fields.-Another way to look at it is that the query produces a Decision Tree, in that Solr tells you "for facet A, the constraints/counts are X/N, Y/M, etc. If you were to constrain A by X, then the constraint counts for B would be S/P, T/Q, etc.". In other words, it tells you in advance what the "next" set of facet results would be for a field if you apply a constraint from the current facet results.+Another way to look at it is that the query produces a Decision Tree, in that Solr tells you "for facet A, the constraints/counts are X/N, Y/M, etc.+If you were to constrain A by X, then the constraint counts for B would be S/P, T/Q, etc.". In other words, it tells you in advance what the "next" set of facet results would be for a field if you apply a constraint from the current facet results.`facet.pivot`::-The `facet.pivot` parameter defines the fields to use for the pivot. Multiple `facet.pivot` values will create multiple "facet_pivot" sections in the response. Separate each list of fields with a comma.+The `facet.pivot` parameter defines the fields to use for the pivot.+Multiple `facet.pivot` values will create multiple "facet_pivot" sections in the response.+Separate each list of fields with a comma.`facet.pivot.mincount`::-The `facet.pivot.mincount` parameter defines the minimum number of documents that need to match in order for the facet to be included in results. The default is 1.+The `facet.pivot.mincount` parameter defines the minimum number of documents that need to match in order for the facet to be included in results.+The default is 1.+Using the "`bin/solr -e techproducts`" example, A query URL like this one will return the data below, with the pivot faceting results found in the section "facet_pivot":@@ -384,7 +445,8 @@ Results:=== Combining Facet Queries And Facet Ranges With Pivot Facets-A `query` local parameter can be used with `facet.pivot` to refer to `facet.query` instances (by tag) that should be computed for each pivot constraint. Similarly, a `range` local parameter can be used with `facet.pivot` to refer to `facet.range` instances.+A `query` local parameter can be used with `facet.pivot` to refer to `facet.query` instances (by tag) that should be computed for each pivot constraint.+Similarly, a `range` local parameter can be used with `facet.pivot` to refer to `facet.range` instances.In the example below, two query facets are computed for h of the `facet.pivot` result hierarchies:@@ -520,11 +582,14 @@ Although `facet.pivot.mincount` deviates in name from the `facet.mincount` param== Interval Faceting-Another supported form of faceting is interval faceting. This sounds similar to range faceting, but the functionality is really closer to doing facet queries with range queries. Interval faceting allows you to set variable intervals and count the number of documents that have values within those intervals in the specified field.+Another supported form of faceting is interval faceting.+This sounds similar to range faceting, but the functionality is really closer to doing facet queries with range queries.+Interval faceting allows you to set variable intervals and count the number of documents that have values within those intervals in the specified field.Even though the same functionality can be achieved by using a facet query with range queries, the implementation of these two methods is very different and will provide different performance depending on the context.-If you are concerned about the performance of your searches you should test with both options. Interval faceting tends to be better with multiple intervals for the same fields, while facet query tend to be better in environments where filter cache is more effective (static indexes for example).+If you are concerned about the performance of your searches you should test with both options.+Interval faceting tends to be better with multiple intervals for the same fields, while facet query tend to be better in environments where filter cache is more effective (static indexes for example).This method will use <<docvalues.adoc#,docValues>> if they are enabled for the field, will use fieldCache otherwise.@@ -532,12 +597,15 @@ Use these parameters for interval faceting:`facet.interval`::-This parameter Indicates the field where interval faceting must be applied. It can be used multiple times in the same request to indicate multiple fields.+This parameter Indicates the field where interval faceting must be applied.+It can be used multiple times in the same request to indicate multiple fields.+`facet.interval=price&facet.interval=size``facet.interval.set`::-This parameter is used to set the intervals for the field, it can be specified multiple times to indicate multiple intervals. This parameter is global, which means that it will be used for all fields indicated with `facet.interval` unless there is an override for a specific field. To override this parameter on a specific field you can use: `f.<fieldname>.facet.interval.set`, for example:+This parameter is used to set the intervals for the field, it can be specified multiple times to indicate multiple intervals.+This parameter is global, which means that it will be used for all fields indicated with `facet.interval` unless there is an override for a specific field.+To override this parameter on a specific field you can use: `f.<fieldname>.facet.interval.set`, for example:+[source,text]f.price.facet.interval.set=[0,10]&f.price.facet.interval.set=(10,100]@@ -555,15 +623,25 @@ For example:The initial and end values cannot be empty.-If the interval needs to be unbounded, the special character `\*` can be used for both, start and end, limits. When using this special character, the start syntax options (`(` and `[`), and end syntax options (`)` and `]`) will be treated the same. `[*,*]` will include all documents with a value in the field.+If the interval needs to be unbounded, the special character `\*` can be used for both, start and end, limits.+When using this special character, the start syntax options (`(` and `[`), and end syntax options (`)` and `]`) will be treated the same.+`[*,*]` will include all documents with a value in the field.-The interval limits may be strings but there is no need to add quotes. All the text until the comma will be treated as the start limit, and the text after that will be the end limit. For example: `[Buenos Aires,New York]`. Keep in mind that a string-like comparison will be done to match documents in string intervals (case-sensitive). The comparator can't be changed.+The interval limits may be strings but there is no need to add quotes.+All the text until the comma will be treated as the start limit, and the text after that will be the end limit.+For example: `[Buenos Aires,New York]`.+Keep in mind that a string-like comparison will be done to match documents in string intervals (case-sensitive).+The comparator can't be changed.-Commas, brackets and square brackets can be escaped by using `\` in front of them. Whitespaces before and after the values will be omitted.+Commas, brackets and square brackets can be escaped by using `\` in front of them.+Whitespaces before and after the values will be omitted.-The start limit can't be grater than the end limit. Equal limits are allowed, this allows you to indicate the specific values that you want to count, like `[A,A]`, `[B,B]` and `[C,Z]`.+The start limit can't be grater than the end limit.+Equal limits are allowed, this allows you to indicate the specific values that you want to count, like `[A,A]`, `[B,B]` and `[C,Z]`.-Interval faceting supports output key replacement described below. Output keys can be replaced in both the `facet.interval parameter` and in the `facet.interval.set parameter`. For example:+Interval faceting supports output key replacement described below.+Output keys can be replaced in both the `facet.interval parameter` and in the `facet.interval.set parameter`.+For example:[source,text]----@@ -575,11 +653,13 @@ Interval faceting supports output key replacement described below. Output keys c== Local Params for Faceting-The <<local-params.adoc#,LocalParams syntax>> allows overriding global settings. It can also provide a method of adding metadata to other parameter values, much like XML attributes.+The <<local-params.adoc#,LocalParams syntax>> allows overriding global settings.+It can also provide a method of adding metadata to other parameter values, much like XML attributes.=== Tagging and Excluding Filters-You can tag specific filters and exclude those filters when faceting. This is useful when doing multi-select faceting.+You can tag specific filters and exclude those filters when faceting.+This is useful when doing multi-select faceting.Consider the following example query with faceting:@@ -587,7 +667,8 @@ Consider the following example query with faceting:Because everything is already constrained by the filter `doctype:pdf`, the `facet.field=doctype` facet command is currently redundant and will return 0 counts for everything except `doctype:pdf`.-To implement a multi-select facet for doctype, a GUI may want to still display the other doctype values and their associated counts, as if the `doctype:pdf` constraint had not yet been applied. For example:+To implement a multi-select facet for doctype, a GUI may want to still display the other doctype values and their associated counts, as if the `doctype:pdf` constraint had not yet been applied.+For example:[source,text]----@@ -602,23 +683,31 @@ To return counts for doctype values that are currently not selected, tag filters`q=mainquery&fq=status:public&fq={!tag=dt}doctype:pdf&facet=true&facet.field={!ex=dt}doctype`-Filter exclusion is supported for all types of facets. Both the `tag` and `ex` local params may specify multiple values by separating them with commas.+Filter exclusion is supported for all types of facets.+Both the `tag` and `ex` local params may specify multiple values by separating them with commas.=== Changing the Output Key-To change the output key for a faceting command, specify a new name with the `key` local parameter. For example:+To change the output key for a faceting command, specify a new name with the `key` local parameter.+For example:`facet.field={!ex=dt key=mylabel}doctype`-The parameter setting above causes the field facet results for the "doctype" field to be returned using the key "mylabel" rather than "doctype" in the response. This can be helpful when faceting on the same field multiple times with different exclusions.+The parameter setting above causes the field facet results for the "doctype" field to be returned using the key "mylabel" rather than "doctype" in the response.+This can be helpful when faceting on the same field multiple times with different exclusions.=== Limiting Facet with Certain Terms-To limit field facet with certain terms specify them comma separated with `terms` local parameter. Commas and quotes in terms can be escaped with backslash, as in `\,`. In this case facet is calculated on a way similar to `facet.method=enum`, but ignores `facet.enum.cache.minDf`. For example:+To limit field facet with certain terms specify them comma separated with `terms` local parameter.+Commas and quotes in terms can be escaped with backslash, as in `\,`.+In this case facet is calculated on a way similar to `facet.method=enum`, but ignores `facet.enum.cache.minDf`.+For example:`facet.field={!terms='alfa,betta,with\,with\',with space'}symbol`-This local parameter overrides default logic for `facet.sort`. if `facet.sort` is omitted, facets are returned in the given terms order that might be changed with `index` and `count` values. Note: other parameters might not be fully supported when this parameter is supplied.+This local parameter overrides default logic for `facet.sort`.+if `facet.sort` is omitted, facets are returned in the given terms order that might be changed with `index` and `count` values.+Note: other parameters might not be fully supported when this parameter is supplied.== Related Topicsdiff --git a/solr/solr-ref-guide/src/field-properties-by-use-case.adoc b/solr/solr-ref-guide/src/field-properties-by-use-case.adocindex 8425b64de33..3b84c6b60df 100644--- a/solr/solr-ref-guide/src/field-properties-by-use-case.adoc+++ b/solr/solr-ref-guide/src/field-properties-by-use-case.adoc@@ -16,7 +16,9 @@// specific language governing permissions and limitations// under the License.-Here is a summary of common use cases, and the attributes the fields or field types should have to support the case. An entry of true or false in the table indicates that the option must be set to the given value for the use case to function correctly. If no entry is provided, the setting of that attribute has no impact on the case.+Here is a summary of common use cases, and the attributes the fields or field types should have to support the case.+An entry of true or false in the table indicates that the option must be set to the given value for the use case to function correctly.+If no entry is provided, the setting of that attribute has no impact on the case.// NOTE: not currently using footnoteref here because:// - it has issues with tables in the PDF@@ -43,7 +45,12 @@ Notes:3. [[fpbuc_3,3]] (if termVectors=true)4. [[fpbuc_4,4]] A tokenizer must be defined for the field, but it doesn't need to be indexed.5. [[fpbuc_5,5]] Described in <<document-analysis.adoc#,Document Analysis in Solr>>.-6. [[fpbuc_6,6]] Term vectors are not mandatory here. If not true, then a stored field is analyzed. So term vectors are recommended, but only required if `stored=false`.-7. [[fpbuc_7,7]] For most field types, either `indexed` or `docValues` must be true, but both are not required. <<docvalues.adoc#,DocValues>> can be more efficient in many cases. For `[Int/Long/Float/Double/Date]PointFields`, `docValues=true` is required.-8. [[fpbuc_8,8]] Stored content will be used by default, but docValues can alternatively be used. See <<docvalues.adoc#,DocValues>>.+6. [[fpbuc_6,6]] Term vectors are not mandatory here.+If not true, then a stored field is analyzed.+So term vectors are recommended, but only required if `stored=false`.+7. [[fpbuc_7,7]] For most field types, either `indexed` or `docValues` must be true, but both are not required.+<<docvalues.adoc#,DocValues>> can be more efficient in many cases.+For `[Int/Long/Float/Double/Date]PointFields`, `docValues=true` is required.+8. [[fpbuc_8,8]] Stored content will be used by default, but docValues can alternatively be used.+See <<docvalues.adoc#,DocValues>>.9. [[fpbuc_9,9]] Multi-valued sorting may be performed on docValues-enabled fields using the two-argument `field()` function, e.g., `field(myfield,min)`; see the <<function-queries.adoc#field-function,field() function in Function Queries>>.diff --git a/solr/solr-ref-guide/src/field-type-definitions-and-properties.adoc b/solr/solr-ref-guide/src/field-type-definitions-and-properties.adocindex 92a85eb8838..a7f55333382 100644--- a/solr/solr-ref-guide/src/field-type-definitions-and-properties.adoc+++ b/solr/solr-ref-guide/src/field-type-definitions-and-properties.adoc@@ -25,9 +25,13 @@ A field type definition can include four types of information:* If the field type is `TextField`, a description of the field analysis for the field type.* Field type properties - depending on the implementation class, some properties may be mandatory.-== Field Type Definitions in schema.xml+== Field Type Definitions in the Schema-Field types are defined in `schema.xml`. Each field type is defined between `fieldType` elements. They can optionally be grouped within a `types` element. Here is an example of a field type definition for a type called `text_general`:+Field types are defined in the collection's <<solr-schema.adoc#,schema>>.+Each field type is defined between `fieldType` elements.+They can optionally be grouped within a `types` element.++Here is an example of a field type definition for a type called `text_general`:[source,xml,subs="verbatim,callouts"]----@@ -52,11 +56,14 @@ Field types are defined in `schema.xml`. Each field type is defined between `fie<1> The first line in the example above contains the field type name, `text_general`, and the name of the implementing class, `solr.TextField`.<2> The rest of the definition is about field analysis, described in <<document-analysis.adoc#,Document Analysis in Solr>>.-The implementing class is responsible for making sure the field is handled correctly. In the class names in `schema.xml`, the string `solr` is shorthand for `org.apache.solr.schema` or `org.apache.solr.analysis`. Therefore, `solr.TextField` is really `org.apache.solr.schema.TextField`.+The implementing class is responsible for making sure the field is handled correctly.+In the class names in `schema.xml`, the string `solr` is shorthand for `org.apache.solr.schema` or `org.apache.solr.analysis`.+Therefore, `solr.TextField` is really `org.apache.solr.schema.TextField`.== Field Type Properties-The field type `class` determines most of the behavior of a field type, but optional properties can also be defined. For example, the following definition of a date field type defines two properties, `sortMissingLast` and `omitNorms`.+The field type `class` determines most of the behavior of a field type, but optional properties can also be defined.+For example, the following definition of a date field type defines two properties, `sortMissingLast` and `omitNorms`.[source,xml]----@@ -75,48 +82,65 @@ The properties that can be specified for a given field type fall into three majoThese are the general properties for fields:`name`::-The name of the fieldType. This value gets used in field definitions, in the "type" attribute. It is strongly recommended that names consist of alphanumeric or underscore characters only and not start with a digit. This is not currently strictly enforced.+The name of the fieldType.+This value gets used in field definitions, in the "type" attribute.+It is strongly recommended that names consist of alphanumeric or underscore characters only and not start with a digit.+This is not currently strictly enforced.`class`::-The class name that gets used to store and index the data for this type. Note that you may prefix included class names with "solr." and Solr will automatically figure out which packages to search for the class - so `solr.TextField` will work.+The class name that gets used to store and index the data for this type.+Note that you may prefix included class names with "solr." and Solr will automatically figure out which packages to search for the class - so `solr.TextField` will work.+-If you are using a third-party class, you will probably need to have a fully qualified class name. The fully qualified equivalent for `solr.TextField` is `org.apache.solr.schema.TextField`.+If you are using a third-party class, you will probably need to have a fully qualified class name.+The fully qualified equivalent for `solr.TextField` is `org.apache.solr.schema.TextField`.`positionIncrementGap`::For multivalued fields, specifies a distance between multiple values, which prevents spurious phrase matches.-`autoGeneratePhraseQueries`:: For text fields. If `true`, Solr automatically generates phrase queries for adjacent terms. If `false`, terms must be enclosed in double-quotes to be treated as phrases.+`autoGeneratePhraseQueries`:: For text fields.+If `true`, Solr automatically generates phrase queries for adjacent terms.+If `false`, terms must be enclosed in double-quotes to be treated as phrases.`synonymQueryStyle`::-Query used to combine scores of overlapping query terms (i.e., synonyms). Consider a search for "blue tee" with query-time synonyms `tshirt,tee`.+Query used to combine scores of overlapping query terms (i.e., synonyms).+Consider a search for "blue tee" with query-time synonyms `tshirt,tee`.+-Use `as_same_term` (default) to blend terms, i.e., `SynonymQuery(tshirt,tee)` where each term will be treated as equally important. Use `pick_best` to select the most significant synonym when scoring `Dismax(tee,tshirt)`. Use `as_distinct_terms` to bias scoring towards the most significant synonym `(pants OR slacks)`.+Use `as_same_term` (default) to blend terms, i.e., `SynonymQuery(tshirt,tee)` where each term will be treated as equally important.+Use `pick_best` to select the most significant synonym when scoring `Dismax(tee,tshirt)`.+Use `as_distinct_terms` to bias scoring towards the most significant synonym `(pants OR slacks)`.+-`as_same_term` is appropriate when terms are true synonyms (television, tv). Use `pick_best` or `as_distinct_terms` when synonyms are expanding to hyponyms `(q=jeans w/ jeans=>jeans,pants)` and you want exact to come before parent and sibling concepts. See this http://opensourceconnections.com/blog/2017/11/21/solr-synonyms-mea-culpa/[blog article].+`as_same_term` is appropriate when terms are true synonyms (television, tv).+Use `pick_best` or `as_distinct_terms` when synonyms are expanding to hyponyms `(q=jeans w/ jeans=>jeans,pants)` and you want exact to come before parent and sibling concepts.+See this http://opensourceconnections.com/blog/2017/11/21/solr-synonyms-mea-culpa/[blog article].`enableGraphQueries`::-For text fields, applicable when querying with <<standard-query-parser.adoc#standard-query-parser-parameters,`sow=false`>> (which is the default for the `sow` parameter). Use `true`, the default, for field types with query analyzers including graph-aware filters, e.g., <<filters.adoc#synonym-graph-filter,Synonym Graph Filter>> and <<filters.adoc#word-delimiter-graph-filter,Word Delimiter Graph Filter>>.+For text fields, applicable when querying with <<standard-query-parser.adoc#standard-query-parser-parameters,`sow=false`>> (which is the default for the `sow` parameter).+Use `true`, the default, for field types with query analyzers including graph-aware filters, e.g., <<filters.adoc#synonym-graph-filter,Synonym Graph Filter>> and <<filters.adoc#word-delimiter-graph-filter,Word Delimiter Graph Filter>>.+Use `false` for field types with query analyzers including filters that can match docs when some tokens are missing, e.g., <<filters.adoc#shingle-filter,Shingle Filter>>.[[docvaluesformat]]`docValuesFormat`::-Defines a custom `DocValuesFormat` to use for fields of this type. This requires that a schema-aware codec, such as the `SchemaCodecFactory`, has been configured in `solrconfig.xml`.+Defines a custom `DocValuesFormat` to use for fields of this type.+This requires that a schema-aware codec, such as the `SchemaCodecFactory`, has been configured in `solrconfig.xml`.`postingsFormat`::-Defines a custom `PostingsFormat` to use for fields of this type. This requires that a schema-aware codec, such as the `SchemaCodecFactory`, has been configured in `solrconfig.xml`.+Defines a custom `PostingsFormat` to use for fields of this type.+This requires that a schema-aware codec, such as the `SchemaCodecFactory`, has been configured in `solrconfig.xml`.[NOTE]====-Lucene index back-compatibility is only supported for the default codec. If you choose to customize the `postingsFormat` or `docValuesFormat` in your `schema.xml`, upgrading to a future version of Solr may require you to either switch back to the default codec and optimize your index to rewrite it into the default codec before upgrading, or re-build your entire index from scratch after upgrading.+Lucene index back-compatibility is only supported for the default codec.+If you choose to customize the `postingsFormat` or `docValuesFormat` in your `schema.xml`, upgrading to a future version of Solr may require you to either switch back to the default codec and optimize your index to rewrite it into the default codec before upgrading, or re-build your entire index from scratch after upgrading.======= Field Default PropertiesThese are properties that can be specified either on the field types, or on individual fields to override the values provided by the field types.-The default values for each property depend on the underlying `FieldType` class, which in turn may depend on the `version` attribute of the `<schema/>`. The table below includes the default value for most `FieldType` implementations provided by Solr, assuming a `schema.xml` that declares `version="1.6"`.+The default values for each property depend on the underlying `FieldType` class, which in turn may depend on the `version` attribute of the `<schema/>`.+The table below includes the default value for most `FieldType` implementations provided by Solr, assuming a `schema.xml` that declares `version="1.6"`.// TODO: SOLR-10655 BEGIN: refactor this into a 'field-default-properties.include.adoc' file for reuse@@ -142,19 +166,27 @@ The default values for each property depend on the underlying `FieldType` class,== Choosing Appropriate Numeric Types-For general numeric needs, consider using one of the `IntPointField`, `LongPointField`, `FloatPointField`, or `DoublePointField` classes, depending on the specific values you expect. These "Dimensional Point" based numeric classes use specially encoded data structures to support efficient range queries regardless of the size of the ranges used. Enable <<docvalues.adoc#,DocValues>> on these fields as needed for sorting and/or faceting.+For general numeric needs, consider using one of the `IntPointField`, `LongPointField`, `FloatPointField`, or `DoublePointField` classes, depending on the specific values you expect.+These "Dimensional Point" based numeric classes use specially encoded data structures to support efficient range queries regardless of the size of the ranges used.+Enable <<docvalues.adoc#,DocValues>> on these fields as needed for sorting and/or faceting.-Some Solr features may not yet work with "Dimensional Points", in which case you may want to consider the equivalent `TrieIntField`, `TrieLongField`, `TrieFloatField`, and `TrieDoubleField` classes. These field types are deprecated and are likely to be removed in a future major Solr release, but they can still be used if necessary. Configure a `precisionStep="0"` if you wish to minimize index size, but if you expect users to make frequent range queries on numeric types, use the default `precisionStep` (by not specifying it) or specify it as `precisionStep="8"` (which is the default). This offers faster speed for range queries at the expense of increasing index size.+Some Solr features may not yet work with "Dimensional Points", in which case you may want to consider the equivalent `TrieIntField`, `TrieLongField`, `TrieFloatField`, and `TrieDoubleField` classes.+These field types are deprecated and are likely to be removed in a future major Solr release, but they can still be used if necessary.+Configure a `precisionStep="0"` if you wish to minimize index size, but if you expect users to make frequent range queries on numeric types, use the default `precisionStep` (by not specifying it) or specify it as `precisionStep="8"` (which is the default).+This offers faster speed for range queries at the expense of increasing index size.== Working With TextHandling text properly will make your users happy by providing them with the best possible results for text searches.-One technique is using a text field as a catch-all for keyword searching. Most users are not sophisticated about their searches and the most common search is likely to be a simple keyword search. You can use `copyField` to take a variety of fields and funnel them all into a single text field for keyword searches.+One technique is using a text field as a catch-all for keyword searching.+Most users are not sophisticated about their searches and the most common search is likely to be a simple keyword search.+You can use `copyField` to take a variety of fields and funnel them all into a single text field for keyword searches.In the `schema.xml` file for the "```techproducts```" example included with Solr, `copyField` declarations are used to dump the contents of `cat`, `name`, `manu`, `features`, and `includes` into a single field, `text`. In addition, it could be a good idea to copy `ID` into `text` in case users wanted to search for a particular product by passing its product number to a keyword search.-Another technique is using `copyField` to use the same field in different ways. Suppose you have a field that is a list of authors, like this:+Another technique is using `copyField` to use the same field in different ways.+Suppose you have a field that is a list of authors, like this:`Schildt, Herbert; Wolpert, Lewis; Davies, P.`diff --git a/solr/solr-ref-guide/src/filters.adoc b/solr/solr-ref-guide/src/filters.adocindex eb72764d2fc..83a153e3450 100644--- a/solr/solr-ref-guide/src/filters.adoc+++ b/solr/solr-ref-guide/src/filters.adoc@@ -191,24 +191,31 @@ This filter converts characters from the following Unicode blocks:== Beider-Morse Filter-Implements the Beider-Morse Phonetic Matching (BMPM) algorithm, which allows identification of similar names, even if they are spelled differently or in different languages. More information about how this works is available in the section on <<phonetic-matching.adoc#beider-morse-phonetic-matching-bmpm,Phonetic Matching>>.+Implements the Beider-Morse Phonetic Matching (BMPM) algorithm, which allows identification of similar names, even if they are spelled differently or in different languages.+More information about how this works is available in the section on <<phonetic-matching.adoc#beider-morse-phonetic-matching-bmpm,Phonetic Matching>>.[IMPORTANT]====-BeiderMorseFilter changed its behavior in Solr 5.0 due to an update to version 3.04 of the BMPM algorithm. Older version of Solr implemented BMPM version 3.00 (see http://stevemorse.org/phoneticinfo.htm). Any index built using this filter with earlier versions of Solr will need to be rebuilt.+BeiderMorseFilter changed its behavior in Solr 5.0 due to an update to version 3.04 of the BMPM algorithm.+Older version of Solr implemented BMPM version 3.00 (see http://stevemorse.org/phoneticinfo.htm).+Any index built using this filter with earlier versions of Solr will need to be rebuilt.====*Factory class:* `solr.BeiderMorseFilterFactory`*Arguments:*-`nameType`:: Types of names. Valid values are GENERIC, ASHKENAZI, or SEPHARDIC. If not processing Ashkenazi or Sephardic names, use GENERIC.+`nameType`:: Types of names.+Valid values are GENERIC, ASHKENAZI, or SEPHARDIC.+If not processing Ashkenazi or Sephardic names, use GENERIC.-`ruleType`:: Types of rules to apply. Valid values are APPROX or EXACT.+`ruleType`:: Types of rules to apply.+Valid values are APPROX or EXACT.`concat`:: Defines if multiple possible matches should be combined with a pipe ("|").-`languageSet`:: The language set to use. The value "auto" will allow the Filter to identify the language, or a comma-separated list can be supplied.+`languageSet`:: The language set to use.+The value "auto" will allow the Filter to identify the language, or a comma-separated list can be supplied.*Example:*@@ -284,7 +291,9 @@ This filter takes the output of the <<tokenizers.adoc#classic-tokenizer,Classic== Common Grams Filter-This filter for use in `index` time analysis creates word shingles by combining common tokens such as stop words with regular tokens. This can result in an index with more unique terms, but is useful for creating phrase queries containing common words, such as "the cat", in a way that will typically be much faster then if the combined tokens are not used, because only the term positions of documents containg both terms in sequence have to be considered. Correct usage requires being paired with <<#common-grams-query-filter,Common Grams Query Filter>> during `query` analysis.+This filter for use in `index` time analysis creates word shingles by combining common tokens such as stop words with regular tokens.+This can result in an index with more unique terms, but is useful for creating phrase queries containing common words, such as "the cat", in a way that will typically be much faster then if the combined tokens are not used, because only the term positions of documents containg both terms in sequence have to be considered.+Correct usage requires being paired with <<#common-grams-query-filter,Common Grams Query Filter>> during `query` analysis.These filters can also be combined with <<#stop-filter,Stop Filter>> so searching for `"the cat"` would match different documents then `"a cat"`, while pathological searches for either `"the"` or `"a"` would not match any documents.@@ -296,7 +305,8 @@ These filters can also be combined with <<#stop-filter,Stop Filter>> so searchin`format`:: (optional) If the stopwords list has been formatted for Snowball, you can specify `format="snowball"` so Solr can read the stopwords file.-`ignoreCase`:: (boolean) If true, the filter ignores the case of words when comparing them to the common word file. The default is false.+`ignoreCase`:: (boolean) If true, the filter ignores the case of words when comparing them to the common word file.+The default is false.*Example:*@@ -348,17 +358,22 @@ This filter is used for the `query` time analysis aspect of <<#common-grams-filt== Collation Key Filter-Collation allows sorting of text in a language-sensitive way. It is usually used for sorting, but can also be used with advanced searches. We've covered this in much more detail in the section on <<language-analysis.adoc#unicode-collation,Unicode Collation>>.+Collation allows sorting of text in a language-sensitive way.+It is usually used for sorting, but can also be used with advanced searches.+We've covered this in much more detail in the section on <<language-analysis.adoc#unicode-collation,Unicode Collation>>.== Daitch-Mokotoff Soundex Filter-Implements the Daitch-Mokotoff Soundex algorithm, which allows identification of similar names, even if they are spelled differently. More information about how this works is available in the section on <<phonetic-matching.adoc#,Phonetic Matching>>.+Implements the Daitch-Mokotoff Soundex algorithm, which allows identification of similar names, even if they are spelled differently.+More information about how this works is available in the section on <<phonetic-matching.adoc#,Phonetic Matching>>.*Factory class:* `solr.DaitchMokotoffSoundexFilterFactory`*Arguments:*-`inject`:: (true/false) If true (the default), then new phonetic tokens are added to the stream. Otherwise, tokens are replaced with the phonetic equivalent. Setting this to false will enable phonetic matching, but the exact spelling of the target word may not match.+`inject`:: (true/false) If true (the default), then new phonetic tokens are added to the stream.+Otherwise, tokens are replaced with the phonetic equivalent.+Setting this to false will enable phonetic matching, but the exact spelling of the target word may not match.*Example:*@@ -390,13 +405,16 @@ Implements the Daitch-Mokotoff Soundex algorithm, which allows identification of== Double Metaphone Filter-This filter creates tokens using the http://commons.apache.org/proper/commons-codec/archives/{ivy-commons-codec-version}/apidocs/org/apache/commons/codec/language/DoubleMetaphone.html[`DoubleMetaphone`] encoding algorithm from commons-codec. For more information, see the <<phonetic-matching.adoc#,Phonetic Matching>> section.+This filter creates tokens using the http://commons.apache.org/proper/commons-codec/archives/{ivy-commons-codec-version}/apidocs/org/apache/commons/codec/language/DoubleMetaphone.html[`DoubleMetaphone`] encoding algorithm from commons-codec.+For more information, see the <<phonetic-matching.adoc#,Phonetic Matching>> section.*Factory class:* `solr.DoubleMetaphoneFilterFactory`*Arguments:*-`inject`:: (true/false) If true (the default), then new phonetic tokens are added to the stream. Otherwise, tokens are replaced with the phonetic equivalent. Setting this to false will enable phonetic matching, but the exact spelling of the target word may not match.+`inject`:: (true/false) If true (the default), then new phonetic tokens are added to the stream.+Otherwise, tokens are replaced with the phonetic equivalent.+Setting this to false will enable phonetic matching, but the exact spelling of the target word may not match.`maxCodeLength`:: (integer) The maximum length of the code to be generated.@@ -436,7 +454,8 @@ Default behavior for inject (true): keep the original token and add phonetic tok*Out:* "four"(1), "FR"(1), "score"(2), "SKR"(2), "and"(3), "ANT"(3), "Kuczewski"(4), "KSSK"(4), "KXFS"(4)-The phonetic tokens have a position increment of 0, which indicates that they are at the same position as the token they were derived from (immediately preceding). Note that "Kuczewski" has two encodings, which are added at the same position.+The phonetic tokens have a position increment of 0, which indicates that they are at the same position as the token they were derived from (immediately preceding).+Note that "Kuczewski" has two encodings, which are added at the same position.*Example:*@@ -466,7 +485,8 @@ This filter adds a numeric floating point boost value to tokens, splitting on a*Arguments:*-`delimiter`:: The character used to separate the token and the boost. Defaults to '|'.+`delimiter`:: The character used to separate the token and the boost.+Defaults to '|'.*Example:*@@ -672,7 +692,8 @@ This filter stems plural English words to their singular form.== English Possessive Filter-This filter removes singular possessives (trailing *'s*) from words. Note that plural possessives, e.g., the *s'* in "divers' snorkels", are not removed by this filter.+This filter removes singular possessives (trailing *'s*) from words.+Note that plural possessives, e.g., the *s'* in "divers' snorkels", are not removed by this filter.*Factory class:* `solr.EnglishPossessiveFilterFactory`@@ -714,15 +735,19 @@ This filter removes singular possessives (trailing *'s*) from words. Note that p== Fingerprint Filter-This filter outputs a single token which is a concatenation of the sorted and de-duplicated set of input tokens. This can be useful for clustering/linking use cases.+This filter outputs a single token which is a concatenation of the sorted and de-duplicated set of input tokens.+This can be useful for clustering/linking use cases.*Factory class:* `solr.FingerprintFilterFactory`*Arguments:*-`separator`:: The character used to separate tokens combined into the single output token. Defaults to " " (a space character).+`separator`:: The character used to separate tokens combined into the single output token.+Defaults to " " (a space character).-`maxOutputTokenSize`:: The maximum length of the summarized output token. If exceeded, no output token is emitted. Defaults to 1024.+`maxOutputTokenSize`:: The maximum length of the summarized output token.+If exceeded, no output token is emitted.+Defaults to 1024.*Example:*@@ -770,9 +795,13 @@ See the examples below for <<Synonym Graph Filter>> and <<Word Delimiter Graph F== Hunspell Stem Filter-The `Hunspell Stem Filter` provides support for several languages. You must provide the dictionary (`.dic`) and rules (`.aff`) files for each language you wish to use with the Hunspell Stem Filter. You can download those language files http://wiki.services.openoffice.org/wiki/Dictionaries[here].+The `Hunspell Stem Filter` provides support for several languages.+You must provide the dictionary (`.dic`) and rules (`.aff`) files for each language you wish to use with the Hunspell Stem Filter.+You can download those language files http://wiki.services.openoffice.org/wiki/Dictionaries[here].-Be aware that your results will vary widely based on the quality of the provided dictionary and rules files. For example, some languages have only a minimal word list with no morphological information. On the other hand, for languages that have no stemmer but do have an extensive dictionary file, the Hunspell stemmer may be a good choice.+Be aware that your results will vary widely based on the quality of the provided dictionary and rules files.+For example, some languages have only a minimal word list with no morphological information.+On the other hand, for languages that have no stemmer but do have an extensive dictionary file, the Hunspell stemmer may be a good choice.*Factory class:* `solr.HunspellStemFilterFactory`@@ -782,9 +811,12 @@ Be aware that your results will vary widely based on the quality of the provided`affix`:: (required) The path of a rules file.-`ignoreCase`:: (boolean) controls whether matching is case sensitive or not. The default is false.+`ignoreCase`:: (boolean) controls whether matching is case sensitive or not.+The default is false.-`strictAffixParsing`:: (boolean) controls whether the affix parsing is strict or not. If true, an error while reading an affix rule causes a ParseException, otherwise is ignored. The default is true.+`strictAffixParsing`:: (boolean) controls whether the affix parsing is strict or not.+If true, an error while reading an affix rule causes a ParseException, otherwise is ignored.+The default is true.*Example:*@@ -830,9 +862,11 @@ Be aware that your results will vary widely based on the quality of the provided== Hyphenated Words Filter-This filter reconstructs hyphenated words that have been tokenized as two tokens because of a line break or other intervening whitespace in the field test. If a token ends with a hyphen, it is joined with the following token and the hyphen is discarded.+This filter reconstructs hyphenated words that have been tokenized as two tokens because of a line break or other intervening whitespace in the field test.+If a token ends with a hyphen, it is joined with the following token and the hyphen is discarded.-Note that for this filter to work properly, the upstream tokenizer must not remove trailing hyphen characters. This filter is generally only useful at index time.+Note that for this filter to work properly, the upstream tokenizer must not remove trailing hyphen characters.+This filter is generally only useful at index time.*Factory class:* `solr.HyphenatedWordsFilterFactory`@@ -876,13 +910,15 @@ Note that for this filter to work properly, the upstream tokenizer must not remoThis filter is a custom Unicode normalization form that applies the foldings specified in http://www.unicode.org/reports/tr30/tr30-4.html[Unicode TR #30: Character Foldings] in addition to the `NFKC_Casefold` normalization form as described in <<ICU Normalizer 2 Filter>>. This filter is a better substitute for the combined behavior of the <<ASCII Folding Filter>>, <<Lower Case Filter>>, and <<ICU Normalizer 2 Filter>>.-To use this filter, you must add additional .jars to Solr's classpath (as described in the section <<solr-plugins.adoc#installing-plugins,Solr Plugins>>). See `solr/contrib/analysis-extras/README.md` for instructions on which jars you need to add.+To use this filter, you must add additional .jars to Solr's classpath (as described in the section <<solr-plugins.adoc#installing-plugins,Solr Plugins>>).+See `solr/contrib/analysis-extras/README.md` for instructions on which jars you need to add.*Factory class:* `solr.ICUFoldingFilterFactory`*Arguments:*-`filter`:: (string, optional) A Unicode set filter that can be used to e.g., exclude a set of characters from being processed. See the http://icu-project.org/apiref/icu4j/com/ibm/icu/text/UnicodeSet.html[UnicodeSet javadocs] for more information.+`filter`:: (string, optional) A Unicode set filter that can be used to e.g., exclude a set of characters from being processed.+See the http://icu-project.org/apiref/icu4j/com/ibm/icu/text/UnicodeSet.html[UnicodeSet javadocs] for more information.*Example without a filter:*@@ -932,17 +968,24 @@ This filter factory normalizes text according to one of five Unicode Normalizati* NFD: (`name="nfc" mode="decompose"`) Normalization Form D, canonical decomposition, followed by canonical composition* NFKC: (`name="nfkc" mode="compose"`) Normalization Form KC, compatibility decomposition* NFKD: (`name="nfkc" mode="decompose"`) Normalization Form KD, compatibility decomposition, followed by canonical composition-* NFKC_Casefold: (`name="nfkc_cf" mode="compose"`) Normalization Form KC, with additional Unicode case folding. Using the ICU Normalizer 2 Filter is a better-performing substitution for the <<Lower Case Filter>> and NFKC normalization.+* NFKC_Casefold: (`name="nfkc_cf" mode="compose"`) Normalization Form KC, with additional Unicode case folding.+Using the ICU Normalizer 2 Filter is a better-performing substitution for the <<Lower Case Filter>> and NFKC normalization.*Factory class:* `solr.ICUNormalizer2FilterFactory`*Arguments:*-`form`:: The name of the normalization form. Valid options are `nfc`, `nfd`, `nfkc`, `nfkd`, or `nfkc_cf` (the default). Required.+`form`:: The name of the normalization form.+Valid options are `nfc`, `nfd`, `nfkc`, `nfkd`, or `nfkc_cf` (the default).+Required.-`mode`:: The mode of Unicode character composition and decomposition. Valid options are: `compose` (the default) or `decompose`. Required.+`mode`:: The mode of Unicode character composition and decomposition.+Valid options are: `compose` (the default) or `decompose`.+Required.-`filter`:: A Unicode set filter that can be used to e.g., exclude a set of characters from being processed. See the http://icu-project.org/apiref/icu4j/com/ibm/icu/text/UnicodeSet.html[UnicodeSet javadocs] for more information. Optional.+`filter`:: A Unicode set filter that can be used to e.g., exclude a set of characters from being processed.+See the http://icu-project.org/apiref/icu4j/com/ibm/icu/text/UnicodeSet.html[UnicodeSet javadocs] for more information.+Optional.*Example with NFKC_Casefold:*@@ -984,17 +1027,21 @@ This filter factory normalizes text according to one of five Unicode NormalizatiFor detailed information about these normalization forms, see http://unicode.org/reports/tr15/[Unicode Normalization Forms].-To use this filter, you must add additional .jars to Solr's classpath (as described in the section <<solr-plugins.adoc#installing-plugins,Solr Plugins>>). See `solr/contrib/analysis-extras/README.md` for instructions on which jars you need to add.+To use this filter, you must add additional .jars to Solr's classpath (as described in the section <<solr-plugins.adoc#installing-plugins,Solr Plugins>>).+See `solr/contrib/analysis-extras/README.md` for instructions on which jars you need to add.== ICU Transform Filter-This filter applies http://userguide.icu-project.org/transforms/general[ICU Tranforms] to text. This filter supports only ICU System Transforms. Custom rule sets are not supported.+This filter applies http://userguide.icu-project.org/transforms/general[ICU Tranforms] to text.+This filter supports only ICU System Transforms.+Custom rule sets are not supported.*Factory class:* `solr.ICUTransformFilterFactory`*Arguments:*-`id`:: (string) The identifier for the ICU System Transform you wish to apply with this filter. For a full list of ICU System Transforms, see http://demo.icu-project.org/icu-bin/translit?TEMPLATE_FILE=data/translit_rule_main.html.+`id`:: (string) The identifier for the ICU System Transform you wish to apply with this filter.+For a full list of ICU System Transforms, see http://demo.icu-project.org/icu-bin/translit?TEMPLATE_FILE=data/translit_rule_main.html.*Example:*@@ -1026,21 +1073,29 @@ This filter applies http://userguide.icu-project.org/transforms/general[ICU TranFor detailed information about ICU Transforms, see http://userguide.icu-project.org/transforms/general.-To use this filter, you must add additional .jars to Solr's classpath (as described in the section <<solr-plugins.adoc#installing-plugins,Solr Plugins>>). See `solr/contrib/analysis-extras/README.md` for instructions on which jars you need to add.+To use this filter, you must add additional .jars to Solr's classpath (as described in the section <<solr-plugins.adoc#installing-plugins,Solr Plugins>>).+See `solr/contrib/analysis-extras/README.md` for instructions on which jars you need to add.== Keep Word Filter-This filter discards all tokens except those that are listed in the given word list. This is the inverse of the Stop Words Filter. This filter can be useful for building specialized indices for a constrained set of terms.+This filter discards all tokens except those that are listed in the given word list.+This is the inverse of the Stop Words Filter.+This filter can be useful for building specialized indices for a constrained set of terms.*Factory class:* `solr.KeepWordFilterFactory`*Arguments:*-`words`:: (required) Path of a text file containing the list of keep words, one per line. Blank lines and lines that begin with "#" are ignored. This may be an absolute path, or a simple filename in the Solr `conf` directory.+`words`:: (required) Path of a text file containing the list of keep words, one per line.+Blank lines and lines that begin with "#" are ignored.+This may be an absolute path, or a simple filename in the Solr `conf` directory.-`ignoreCase`:: (true/false) If *true* then comparisons are done case-insensitively. If this argument is true, then the words file is assumed to contain only lowercase words. The default is *false*.+`ignoreCase`:: (true/false) If *true* then comparisons are done case-insensitively.+If this argument is true, then the words file is assumed to contain only lowercase words.+The default is *false*.-`enablePositionIncrements`:: if `luceneMatchVersion` is `4.3` or earlier and `enablePositionIncrements="false"`, no position holes will be left by this filter when it removes tokens. *This argument is invalid if `luceneMatchVersion` is `5.0` or later.*+`enablePositionIncrements`:: if `luceneMatchVersion` is `4.3` or earlier and `enablePositionIncrements="false"`, no position holes will be left by this filter when it removes tokens.+*This argument is invalid if `luceneMatchVersion` is `5.0` or later.**Example:*@@ -1121,7 +1176,9 @@ Using LowerCaseFilterFactory before filtering for keep words, no `ignoreCase` fl== KStem Filter-KStem is an alternative to the Porter Stem Filter for developers looking for a less aggressive stemmer. KStem was written by Bob Krovetz, ported to Lucene by Sergio Guzman-Lara (UMASS Amherst). This stemmer is only appropriate for English language text.+KStem is an alternative to the Porter Stem Filter for developers looking for a less aggressive stemmer.+KStem was written by Bob Krovetz, ported to Lucene by Sergio Guzman-Lara (UMASS Amherst).+This stemmer is only appropriate for English language text.*Factory class:* `solr.KStemFilterFactory`@@ -1163,17 +1220,21 @@ KStem is an alternative to the Porter Stem Filter for developers looking for a l== Length Filter-This filter passes tokens whose length falls within the min/max limit specified. All other tokens are discarded.+This filter passes tokens whose length falls within the min/max limit specified.+All other tokens are discarded.*Factory class:* `solr.LengthFilterFactory`*Arguments:*-`min`:: (integer, required) Minimum token length. Tokens shorter than this are discarded.+`min`:: (integer, required) Minimum token length.+Tokens shorter than this are discarded.-`max`:: (integer, required, must be >= min) Maximum token length. Tokens longer than this are discarded.+`max`:: (integer, required, must be >= min) Maximum token length.+Tokens longer than this are discarded.-`enablePositionIncrements`:: if `luceneMatchVersion` is `4.3` or earlier and `enablePositionIncrements="false"`, no position holes will be left by this filter when it removes tokens. *This argument is invalid if `luceneMatchVersion` is `5.0` or later.*+`enablePositionIncrements`:: if `luceneMatchVersion` is `4.3` or earlier and `enablePositionIncrements="false"`, no position holes will be left by this filter when it removes tokens.+*This argument is invalid if `luceneMatchVersion` is `5.0` or later.**Example:*@@ -1213,15 +1274,19 @@ This filter passes tokens whose length falls within the min/max limit specified.This filter limits the number of accepted tokens, typically useful for index analysis.-By default, this filter ignores any tokens in the wrapped `TokenStream` once the limit has been reached, which can result in `reset()` being called prior to `incrementToken()` returning `false`. For most `TokenStream` implementations this should be acceptable, and faster then consuming the full stream. If you are wrapping a `TokenStream` which requires that the full stream of tokens be exhausted in order to function properly, use the `consumeAllTokens="true"` option.+By default, this filter ignores any tokens in the wrapped `TokenStream` once the limit has been reached, which can result in `reset()` being called prior to `incrementToken()` returning `false`.+For most `TokenStream` implementations this should be acceptable, and faster then consuming the full stream.+If you are wrapping a `TokenStream` which requires that the full stream of tokens be exhausted in order to function properly, use the `consumeAllTokens="true"` option.*Factory class:* `solr.LimitTokenCountFilterFactory`*Arguments:*-`maxTokenCount`:: (integer, required) Maximum token count. After this limit has been reached, tokens are discarded.+`maxTokenCount`:: (integer, required) Maximum token count.+After this limit has been reached, tokens are discarded.-`consumeAllTokens`:: (boolean, defaults to false) Whether to consume (and discard) previous token filters' tokens after the maximum token count has been reached. See description above.+`consumeAllTokens`:: (boolean, defaults to false) Whether to consume (and discard) previous token filters' tokens after the maximum token count has been reached.+See description above.*Example:*@@ -1261,17 +1326,22 @@ By default, this filter ignores any tokens in the wrapped `TokenStream` once the== Limit Token Offset Filter-This filter limits tokens to those before a configured maximum start character offset. This can be useful to limit highlighting, for example.+This filter limits tokens to those before a configured maximum start character offset.+This can be useful to limit highlighting, for example.-By default, this filter ignores any tokens in the wrapped `TokenStream` once the limit has been reached, which can result in `reset()` being called prior to `incrementToken()` returning `false`. For most `TokenStream` implementations this should be acceptable, and faster then consuming the full stream. If you are wrapping a `TokenStream` which requires that the full stream of tokens be exhausted in order to function properly, use the `consumeAllTokens="true"` option.+By default, this filter ignores any tokens in the wrapped `TokenStream` once the limit has been reached, which can result in `reset()` being called prior to `incrementToken()` returning `false`.+For most `TokenStream` implementations this should be acceptable, and faster then consuming the full stream.+If you are wrapping a `TokenStream` which requires that the full stream of tokens be exhausted in order to function properly, use the `consumeAllTokens="true"` option.*Factory class:* `solr.LimitTokenOffsetFilterFactory`*Arguments:*-`maxStartOffset`:: (integer, required) Maximum token start character offset. After this limit has been reached, tokens are discarded.+`maxStartOffset`:: (integer, required) Maximum token start character offset.+After this limit has been reached, tokens are discarded.-`consumeAllTokens`:: (boolean, defaults to false) Whether to consume (and discard) previous token filters' tokens after the maximum start offset has been reached. See description above.+`consumeAllTokens`:: (boolean, defaults to false) Whether to consume (and discard) previous token filters' tokens after the maximum start offset has been reached.+See description above.*Example:*@@ -1313,15 +1383,19 @@ By default, this filter ignores any tokens in the wrapped `TokenStream` once theThis filter limits tokens to those before a configured maximum token position.-By default, this filter ignores any tokens in the wrapped `TokenStream` once the limit has been reached, which can result in `reset()` being called prior to `incrementToken()` returning `false`. For most `TokenStream` implementations this should be acceptable, and faster then consuming the full stream. If you are wrapping a `TokenStream` which requires that the full stream of tokens be exhausted in order to function properly, use the `consumeAllTokens="true"` option.+By default, this filter ignores any tokens in the wrapped `TokenStream` once the limit has been reached, which can result in `reset()` being called prior to `incrementToken()` returning `false`.+For most `TokenStream` implementations this should be acceptable, and faster then consuming the full stream.+If you are wrapping a `TokenStream` which requires that the full stream of tokens be exhausted in order to function properly, use the `consumeAllTokens="true"` option.*Factory class:* `solr.LimitTokenPositionFilterFactory`*Arguments:*-`maxTokenPosition`:: (integer, required) Maximum token position. After this limit has been reached, tokens are discarded.+`maxTokenPosition`:: (integer, required) Maximum token position.+After this limit has been reached, tokens are discarded.-`consumeAllTokens`:: (boolean, defaults to false) Whether to consume (and discard) previous token filters' tokens after the maximum start offset has been reached. See description above.+`consumeAllTokens`:: (boolean, defaults to false) Whether to consume (and discard) previous token filters' tokens after the maximum start offset has been reached.+See description above.*Example:*@@ -1361,7 +1435,8 @@ By default, this filter ignores any tokens in the wrapped `TokenStream` once the== Lower Case Filter-Converts any uppercase letters in a token to the equivalent lowercase token. All other characters are left unchanged.+Converts any uppercase letters in a token to the equivalent lowercase token.+All other characters are left unchanged.*Factory class:* `solr.LowerCaseFilterFactory`@@ -1459,7 +1534,8 @@ For arguments and examples, see the <<Synonym Graph Filter>> below.This is specialized version of the <<Synonym Graph Filter>> that uses a mapping on synonyms that is <<managed-resources.adoc#,managed from a REST API.>>-This filter maps single- or multi-token synonyms, producing a fully correct graph output. This filter is a replacement for the Managed Synonym Filter, which produces incorrect graphs for multi-token synonyms.+This filter maps single- or multi-token synonyms, producing a fully correct graph output.+This filter is a replacement for the Managed Synonym Filter, which produces incorrect graphs for multi-token synonyms.NOTE: Although this filter produces correct token graphs, it cannot consume an input token graph correctly.@@ -1511,7 +1587,8 @@ See <<Synonym Graph Filter>> below for example input/output.== N-Gram Filter-Generates n-gram tokens of sizes in the given range. Note that tokens are ordered by position and then by gram size.+Generates n-gram tokens of sizes in the given range.+Note that tokens are ordered by position and then by gram size.*Factory class:* `solr.NGramFilterFactory`@@ -1615,7 +1692,8 @@ Preserve original term.== Numeric Payload Token Filter-This filter adds a numeric floating point payload value to tokens that match a given type. Refer to the Javadoc for the `org.apache.lucene.analysis.Token` class for more information about token types and payloads.+This filter adds a numeric floating point payload value to tokens that match a given type.+Refer to the Javadoc for the `org.apache.lucene.analysis.Token` class for more information about token types and payloads.*Factory class:* `solr.NumericPayloadTokenFilterFactory`@@ -1623,7 +1701,8 @@ This filter adds a numeric floating point payload value to tokens that match a g`payload`:: (required) A floating point value that will be added to all matching tokens.-`typeMatch`:: (required) A token type name string. Tokens with a matching type name will have their payload set to the above floating point value.+`typeMatch`:: (required) A token type name string.+Tokens with a matching type name will have their payload set to the above floating point value.*Example:*@@ -1661,7 +1740,8 @@ This filter adds a numeric floating point payload value to tokens that match a g== Pattern Replace Filter-This filter applies a regular expression to each token and, for those that match, substitutes the given replacement string in place of the matched pattern. Tokens which do not match are passed though unchanged.+This filter applies a regular expression to each token and, for those that match, substitutes the given replacement string in place of the matched pattern.+Tokens which do not match are passed though unchanged.*Factory class:* `solr.PatternReplaceFilterFactory`@@ -1669,7 +1749,9 @@ This filter applies a regular expression to each token and, for those that match`pattern`:: (required) The regular expression to test against each token, as per `java.util.regex.Pattern`.-`replacement`:: (required) A string to substitute in place of the matched pattern. This string may contain references to capture groups in the regex pattern. See the Javadoc for `java.util.regex.Matcher`.+`replacement`:: (required) A string to substitute in place of the matched pattern.+This string may contain references to capture groups in the regex pattern.+See the Javadoc for `java.util.regex.Matcher`.`replace`:: ("all" or "first", default "all") Indicates whether all occurrences of the pattern in the token should be replaced, or only the first.@@ -1729,7 +1811,9 @@ String replacement, first occurrence only:*Example:*-More complex pattern with capture group reference in the replacement. Tokens that start with non-numeric characters and end with digits will have an underscore inserted before the numbers. Otherwise the token is passed through.+More complex pattern with capture group reference in the replacement.+Tokens that start with non-numeric characters and end with digits will have an underscore inserted before the numbers.+Otherwise the token is passed through.[source,xml]----@@ -1747,15 +1831,19 @@ More complex pattern with capture group reference in the replacement. Tokens tha== Phonetic Filter-This filter creates tokens using one of the phonetic encoding algorithms in the `org.apache.commons.codec.language` package. For more information, see the section on <<phonetic-matching.adoc#,Phonetic Matching>>.+This filter creates tokens using one of the phonetic encoding algorithms in the `org.apache.commons.codec.language` package.+For more information, see the section on <<phonetic-matching.adoc#,Phonetic Matching>>.*Factory class:* `solr.PhoneticFilterFactory`*Arguments:*-`encoder`:: (required) The name of the encoder to use. The encoder name must be one of the following (case insensitive): `http://commons.apache.org/proper/commons-codec/archives/{ivy-commons-codec-version}/apidocs/org/apache/commons/codec/language/DoubleMetaphone.html[DoubleMetaphone]`, `http://commons.apache.org/proper/commons-codec/archives/{ivy-commons-codec-version}/apidocs/org/apache/commons/codec/language/Metaphone.html[Metaphone]`, `http://commons.apache.org/proper/commons-codec/archives/{ivy-commons-codec-version}/apidocs/org/apache/commons/codec/language/Soundex.html[Soundex]`, `http://commons.apache.org/proper/commons-codec/archives/{ivy-commons-codec-version}/apidocs/org/apache/commons/codec/language/RefinedSoundex.html[RefinedSoundex]`, `http://commons.apache.org/proper/commons-codec/archives/{ivy-commons-codec-version}/apidocs/org/apache/commons/codec/language/Caverphone.html[Caverphone]` (v2.0), `http://commons.apache.org/proper/commons-codec/archives/{ivy-commons-codec-version}/apidocs/org/apache/commons/codec/language/ColognePhonetic.html[ColognePhonetic]`, or `http://commons.apache.org/proper/commons-codec/archives/{ivy-commons-codec-version}/apidocs/org/apache/commons/codec/language/Nysiis.html[Nysiis]`.+`encoder`:: (required) The name of the encoder to use.+The encoder name must be one of the following (case insensitive): `http://commons.apache.org/proper/commons-codec/archives/{ivy-commons-codec-version}/apidocs/org/apache/commons/codec/language/DoubleMetaphone.html[DoubleMetaphone]`, `http://commons.apache.org/proper/commons-codec/archives/{ivy-commons-codec-version}/apidocs/org/apache/commons/codec/language/Metaphone.html[Metaphone]`, `http://commons.apache.org/proper/commons-codec/archives/{ivy-commons-codec-version}/apidocs/org/apache/commons/codec/language/Soundex.html[Soundex]`, `http://commons.apache.org/proper/commons-codec/archives/{ivy-commons-codec-version}/apidocs/org/apache/commons/codec/language/RefinedSoundex.html[RefinedSoundex]`, `http://commons.apache.org/proper/commons-codec/archives/{ivy-commons-codec-version}/apidocs/org/apache/commons/codec/language/Caverphone.html[Caverphone]` (v2.0), `http://commons.apache.org/proper/commons-codec/archives/{ivy-commons-codec-version}/apidocs/org/apache/commons/codec/language/ColognePhonetic.html[ColognePhonetic]`, or `http://commons.apache.org/proper/commons-codec/archives/{ivy-commons-codec-version}/apidocs/org/apache/commons/codec/language/Nysiis.html[Nysiis]`.-`inject`:: (true/false) If true (the default), then new phonetic tokens are added to the stream. Otherwise, tokens are replaced with the phonetic equivalent. Setting this to false will enable phonetic matching, but the exact spelling of the target word may not match.+`inject`:: (true/false) If true (the default), then new phonetic tokens are added to the stream.+Otherwise, tokens are replaced with the phonetic equivalent.+Setting this to false will enable phonetic matching, but the exact spelling of the target word may not match.`maxCodeLength`:: (integer) The maximum length of the code to be generated by the Metaphone or Double Metaphone encoders.@@ -1835,7 +1923,11 @@ Default Soundex encoder.== Porter Stem Filter-This filter applies the Porter Stemming Algorithm for English. The results are similar to using the Snowball Porter Stemmer with the `language="English"` argument. But this stemmer is coded directly in Java and is not based on Snowball. It does not accept a list of protected words and is only appropriate for English language text. However, it has been benchmarked as http://markmail.org/thread/d2c443z63z37rwf6[four times faster] than the English Snowball stemmer, so can provide a performance enhancement.+This filter applies the Porter Stemming Algorithm for English.+The results are similar to using the Snowball Porter Stemmer with the `language="English"` argument.+But this stemmer is coded directly in Java and is not based on Snowball.+It does not accept a list of protected words and is only appropriate for English language text.+However, it has been benchmarked as http://markmail.org/thread/d2c443z63z37rwf6[four times faster] than the English Snowball stemmer, so can provide a performance enhancement.*Factory class:* `solr.PorterStemFilterFactory`@@ -1885,9 +1977,11 @@ This filter enables a form of conditional filtering: it only applies its wrapped`protected`:: (required) Comma-separated list of files containing protected terms, one per line.-`wrappedFilters`:: (required) Case-insensitive comma-separated list of `TokenFilterFactory` SPI names (strip trailing `(Token)FilterFactory` from the factory name - see the https://docs.oracle.com/javase/8/docs/api/java/util/ServiceLoader.html[java.util.ServiceLoader interface]). Each filter name must be unique, so if you need to specify the same filter more than once, you must add case-insensitive unique `-id` suffixes to each same-SPI-named filter (note that the `-id` suffix is stripped prior to SPI lookup).+`wrappedFilters`:: (required) Case-insensitive comma-separated list of `TokenFilterFactory` SPI names (strip trailing `(Token)FilterFactory` from the factory name - see the https://docs.oracle.com/javase/8/docs/api/java/util/ServiceLoader.html[java.util.ServiceLoader interface]).+Each filter name must be unique, so if you need to specify the same filter more than once, you must add case-insensitive unique `-id` suffixes to each same-SPI-named filter (note that the `-id` suffix is stripped prior to SPI lookup).-`ignoreCase`:: (true/false, default false) Ignore case when testing for protected words. If true, the protected list should contain lowercase words.+`ignoreCase`:: (true/false, default false) Ignore case when testing for protected words.+If true, the protected list should contain lowercase words.*Example:*@@ -1927,7 +2021,8 @@ All terms except those in `protectedTerms.txt` are truncated at 4 characters and*Example:*-This example includes multiple same-named wrapped filters with unique `-id` suffixes. Note that both the filter SPI names and `-id` suffixes are treated case-insensitively.+This example includes multiple same-named wrapped filters with unique `-id` suffixes.+Note that both the filter SPI names and `-id` suffixes are treated case-insensitively.For all terms except those in `protectedTerms.txt`, synonyms are added, terms are reversed, and then synonyms are added for the reversed terms:@@ -1945,9 +2040,12 @@ For all terms except those in `protectedTerms.txt`, synonyms are added, terms ar== Remove Duplicates Token Filter-The filter removes duplicate tokens in the stream. Tokens are considered to be duplicates ONLY if they have the same text and position values.+The filter removes duplicate tokens in the stream.+Tokens are considered to be duplicates ONLY if they have the same text and position values.-Because positions must be the same, this filter might not do what a user expects it to do based on its name. It is a very specialized filter that is only useful in very specific circumstances. It has been so named for brevity, even though it is potentially misleading.+Because positions must be the same, this filter might not do what a user expects it to do based on its name.+It is a very specialized filter that is only useful in very specific circumstances.+It has been so named for brevity, even though it is potentially misleading.*Factory class:* `solr.RemoveDuplicatesTokenFilterFactory`@@ -1955,7 +2053,8 @@ Because positions must be the same, this filter might not do what a user expects*Example:*-One example of where `RemoveDuplicatesTokenFilterFactory` is useful in situations where a synonym file is being used in conjunction with a stemmer. In these situations, both the stemmer and the synonym filter can cause completely identical terms with the same positions to end up in the stream, increasing index size with no benefit.+One example of where `RemoveDuplicatesTokenFilterFactory` is useful in situations where a synonym file is being used in conjunction with a stemmer.+In these situations, both the stemmer and the synonym filter can cause completely identical terms with the same positions to end up in the stream, increasing index size with no benefit.Consider the following entry from a `synonyms.txt` file:@@ -2008,21 +2107,26 @@ When used in the following configuration:== Reversed Wildcard Filter-This filter reverses tokens to provide faster leading wildcard and prefix queries. Tokens without wildcards are not reversed.+This filter reverses tokens to provide faster leading wildcard and prefix queries.+Tokens without wildcards are not reversed.*Factory class:* `solr.ReversedWildcardFilterFactory`*Arguments:*-`withOriginal`:: (boolean) If true, the filter produces both original and reversed tokens at the same positions. If false, produces only reversed tokens.+`withOriginal`:: (boolean) If true, the filter produces both original and reversed tokens at the same positions.+If false, produces only reversed tokens.-`maxPosAsterisk`:: (integer, default = 2) The maximum position of the asterisk wildcard ('*') that triggers the reversal of the query term. Terms with asterisks at positions above this value are not reversed.+`maxPosAsterisk`:: (integer, default = 2) The maximum position of the asterisk wildcard ('*') that triggers the reversal of the query term.+Terms with asterisks at positions above this value are not reversed.-`maxPosQuestion`:: (integer, default = 1) The maximum position of the question mark wildcard ('?') that triggers the reversal of query term. To reverse only pure suffix queries (queries with a single leading asterisk), set this to 0 and `maxPosAsterisk` to 1.+`maxPosQuestion`:: (integer, default = 1) The maximum position of the question mark wildcard ('?') that triggers the reversal of query term.+To reverse only pure suffix queries (queries with a single leading asterisk), set this to 0 and `maxPosAsterisk` to 1.`maxFractionAsterisk`:: (float, default = 0.0) An additional parameter that triggers the reversal if asterisk ('*') position is less than this fraction of the query token length.-`minTrailing`:: (integer, default = 2) The minimum number of trailing characters in a query token after the last wildcard character. For good performance this should be set to a value larger than 1.+`minTrailing`:: (integer, default = 2) The minimum number of trailing characters in a query token after the last wildcard character.+For good performance this should be set to a value larger than 1.*Example:*@@ -2062,7 +2166,8 @@ This filter reverses tokens to provide faster leading wildcard and prefix querie== Shingle Filter-This filter constructs shingles, which are token n-grams, from the token stream. It combines runs of tokens into a single token.+This filter constructs shingles, which are token n-grams, from the token stream.+It combines runs of tokens into a single token.*Factory class:* `solr.ShingleFilterFactory`@@ -2134,9 +2239,13 @@ A shingle size of four, do not include original token.== Snowball Porter Stemmer Filter-This filter factory instantiates a language-specific stemmer generated by Snowball. Snowball is a software package that generates pattern-based word stemmers. This type of stemmer is not as accurate as a table-based stemmer, but is faster and less complex. Table-driven stemmers are labor intensive to create and maintain and so are typically commercial products.+This filter factory instantiates a language-specific stemmer generated by Snowball.+Snowball is a software package that generates pattern-based word stemmers.+This type of stemmer is not as accurate as a table-based stemmer, but is faster and less complex.+Table-driven stemmers are labor intensive to create and maintain and so are typically commercial products.-Solr contains Snowball stemmers for Armenian, Basque, Catalan, Danish, Dutch, English, Finnish, French, German, Hungarian, Italian, Norwegian, Portuguese, Romanian, Russian, Spanish, Swedish and Turkish. For more information on Snowball, visit http://snowball.tartarus.org/.+Solr contains Snowball stemmers for Armenian, Basque, Catalan, Danish, Dutch, English, Finnish, French, German, Hungarian, Italian, Norwegian, Portuguese, Romanian, Russian, Spanish, Swedish and Turkish.+For more information on Snowball, visit http://snowball.tartarus.org/.`StopFilterFactory`, `CommonGramsFilterFactory`, and `CommonGramsQueryFilterFactory` can optionally read stopwords in Snowball format (specify `format="snowball"` in the configuration of those FilterFactories).@@ -2144,9 +2253,14 @@ Solr contains Snowball stemmers for Armenian, Basque, Catalan, Danish, Dutch, En*Arguments:*-`language`:: (default "English") The name of a language, used to select the appropriate Porter stemmer to use. Case is significant. This string is used to select a package name in the `org.tartarus.snowball.ext` class hierarchy.+`language`:: (default "English") The name of a language, used to select the appropriate Porter stemmer to use.+Case is significant.+This string is used to select a package name in the `org.tartarus.snowball.ext` class hierarchy.-`protected`:: Path of a text file containing a list of protected words, one per line. Protected words will not be stemmed. Blank lines and lines that begin with "#" are ignored. This may be an absolute path, or a simple file name in the Solr `conf` directory.+`protected`:: Path of a text file containing a list of protected words, one per line.+Protected words will not be stemmed.+Blank lines and lines that begin with "#" are ignored.+This may be an absolute path, or a simple file name in the Solr `conf` directory.*Example:*@@ -2222,23 +2336,29 @@ Spanish stemmer, Spanish words:== Stop Filter-This filter discards, or _stops_ analysis of, tokens that are on the given stop words list. A standard stop words list is included in the Solr `conf` directory, named `stopwords.txt`, which is appropriate for typical English language text.+This filter discards, or _stops_ analysis of, tokens that are on the given stop words list.+A standard stop words list is included in the Solr `conf` directory, named `stopwords.txt`, which is appropriate for typical English language text.*Factory class:* `solr.StopFilterFactory`*Arguments:*-`words`:: (optional) The path to a file that contains a list of stop words, one per line. Blank lines and lines that begin with "#" are ignored. This may be an absolute path, or path relative to the Solr `conf` directory.+`words`:: (optional) The path to a file that contains a list of stop words, one per line.+Blank lines and lines that begin with "#" are ignored.+This may be an absolute path, or path relative to the Solr `conf` directory.`format`:: (optional) If the stopwords list has been formatted for Snowball, you can specify `format="snowball"` so Solr can read the stopwords file.-`ignoreCase`:: (true/false, default false) Ignore case when testing for stop words. If true, the stop list should contain lowercase words.+`ignoreCase`:: (true/false, default false) Ignore case when testing for stop words.+If true, the stop list should contain lowercase words.-`enablePositionIncrements`:: if `luceneMatchVersion` is `4.4` or earlier and `enablePositionIncrements="false"`, no position holes will be left by this filter when it removes tokens. *This argument is invalid if `luceneMatchVersion` is `5.0` or later.*+`enablePositionIncrements`:: if `luceneMatchVersion` is `4.4` or earlier and `enablePositionIncrements="false"`, no position holes will be left by this filter when it removes tokens.+*This argument is invalid if `luceneMatchVersion` is `5.0` or later.**Example:*-Case-sensitive matching, capitalized words not stopped. Token positions skip stopped words.+Case-sensitive matching, capitalized words not stopped.+Token positions skip stopped words.[.dynamic-tabs]--@@ -2292,9 +2412,11 @@ Case-sensitive matching, capitalized words not stopped. Token positions skip stoLike <<Stop Filter>>, this filter discards, or _stops_ analysis of, tokens that are on the given stop words list.-Suggest Stop Filter differs from Stop Filter in that it will not remove the last token unless it is followed by a token separator. For example, a query `"find the"` would preserve the `'the'` since it was not followed by a space, punctuation, etc., and mark it as a `KEYWORD` so that following filters will not change or remove it.+Suggest Stop Filter differs from Stop Filter in that it will not remove the last token unless it is followed by a token separator.+For example, a query `"find the"` would preserve the `'the'` since it was not followed by a space, punctuation, etc., and mark it as a `KEYWORD` so that following filters will not change or remove it.-By contrast, a query like "`find the popsicle`" would remove '`the`' as a stopword, since it's followed by a space. When using one of the analyzing suggesters, you would normally use the ordinary `StopFilterFactory` in your index analyzer and then SuggestStopFilter in your query analyzer.+By contrast, a query like "`find the popsicle`" would remove '`the`' as a stopword, since it's followed by a space.+When using one of the analyzing suggesters, you would normally use the ordinary `StopFilterFactory` in your index analyzer and then SuggestStopFilter in your query analyzer.*Factory class:* `solr.SuggestStopFilterFactory`@@ -2302,11 +2424,15 @@ By contrast, a query like "`find the popsicle`" would remove '`the`' as a stopwo`words`:: (optional; default: {lucene-javadocs}/analysis/common/org/apache/lucene/analysis/core/StopAnalyzer.html[`StopAnalyzer#ENGLISH_STOP_WORDS_SET`] ) The name of a stopwords file to parse.-`format`:: (optional; default: `wordset`) Defines how the words file will be parsed. If `words` is not specified, then `format` must not be specified. The valid values for the format option are:+`format`:: (optional; default: `wordset`) Defines how the words file will be parsed.+If `words` is not specified, then `format` must not be specified.+The valid values for the format option are:-`wordset`:: This is the default format, which supports one word per line (including any intra-word whitespace) and allows whole line comments beginning with the `#` character. Blank lines are ignored.+`wordset`:: This is the default format, which supports one word per line (including any intra-word whitespace) and allows whole line comments beginning with the `#` character.+Blank lines are ignored.-`snowball`:: This format allows for multiple words specified on each line, and trailing comments may be specified using the vertical line (`|`). Blank lines are ignored.+`snowball`:: This format allows for multiple words specified on each line, and trailing comments may be specified using the vertical line (`|`).+Blank lines are ignored.`ignoreCase`:: (optional; default: *false*) If *true*, matching is case-insensitive.@@ -2350,7 +2476,9 @@ By contrast, a query like "`find the popsicle`" would remove '`the`' as a stopwo== Synonym Filter-This filter does synonym mapping. Each token is looked up in the list of synonyms and if a match is found, then the synonym is emitted in place of the token. The position value of the new tokens are set such they all occur at the same position as the original token.+This filter does synonym mapping.+Each token is looked up in the list of synonyms and if a match is found, then the synonym is emitted in place of the token.+The position value of the new tokens are set such they all occur at the same position as the original token..Synonym Filter has been Deprecated[WARNING]@@ -2364,9 +2492,11 @@ For arguments and examples, see the Synonym Graph Filter below.== Synonym Graph Filter-This filter maps single- or multi-token synonyms, producing a fully correct graph output. This filter is a replacement for the Synonym Filter, which produces incorrect graphs for multi-token synonyms.+This filter maps single- or multi-token synonyms, producing a fully correct graph output.+This filter is a replacement for the Synonym Filter, which produces incorrect graphs for multi-token synonyms.-If you use this filter during indexing, you must follow it with a Flatten Graph Filter to squash tokens on top of one another like the Synonym Filter, because the indexer can't directly consume a graph. To get fully correct positional queries when your synonym replacements are multiple tokens, you should instead apply synonyms using this filter at query time.+If you use this filter during indexing, you must follow it with a Flatten Graph Filter to squash tokens on top of one another like the Synonym Filter, because the indexer can't directly consume a graph.+To get fully correct positional queries when your synonym replacements are multiple tokens, you should instead apply synonyms using this filter at query time.NOTE: Although this filter produces correct token graphs, it cannot consume an input token graph correctly.@@ -2374,27 +2504,37 @@ NOTE: Although this filter produces correct token graphs, it cannot consume an i*Arguments:*-`synonyms`:: (required) The path of a file that contains a list of synonyms, one per line. In the (default) `solr` format - see the `format` argument below for alternatives - blank lines and lines that begin with "`#`" are ignored. This may be a comma-separated list of paths. See <<resource-loading.adoc#,Resource Loading>> for more information.+`synonyms`:: (required) The path of a file that contains a list of synonyms, one per line.+In the (default) `solr` format - see the `format` argument below for alternatives - blank lines and lines that begin with "`#`" are ignored.+This may be a comma-separated list of paths.+See <<resource-loading.adoc#,Resource Loading>> for more information.+There are two ways to specify synonym mappings:+-* A comma-separated list of words. If the token matches any of the words, then all the words in the list are substituted, which will include the original token.+* A comma-separated list of words.+If the token matches any of the words, then all the words in the list are substituted, which will include the original token.+-* Two comma-separated lists of words with the symbol "\=>" between them. If the token matches any word on the left, then the list on the right is substituted. The original token will not be included unless it is also in the list on the right.+* Two comma-separated lists of words with the symbol "\=>" between them.+If the token matches any word on the left, then the list on the right is substituted.+The original token will not be included unless it is also in the list on the right.`ignoreCase`:: (optional; default: `false`) If `true`, synonyms will be matched case-insensitively.-`expand`:: (optional; default: `true`) If `true`, a synonym will be expanded to all equivalent synonyms. If `false`, all equivalent synonyms will be reduced to the first in the list.+`expand`:: (optional; default: `true`) If `true`, a synonym will be expanded to all equivalent synonyms.+If `false`, all equivalent synonyms will be reduced to the first in the list.-`format`:: (optional; default: `solr`) Controls how the synonyms will be parsed. The short names `solr` (for {lucene-javadocs}/analysis/common/org/apache/lucene/analysis/synonym/SolrSynonymParser.html[`SolrSynonymParser)`] and `wordnet` (for {lucene-javadocs}/analysis/common/org/apache/lucene/analysis/synonym/WordnetSynonymParser.html[`WordnetSynonymParser`] ) are supported, or you may alternatively supply the name of your own {lucene-javadocs}/analysis/common/org/apache/lucene/analysis/synonym/SynonymMap.Builder.html[`SynonymMap.Builder`] subclass.+`format`:: (optional; default: `solr`) Controls how the synonyms will be parsed.+The short names `solr` (for {lucene-javadocs}/analysis/common/org/apache/lucene/analysis/synonym/SolrSynonymParser.html[`SolrSynonymParser)`] and `wordnet` (for {lucene-javadocs}/analysis/common/org/apache/lucene/analysis/synonym/WordnetSynonymParser.html[`WordnetSynonymParser`] ) are supported, or you may alternatively supply the name of your own {lucene-javadocs}/analysis/common/org/apache/lucene/analysis/synonym/SynonymMap.Builder.html[`SynonymMap.Builder`] subclass.-`tokenizerFactory`:: (optional; default: `WhitespaceTokenizerFactory`) The name of the tokenizer factory to use when parsing the synonyms file. Arguments with the name prefix `tokenizerFactory.*` will be supplied as init params to the specified tokenizer factory.+`tokenizerFactory`:: (optional; default: `WhitespaceTokenizerFactory`) The name of the tokenizer factory to use when parsing the synonyms file.+Arguments with the name prefix `tokenizerFactory.*` will be supplied as init params to the specified tokenizer factory.+Any arguments not consumed by the synonym filter factory, including those without the `tokenizerFactory.*` prefix, will also be supplied as init params to the tokenizer factory.+If `tokenizerFactory` is specified, then `analyzer` may not be, and vice versa.-`analyzer`:: (optional; default: `WhitespaceTokenizerFactory`) The name of the analyzer class to use when parsing the synonyms file. If `analyzer` is specified, then `tokenizerFactory` may not be, and vice versa.+`analyzer`:: (optional; default: `WhitespaceTokenizerFactory`) The name of the analyzer class to use when parsing the synonyms file.+If `analyzer` is specified, then `tokenizerFactory` may not be, and vice versa.For the following examples, assume a synonyms file named `mysynonyms.txt`:@@ -2535,13 +2675,15 @@ This filter adds the numeric character offsets of the token as a payload value f== Trim Filter-This filter trims leading and/or trailing whitespace from tokens. Most tokenizers break tokens at whitespace, so this filter is most often used for special situations.+This filter trims leading and/or trailing whitespace from tokens.+Most tokenizers break tokens at whitespace, so this filter is most often used for special situations.*Factory class:* `solr.TrimFilterFactory`*Arguments:*-`updateOffsets`:: if `luceneMatchVersion` is `4.3` or earlier and `updateOffsets="true"`, trimmed tokens' start and end offsets will be updated to those of the first and last characters (plus one) remaining in the token. *This argument is invalid if `luceneMatchVersion` is `5.0` or later.*+`updateOffsets`:: if `luceneMatchVersion` is `4.3` or earlier and `updateOffsets="true"`, trimmed tokens' start and end offsets will be updated to those of the first and last characters (plus one) remaining in the token.+*This argument is invalid if `luceneMatchVersion` is `5.0` or later.**Example:*@@ -2691,7 +2833,9 @@ With the example below, for a token "example.com" with type `<URL>`, the token e== Type Token Filter-This filter blacklists or whitelists a specified list of token types, assuming the tokens have type metadata associated with them. For example, the <<tokenizers.adoc#uax29-url-email-tokenizer,UAX29 URL Email Tokenizer>> emits "<URL>" and "<EMAIL>" typed tokens, as well as other types. This filter would allow you to pull out only e-mail addresses from text as tokens, if you wish.+This filter blacklists or whitelists a specified list of token types, assuming the tokens have type metadata associated with them.+For example, the <<tokenizers.adoc#uax29-url-email-tokenizer,UAX29 URL Email Tokenizer>> emits "<URL>" and "<EMAIL>" typed tokens, as well as other types.+This filter would allow you to pull out only e-mail addresses from text as tokens, if you wish.*Factory class:* `solr.TypeTokenFilterFactory`@@ -2699,9 +2843,11 @@ This filter blacklists or whitelists a specified list of token types, assuming t`types`:: Defines the location of a file of types to filter.-`useWhitelist`:: If *true*, the file defined in `types` should be used as include list. If *false*, or undefined, the file defined in `types` is used as a blacklist.+`useWhitelist`:: If *true*, the file defined in `types` should be used as include list.+If *false*, or undefined, the file defined in `types` is used as a blacklist.-`enablePositionIncrements`:: if `luceneMatchVersion` is `4.3` or earlier and `enablePositionIncrements="false"`, no position holes will be left by this filter when it removes tokens. *This argument is invalid if `luceneMatchVersion` is `5.0` or later.*+`enablePositionIncrements`:: if `luceneMatchVersion` is `4.3` or earlier and `enablePositionIncrements="false"`, no position holes will be left by this filter when it removes tokens.+*This argument is invalid if `luceneMatchVersion` is `5.0` or later.**Example:*@@ -2747,7 +2893,8 @@ For a full description, including arguments and examples, see the Word DelimiterThis filter splits tokens at word delimiters.-If you use this filter during indexing, you must follow it with a Flatten Graph Filter to squash tokens on top of one another like the Word Delimiter Filter, because the indexer can't directly consume a graph. To get fully correct positional queries when tokens are split, you should instead use this filter at query time.+If you use this filter during indexing, you must follow it with a Flatten Graph Filter to squash tokens on top of one another like the Word Delimiter Filter, because the indexer can't directly consume a graph.+To get fully correct positional queries when tokens are split, you should instead use this filter at query time.Note: although this filter produces correct token graphs, it cannot consume an input token graph correctly.@@ -2767,7 +2914,8 @@ The rules for determining delimiters are determined as follows:*Arguments:*-`generateWordParts`:: (integer, default 1) If non-zero, splits words at delimiters. For example:"CamelCase", "hot-spot" -> "Camel", "Case", "hot", "spot"+`generateWordParts`:: (integer, default 1) If non-zero, splits words at delimiters.+For example:"CamelCase", "hot-spot" -> "Camel", "Case", "hot", "spot"`generateNumberParts`:: (integer, default 1) If non-zero, splits numeric strings at delimiters:"1947-32" -> *"1947", "32"@@ -2787,9 +2935,12 @@ The rules for determining delimiters are determined as follows:`stemEnglishPossessive`:: (integer, default 1) If 1, strips the possessive `'s` from each subword.-`types`:: (optional) The pathname of a file that contains *character \=> type* mappings, which enable customization of this filter's splitting behavior. Recognized character types: `LOWER`, `UPPER`, `ALPHA`, `DIGIT`, `ALPHANUM`, and `SUBWORD_DELIM`.+`types`:: (optional) The pathname of a file that contains *character \=> type* mappings, which enable customization of this filter's splitting behavior.+Recognized character types: `LOWER`, `UPPER`, `ALPHA`, `DIGIT`, `ALPHANUM`, and `SUBWORD_DELIM`.+-The default for any character without a customized mapping is computed from Unicode character properties. Blank lines and comment lines starting with '#' are ignored. An example file:+The default for any character without a customized mapping is computed from Unicode character properties.+Blank lines and comment lines starting with '#' are ignored.+An example file:+[source,text]----@@ -2804,7 +2955,8 @@ $ => DIGIT*Example:*-Default behavior. The whitespace tokenizer is used here to preserve non-alphanumeric characters.+Default behavior.+The whitespace tokenizer is used here to preserve non-alphanumeric characters.[.dynamic-tabs]--@@ -2850,7 +3002,8 @@ Default behavior. The whitespace tokenizer is used here to preserve non-alphanum*Example:*-Do not split on case changes, and do not generate number parts. Note that by not generating number parts, tokens containing only numeric parts are ultimately discarded.+Do not split on case changes, and do not generate number parts.+Note that by not generating number parts, tokens containing only numeric parts are ultimately discarded.[source,xml]----@@ -2886,7 +3039,8 @@ Concatenate word parts and number parts, but not word and number parts that occu*Example:*-Concatenate all. Word and/or number parts are joined together.+Concatenate all.+Word and/or number parts are joined together.[source,xml]----diff --git a/solr/solr-ref-guide/src/function-queries.adoc b/solr/solr-ref-guide/src/function-queries.adocindex 3a57378e0c9..e0df3163d4f 100644--- a/solr/solr-ref-guide/src/function-queries.adoc+++ b/solr/solr-ref-guide/src/function-queries.adoc@@ -20,7 +20,9 @@ Function queries enable you to generate a relevancy score using the actual valueFunction queries are supported by the <<dismax-query-parser.adoc#,DisMax>>, <<edismax-query-parser.adoc#,Extended DisMax>>, and <<standard-query-parser.adoc#,standard>> query parsers.-Function queries use _functions_. The functions can be a constant (numeric or string literal), a field, another function or a parameter substitution argument. You can use these functions to modify the ranking of results for users. These could be used to change the ranking of results based on a user's location, or some other calculation.+Function queries use _functions_. The functions can be a constant (numeric or string literal), a field, another function or a parameter substitution argument.+You can use these functions to modify the ranking of results for users.+These could be used to change the ranking of results based on a user's location, or some other calculation.== Using Function Query@@ -34,13 +36,15 @@ There are several ways of using function queries in a Solr query:----q={!func}div(popularity,price)&fq={!frange l=1000}customer_ratings-----* In a Sort expression. For example:+* In a Sort expression.+For example:+[source,text]----sort=div(popularity,price) desc, score desc-----* Add the results of functions as pseudo-fields to documents in query results. For instance, for:+* Add the results of functions as pseudo-fields to documents in query results.+For instance, for:+[source,text]----@@ -57,13 +61,16 @@ the output would be:<float name="score">0.343</float>...-----* Use in a parameter that is explicitly for specifying functions, such as the eDisMax query parser's <<edismax-query-parser.adoc#extended-dismax-parameters,`boost` parameter>>, or the DisMax query parser's <<dismax-query-parser.adoc#bf-boost-functions-parameter,`bf` (boost function) parameter>>. (Note that the `bf` parameter actually takes a list of function queries separated by white space and each with an optional boost. Make sure you eliminate any internal white space in single function queries when using `bf`). For example:+* Use in a parameter that is explicitly for specifying functions, such as the eDisMax query parser's <<edismax-query-parser.adoc#extended-dismax-parameters,`boost` parameter>>, or the DisMax query parser's <<dismax-query-parser.adoc#bf-boost-functions-parameter,`bf` (boost function) parameter>>. (Note that the `bf` parameter actually takes a list of function queries separated by white space and each with an optional boost.+Make sure you eliminate any internal white space in single function queries when using `bf`).+For example:+[source,text]----q=dismax&bf="ord(popularity)^0.5 recip(rord(price),1,1000,1000)^0.3"-----* Introduce a function query inline in the Lucene query parser with the `\_val_` keyword. For example:+* Introduce a function query inline in the Lucene query parser with the `\_val_` keyword.+For example:+[source,text]----@@ -109,7 +116,9 @@ Specifies a floating point constant.* `1.5`=== def Function-`def` is short for default. Returns the value of field "field", or if the field does not exist, returns the default value specified. Yields the first value where `exists()==true`.+`def` is short for default.+Returns the value of field "field", or if the field does not exist, returns the default value specified.+Yields the first value where `exists()==true`.*Syntax Examples*@@ -117,7 +126,8 @@ Specifies a floating point constant.* `def(myfield, 1.0):` equivalent to `if(exists(myfield),myfield,1.0)`=== div Function-Divides one value or function by another. `div(x,y)` divides `x` by `y`.+Divides one value or function by another.+`div(x,y)` divides `x` by `y`.*Syntax Examples*@@ -125,7 +135,9 @@ Divides one value or function by another. `div(x,y)` divides `x` by `y`.* `div(sum(x,100),max(y,1))`=== dist Function-Returns the distance between two vectors (points) in an n-dimensional space. Takes in the power, plus two or more ValueSource instances and calculates the distances between the two vectors. Each ValueSource must be a number.+Returns the distance between two vectors (points) in an n-dimensional space.+Takes in the power, plus two or more ValueSource instances and calculates the distances between the two vectors.+Each ValueSource must be a number.There must be an even number of ValueSource instances passed in and the method assumes that the first half represent the first vector and the second half represent the second vector.@@ -137,7 +149,8 @@ There must be an even number of ValueSource instances passed in and the method a* `dist(1,x,y,z,e,f,g)`: Manhattan distance between (x,y,z) and (e,f,g) where each letter is a field name.=== docfreq(field,val) Function-Returns the number of documents that contain the term in the field. This is a constant (the same value for all documents in the index).+Returns the number of documents that contain the term in the field.+This is a constant (the same value for all documents in the index).You can quote the term if it's more complex, or do parameter substitution for the term value.@@ -147,7 +160,8 @@ You can quote the term if it's more complex, or do parameter substitution for th* `...&defType=func` `&q=docfreq(text,$myterm)&myterm=solr`=== field Function-Returns the numeric docValues or indexed value of the field with the specified name. In its simplest (single argument) form, this function can only be used on single valued fields, and can be called using the name of the field as a string, or for most conventional field names simply use the field name by itself without using the `field(...)` syntax.+Returns the numeric docValues or indexed value of the field with the specified name.+In its simplest (single argument) form, this function can only be used on single valued fields, and can be called using the name of the field as a string, or for most conventional field names simply use the field name by itself without using the `field(...)` syntax.When using docValues, an optional 2nd argument can be specified to select the `min` or `max` value of multivalued fields.@@ -170,21 +184,26 @@ For multivalued docValues fields:* `field(myMultiValuedFloatField,max)`=== hsin Function-The Haversine distance calculates the distance between two points on a sphere when traveling along the sphere. The values must be in radians. `hsin` also take a Boolean argument to specify whether the function should convert its output to radians.+The Haversine distance calculates the distance between two points on a sphere when traveling along the sphere.+The values must be in radians.+`hsin` also take a Boolean argument to specify whether the function should convert its output to radians.*Syntax Example** `hsin(2, true, x, y, 0, 0)`=== idf Function-Inverse document frequency; a measure of whether the term is common or rare across all documents. Obtained by dividing the total number of documents by the number of documents containing the term, and then taking the logarithm of that quotient. See also `tf`.+Inverse document frequency; a measure of whether the term is common or rare across all documents.+Obtained by dividing the total number of documents by the number of documents containing the term, and then taking the logarithm of that quotient.+See also `tf`.*Syntax Example** `idf(fieldName,'solr')`: measures the inverse of the frequency of the occurrence of the term `'solr'` in `fieldName`.=== if Function-Enables conditional function queries. In `if(test,value1,value2)`:+Enables conditional function queries.+In `if(test,value1,value2)`:* `test` is or refers to a logical value or expression that returns a logical value (TRUE or FALSE).* `value1` is the value that is returned by the function if `test` yields TRUE.@@ -194,10 +213,12 @@ An expression can be any function which outputs boolean values, or even function*Syntax Example*-* `if(termfreq (cat,'electronics'),popularity,42)`: This function checks each document for to see if it contains the term "electronics" in the `cat` field. If it does, then the value of the `popularity` field is returned, otherwise the value of `42` is returned.+* `if(termfreq (cat,'electronics'),popularity,42)`: This function checks each document for to see if it contains the term "electronics" in the `cat` field.+If it does, then the value of the `popularity` field is returned, otherwise the value of `42` is returned.=== linear Function-Implements `m*x+c` where `m` and `c` are constants and `x` is an arbitrary function. This is equivalent to `sum(product(m,x),c)`, but slightly more efficient as it is implemented as a single function.+Implements `m*x+c` where `m` and `c` are constants and `x` is an arbitrary function.+This is equivalent to `sum(product(m,x),c)`, but slightly more efficient as it is implemented as a single function.*Syntax Examples*@@ -213,20 +234,24 @@ Returns the log base 10 of the specified function.* `log(sum(x,100))`=== map Function-Maps any values of an input function `x` that fall within `min` and `max` inclusive to the specified `target`. The arguments `min` and `max` must be constants. The arguments `target` and `default` can be constants or functions.+Maps any values of an input function `x` that fall within `min` and `max` inclusive to the specified `target`.+The arguments `min` and `max` must be constants.+The arguments `target` and `default` can be constants or functions.If the value of `x` does not fall between `min` and `max`, then either the value of `x` is returned, or a default value is returned if specified as a 5th argument.*Syntax Examples** `map(x,min,max,target)`-** `map(x,0,0,1)`: Changes any values of 0 to 1. This can be useful in handling default 0 values.+** `map(x,0,0,1)`: Changes any values of 0 to 1.+This can be useful in handling default 0 values.* `map(x,min,max,target,default)`** `map(x,0,100,1,-1)`: Changes any values between `0` and `100` to `1`, and all other values to` -1`.** `map(x,0,100,sum(x,599),docfreq(text,solr))`: Changes any values between `0` and `100` to x+599, and all other values to frequency of the term 'solr' in the field text.=== max Function-Returns the maximum numeric value of multiple nested functions or constants, which are specified as arguments: `max(x,y,...)`. The `max` function can also be useful for "bottoming out" another function or field at some specified constant.+Returns the maximum numeric value of multiple nested functions or constants, which are specified as arguments: `max(x,y,...)`.+The `max` function can also be useful for "bottoming out" another function or field at some specified constant.Use the `field(myfield,max)` syntax for <<field Function,selecting the maximum value of a single multivalued field>>.@@ -235,14 +260,16 @@ Use the `field(myfield,max)` syntax for <<field Function,selecting the maximum v* `max(myfield,myotherfield,0)`=== maxdoc Function-Returns the number of documents in the index, including those that are marked as deleted but have not yet been purged. This is a constant (the same value for all documents in the index).+Returns the number of documents in the index, including those that are marked as deleted but have not yet been purged.+This is a constant (the same value for all documents in the index).*Syntax Example** `maxdoc()`=== min Function-Returns the minimum numeric value of multiple nested functions of constants, which are specified as arguments: `min(x,y,...)`. The `min` function can also be useful for providing an "upper bound" on a function using a constant.+Returns the minimum numeric value of multiple nested functions of constants, which are specified as arguments: `min(x,y,...)`.+The `min` function can also be useful for providing an "upper bound" on a function using a constant.Use the `field(myfield,min)` <<field Function,syntax for selecting the minimum value of a single multivalued field>>.@@ -251,7 +278,8 @@ Use the `field(myfield,min)` <<field Function,syntax for selecting the minimum v* `min(myfield,myotherfield,0)`=== ms Function-Returns milliseconds of difference between its arguments. Dates are relative to the Unix or POSIX time epoch, midnight, January 1, 1970 UTC.+Returns milliseconds of difference between its arguments.+Dates are relative to the Unix or POSIX time epoch, midnight, January 1, 1970 UTC.Arguments may be the name of a `DatePointField`, `TrieDateField`, or date math based on a <<date-formatting-math.adoc#,constant date or `NOW`>>.@@ -269,14 +297,16 @@ Arguments may be the name of a `DatePointField`, `TrieDateField`, or date math b* `ms(datefield1, datefield2)`=== norm(_field_) Function-Returns the "norm" stored in the index for the specified field. This is the product of the index time boost and the length normalization factor, according to the {lucene-javadocs}/core/org/apache/lucene/search/similarities/Similarity.html[Similarity] for the field.+Returns the "norm" stored in the index for the specified field.+This is the product of the index time boost and the length normalization factor, according to the {lucene-javadocs}/core/org/apache/lucene/search/similarities/Similarity.html[Similarity] for the field.*Syntax Example** `norm(fieldName)`=== numdocs Function-Returns the number of documents in the index, not including those that are marked as deleted but have not yet been purged. This is a constant (the same value for all documents in the index).+Returns the number of documents in the index, not including those that are marked as deleted but have not yet been purged.+This is a constant (the same value for all documents in the index).*Syntax Example*@@ -285,7 +315,9 @@ Returns the number of documents in the index, not including those that are marke=== ord FunctionReturns the ordinal of the indexed field value within the indexed list of terms for that field in Lucene index order (lexicographically ordered by unicode value), starting at 1.-In other words, for a given field, all values are ordered lexicographically; this function then returns the offset of a particular value in that ordering. The field must have a maximum of one value per document (not multi-valued). `0` is returned for documents without a value in the field.+In other words, for a given field, all values are ordered lexicographically; this function then returns the offset of a particular value in that ordering.+The field must have a maximum of one value per document (not multi-valued).+`0` is returned for documents without a value in the field.IMPORTANT: `ord()` depends on the position in an index and can change when other documents are inserted or deleted.@@ -300,12 +332,15 @@ See also `rord` below.=== payload FunctionReturns the float value computed from the decoded payloads of the term specified.-The return value is computed using the `min`, `max`, or `average` of the decoded payloads. A special `first` function can be used instead of the others, to short-circuit term enumeration and return only the decoded payload of the first term.+The return value is computed using the `min`, `max`, or `average` of the decoded payloads.+A special `first` function can be used instead of the others, to short-circuit term enumeration and return only the decoded payload of the first term.-The field specified must have float or integer payload encoding capability (via `DelimitedPayloadTokenFilter` or `NumericPayloadTokenFilter`). If no payload is found for the term, the default value is returned.+The field specified must have float or integer payload encoding capability (via `DelimitedPayloadTokenFilter` or `NumericPayloadTokenFilter`).+If no payload is found for the term, the default value is returned.* `payload(field_name,term)`: default value is 0.0, `average` function is used.-* `payload(field_name,term,default_value)`: default value can be a constant, field name, or another float returning function. `average` function used.+* `payload(field_name,term,default_value)`: default value can be a constant, field name, or another float returning function.+`average` function used.* `payload(field_name,term,default_value,function)`: function values can be `min`, `max`, `average`, or `first`.*Syntax Example*@@ -314,7 +349,8 @@ The field specified must have float or integer payload encoding capability (via=== pow Function-Raises the specified base to the specified power. `pow(x,y)` raises `x` to the power of `y`.+Raises the specified base to the specified power.+`pow(x,y)` raises `x` to the power of `y`.*Syntax Examples*@@ -323,7 +359,8 @@ Raises the specified base to the specified power. `pow(x,y)` raises `x` to the p* `pow(x,0.5):` the same as `sqrt`=== product Function-Returns the product of multiple values or functions, which are specified in a comma-separated list. `mul(...)` may also be used as an alias for this function.+Returns the product of multiple values or functions, which are specified in a comma-separated list.+`mul(...)` may also be used as an alias for this function.*Syntax Examples*@@ -332,7 +369,8 @@ Returns the product of multiple values or functions, which are specified in a co* `mul(x,y)`=== query Function-Returns the score for the given subquery, or the default value for documents not matching the query. Any type of subquery is supported through either parameter de-referencing `$otherparam` or direct specification of the query string in the <<local-params.adoc#,local params>> through the `v` key.+Returns the score for the given subquery, or the default value for documents not matching the query.+Any type of subquery is supported through either parameter de-referencing `$otherparam` or direct specification of the query string in the <<local-params.adoc#,local params>> through the `v` key.*Syntax Examples*@@ -344,7 +382,9 @@ Returns the score for the given subquery, or the default value for documents not=== recip FunctionPerforms a reciprocal function with `recip(x,m,a,b)` implementing `a/(m*x+b)` where `m,a,b` are constants, and `x` is any arbitrarily complex function.-When `a` and `b` are equal, and `x>=0`, this function has a maximum value of `1` that drops as `x` increases. Increasing the value of `a` and `b` together results in a movement of the entire function to a flatter part of the curve. These properties can make this an ideal function for boosting more recent documents when x is `rord(datefield)`.+When `a` and `b` are equal, and `x>=0`, this function has a maximum value of `1` that drops as `x` increases.+Increasing the value of `a` and `b` together results in a movement of the entire function to a flatter part of the curve.+These properties can make this an ideal function for boosting more recent documents when x is `rord(datefield)`.*Syntax Examples*@@ -359,9 +399,13 @@ Returns the reverse ordering of that returned by `ord`.* `rord(myDateField)`=== scale Function-Scales values of the function `x` such that they fall between the specified `minTarget` and `maxTarget` inclusive. The current implementation traverses all of the function values to obtain the min and max, so it can pick the correct scale.+Scales values of the function `x` such that they fall between the specified `minTarget` and `maxTarget` inclusive.+The current implementation traverses all of the function values to obtain the min and max, so it can pick the correct scale.-The current implementation cannot distinguish when documents have been deleted or documents that have no value. It uses `0.0` values for these cases. This means that if values are normally all greater than `0.0`, one can still end up with `0.0` as the `min` value to map from. In these cases, an appropriate `map()` function could be used as a workaround to change `0.0` to a value in the real range, as shown here: `scale(map(x,0,0,5),1,2)`+The current implementation cannot distinguish when documents have been deleted or documents that have no value.+It uses `0.0` values for these cases.+This means that if values are normally all greater than `0.0`, one can still end up with `0.0` as the `min` value to map from.+In these cases, an appropriate `map()` function could be used as a workaround to change `0.0` to a value in the real range, as shown here: `scale(map(x,0,0,5),1,2)`*Syntax Examples*@@ -369,7 +413,9 @@ The current implementation cannot distinguish when documents have been deleted o* `scale(x,1,2)`: scales the values of x such that all values will be between 1 and 2 inclusive.=== sqedist Function-The Square Euclidean distance calculates the 2-norm (Euclidean distance) but does not take the square root, thus saving a fairly expensive operation. It is often the case that applications that care about Euclidean distance do not need the actual distance, but instead can use the square of the distance. There must be an even number of ValueSource instances passed in and the method assumes that the first half represent the first vector and the second half represent the second vector.+The Square Euclidean distance calculates the 2-norm (Euclidean distance) but does not take the square root, thus saving a fairly expensive operation.+It is often the case that applications that care about Euclidean distance do not need the actual distance, but instead can use the square of the distance.+There must be an even number of ValueSource instances passed in and the method assumes that the first half represent the first vector and the second half represent the second vector.*Syntax Example*@@ -385,14 +431,18 @@ Returns the square root of the specified value or function.* `sqrt(sum(x,100))`=== strdist Function-Calculate the distance between two strings. Uses the Lucene spell checker `StringDistance` interface and supports all of the implementations available in that package, plus allows applications to plug in their own via Solr's resource loading capabilities. `strdist` takes (string1, string2, distance measure).+Calculate the distance between two strings.+Uses the Lucene spell checker `StringDistance` interface and supports all of the implementations available in that package, plus allows applications to plug in their own via Solr's resource loading capabilities.+`strdist` takes (string1, string2, distance measure).Possible values for distance measure are:* jw: Jaro-Winkler* edit: Levenstein or Edit distance-* ngram: The NGramDistance, if specified, can optionally pass in the ngram size too. Default is 2.-* FQN: Fully Qualified class Name for an implementation of the StringDistance interface. Must have a no-arg constructor.+* ngram: The NGramDistance, if specified, can optionally pass in the ngram size too.+Default is 2.+* FQN: Fully Qualified class Name for an implementation of the StringDistance interface.+Must have a no-arg constructor.*Syntax Example*@@ -407,7 +457,8 @@ Returns `x-y` from `sub(x,y)`.* `sub(100, sqrt(myfield))`=== sum Function-Returns the sum of multiple values or functions, which are specified in a comma-separated list. `add(...)` may be used as an alias for this function.+Returns the sum of multiple values or functions, which are specified in a comma-separated list.+`add(...)` may be used as an alias for this function.*Syntax Examples*@@ -417,7 +468,8 @@ Returns the sum of multiple values or functions, which are specified in a comma-* `add(x,y)`=== sumtotaltermfreq Function-Returns the sum of `totaltermfreq` values for all terms in the field in the entire index (i.e., the number of indexed tokens for that field). (Aliases `sumtotaltermfreq` to `sttf`.)+Returns the sum of `totaltermfreq` values for all terms in the field in the entire index (i.e., the number of indexed tokens for that field).+(Aliases `sumtotaltermfreq` to `sttf`.)*Syntax Example*If doc1:(fieldX:A B C) and doc2:(fieldX:A A A A):@@ -435,26 +487,32 @@ Returns the number of times the term appears in the field for that document.* `termfreq(text,'memory')`=== tf Function-Term frequency; returns the term frequency factor for the given term, using the {lucene-javadocs}/core/org/apache/lucene/search/similarities/Similarity.html[Similarity] for the field. The `tf-idf` value increases proportionally to the number of times a word appears in the document, but is offset by the frequency of the word in the document, which helps to control for the fact that some words are generally more common than others. See also `idf`.+Term frequency; returns the term frequency factor for the given term, using the {lucene-javadocs}/core/org/apache/lucene/search/similarities/Similarity.html[Similarity] for the field.+The `tf-idf` value increases proportionally to the number of times a word appears in the document, but is offset by the frequency of the word in the document, which helps to control for the fact that some words are generally more common than others.+See also `idf`.*Syntax Examples** `tf(text,'solr')`=== top Function-Causes the function query argument to derive its values from the top-level IndexReader containing all parts of an index. For example, the ordinal of a value in a single segment will be different from the ordinal of that same value in the complete index.+Causes the function query argument to derive its values from the top-level IndexReader containing all parts of an index.+For example, the ordinal of a value in a single segment will be different from the ordinal of that same value in the complete index.The `ord()` and `rord()` functions implicitly use `top()`, and hence `ord(foo)` is equivalent to `top(ord(foo))`.=== totaltermfreq Function-Returns the number of times the term appears in the field in the entire index. (Aliases `totaltermfreq` to `ttf`.)+Returns the number of times the term appears in the field in the entire index.+(Aliases `totaltermfreq` to `ttf`.)*Syntax Example** `ttf(text,'memory')`== Boolean Functions-The following functions are boolean – they return true or false. They are mostly useful as the first argument of the `if` function, and some of these can be combined. If used somewhere else, it will yield a '1' or '0'.+The following functions are boolean – they return true or false.+They are mostly useful as the first argument of the `if` function, and some of these can be combined.+If used somewhere else, it will yield a '1' or '0'.=== and FunctionReturns a value of true if and only if all of its operands evaluate to true.@@ -504,17 +562,21 @@ Returns `true` if any member of the field exists.== Example Function Queries-To give you a better understanding of how function queries can be used in Solr, suppose an index stores the dimensions in meters x,y,z of some hypothetical boxes with arbitrary names stored in field `boxname`. Suppose we want to search for box matching name `findbox` but ranked according to volumes of boxes. The query parameters would be:+To give you a better understanding of how function queries can be used in Solr, suppose an index stores the dimensions in meters x,y,z of some hypothetical boxes with arbitrary names stored in field `boxname`.+Suppose we want to search for box matching name `findbox` but ranked according to volumes of boxes.+The query parameters would be:[source,text]q=boxname:findbox _val_:"product(x,y,z)"-This query will rank the results based on volumes. In order to get the computed volume, you will need to request the `score`, which will contain the resultant volume:+This query will rank the results based on volumes.+In order to get the computed volume, you will need to request the `score`, which will contain the resultant volume:[source,text]&fl=*, score-Suppose that you also have a field storing the weight of the box as `weight`. To sort by the density of the box and return the value of the density in score, you would submit the following query:+Suppose that you also have a field storing the weight of the box as `weight`.+To sort by the density of the box and return the value of the density in score, you would submit the following query:[source,text]----@@ -523,14 +585,16 @@ http://localhost:8983/solr/collection_name/select?q=boxname:findbox _val_:"div(w== Sort By Function-You can sort your query results by the output of a function. For example, to sort results by distance, you could enter:+You can sort your query results by the output of a function.+For example, to sort results by distance, you could enter:[source,text]----http://localhost:8983/solr/collection_name/select?q=*:*&sort=dist(2, point1, point2) desc-----Sort by function also supports pseudo-fields: fields can be generated dynamically and return results as though it was normal field in the index. For example,+Sort by function also supports pseudo-fields: fields can be generated dynamically and return results as though it was normal field in the index.+For example,`&fl=id,sum(x, y),score&wt=xml`diff --git a/solr/solr-ref-guide/src/graph-traversal.adoc b/solr/solr-ref-guide/src/graph-traversal.adocindex 23b9737ced6..ce210f9d57f 100644--- a/solr/solr-ref-guide/src/graph-traversal.adoc+++ b/solr/solr-ref-guide/src/graph-traversal.adoc@@ -18,20 +18,25 @@Graph traversal with streaming expressions uses the `nodes` function to perform a breadth-first graph traversal.-The `nodes` function can be combined with the `scoreNodes` function to provide recommendations. `nodes` can also be combined with the wider streaming expression library to perform complex operations on gathered node sets.+The `nodes` function can be combined with the `scoreNodes` function to provide recommendations.+`nodes` can also be combined with the wider streaming expression library to perform complex operations on gathered node sets.`nodes` traversals are distributed within a SolrCloud collection and can span collections.-`nodes` is designed for use cases that involve zooming into a neighborhood in the graph and performing precise traversals to gather node sets and aggregations. In these types of use cases `nodes` will often provide sub-second performance. Some sample use cases are provided later in the document.+`nodes` is designed for use cases that involve zooming into a neighborhood in the graph and performing precise traversals to gather node sets and aggregations.+In these types of use cases `nodes` will often provide sub-second performance.+Some sample use cases are provided later in the document.[IMPORTANT]====-This document assumes a basic understanding of graph terminology and streaming expressions. You can begin exploring graph traversal concepts with this https://en.wikipedia.org/wiki/Graph_traversal[Wikipedia article]. More details about streaming expressions are available in this Guide, in the section <<streaming-expressions.adoc#,Streaming Expressions>>.+This document assumes a basic understanding of graph terminology and streaming expressions.+You can begin exploring graph traversal concepts with this https://en.wikipedia.org/wiki/Graph_traversal[Wikipedia article]. More details about streaming expressions are available in this Guide, in the section <<streaming-expressions.adoc#,Streaming Expressions>>.====== Basic Syntax-We'll start with the most basic syntax and slowly build up more complexity. The most basic syntax for `nodes` is:+We'll start with the most basic syntax and slowly build up more complexity.+The most basic syntax for `nodes` is:[source,plain]----@@ -42,9 +47,12 @@ nodes(emails,Let's break down this simple expression.-The first parameter, `emails`, is the collection being traversed. The second parameter, `walk`, maps a hard-coded node ID ("\[email protected]") to a field in the index (`from`). This will return all the *edges* in the index that have `[email protected]` in the `from` field.+The first parameter, `emails`, is the collection being traversed.+The second parameter, `walk`, maps a hard-coded node ID ("\[email protected]") to a field in the index (`from`).+This will return all the *edges* in the index that have `[email protected]` in the `from` field.-The `gather` parameter tells the function to gather the values in the `to `field. The values that are gathered are the node IDs emitted by the function.+The `gather` parameter tells the function to gather the values in the `to `field.+The values that are gathered are the node IDs emitted by the function.In the example above the nodes emitted will be all of the people that "[email protected]" has emailed.@@ -59,7 +67,8 @@ nodes(emails,The `nodes` function above finds all the edges with "[email protected]" or "[email protected]" in the `from` field and gathers the `to` field.-Like all <<streaming-expressions.adoc#,Streaming Expressions>>, you can execute a `nodes` expression by sending it to the `/stream` handler. For example:+Like all <<streaming-expressions.adoc#,Streaming Expressions>>, you can execute a `nodes` expression by sending it to the `/stream` handler.+For example:[source,bash]----@@ -102,9 +111,13 @@ The output of this expression would look like this:}-----All of the tuples returned have the `node` field. The `node` field contains the node IDs gathered by the function. The `collection`, `field`, and `level` of the traversal are also included in the output.+All of the tuples returned have the `node` field.+The `node` field contains the node IDs gathered by the function.+The `collection`, `field`, and `level` of the traversal are also included in the output.-Notice that the level is "1" for each tuple in the example. The root nodes are level 0 (in the example above, the root nodes are "[email protected], [email protected]") By default the `nodes` function emits only the _*leaf nodes*_ of the traversal, which is the outer-most node set. To emit the root nodes you can specify the `scatter` parameter:+Notice that the level is "1" for each tuple in the example.+The root nodes are level 0 (in the example above, the root nodes are "[email protected], [email protected]") By default the `nodes` function emits only the _*leaf nodes*_ of the traversal, which is the outer-most node set.+To emit the root nodes you can specify the `scatter` parameter:[source,plain]----@@ -160,7 +173,8 @@ Now the level 0 root node is included in the output.== Aggregations-`nodes` also supports aggregations. For example:+`nodes` also supports aggregations.+For example:[source,plain]----@@ -170,17 +184,25 @@ nodes(emails,count(*))-----The expression above finds the edges with "\[email protected]" or "\[email protected]" in the `from` field and gathers the values from the `to` field. It also aggregates the count for each node ID gathered.+The expression above finds the edges with "\[email protected]" or "\[email protected]" in the `from` field and gathers the values from the `to` field.+It also aggregates the count for each node ID gathered.-A gathered node could have a count of 2 if both "\[email protected]" and "\[email protected]" have emailed the same person. Node sets contain a unique set of nodes, so the same person won't appear twice in the node set, but the count will reflect that it appeared twice during the traversal.+A gathered node could have a count of 2 if both "\[email protected]" and "\[email protected]" have emailed the same person.+Node sets contain a unique set of nodes, so the same person won't appear twice in the node set, but the count will reflect that it appeared twice during the traversal.-Edges are uniqued as part of the traversal so the count will *not* reflect the number of times "\[email protected]" emailed the same person. For example, personA might have emailed personB 100 times. These edges would get uniqued and only be counted once. But if person personC also emailed personB this would increment the count for personB.+Edges are uniqued as part of the traversal so the count will *not* reflect the number of times "\[email protected]" emailed the same person.+For example, personA might have emailed personB 100 times.+These edges would get uniqued and only be counted once.+But if person personC also emailed personB this would increment the count for personB.-The aggregation functions supported are `count(*)`, `sum(field)`, `min(field)`, `max(field)`, and `avg(field)`. The fields being aggregated should be present in the edges collected during the traversal. Later examples (below) will show aggregations can be a powerful tool for providing recommendations and limiting the scope of traversals.+The aggregation functions supported are `count(*)`, `sum(field)`, `min(field)`, `max(field)`, and `avg(field)`.+The fields being aggregated should be present in the edges collected during the traversal.+Later examples (below) will show aggregations can be a powerful tool for providing recommendations and limiting the scope of traversals.== Nesting nodes Functions-The `nodes` function can be nested to traverse deeper into the graph. For example:+The `nodes` function can be nested to traverse deeper into the graph.+For example:[source,plain]----@@ -194,23 +216,36 @@ nodes(emails,In the example above the outer `nodes` function operates on the node set collected from the inner `nodes` function.-Notice that the inner `nodes` function behaves exactly as the examples already discussed. But the `walk` parameter of the outer `nodes` function behaves differently.+Notice that the inner `nodes` function behaves exactly as the examples already discussed.+But the `walk` parameter of the outer `nodes` function behaves differently.-In the outer `nodes` function the `walk` parameter works with tuples coming from an internal streaming expression. In this scenario the `walk` parameter maps the `node` field to the `from` field. Remember that the node IDs collected from the inner `nodes` expression are placed in the `node` field.+In the outer `nodes` function the `walk` parameter works with tuples coming from an internal streaming expression.+In this scenario the `walk` parameter maps the `node` field to the `from` field.+Remember that the node IDs collected from the inner `nodes` expression are placed in the `node` field.-Put more simply, the inner expression gathers all the people that "\[email protected]" has emailed. We can call this group the "friends of \[email protected]". The outer expression gathers all the people that the "friends of \[email protected]" have emailed. This is a basic friends-of-friends traversal.+Put more simply, the inner expression gathers all the people that "\[email protected]" has emailed.+We can call this group the "friends of \[email protected]". The outer expression gathers all the people that the "friends of \[email protected]" have emailed.+This is a basic friends-of-friends traversal.This construct of nesting `nodes` functions is the basic technique for doing a controlled traversal through the graph.== Cycle Detection-The `nodes` function performs cycle detection across the entire traversal. This ensures that nodes that have already been visited are not traversed again. Cycle detection is important for both limiting the size of traversals and gathering accurate aggregations. Without cycle detection the size of the traversal could grow exponentially with each hop in the traversal. With cycle detection only new nodes encountered are traversed.+The `nodes` function performs cycle detection across the entire traversal.+This ensures that nodes that have already been visited are not traversed again.+Cycle detection is important for both limiting the size of traversals and gathering accurate aggregations.+Without cycle detection the size of the traversal could grow exponentially with each hop in the traversal.+With cycle detection only new nodes encountered are traversed.-Cycle detection *does not* cross collection boundaries. This is because internally the collection name is part of the node ID. For example the node ID "\[email protected]", is really `emails/[email protected]`. When traversing to another collection "\[email protected]" will be traversed.+Cycle detection *does not* cross collection boundaries.+This is because internally the collection name is part of the node ID.+For example the node ID "\[email protected]", is really `emails/[email protected]`.+When traversing to another collection "\[email protected]" will be traversed.== Filtering the Traversal-Each level in the traversal can be filtered with a filter query. For example:+Each level in the traversal can be filtered with a filter query.+For example:[source,plain]----@@ -220,11 +255,14 @@ nodes(emails,gather="to")-----In the example above only emails that match the filter query will be included in the traversal. Any Solr query can be included here. So you can do fun things like <<spatial-search.adoc#,geospatial queries>>, apply any of the available <<query-syntax-and-parsers.adoc#,query parsers>>, or even write custom query parsers to limit the traversal.+In the example above only emails that match the filter query will be included in the traversal.+Any Solr query can be included here.+So you can do fun things like <<spatial-search.adoc#,geospatial queries>>, apply any of the available <<query-syntax-and-parsers.adoc#,query parsers>>, or even write custom query parsers to limit the traversal.== Root Streams-Any streaming expression can be used to provide the root nodes for a traversal. For example:+Any streaming expression can be used to provide the root nodes for a traversal.+For example:[source,plain]----@@ -234,26 +272,37 @@ nodes(emails,gather="to")-----The example above provides the root nodes through a search expression. You can also provide arbitrarily complex, nested streaming expressions with joins, etc., to specify the root nodes.+The example above provides the root nodes through a search expression.+You can also provide arbitrarily complex, nested streaming expressions with joins, etc., to specify the root nodes.-Notice that the `walk` parameter maps a field from the tuples generated by the inner stream. In this case it maps the `to` field from the inner stream to the `from` field.+Notice that the `walk` parameter maps a field from the tuples generated by the inner stream.+In this case it maps the `to` field from the inner stream to the `from` field.== Skipping High Frequency Nodes-It's often desirable to skip traversing high frequency nodes in the graph. This is similar in nature to a search term stop list. The best way to describe this is through an example use case.+It's often desirable to skip traversing high frequency nodes in the graph.+This is similar in nature to a search term stop list.+The best way to describe this is through an example use case.-Let's say that you want to recommend content for a user based on a collaborative filter. Below is one approach for a simple collaborative filter:+Let's say that you want to recommend content for a user based on a collaborative filter.+Below is one approach for a simple collaborative filter:. Find all content userA has read.-. Find users whose reading list is closest to userA. These are users with similar tastes as userA.+. Find users whose reading list is closest to userA.+These are users with similar tastes as userA.. Recommend content based on what the users in step 2 have read, that userA has not yet read.-Look closely at step 2. In large graphs, step 2 can lead to a very large traversal. This is because userA may have viewed content that has been viewed by millions of other people. We may want to skip these high frequency nodes for two reasons:+Look closely at step 2.+In large graphs, step 2 can lead to a very large traversal.+This is because userA may have viewed content that has been viewed by millions of other people.+We may want to skip these high frequency nodes for two reasons:. A large traversal that visit millions of unique nodes is slow and takes a lot of memory because cycle detection is tracked in memory.-. High frequency nodes are also not useful in determining users with similar tastes. The content that fewer people have viewed provides a more precise recommendation.+. High frequency nodes are also not useful in determining users with similar tastes.+The content that fewer people have viewed provides a more precise recommendation.-The `nodes` function has the `maxDocFreq` parameter to allow for filtering out high frequency nodes. The sample code below shows steps 1 and 2 of the recommendation:+The `nodes` function has the `maxDocFreq` parameter to allow for filtering out high frequency nodes.+The sample code below shows steps 1 and 2 of the recommendation:[source,plain]----@@ -266,13 +315,20 @@ The `nodes` function has the `maxDocFreq` parameter to allow for filtering out hcount(*)))-----In the example above, the inner search expression searches the `logs` collection and returning all the articles viewed by "user1". The outer `nodes` expression takes all the articles emitted from the inner search expression and finds all the records in the logs collection for those articles. It then gathers and aggregates the users that have read the articles. The `maxDocFreq` parameter limits the articles returned to those that appear in no more then 10,000 log records (per shard). This guards against returning articles that have been viewed by millions of users.+In the example above, the inner search expression searches the `logs` collection and returning all the articles viewed by "user1". The outer `nodes` expression takes all the articles emitted from the inner search expression and finds all the records in the logs collection for those articles.+It then gathers and aggregates the users that have read the articles.+The `maxDocFreq` parameter limits the articles returned to those that appear in no more then 10,000 log records (per shard).+This guards against returning articles that have been viewed by millions of users.== Tracking the Traversal-By default the `nodes` function only tracks enough information to do cycle detection. This provides enough information to output the nodes and aggregations in the graph.+By default the `nodes` function only tracks enough information to do cycle detection.+This provides enough information to output the nodes and aggregations in the graph.-For some use cases, such as graph visualization, we also need to output the edges. Setting `trackTraversal="true"` tells `nodes` to track the connections between nodes, so the edges can be constructed. When `trackTraversal` is enabled a new `ancestors` property will appear with each node. The `ancestors` property contains a list of node IDs that pointed to the node.+For some use cases, such as graph visualization, we also need to output the edges.+Setting `trackTraversal="true"` tells `nodes` to track the connections between nodes, so the edges can be constructed.+When `trackTraversal` is enabled a new `ancestors` property will appear with each node.+The `ancestors` property contains a list of node IDs that pointed to the node.Below is a sample `nodes` expression with `trackTraversal` set to true:@@ -290,7 +346,11 @@ nodes(emails,== Cross-Collection Traversals-Nested `nodes` functions can operate on different SolrCloud collections. This allow traversals to "walk" from one collection to another to gather nodes. Cycle detection does not cross collection boundaries, so nodes collected in one collection will be traversed in a different collection. This was done deliberately to support cross-collection traversals. Note that the output from a cross-collection traversal will likely contain duplicate nodes with different collection attributes.+Nested `nodes` functions can operate on different SolrCloud collections.+This allow traversals to "walk" from one collection to another to gather nodes.+Cycle detection does not cross collection boundaries, so nodes collected in one collection will be traversed in a different collection.+This was done deliberately to support cross-collection traversals.+Note that the output from a cross-collection traversal will likely contain duplicate nodes with different collection attributes.Below is a sample `nodes` expression that traverses from the "emails" collection to the "logs" collection:@@ -307,11 +367,14 @@ nodes(logs,gather="contentID")-----The example above finds all people who sent emails with a body that contains "solr rocks". It then finds all the people these people have emailed. Then it traverses to the logs collection and gathers all the content IDs that these people have edited.+The example above finds all people who sent emails with a body that contains "solr rocks". It then finds all the people these people have emailed.+Then it traverses to the logs collection and gathers all the content IDs that these people have edited.== Combining nodes With Other Streaming Expressions-The `nodes` function can act as both a stream source and a stream decorator. The connection with the wider stream expression library provides tremendous power and flexibility when performing graph traversals. Here is an example of using the streaming expression library to intersect two friend networks:+The `nodes` function can act as both a stream source and a stream decorator.+The connection with the wider stream expression library provides tremendous power and flexibility when performing graph traversals.+Here is an example of using the streaming expression library to intersect two friend networks:[source,plain]----@@ -334,13 +397,18 @@ The `nodes` function can act as both a stream source and a stream decorator. Thescatter="branches,leaves")))-----The example above gathers two separate friend networks, one rooted with "\[email protected]" and another rooted with "\[email protected]". The friend networks are then sorted by the `node` field, and intersected. The resulting node set will be the intersection of the two friend networks.+The example above gathers two separate friend networks, one rooted with "\[email protected]" and another rooted with "\[email protected]". The friend networks are then sorted by the `node` field, and intersected.+The resulting node set will be the intersection of the two friend networks.== Sample Use Cases for Graph Traversal=== Calculate Market Basket Co-occurrence-It is often useful to know which products are most frequently purchased with a particular product. This example uses a simple market basket table (indexed in Solr) to store past shopping baskets. The schema for the table is very simple with each row containing a `basketID` and a `productID`. This can be seen as a graph with each row in the table representing an edge. And it can be traversed very quickly to calculate basket co-occurrence, even when the graph contains billions of edges.+It is often useful to know which products are most frequently purchased with a particular product.+This example uses a simple market basket table (indexed in Solr) to store past shopping baskets.+The schema for the table is very simple with each row containing a `basketID` and a `productID`.+This can be seen as a graph with each row in the table representing an edge.+And it can be traversed very quickly to calculate basket co-occurrence, even when the graph contains billions of edges.Here is the sample syntax:@@ -358,21 +426,32 @@ top(n="5",Let's break down exactly what this traversal is doing.-. The first expression evaluated is the inner `random` expression, which returns 500 random basketIDs, from the `baskets` collection, that have the `productID` "ABC". The `random` expression is very useful for recommendations because it limits the traversal to a fixed set of baskets, and because it adds the element of surprise into the recommendation. Using the `random` function you can provide fast sample sets from very large graphs.-. The outer `nodes` expression finds all the records in the `baskets` collection for the basketIDs generated in step 1. It also filters out `productID` "ABC" so it doesn't show up in the results. It then gathers and counts the productID's across these baskets.+. The first expression evaluated is the inner `random` expression, which returns 500 random basketIDs, from the `baskets` collection, that have the `productID` "ABC". The `random` expression is very useful for recommendations because it limits the traversal to a fixed set of baskets, and because it adds the element of surprise into the recommendation.+Using the `random` function you can provide fast sample sets from very large graphs.+. The outer `nodes` expression finds all the records in the `baskets` collection for the basketIDs generated in step 1.+It also filters out `productID` "ABC" so it doesn't show up in the results.+It then gathers and counts the productID's across these baskets.. The outer `top` expression ranks the productIDs emitted in step 2 by the count and selects the top 5.In a nutshell this expression finds the products that most frequently co-occur with product "ABC" in past shopping baskets.=== Using the scoreNodes Function to Make a Recommendation-This use case builds on the market basket example <<Calculate Market Basket Co-occurrence,above>> that calculates which products co-occur most frequently with productID:ABC. The ranked co-occurrence counts provide candidates for a recommendation. The `scoreNodes` function can be used to score the candidates to find the best recommendation.+This use case builds on the market basket example <<Calculate Market Basket Co-occurrence,above>> that calculates which products co-occur most frequently with productID:ABC.+The ranked co-occurrence counts provide candidates for a recommendation.+The `scoreNodes` function can be used to score the candidates to find the best recommendation.-Before diving into the syntax of the `scoreNodes` function it's useful to understand why the raw co-occurrence counts may not produce the best recommendation. The reason is that raw co-occurrence counts favor items that occur frequently across all baskets. A better recommendation would find the product that has the most significant relationship with productID ABC. The `scoreNodes` function uses a term frequency-inverse document frequency (TF-IDF) algorithm to find the most significant relationship.+Before diving into the syntax of the `scoreNodes` function it's useful to understand why the raw co-occurrence counts may not produce the best recommendation.+The reason is that raw co-occurrence counts favor items that occur frequently across all baskets.+A better recommendation would find the product that has the most significant relationship with productID ABC.+The `scoreNodes` function uses a term frequency-inverse document frequency (TF-IDF) algorithm to find the most significant relationship.==== How scoreNodes Works-The `scoreNodes` function assigns a score to each node emitted by the nodes expression. By default the `scoreNodes` function uses the `count(*)` aggregation, which is the co-occurrence count, as the TF value. The IDF value for each node is fetched from the collection where the node was gathered. Each node is then scored using the TF*IDF formula, which provides a boost to nodes with a lower frequency across all market baskets.+The `scoreNodes` function assigns a score to each node emitted by the nodes expression.+By default the `scoreNodes` function uses the `count(*)` aggregation, which is the co-occurrence count, as the TF value.+The IDF value for each node is fetched from the collection where the node was gathered.+Each node is then scored using the TF*IDF formula, which provides a boost to nodes with a lower frequency across all market baskets.Combining the co-occurrence count with the IDF provides a score that shows how important the relationship is between productID ABC and the recommendation candidates.@@ -396,13 +475,18 @@ top(n="1",This example builds on the earlier example "Calculate market basket co-occurrence".-. Notice that the inner-most `top` function is taking the top 50 products that co-occur most frequently with productID ABC. This provides 50 candidate recommendations.+. Notice that the inner-most `top` function is taking the top 50 products that co-occur most frequently with productID ABC.+This provides 50 candidate recommendations.. The `scoreNodes` function then assigns a score to the candidates based on the TF*IDF of each node.-. The outer `top` expression selects the highest scoring node. This is the recommendation.+. The outer `top` expression selects the highest scoring node.+This is the recommendation.=== Recommend Content Based on Collaborative Filter-In this example we'll recommend content for a user based on a collaborative filter. This recommendation is made using log records that contain the `userID` and `articleID` and the action performed. In this scenario each log record can be viewed as an edge in a graph. The userID and articleID are the nodes and the action is an edge property used to filter the traversal.+In this example we'll recommend content for a user based on a collaborative filter.+This recommendation is made using log records that contain the `userID` and `articleID` and the action performed.+In this scenario each log record can be viewed as an edge in a graph.+The userID and articleID are the nodes and the action is an edge property used to filter the traversal.Here is the sample syntax:@@ -428,21 +512,34 @@ top(n="5",Let's break down the expression above step-by-step.-. The first expression evaluated is the inner `search` expression. This expression searches the `logs` collection for all records matching "user1". This is the user we are making the recommendation for.+. The first expression evaluated is the inner `search` expression.+This expression searches the `logs` collection for all records matching "user1". This is the user we are making the recommendation for.+-There is a filter applied to pull back only records where the "action:read". It returns the `articleID` for each record found. In other words, this expression returns all the articles "user1" has read.-. The inner `nodes` expression operates over the articleIDs returned from step 1. It takes each `articleID` found and searches them against the `articleID` field.+There is a filter applied to pull back only records where the "action:read". It returns the `articleID` for each record found.+In other words, this expression returns all the articles "user1" has read.+. The inner `nodes` expression operates over the articleIDs returned from step 1.+It takes each `articleID` found and searches them against the `articleID` field.+-Note that it skips high frequency nodes using the `maxDocFreq` parameter to filter out articles that appear over 10,000 times in the logs. It gathers userIDs and aggregates the counts for each user. This step finds the users that have read the same articles that "user1" has read and counts how many of the same articles they have read.-. The inner `top` expression ranks the users emitted from step 2. It will emit the top 30 users who have the most overlap with user1's reading list.-. The outer `nodes` expression gathers the reading list for the users emitted from step 3. It counts the articleIDs that are gathered.+Note that it skips high frequency nodes using the `maxDocFreq` parameter to filter out articles that appear over 10,000 times in the logs.+It gathers userIDs and aggregates the counts for each user.+This step finds the users that have read the same articles that "user1" has read and counts how many of the same articles they have read.+. The inner `top` expression ranks the users emitted from step 2.+It will emit the top 30 users who have the most overlap with user1's reading list.+. The outer `nodes` expression gathers the reading list for the users emitted from step 3.+It counts the articleIDs that are gathered.+-Any article selected in step 1 (user1 reading list), will not appear in this step due to cycle detection. So this step returns the articles read by the users with the most similar readings habits to "user1" that "user1" has not read yet. It also counts the number of times each article has been read across this user group.-. The outer `top` expression takes the top articles emitted from step 4. This is the recommendation.+Any article selected in step 1 (user1 reading list), will not appear in this step due to cycle detection.+So this step returns the articles read by the users with the most similar readings habits to "user1" that "user1" has not read yet.+It also counts the number of times each article has been read across this user group.+. The outer `top` expression takes the top articles emitted from step 4.+This is the recommendation.=== Protein Pathway Traversal-In recent years, scientists have become increasingly able to rationally design drugs that target the mutated proteins, called oncogenes, responsible for some cancers. Proteins typically act through long chains of chemical interactions between multiple proteins, called pathways, and, while the oncogene in the pathway may not have a corresponding drug, another protein in the pathway may. Graph traversal on a protein collection that records protein interactions and drugs may yield possible candidates. (Thanks to Lewis Geer of the NCBI, for providing this example).+In recent years, scientists have become increasingly able to rationally design drugs that target the mutated proteins, called oncogenes, responsible for some cancers.+Proteins typically act through long chains of chemical interactions between multiple proteins, called pathways, and, while the oncogene in the pathway may not have a corresponding drug, another protein in the pathway may.+Graph traversal on a protein collection that records protein interactions and drugs may yield possible candidates.+(Thanks to Lewis Geer of the NCBI, for providing this example).The example below illustrates a protein pathway traversal:@@ -458,23 +555,36 @@ nodes(proteins,Let's break down exactly what this traversal is doing.-. The inner `nodes` expression traverses in the `proteins` collection. It finds all the edges in the graph where the name of the protein is "NRAS". Then it gathers the proteins in the `interacts` field. This gathers all the proteins that "NRAS" interactions with.-. The outer `nodes` expression also works with the `proteins` collection. It gathers all the drugs that correspond to proteins emitted from step 1.+. The inner `nodes` expression traverses in the `proteins` collection.+It finds all the edges in the graph where the name of the protein is "NRAS". Then it gathers the proteins in the `interacts` field.+This gathers all the proteins that "NRAS" interactions with.+. The outer `nodes` expression also works with the `proteins` collection.+It gathers all the drugs that correspond to proteins emitted from step 1.. Using this stepwise approach you can gather the drugs along the pathway of interactions any number of steps away from the root protein.== Exporting GraphML to Support Graph Visualization-In the examples above, the `nodes` expression was sent to Solr's `/stream` handler like any other streaming expression. This approach outputs the nodes in the same JSON tuple format as other streaming expressions so that it can be treated like any other streaming expression. You can use the `/stream` handler when you need to operate directly on the tuples, such as in the recommendation use cases above.+In the examples above, the `nodes` expression was sent to Solr's `/stream` handler like any other streaming expression.+This approach outputs the nodes in the same JSON tuple format as other streaming expressions so that it can be treated like any other streaming expression.+You can use the `/stream` handler when you need to operate directly on the tuples, such as in the recommendation use cases above.-There are other graph traversal use cases that involve graph visualization. Solr supports these use cases with the introduction of the `/graph` request handler, which takes a `nodes` expression and outputs the results in GraphML.+There are other graph traversal use cases that involve graph visualization.+Solr supports these use cases with the introduction of the `/graph` request handler, which takes a `nodes` expression and outputs the results in GraphML.-http://graphml.graphdrawing.org/[GraphML] is an XML format supported by graph visualization tools such as https://gephi.org/[Gephi], which is a sophisticated open source tool for statistically analyzing and visualizing graphs. Using a `nodes` expression, parts of a larger graph can be exported in GraphML and then imported into tools like Gephi.+http://graphml.graphdrawing.org/[GraphML] is an XML format supported by graph visualization tools such as https://gephi.org/[Gephi], which is a sophisticated open source tool for statistically analyzing and visualizing graphs.+Using a `nodes` expression, parts of a larger graph can be exported in GraphML and then imported into tools like Gephi.There are a few things to keep mind when exporting a graph in GraphML:-. The `/graph` handler can export both the nodes and edges in the graph. By default, it only exports the nodes. To export the edges you must set `trackTraversal="true"` in the `nodes` expression.-. The `/graph` handler currently accepts an arbitrarily complex streaming expression which includes a `nodes` expression. If the streaming expression doesn't include a `nodes` expression, the `/graph` handler will not properly output GraphML.-. The `/graph` handler currently accepts a single arbitrarily complex, nested `nodes` expression per request. This means you cannot send in a streaming expression that joins or intersects the node sets from multiple `nodes` expressions. The `/graph` handler does support any level of nesting within a single `nodes` expression. The `/stream` handler does support joining and intersecting node sets, but the `/graph` handler currently does not.+. The `/graph` handler can export both the nodes and edges in the graph.+By default, it only exports the nodes.+To export the edges you must set `trackTraversal="true"` in the `nodes` expression.+. The `/graph` handler currently accepts an arbitrarily complex streaming expression which includes a `nodes` expression.+If the streaming expression doesn't include a `nodes` expression, the `/graph` handler will not properly output GraphML.+. The `/graph` handler currently accepts a single arbitrarily complex, nested `nodes` expression per request.+This means you cannot send in a streaming expression that joins or intersects the node sets from multiple `nodes` expressions.+The `/graph` handler does support any level of nesting within a single `nodes` expression.+The `/stream` handler does support joining and intersecting node sets, but the `/graph` handler currently does not.=== Sample GraphML Requestdiff --git a/solr/solr-ref-guide/src/graph.adoc b/solr/solr-ref-guide/src/graph.adocindex 8d7294f7912..f91506fa3e4 100644--- a/solr/solr-ref-guide/src/graph.adoc+++ b/solr/solr-ref-guide/src/graph.adoc@@ -17,24 +17,21 @@// under the License.-This section of the user guide covers the syntax and theory behind *graph expressions*. Examples are presented for two key graph use cases: *bipartite graph recommenders* and *event correlation* with-*temporal graph queries*.+This section of the user guide covers the syntax and theory behind *graph expressions*.+Examples are presented for two key graph use cases: *bipartite graph recommenders* and *event correlation* with *temporal graph queries*.== GraphsLog records and other data indexed in Solr have connections between them that can be seen as a distributed graph.Graph expressions provide a mechanism for identifying root nodes in the graph and walking their connections.-The general goal of the graph walk is to materialize a specific *subgraph* and perform *link analysis* to understand-the connections between nodes.+The general goal of the graph walk is to materialize a specific *subgraph* and perform *link analysis* to understand the connections between nodes.In the next few sections below we'll review the graph theory behind Solr's graph expressions.=== Subgraphs-A subgraph is a smaller subset of the nodes and connections of the-larger graph.-Graph expressions allow you to flexibly define and materialize a subgraph from the larger graph-stored in the distributed index.+A subgraph is a smaller subset of the nodes and connections of the larger graph.+Graph expressions allow you to flexibly define and materialize a subgraph from the larger graph stored in the distributed index.Subgraphs play two important roles:@@ -46,19 +43,14 @@ The design of the subgraph defines the meaning of the link analysis.=== Bipartite SubgraphsGraph expressions can be used to materialize *bipartite subgraphs*.-A bipartite graph is a graph where the nodes are split into two-distinct categories.-The links between those two categories can then-be analyzed to study how they relate.-Bipartite graphs are often discussed-in the context of collaborative filter recommender systems.+A bipartite graph is a graph where the nodes are split into two distinct categories.+The links between those two categories can then be analyzed to study how they relate.+Bipartite graphs are often discussed in the context of collaborative filter recommender systems.A bipartite graph between *shopping baskets* and *products* is a useful example.-Through link analysis between the shopping baskets and products-we can determine which products are most often purchased within the same shopping baskets.+Through link analysis between the shopping baskets and products we can determine which products are most often purchased within the same shopping baskets.-In the example below there is a Solr collection called baskets-with three fields:+In the example below there is a Solr collection called baskets with three fields:*id*: Unique ID@@ -70,10 +62,9 @@ Each record in the collection represents a product in a shopping basket.All products in the same basket share the same basket ID.Let's consider a simple example where we want to find a product-that is often sold with *butter*. In order to do this we could create a-*bipartite subgraph* of shopping baskets that contain *butter*.-We won't include butter itself in the graph as it doesn't help with-finding a complementary product for butter.+that is often sold with *butter*.+In order to do this we could create a *bipartite subgraph* of shopping baskets that contain *butter*.+We won't include butter itself in the graph as it doesn't help with finding a complementary product for butter.Below is an example of this bipartite subgraph represented as a matrix:@@ -127,8 +118,7 @@ The `trackTraversal` flag tells the nodes expression to track the links betweenThe output of the nodes function is a *node set* that represents the subgraph specified by the nodes function.The node set contains a unique set of nodes that are gathered during the graph walk.The `node` property in the result is the value of the gathered node.-In the shopping basket example the `product_s` field is in the node property-because that was what was specified to be gathered in the nodes expression.+In the shopping basket example the `product_s` field is in the node property because that was what was specified to be gathered in the nodes expression.The output of the shopping basket graph expression is as follows:[source,json]@@ -174,23 +164,19 @@ The output of the shopping basket graph expression is as follows:}-----The `ancestors` property in the result contains a unique, alphabetically sorted set of all the *inbound links*-to the node in the subgraph.+The `ancestors` property in the result contains a unique, alphabetically sorted set of all the *inbound links* to the node in the subgraph.In this case it shows the baskets that are linked to each product.The ancestor links will only be tracked when the trackTraversal flag is turned on in the nodes expression.=== Link Analysis and Degree Centrality-Link analysis is often performed to determine *node centrality*. When analyzing for centrality the-goal is to assign a weight to each node based on how connected it is in the subgraph.+Link analysis is often performed to determine *node centrality*.+When analyzing for centrality the goal is to assign a weight to each node based on how connected it is in the subgraph.There are different types of node centrality.-Graph expressions very efficiently calculates-*inbound degree centrality* (in-degree).+Graph expressions very efficiently calculates *inbound degree centrality* (in-degree).-Inbound degree centrality is calculated by counting the number of inbound-links to each node.-For simplicity this document will sometimes refer-to inbound degree simply as degree.+Inbound degree centrality is calculated by counting the number of inbound links to each node.+For simplicity this document will sometimes refer to inbound degree simply as degree.Back to the shopping basket example:@@ -204,8 +190,7 @@ eggs: 2milk: 2-----From the degree calculation we know that *eggs* and *milk* appear more frequently in shopping baskets with-butter than *cheese* does.+From the degree calculation we know that *eggs* and *milk* appear more frequently in shopping baskets with butter than *cheese* does.The nodes function can calculate degree centrality by adding the `count(*)` aggregation as shown below:@@ -270,10 +255,8 @@ The output of this graph expression is as follows:The `count(+++*+++)` aggregation counts the "gathered" nodes, in this case the values in the `product_s` field.Notice that the `count(+++*+++)` result is the same as the number of ancestors.-This will always be the case because the nodes function first deduplicates the edges before-counting the gathered nodes.-Because of this the `count(+++*+++)` aggregation always calculates the-inbound degree centrality for the gathered nodes.+This will always be the case because the nodes function first deduplicates the edges before counting the gathered nodes.+Because of this the `count(+++*+++)` aggregation always calculates the inbound degree centrality for the gathered nodes.=== Dot Product@@ -288,20 +271,16 @@ This tells us that a nearest neighbor search, using a maximum inner product simi=== Limiting Basket Out-DegreeThe recommendation can be made stronger by limiting the *out-degree* of the baskets.-The out-degree is the-number of outbound links of a node in a graph.-In the shopping basket example the outbound links-from the baskets link to products.+The out-degree is the number of outbound links of a node in a graph.+In the shopping basket example the outbound links from the baskets link to products.So limiting the out-degree will limit the size of the baskets.-Why does limiting the size of the shopping baskets make a stronger recommendation? To answer this question it helps-to think about each shopping basket as *voting* for products that go with *butter*. In an election with two candidates-if you were to vote for both candidates the votes would cancel each other out and have no effect.+Why does limiting the size of the shopping baskets make a stronger recommendation?+To answer this question it helps to think about each shopping basket as *voting* for products that go with *butter*.+In an election with two candidates if you were to vote for both candidates the votes would cancel each other out and have no effect.But if you vote for only one candidate your vote will affect the outcome.-The same principle holds true-for recommendations.-As a basket votes for more products it dilutes the strength of its recommendation for any-one product.+The same principle holds true for recommendations.+As a basket votes for more products it dilutes the strength of its recommendation for any one product.A basket with just butter and one other item more strongly recommends that item.The `maxDocFreq` parameter can be used to limit the graph "walk" to only include baskets that appear in the index a certain number of times.@@ -327,18 +306,14 @@ nodes(baskets,=== Node ScoringThe degree of the node describes how many nodes in the subgraph link to it.-But this does not tell us if the node is particularly central to this subgraph or if it is just a-very frequent node in the entire graph.-Nodes that appear frequently in the subgraph but-infrequently in the entire graph can be considered more *relevant* to the subgraph.+But this does not tell us if the node is particularly central to this subgraph or if it is just a very frequent node in the entire graph.+Nodes that appear frequently in the subgraph but infrequently in the entire graph can be considered more *relevant* to the subgraph.The search index contains information about how frequently each node appears in the entire index.-Using a technique similar to *tf-idf* document scoring, graph expressions can combine the-degree of the node with its inverse document frequency in the index to determine a relevancy score.+Using a technique similar to *tf-idf* document scoring, graph expressions can combine the degree of the node with its inverse document frequency in the index to determine a relevancy score.The `scoreNodes` function scores the nodes.-Below is an example of the scoreNodes function applied to-the shopping basket node set.+Below is an example of the scoreNodes function applied to the shopping basket node set.[source,text]----@@ -352,13 +327,10 @@ scoreNodes(nodes(baskets,----The output now includes a `nodeScore` property.-In the output below notice how *eggs* has a higher-nodeScore than *milk* even though they have the same `count(+++*+++)`. This is because milk appears more-frequently in the entire index than eggs does.-The `docFreq` property added by the `nodeScore` function-shows the document frequency in the index.-Because of the lower `docFreq` eggs is considered more relevant-to this subgraph, and a better recommendation to be paired with butter.+In the output below notice how *eggs* has a higher nodeScore than *milk* even though they have the same `count(+++*+++)`.+This is because milk appears more frequently in the entire index than eggs does.+The `docFreq` property added by the `nodeScore` function shows the document frequency in the index.+Because of the lower `docFreq` eggs is considered more relevant to this subgraph, and a better recommendation to be paired with butter.[source,json]----@@ -418,10 +390,8 @@ to this subgraph, and a better recommendation to be paired with butter.== Temporal Graph ExpressionsThe examples above lay the groundwork for temporal graph queries.-Temporal graph queries allow the `nodes` function to walk the graph using *windows of time* to surface-*cross-correlations* within the data.-The nodes function currently supports graph walks using *ten second increments*-which is useful for *event correlation* and *root cause analysis* in log analytics.+Temporal graph queries allow the `nodes` function to walk the graph using *windows of time* to surface *cross-correlations* within the data.+The nodes function currently supports graph walks using *ten second increments* which is useful for *event correlation* and *root cause analysis* in log analytics.In order to support temporal graph queries a ten second truncated timestamp in *ISO 8601* format must be added to the log records as a string field at indexing time.Here is a sample ten second truncated timestamp: `2021-02-10T20:51:30Z`.@@ -432,13 +402,11 @@ So those using Solr to analyze Solr logs get temporal graph expressions for free=== Root Events-Once the ten second windows have been indexed with the log records we can devise a query that-creates a set of *root events*. We can demonstrate this with an example using Solr log records.+Once the ten second windows have been indexed with the log records we can devise a query that creates a set of *root events*.+We can demonstrate this with an example using Solr log records.-In this example we'll perform a Streaming Expression `facet` aggregation that finds the top 10, ten second windows-with the highest average query time.-These time windows can be used to represent *slow query events* in a temporal-graph query.+In this example we'll perform a Streaming Expression `facet` aggregation that finds the top 10, ten second windows with the highest average query time.+These time windows can be used to represent *slow query events* in a temporal graph query.Here is the facet function:@@ -494,12 +462,10 @@ Below is a snippet of the results with the 25 windows with the highest average q----=== Temporal Bipartite Subgraphs-Once we've identified a set of root events it's easy to perform a graph query that creates a-bipartite graph of the log events types that occurred within the same ten second windows.+Once we've identified a set of root events it's easy to perform a graph query that creates a bipartite graph of the log events types that occurred within the same ten second windows.With Solr logs there is a field called `type_s` which is the type of log event.-In order to see what log events happened in the same ten second window of our root events we can "walk" the-ten second windows and gather the `type_s` field.+In order to see what log events happened in the same ten second window of our root events we can "walk" the ten second windows and gather the `type_s` field.[source,text]----@@ -565,25 +531,18 @@ Below is the resulting node set:}-----In this result set the `node` field holds the type of log events that occurred within the-same ten second windows as the root events.-Notice that the event types include:-query, admin, update and error.-The `count(+++*+++)` shows the degree centrality of the different-log event types.+In this result set the `node` field holds the type of log events that occurred within the same ten second windows as the root events.+Notice that the event types include: query, admin, update and error.+The `count(+++*+++)` shows the degree centrality of the different log event types.Notice that there is only one *error* event within the same ten second windows of the slow query events.=== Window Parameter-For event correlation and root cause analysis it's not enough to find events that occur-within the *same* ten second root event windows.-What's needed is to find events that occur-within a window of time *prior to each root event*. The `window` parameter allows you to-specify this prior window of time as part of the query.-The window parameter is an integer-which specifies the number of ten second time windows, prior to each root event window,-to include in the graph walk.+For event correlation and root cause analysis it's not enough to find events that occur within the *same* ten second root event windows.+What's needed is to find events that occur within a window of time *prior to each root event*.+The `window` parameter allows you to specify this prior window of time as part of the query.+The window parameter is an integer which specifies the number of ten second time windows, prior to each root event window, to include in the graph walk.[source,text]----@@ -652,42 +611,29 @@ Notice that there are *now 29 error* events within the 3 ten second windows prio=== Degree as a Representation of Correlation-By performing link analysis on the temporal bipartite graph we can calculate the-degree of each event type that occurs in the specified time windows.-We established in the bipartite graph recommender example the direct relationship between-*inbound degree* and the *dot product*. In the field of digital signal processing the-dot product is used to represent *correlation*.-In our temporal graph queries we can then view the inbound degree as a-representation of correlation between the root events and the events that-occur within the specified time windows.+By performing link analysis on the temporal bipartite graph we can calculate the degree of each event type that occurs in the specified time windows.+We established in the bipartite graph recommender example the direct relationship between *inbound degree* and the *dot product*.+In the field of digital signal processing the dot product is used to represent *correlation*.+In our temporal graph queries we can then view the inbound degree as a representation of correlation between the root events and the events that occur within the specified time windows.=== Lag ParameterUnderstanding the *lag* in the correlation is important for certain use cases.In a lagged correlation an event occurs and following a *delay* another event occurs.-The window parameter doesn't capture the delay as we only know that an event-occurred somewhere within a prior window.--The `lag` parameter can be used to start calculating the window parameter a-number of ten second windows in the past.-For example we could walk the graph in 20 second-windows starting from 30 seconds prior to a set of root events.-By adjusting the lag and re-running the query we can determine which lagged-window has the highest degree.+The window parameter doesn't capture the delay as we only know that an event occurred somewhere within a prior window.++The `lag` parameter can be used to start calculating the window parameter a number of ten second windows in the past.+For example we could walk the graph in 20 second windows starting from 30 seconds prior to a set of root events.+By adjusting the lag and re-running the query we can determine which lagged window has the highest degree.From this we can determine the delay.=== Node Scoring and Temporal Anomaly Detection-The concept of node scoring can be applied to temporal graph queries to find events that are-both *correlated* with a set of root events and *anomalous* to the root events.-The degree calculation establishes the correlation between events-but it does not establish if the event is a very common occurrence in-the entire graph or specific to the subgraph.+The concept of node scoring can be applied to temporal graph queries to find events that are both *correlated* with a set of root events and *anomalous* to the root events.+The degree calculation establishes the correlation between events but it does not establish if the event is a very common occurrence in the entire graph or specific to the subgraph.-The `scoreNodes` functions can be applied to score the nodes based on the degree and the-commonality of the node's term in the index.-This will establish whether the event is anomalous to-the root events.+The `scoreNodes` functions can be applied to score the nodes based on the degree and the commonality of the node's term in the index.+This will establish whether the event is anomalous to the root events.[source,text]----diff --git a/solr/solr-ref-guide/src/hadoop-authentication-plugin.adoc b/solr/solr-ref-guide/src/hadoop-authentication-plugin.adocindex 532c99bcbf0..46c9fe93027 100644--- a/solr/solr-ref-guide/src/hadoop-authentication-plugin.adoc+++ b/solr/solr-ref-guide/src/hadoop-authentication-plugin.adoc@@ -149,7 +149,8 @@ The example below uses `ConfigurableInternodeAuthHadoopPlugin`, and hence you muAs a result, all internode communication will use the Kerberos mechanism, instead of PKI authentication.This configuration assumes that your servers are using the `solr` principal, and will be allowed to impersonate any other user with requests coming from any other host.-For additional security, consider setting the host list to match your cluster nodes. The Hadoop https://hadoop.apache.org/docs/stable/hadoop-project-dist/hadoop-common/Superusers.html[proxy users documentation] contains more detail about available configuration options.+For additional security, consider setting the host list to match your cluster nodes.+The Hadoop https://hadoop.apache.org/docs/stable/hadoop-project-dist/hadoop-common/Superusers.html[proxy users documentation] contains more detail about available configuration options.[source,json]----diff --git a/solr/solr-ref-guide/src/highlighting.adoc b/solr/solr-ref-guide/src/highlighting.adocindex 32dc185f7f5..950a56f3226 100644--- a/solr/solr-ref-guide/src/highlighting.adoc+++ b/solr/solr-ref-guide/src/highlighting.adoc@@ -18,25 +18,35 @@Highlighting in Solr allows fragments of documents that match the user's query to be included with the query response.-The fragments are included in a special section of the query response (the `highlighting` section), and the client uses the formatting clues also included to determine how to present the snippets to users. Fragments are a portion of a document field that contains matches from the query and are sometimes also referred to as "snippets" or "passages".+The fragments are included in a special section of the query response (the `highlighting` section), and the client uses the formatting clues also included to determine how to present the snippets to users.+Fragments are a portion of a document field that contains matches from the query and are sometimes also referred to as "snippets" or "passages".-Highlighting is extremely configurable, perhaps more than any other part of Solr. There are many parameters each for fragment sizing, formatting, ordering, backup/alternate behavior, and more options that are hard to categorize. Nonetheless, highlighting is very simple to use.+Highlighting is extremely configurable, perhaps more than any other part of Solr.+There are many parameters each for fragment sizing, formatting, ordering, backup/alternate behavior, and more options that are hard to categorize.+Nonetheless, highlighting is very simple to use.== Usage=== Common Highlighter Parameters-You only need to set the `hl` and often `hl.fl` parameters to get results. The following table documents these and some other supported parameters. Note that many highlighting parameters support per-field overrides, such as: `f._title_txt_.hl.snippets`+You only need to set the `hl` and often `hl.fl` parameters to get results.+The following table documents these and some other supported parameters.+Note that many highlighting parameters support per-field overrides, such as: `f._title_txt_.hl.snippets``hl`::-Use this parameter to enable or disable highlighting. The default is `false`. If you want to use highlighting, you must set this to `true`.+Use this parameter to enable or disable highlighting.+The default is `false`.+If you want to use highlighting, you must set this to `true`.`hl.method`::-The highlighting implementation to use. Acceptable values are: `unified`, `original`, `fastVector`. The default is `original`.+The highlighting implementation to use.+Acceptable values are: `unified`, `original`, `fastVector`.+The default is `original`.+See the <<Choosing a Highlighter>> section below for more details on the differences between the available highlighters.`hl.fl`::-Specifies a list of fields to highlight, either comma- or space-delimited. These must be "stored".+Specifies a list of fields to highlight, either comma- or space-delimited.+ These must be "stored".A wildcard of `\*` (asterisk) can be used to match field globs, such as `text_*` or even `\*` to highlight on all fields where highlighting is possible.When using `*`, consider adding `hl.requireFieldMatch=true`.+@@ -55,42 +65,57 @@ When setting this, you might also need to set `hl.qparser`.The default is the value of the `q` parameter (already parsed).`hl.qparser`::-The <<query-syntax-and-parsers.adoc#,query parser>> to use for the `hl.q` query. It only applies when `hl.q` is set.+The <<query-syntax-and-parsers.adoc#,query parser>> to use for the `hl.q` query.+It only applies when `hl.q` is set.+The default is the value of the `defType` parameter which in turn defaults to `lucene`.`hl.requireFieldMatch`::-By default, `false`, all query terms will be highlighted for each field to be highlighted (`hl.fl`) no matter what fields the parsed query refer to. If set to `true`, only query terms aligning with the field being highlighted will in turn be highlighted.+By default, `false`, all query terms will be highlighted for each field to be highlighted (`hl.fl`) no matter what fields the parsed query refer to.+If set to `true`, only query terms aligning with the field being highlighted will in turn be highlighted.+-If the query references fields different from the field being highlighted and they have different text analysis, the query may not highlight query terms it should have and vice versa. The analysis used is that of the field being highlighted (`hl.fl`), not the query fields.+If the query references fields different from the field being highlighted and they have different text analysis, the query may not highlight query terms it should have and vice versa.+The analysis used is that of the field being highlighted (`hl.fl`), not the query fields.`hl.usePhraseHighlighter`::-If set to `true`, the default, Solr will highlight phrase queries (and other advanced position-sensitive queries) accurately – as phrases. If `false`, the parts of the phrase will be highlighted everywhere instead of only when it forms the given phrase.+If set to `true`, the default, Solr will highlight phrase queries (and other advanced position-sensitive queries) accurately – as phrases.+If `false`, the parts of the phrase will be highlighted everywhere instead of only when it forms the given phrase.`hl.highlightMultiTerm`::-If set to `true`, the default, Solr will highlight wildcard queries (and other `MultiTermQuery` subclasses). If `false`, they won't be highlighted at all.+If set to `true`, the default, Solr will highlight wildcard queries (and other `MultiTermQuery` subclasses).+If `false`, they won't be highlighted at all.`hl.snippets`::-Specifies maximum number of highlighted snippets to generate per field. It is possible for any number of snippets from zero to this value to be generated. The default is `1`.+Specifies maximum number of highlighted snippets to generate per field.+It is possible for any number of snippets from zero to this value to be generated.+The default is `1`.`hl.fragsize`::-Specifies the approximate size, in characters, of fragments to consider for highlighting. The default is `100`. Using `0` indicates that no fragmenting should be considered and the whole field value should be used.+Specifies the approximate size, in characters, of fragments to consider for highlighting.+The default is `100`.+Using `0` indicates that no fragmenting should be considered and the whole field value should be used.`hl.tag.pre`::-(`hl.simple.pre` for the Original Highlighter) Specifies the “tag” to use before a highlighted term. This can be any string, but is most often an HTML or XML tag.+(`hl.simple.pre` for the Original Highlighter) Specifies the “tag” to use before a highlighted term.+This can be any string, but is most often an HTML or XML tag.+The default is `<em>`.`hl.tag.post`::-(`hl.simple.post` for the Original Highlighter) Specifies the “tag” to use after a highlighted term. This can be any string, but is most often an HTML or XML tag.+(`hl.simple.post` for the Original Highlighter) Specifies the “tag” to use after a highlighted term.+This can be any string, but is most often an HTML or XML tag.+The default is `</em>`.`hl.encoder`::-If blank, the default, then the stored text will be returned without any escaping/encoding performed by the highlighter. If set to `html` then special HTML/XML characters will be encoded (e.g., `&` becomes `\&`). The pre/post snippet characters are never encoded.+If blank, the default, then the stored text will be returned without any escaping/encoding performed by the highlighter.+If set to `html` then special HTML/XML characters will be encoded (e.g., `&` becomes `\&`).+The pre/post snippet characters are never encoded.`hl.maxAnalyzedChars`::-The character limit to look for highlights, after which no highlighting will be done. This is mostly only a performance concern for an _analysis_ based offset source since it's the slowest. See <<Schema Options and Performance Considerations>>.+The character limit to look for highlights, after which no highlighting will be done.+This is mostly only a performance concern for an _analysis_ based offset source since it's the slowest.+See <<Schema Options and Performance Considerations>>.+The default is `51200` characters.@@ -98,7 +123,8 @@ There are more parameters supported as well depending on the highlighter (via `h=== Highlighting in the Query Response-In the response to a query, Solr includes highlighting data in a section separate from the documents. It is up to a client to determine how to process this response and display the highlights to users.+In the response to a query, Solr includes highlighting data in a section separate from the documents.+It is up to a client to determine how to process this response and display the highlights to users.Using the example documents included with Solr, we can see how this might work:@@ -135,15 +161,20 @@ we get a response such as this (truncated slightly for space):}-----Note the two sections `docs` and `highlighting`. The `docs` section contains the fields of the document requested with the `fl` parameter of the query (only "id", "name", "manu", and "cat").+Note the two sections `docs` and `highlighting`.+The `docs` section contains the fields of the document requested with the `fl` parameter of the query (only "id", "name", "manu", and "cat").-The `highlighting` section includes the ID of each document, and the field that contains the highlighted portion. In this example, we used the `hl.fl` parameter to say we wanted query terms highlighted in the "manu" field. When there is a match to the query term in that field, it will be included for each document ID in the list.+The `highlighting` section includes the ID of each document, and the field that contains the highlighted portion.+In this example, we used the `hl.fl` parameter to say we wanted query terms highlighted in the "manu" field.+When there is a match to the query term in that field, it will be included for each document ID in the list.== Choosing a Highlighter-Solr provides a `HighlightComponent` (a <<requesthandlers-searchcomponents.adoc#defining-search-components,`SearchComponent`>>) and it's in the default list of components for search handlers. It offers a somewhat unified API over multiple actual highlighting implementations (or simply "highlighters") that do the business of highlighting.+Solr provides a `HighlightComponent` (a <<requesthandlers-searchcomponents.adoc#defining-search-components,`SearchComponent`>>) and it's in the default list of components for search handlers.+It offers a somewhat unified API over multiple actual highlighting implementations (or simply "highlighters") that do the business of highlighting.-There are many parameters supported by more than one highlighter, and sometimes the implementation details and semantics will be a bit different, so don't expect identical results when switching highlighters. You should use the `hl.method` parameter to choose a highlighter but it's also possible to explicitly configure an implementation by class name in `solrconfig.xml`.+There are many parameters supported by more than one highlighter, and sometimes the implementation details and semantics will be a bit different, so don't expect identical results when switching highlighters.+You should use the `hl.method` parameter to choose a highlighter but it's also possible to explicitly configure an implementation by class name in `solrconfig.xml`.There are four highlighters available that can be chosen at runtime with the `hl.method` parameter, in order of general recommendation:@@ -172,57 +203,74 @@ The "alternate" fallback options are more primitive.The Original Highlighter, sometimes called the "Standard Highlighter" or "Default Highlighter", is Lucene's original highlighter – a venerable option with a high degree of customization options.Its query accuracy is good enough for most needs, although it's not quite as good/perfect as the Unified Highlighter.+-The Original Highlighter will normally analyze stored text on the fly in order to highlight. It will use full term vectors if available.+The Original Highlighter will normally analyze stored text on the fly in order to highlight.+It will use full term vectors if available.If the text isn't "stored" but is in doc values (`docValues="true"`), this highlighter can work with it.+-Where this highlighter falls short is performance; it's often twice as slow as the Unified Highlighter. And despite being the most customizable, it doesn't have a BreakIterator based fragmenter (all the others do), which could pose a challenge for some languages.+Where this highlighter falls short is performance; it's often twice as slow as the Unified Highlighter.+And despite being the most customizable, it doesn't have a BreakIterator based fragmenter (all the others do), which could pose a challenge for some languages.<<The FastVector Highlighter,FastVector Highlighter>>:: (`hl.method=fastVector`)+-The FastVector Highlighter _requires_ full term vector options (`termVectors`, `termPositions`, and `termOffsets`) on the field, and is optimized with that in mind. It is nearly as configurable as the Original Highlighter with some variability.+The FastVector Highlighter _requires_ full term vector options (`termVectors`, `termPositions`, and `termOffsets`) on the field, and is optimized with that in mind.+It is nearly as configurable as the Original Highlighter with some variability.+This highlighter notably supports multi-colored highlighting such that different query words can be denoted in the fragment with different marking, usually expressed as an HTML tag with a unique color.+This highlighter's query-representation is less advanced than the Original or Unified Highlighters: for example it will not work well with the `surround` parser, and there are multiple reported bugs pertaining to queries with stop-words.-Both the FastVector and Original Highlighters can be used in conjunction in a search request to highlight some fields with one and some the other. In contrast, the Unified Highlighter can only be chosen exclusively.+Both the FastVector and Original Highlighters can be used in conjunction in a search request to highlight some fields with one and some the other.+In contrast, the Unified Highlighter can only be chosen exclusively.-The Unified Highlighter is exclusively configured via search parameters. In contrast, some settings for the Original and FastVector Highlighters are set in `solrconfig.xml`. There's a robust example of the latter in the "```techproducts```" configset.+The Unified Highlighter is exclusively configured via search parameters.+In contrast, some settings for the Original and FastVector Highlighters are set in `solrconfig.xml`.+There's a robust example of the latter in the "```techproducts```" configset.In addition to further information below, more information can be found in the {solr-javadocs}/core/org/apache/solr/highlight/package-summary.html[Solr javadocs].=== Schema Options and Performance Considerations-Fundamental to the internals of highlighting are detecting the _offsets_ of the individual words that match the query. Some of the highlighters can run the stored text through the analysis chain defined in the schema, some can look them up from _postings_, and some can look them up from _term vectors._ These choices have different trade-offs:+Fundamental to the internals of highlighting are detecting the _offsets_ of the individual words that match the query.+Some of the highlighters can run the stored text through the analysis chain defined in the schema, some can look them up from _postings_, and some can look them up from _term vectors._ These choices have different trade-offs:-* *Analysis*: Supported by the Unified and Original Highlighters. If you don't go out of your way to configure the other options below, the highlighter will analyze the stored text on the fly (during highlighting) to calculate offsets.+* *Analysis*: Supported by the Unified and Original Highlighters.+If you don't go out of your way to configure the other options below, the highlighter will analyze the stored text on the fly (during highlighting) to calculate offsets.+The benefit of this approach is that your index won't grow larger with any extra data that isn't strictly necessary for highlighting.+The down side is that highlighting speed is roughly linear with the amount of text to process, with a large factor being the complexity of your analysis chain.+-For "short" text, this is a good choice. Or maybe it's not short but you're prioritizing a smaller index and indexing speed over highlighting performance.-* *Postings*: Supported by the Unified Highlighter. Set `storeOffsetsWithPositions` to `true`. This adds a moderate amount of extra data to the index but it speeds up highlighting tremendously, especially compared to analysis with longer text fields.+For "short" text, this is a good choice.+Or maybe it's not short but you're prioritizing a smaller index and indexing speed over highlighting performance.+* *Postings*: Supported by the Unified Highlighter.+Set `storeOffsetsWithPositions` to `true`.+This adds a moderate amount of extra data to the index but it speeds up highlighting tremendously, especially compared to analysis with longer text fields.+However, wildcard queries will fall back to analysis unless "light" term vectors are added.-** *with Term Vectors (light)*: Supported only by the Unified Highlighter. To enable this mode set `termVectors` to `true` but no other term vector related options on the field being highlighted.+** *with Term Vectors (light)*: Supported only by the Unified Highlighter.+To enable this mode set `termVectors` to `true` but no other term vector related options on the field being highlighted.+-This adds even more data to the index than just `storeOffsetsWithPositions` but not as much as enabling all the extra term vector options. Term Vectors are only accessed by the highlighter when a wildcard query is used and will prevent a fall back to analysis of the stored text.+This adds even more data to the index than just `storeOffsetsWithPositions` but not as much as enabling all the extra term vector options.+Term Vectors are only accessed by the highlighter when a wildcard query is used and will prevent a fall back to analysis of the stored text.+This is definitely the fastest option for highlighting wildcard queries on large text fields.-* *Term Vectors (full)*: Supported by the Unified, FastVector, and Original Highlighters. Set `termVectors`, `termPositions`, and `termOffsets` to `true`, and potentially `termPayloads` for advanced use cases.+* *Term Vectors (full)*: Supported by the Unified, FastVector, and Original Highlighters.+Set `termVectors`, `termPositions`, and `termOffsets` to `true`, and potentially `termPayloads` for advanced use cases.+-This adds substantial weight to the index – similar in size to the compressed stored text. If you are using the Unified Highlighter then this is not a recommended configuration since it's slower and heavier than postings with light term vectors. However, this could make sense if full term vectors are already needed for another use-case.+This adds substantial weight to the index – similar in size to the compressed stored text.+If you are using the Unified Highlighter then this is not a recommended configuration since it's slower and heavier than postings with light term vectors.+However, this could make sense if full term vectors are already needed for another use-case.== The Unified HighlighterThe Unified Highlighter supports these following additional parameters to the ones listed earlier:`hl.offsetSource`::-By default, the Unified Highlighter will usually pick the right offset source (see above). However it may be ambiguous such as during a migration from one offset source to another that hasn't completed.+By default, the Unified Highlighter will usually pick the right offset source (see above).+However it may be ambiguous such as during a migration from one offset source to another that hasn't completed.+The offset source can be explicitly configured to one of: `ANALYSIS`, `POSTINGS`, `POSTINGS_WITH_TERM_VECTORS`, or `TERM_VECTORS`.@@ -240,19 +288,25 @@ When `false`, it's an optimal target -- the highlighter will _on average_ producA `false` setting is slower, particularly when there's lots of text and `hl.bs.type=SENTENCE`.`hl.tag.ellipsis`::-By default, each snippet is returned as a separate value (as is done with the other highlighters). Set this parameter to instead return one string with this text as the delimiter. _Note: this is likely to be removed in the future._+By default, each snippet is returned as a separate value (as is done with the other highlighters).+Set this parameter to instead return one string with this text as the delimiter.+_Note: this is likely to be removed in the future._`hl.defaultSummary`::-If `true`, use the leading portion of the text as a snippet if a proper highlighted snippet can't otherwise be generated. The default is `false`.+If `true`, use the leading portion of the text as a snippet if a proper highlighted snippet can't otherwise be generated.+The default is `false`.`hl.score.k1`::-Specifies BM25 term frequency normalization parameter 'k1'. For example, it can be set to `0` to rank passages solely based on the number of query terms that match. The default is `1.2`.+Specifies BM25 term frequency normalization parameter 'k1'. For example, it can be set to `0` to rank passages solely based on the number of query terms that match.+The default is `1.2`.`hl.score.b`::-Specifies BM25 length normalization parameter 'b'. For example, it can be set to "0" to ignore the length of passages entirely when ranking. The default is `0.75`.+Specifies BM25 length normalization parameter 'b'. For example, it can be set to "0" to ignore the length of passages entirely when ranking.+The default is `0.75`.`hl.score.pivot`::-Specifies BM25 average passage length in characters. The default is `87`.+Specifies BM25 average passage length in characters.+The default is `87`.`hl.bs.language`::Specifies the breakiterator language for dividing the document into passages.@@ -264,14 +318,18 @@ Specifies the breakiterator country for dividing the document into passages.Specifies the breakiterator variant for dividing the document into passages.`hl.bs.type`::-Specifies the breakiterator type for dividing the document into passages. Can be `SEPARATOR`, `SENTENCE`, `WORD`*, `CHARACTER`, `LINE`, or `WHOLE`. `SEPARATOR` is special value that splits text on a user-provided character in `hl.bs.separator`.+Specifies the breakiterator type for dividing the document into passages.+Can be `SEPARATOR`, `SENTENCE`, `WORD`*, `CHARACTER`, `LINE`, or `WHOLE`.+`SEPARATOR` is special value that splits text on a user-provided character in `hl.bs.separator`.+The default is `SENTENCE`.`hl.bs.separator`::-Indicates which character to break the text on. Use only if you have defined `hl.bs.type=SEPARATOR`.+Indicates which character to break the text on.+Use only if you have defined `hl.bs.type=SEPARATOR`.+-This is useful when the text has already been manipulated in advance to have a special delineation character at desired highlight passage boundaries. This character will still appear in the text as the last character of a passage.+This is useful when the text has already been manipulated in advance to have a special delineation character at desired highlight passage boundaries.+This character will still appear in the text as the last character of a passage.`hl.weightMatches`::Tells the UH to use Lucene's new "Weight Matches" API instead of doing SpanQuery conversion.@@ -286,10 +344,13 @@ However if either `hl.usePhraseHighlighter` or `hl.multiTermQuery` are set to faThe Original Highlighter supports these following additional parameters to the ones listed earlier:`hl.mergeContiguous`::-Instructs Solr to collapse contiguous fragments into a single fragment. A value of `true` indicates contiguous fragments will be collapsed into single fragment. The default value, `false`, is also the backward-compatible setting.+Instructs Solr to collapse contiguous fragments into a single fragment.+A value of `true` indicates contiguous fragments will be collapsed into single fragment.+The default value, `false`, is also the backward-compatible setting.`hl.maxMultiValuedToExamine`::-Specifies the maximum number of entries in a multi-valued field to examine before stopping. This can potentially return zero results if the limit is reached before any matches are found.+Specifies the maximum number of entries in a multi-valued field to examine before stopping.+This can potentially return zero results if the limit is reached before any matches are found.+If used with the `maxMultiValuedToMatch`, whichever limit is reached first will determine when to stop looking.+@@ -306,54 +367,67 @@ The default is `Integer.MAX_VALUE`.Specifies a field to be used as a backup default summary if Solr cannot generate a snippet (i.e., because no terms match).`hl.maxAlternateFieldLength`::-Specifies the maximum number of characters of the field to return. Any value less than or equal to `0` means the field's length is unlimited (the default behavior).+Specifies the maximum number of characters of the field to return.+Any value less than or equal to `0` means the field's length is unlimited (the default behavior).+This parameter is only used in conjunction with the `hl.alternateField` parameter.`hl.highlightAlternate`::-If set to `true`, the default, and `hl.alternateFieldName` is active, Solr will show the entire alternate field, with highlighting of occurrences. If `hl.maxAlternateFieldLength=N` is used, Solr returns max `N` characters surrounding the best matching fragment.+If set to `true`, the default, and `hl.alternateFieldName` is active, Solr will show the entire alternate field, with highlighting of occurrences.+If `hl.maxAlternateFieldLength=N` is used, Solr returns max `N` characters surrounding the best matching fragment.+If set to `false`, or if there is no match in the alternate field either, the alternate field will be shown without highlighting.`hl.formatter`::-Selects a formatter for the highlighted output. Currently the only legal value is `simple`, which surrounds a highlighted term with a customizable pre- and post-text snippet.+Selects a formatter for the highlighted output.+Currently the only legal value is `simple`, which surrounds a highlighted term with a customizable pre- and post-text snippet.`hl.simple.pre`, `hl.simple.post`::-Specifies the text that should appear before (`hl.simple.pre`) and after (`hl.simple.post`) a highlighted term, when using the `simple` formatter. The default is `<em>` and `</em>`.+Specifies the text that should appear before (`hl.simple.pre`) and after (`hl.simple.post`) a highlighted term, when using the `simple` formatter.+The default is `<em>` and `</em>`.`hl.fragmenter`::-Specifies a text snippet generator for highlighted text. The standard (default) fragmenter is `gap`, which creates fixed-sized fragments with gaps for multi-valued fields.+Specifies a text snippet generator for highlighted text.+The standard (default) fragmenter is `gap`, which creates fixed-sized fragments with gaps for multi-valued fields.+Another option is `regex`, which tries to create fragments that resemble a specified regular expression.`hl.regex.slop`::When using the regex fragmenter (`hl.fragmenter=regex`), this parameter defines the factor by which the fragmenter can stray from the ideal fragment size (given by `hl.fragsize`) to accommodate a regular expression.+-For instance, a slop of `0.2` with `hl.fragsize=100` should yield fragments between 80 and 120 characters in length. It is usually good to provide a slightly smaller `hl.fragsize` value when using the regex fragmenter.+For instance, a slop of `0.2` with `hl.fragsize=100` should yield fragments between 80 and 120 characters in length.+It is usually good to provide a slightly smaller `hl.fragsize` value when using the regex fragmenter.+The default is `0.6`.`hl.regex.pattern`::-Specifies the regular expression for fragmenting. This could be used to extract sentences.+Specifies the regular expression for fragmenting.+This could be used to extract sentences.`hl.regex.maxAnalyzedChars`::-Instructs Solr to analyze only this many characters from a field when using the regex fragmenter (after which, the fragmenter produces fixed-sized fragments). The default is `10000`.+Instructs Solr to analyze only this many characters from a field when using the regex fragmenter (after which, the fragmenter produces fixed-sized fragments).+The default is `10000`.+Note, applying a complicated regex to a huge field is computationally expensive.`hl.preserveMulti`::-If `true`, multi-valued fields will return all values in the order they were saved in the index. If `false`, the default, only values that match the highlight request will be returned.+If `true`, multi-valued fields will return all values in the order they were saved in the index.+If `false`, the default, only values that match the highlight request will be returned.`hl.payloads`::When `hl.usePhraseHighlighter` is `true` and the indexed field has payloads but not term vectors (generally quite rare), the index's payloads will be read into the highlighter's memory index along with the postings.+If this may happen and you know you don't need them for highlighting (i.e., your queries don't filter by payload) then you can save a little memory by setting this to false.-The Original Highlighter has a plugin architecture that enables new functionality to be registered in `solrconfig.xml`. The "```techproducts```" configset shows most of these settings explicitly. You can use it as a guide to provide your own components to include a `SolrFormatter`, `SolrEncoder`, and `SolrFragmenter.`+The Original Highlighter has a plugin architecture that enables new functionality to be registered in `solrconfig.xml`.+The "```techproducts```" configset shows most of these settings explicitly.+You can use it as a guide to provide your own components to include a `SolrFormatter`, `SolrEncoder`, and `SolrFragmenter.`== The FastVector Highlighter-The FastVector Highlighter (FVH) can be used in conjunction with the Original Highlighter if not all fields should be highlighted with the FVH. In such a mode, set `hl.method=original` and `f.yourTermVecField.hl.method=fastVector` for all fields that should use the FVH. One annoyance to keep in mind is that the Original Highlighter uses `hl.simple.pre` whereas the FVH (and other highlighters) use `hl.tag.pre`.+The FastVector Highlighter (FVH) can be used in conjunction with the Original Highlighter if not all fields should be highlighted with the FVH.+In such a mode, set `hl.method=original` and `f.yourTermVecField.hl.method=fastVector` for all fields that should use the FVH.+One annoyance to keep in mind is that the Original Highlighter uses `hl.simple.pre` whereas the FVH (and other highlighters) use `hl.tag.pre`.In addition to the initial listed parameters, the following parameters documented for the Original Highlighter above are also supported by the FVH:@@ -364,14 +438,19 @@ In addition to the initial listed parameters, the following parameters documenteAnd here are additional parameters supported by the FVH:`hl.fragListBuilder`::-The snippet fragmenting algorithm. The `weighted` fragListBuilder uses IDF-weights to order fragments. This fragListBuilder is the default.+The snippet fragmenting algorithm.+The `weighted` fragListBuilder uses IDF-weights to order fragments.+This fragListBuilder is the default.+-Other options are `single`, which returns the entire field contents as one snippet, or `simple`. You can select a fragListBuilder with this parameter, or modify an existing implementation in `solrconfig.xml` to be the default by adding "default=true".+Other options are `single`, which returns the entire field contents as one snippet, or `simple`.+You can select a fragListBuilder with this parameter, or modify an existing implementation in `solrconfig.xml` to be the default by adding "default=true".`hl.fragmentsBuilder`::The fragments builder is responsible for formatting the fragments, which uses `<em>` and `</em>` markup by default (if `hl.tag.pre` and `hl.tag.post` are not defined).+-Another pre-configured choice is `colored`, which is an example of how to use the fragments builder to insert HTML into the snippets for colored highlights if you choose. You can also implement your own if you'd like. You can select a fragments builder with this parameter, or modify an existing implementation in `solrconfig.xml` to be the default by adding "default=true".+Another pre-configured choice is `colored`, which is an example of how to use the fragments builder to insert HTML into the snippets for colored highlights if you choose.+You can also implement your own if you'd like.+You can select a fragments builder with this parameter, or modify an existing implementation in `solrconfig.xml` to be the default by adding "default=true".`hl.boundaryScanner`::See <<Using Boundary Scanners with the FastVector Highlighter>> below.@@ -380,20 +459,25 @@ See <<Using Boundary Scanners with the FastVector Highlighter>> below.See <<Using Boundary Scanners with the FastVector Highlighter>> below.`hl.phraseLimit`::-The maximum number of phrases to analyze when searching for the highest-scoring phrase. The default is `5000`.+The maximum number of phrases to analyze when searching for the highest-scoring phrase.+The default is `5000`.`hl.multiValuedSeparatorChar`::-Text to use to separate one value from the next for a multi-valued field. The default is " " (a space).+Text to use to separate one value from the next for a multi-valued field.+The default is " " (a space).=== Using Boundary Scanners with the FastVector Highlighter-The FastVector Highlighter will occasionally truncate highlighted words. To prevent this, implement a boundary scanner in `solrconfig.xml`, then use the `hl.boundaryScanner` parameter to specify the boundary scanner for highlighting.+The FastVector Highlighter will occasionally truncate highlighted words.+To prevent this, implement a boundary scanner in `solrconfig.xml`, then use the `hl.boundaryScanner` parameter to specify the boundary scanner for highlighting.Solr supports two boundary scanners: `breakIterator` and `simple`.==== The breakIterator Boundary Scanner-The `breakIterator` boundary scanner offers excellent performance right out of the box by taking locale and boundary type into account. In most cases you will want to use the `breakIterator` boundary scanner. To implement the `breakIterator` boundary scanner, add this code to the `highlighting` section of your `solrconfig.xml` file, adjusting the type, language, and country values as appropriate to your application:+The `breakIterator` boundary scanner offers excellent performance right out of the box by taking locale and boundary type into account.+In most cases you will want to use the `breakIterator` boundary scanner.+To implement the `breakIterator` boundary scanner, add this code to the `highlighting` section of your `solrconfig.xml` file, adjusting the type, language, and country values as appropriate to your application:[source,xml]----@@ -410,7 +494,8 @@ Possible values for the `hl.bs.type` parameter are WORD, LINE, SENTENCE, and CHA==== The simple Boundary Scanner-The `simple` boundary scanner scans term boundaries for a specified maximum character value (`hl.bs.maxScan`) and for common delimiters such as punctuation marks (`hl.bs.chars`). To implement the `simple` boundary scanner, add this code to the `highlighting` section of your `solrconfig.xml` file, adjusting the values as appropriate to your application:+The `simple` boundary scanner scans term boundaries for a specified maximum character value (`hl.bs.maxScan`) and for common delimiters such as punctuation marks (`hl.bs.chars`).+To implement the `simple` boundary scanner, add this code to the `highlighting` section of your `solrconfig.xml` file, adjusting the values as appropriate to your application:[source,xml]----diff --git a/solr/solr-ref-guide/src/implicit-requesthandlers.adoc b/solr/solr-ref-guide/src/implicit-requesthandlers.adocindex 0c5c9d099d0..f53ef8cdfd6 100644--- a/solr/solr-ref-guide/src/implicit-requesthandlers.adoc+++ b/solr/solr-ref-guide/src/implicit-requesthandlers.adoc@@ -29,7 +29,8 @@ NOTE: All endpoint paths listed below should be placed after Solr's host and porMany of these handlers are used throughout the Admin UI to show information about Solr.[horizontal]-File:: Return content of files in `${solr.home}/conf/`. This handler must have a collection name in the path to the endpoint.+File:: Return content of files in `${solr.home}/conf/`.+This handler must have a collection name in the path to the endpoint.+[cols="3*.",frame=none,grid=cols,options="header"]|===@@ -150,7 +151,8 @@ Document Analysis:: Return a breakdown of the analysis process of the given docu|`solr/<collection>/analysis/document` |{solr-javadocs}/core/org/apache/solr/handler/DocumentAnalysisRequestHandler.html[DocumentAnalysisRequestHandler] |`_ANALYSIS_DOCUMENT`|===-Field Analysis:: Return index- and query-time analysis over the given field(s)/field type(s). This handler drives the <<analysis-screen.adoc#,Analysis screen>> in Solr's Admin UI.+Field Analysis:: Return index- and query-time analysis over the given field(s)/field type(s).+This handler drives the <<analysis-screen.adoc#,Analysis screen>> in Solr's Admin UI.+[cols="3*.",frame=none,grid=cols,options="header"]|===diff --git a/solr/solr-ref-guide/src/index-location-format.adoc b/solr/solr-ref-guide/src/index-location-format.adocindex 4370356ccd1..7e0d6cd0936 100644--- a/solr/solr-ref-guide/src/index-location-format.adoc+++ b/solr/solr-ref-guide/src/index-location-format.adoc@@ -20,7 +20,10 @@ Where and how Solr stores its indexes are configurable options.== Specifying a Location for Index Data with the dataDir Parameter-By default, Solr stores its index data in a directory called `/data` under the core's instance directory (`instanceDir`). If you would like to specify a different directory for storing index data, you can configure the `dataDir` in the `core.properties` file for the core, or use the `<dataDir>` parameter in the `solrconfig.xml` file. You can specify another directory either with an absolute path or a pathname relative to the instanceDir of the SolrCore. For example:+By default, Solr stores its index data in a directory called `/data` under the core's instance directory (`instanceDir`).+If you would like to specify a different directory for storing index data, you can configure the `dataDir` in the `core.properties` file for the core, or use the `<dataDir>` parameter in the `solrconfig.xml` file.+You can specify another directory either with an absolute path or a pathname relative to the instanceDir of the SolrCore.+For example:[source,xml]----@@ -36,7 +39,8 @@ element `<solrDataHome>` then the location of data directory will be `<SOLR_DATA== Specifying the DirectoryFactory For Your Index-The default {solr-javadocs}/core/org/apache/solr/core/NRTCachingDirectoryFactory.html[`solr.NRTCachingDirectoryFactory`] is filesystem based, and tries to pick the best implementation for the current JVM and platform. You can force a particular implementation and/or configuration options by specifying {solr-javadocs}/core/org/apache/solr/core/MMapDirectoryFactory.html[`solr.MMapDirectoryFactory`] or {solr-javadocs}/core/org/apache/solr/core/NIOFSDirectoryFactory.html[`solr.NIOFSDirectoryFactory`].+The default {solr-javadocs}/core/org/apache/solr/core/NRTCachingDirectoryFactory.html[`solr.NRTCachingDirectoryFactory`] is filesystem based, and tries to pick the best implementation for the current JVM and platform.+You can force a particular implementation and/or configuration options by specifying {solr-javadocs}/core/org/apache/solr/core/MMapDirectoryFactory.html[`solr.MMapDirectoryFactory`] or {solr-javadocs}/core/org/apache/solr/core/NIOFSDirectoryFactory.html[`solr.NIOFSDirectoryFactory`].[source,xml]----@@ -46,7 +50,8 @@ The default {solr-javadocs}/core/org/apache/solr/core/NRTCachingDirectoryFactory</directoryFactory>-----The {solr-javadocs}/core/org/apache/solr/core/RAMDirectoryFactory.html[`solr.RAMDirectoryFactory`] is memory based, not persistent, and does not work with replication. Use this DirectoryFactory to store your index in RAM.+The {solr-javadocs}/core/org/apache/solr/core/RAMDirectoryFactory.html[`solr.RAMDirectoryFactory`] is memory based, not persistent, and does not work with replication.+Use this DirectoryFactory to store your index in RAM.[source,xml]----@@ -55,5 +60,6 @@ The {solr-javadocs}/core/org/apache/solr/core/RAMDirectoryFactory.html[`solr.RAM[NOTE]====-If you are using Hadoop and would like to store your indexes in HDFS, you should use the {solr-javadocs}/core/org/apache/solr/core/HdfsDirectoryFactory.html[`solr.HdfsDirectoryFactory`] instead of either of the above implementations. For more details, see the section <<solr-on-hdfs.adoc#,Solr on HDFS>>.+If you are using Hadoop and would like to store your indexes in HDFS, you should use the {solr-javadocs}/core/org/apache/solr/core/HdfsDirectoryFactory.html[`solr.HdfsDirectoryFactory`] instead of either of the above implementations.+For more details, see the section <<solr-on-hdfs.adoc#,Solr on HDFS>>.====diff --git a/solr/solr-ref-guide/src/index-segments-merging.adoc b/solr/solr-ref-guide/src/index-segments-merging.adocindex 7ecb000d0ab..98a9adb000f 100644--- a/solr/solr-ref-guide/src/index-segments-merging.adoc+++ b/solr/solr-ref-guide/src/index-segments-merging.adoc@@ -147,7 +147,7 @@ There are two parameters that can can be adjusted when using the default TieredMA value of `0.0` will make expungeDeletes behave essentially identically to `optimize`.`deletesPctAllowed`::-(default `33.0`). During normal segment merging, a best effort is made to insure that the total percentage of deleted documents in the index is below this threshold.+(default `33.0`) During normal segment merging, a best effort is made to insure that the total percentage of deleted documents in the index is below this threshold.Valid settings are between 20% and 50%.33% was chosen as the default because as this setting approaches 20%, considerable load is added to the system.@@ -183,7 +183,8 @@ If a merge is necessary yet we already have this many threads running, the indexNote that Solr will only run the smallest `maxThreadCount` merges at a time.`maxThreadCount`::-The maximum number of simultaneous merge threads that should be running at once. This must be less than `maxMergeCount`.+The maximum number of simultaneous merge threads that should be running at once.+This must be less than `maxMergeCount`.`ioThrottle`::A Boolean value (`true` or `false`) to explicitly control I/O throttling.@@ -293,7 +294,8 @@ The default is `1000`, expressed in milliseconds.== Other Indexing Settings-There are a few other parameters that may be important to configure for your implementation. These settings affect how or when updates are made to an index.+There are a few other parameters that may be important to configure for your implementation.+These settings affect how or when updates are made to an index.=== deletionPolicydiff --git a/solr/solr-ref-guide/src/index.adoc b/solr/solr-ref-guide/src/index.adocindex 2805c849dd2..010cdda289f 100644--- a/solr/solr-ref-guide/src/index.adoc+++ b/solr/solr-ref-guide/src/index.adoc@@ -33,7 +33,8 @@[.lead-homepage]Welcome to Apache Solr(TM), the open source solution for search and analytics.-Solr is the fast open source search platform built on Apache Lucene(TM) that provides scalable indexing and search, as well as faceting, hit highlighting and advanced analysis/tokenization capabilities. Solr and Lucene are managed by the http://www.apache.org/[Apache Software Foundation].+Solr is the fast open source search platform built on Apache Lucene(TM) that provides scalable indexing and search, as well as faceting, hit highlighting and advanced analysis/tokenization capabilities.+Solr and Lucene are managed by the http://www.apache.org/[Apache Software Foundation].This Reference Guide is the official Solr documentation, written and published by Lucene/Solr committers.****diff --git a/solr/solr-ref-guide/src/indexing-data-operations.adoc b/solr/solr-ref-guide/src/indexing-data-operations.adocindex 6245d113460..e33fe91209d 100644--- a/solr/solr-ref-guide/src/indexing-data-operations.adoc+++ b/solr/solr-ref-guide/src/indexing-data-operations.adoc@@ -27,7 +27,8 @@// specific language governing permissions and limitations// under the License.-This section describes how Solr adds data to its index. It covers the following topics:+This section describes how Solr adds data to its index.+It covers the following topics:****// This tags the below list so it can be used in the parent page section list@@ -52,4 +53,5 @@ This section describes how Solr adds data to its index. It covers the following== Indexing Using Client APIs-Using client APIs, such as <<solrj.adoc#,SolrJ>>, from your applications is an important option for updating Solr indexes. See the <<client-apis.adoc#,Client APIs>> section for more information.+Using client APIs, such as <<solrj.adoc#,SolrJ>>, from your applications is an important option for updating Solr indexes.+See the <<client-apis.adoc#,Client APIs>> section for more information.diff --git a/solr/solr-ref-guide/src/indexing-nested-documents.adoc b/solr/solr-ref-guide/src/indexing-nested-documents.adocindex 21f02938db4..30030187023 100644--- a/solr/solr-ref-guide/src/indexing-nested-documents.adoc+++ b/solr/solr-ref-guide/src/indexing-nested-documents.adoc@@ -423,17 +423,13 @@ include::{example-source-dir}IndexingNestedDocuments.java[tag=anon-kids]---This simplified approach was common in older versions of Solr, and can still be used with "Root-Only" schemas that do not contain any other nested related fields apart from `\_root_`. (Many schemas in existence are this way simply because default configsets are this way, even if the application isn't using nested documents.)+This simplified approach was common in older versions of Solr, and can still be used with "Root-Only" schemas that do not contain any other nested related fields apart from `\_root_`.+Many schemas in existence are this way simply because default configsets are this way, even if the application isn't using nested documents.This approach should *NOT* be used when schemas include a `\_nest_path_` field, as the existence of that field triggers assumptions and changes in behavior in various query time functionality, such as the <<searching-nested-documents.adoc#child-doc-transformer,[child]>>, that will not work when nested documents do not have any intrinsic "nested path" information.-The results of indexing anonymous nested children with a "Root-Only" schema are similar to what-happens if you attempt to index "pseudo field" nested documents using a "Root-Only" schema.-Notably: since there is no nested path information for the-<<searching-nested-documents.adoc#child-doc-transformer,[child]>> transformer to use to reconstruct the structure of a nest-of documents, it returns all matching children as a flat list, similar in structure to how they were originally indexed:--+The results of indexing anonymous nested children with a "Root-Only" schema are similar to what happens if you attempt to index "pseudo field" nested documents using a "Root-Only" schema.+Notably: since there is no nested path information for the <<searching-nested-documents.adoc#child-doc-transformer,[child]>> transformer to use to reconstruct the structure of a nest of documents, it returns all matching children as a flat list, similar in structure to how they were originally indexed:[.dynamic-tabs]--diff --git a/solr/solr-ref-guide/src/indexing-with-tika.adoc b/solr/solr-ref-guide/src/indexing-with-tika.adocindex e3c97239995..0086acbcb0e 100644--- a/solr/solr-ref-guide/src/indexing-with-tika.adoc+++ b/solr/solr-ref-guide/src/indexing-with-tika.adoc@@ -21,9 +21,11 @@ If the documents you need to index are in a binary format, such as Word, Excel,Solr uses code from the Tika project to provide a framework for incorporating many different file-format parsers such as http://pdfbox.apache.org/[Apache PDFBox] and http://poi.apache.org/index.html[Apache POI] into Solr itself.Working with this framework, Solr's `ExtractingRequestHandler` uses Tika internally to support uploading binary files-for data extraction and indexing. Downloading Tika is not required to use Solr Cell.+for data extraction and indexing.+Downloading Tika is not required to use Solr Cell.-When this framework was under development, it was called the Solr _Content Extraction Library_, or _CEL_; from that abbreviation came this framework's name: Solr Cell. The names Solr Cell and `ExtractingRequestHandler` are used+When this framework was under development, it was called the Solr _Content Extraction Library_, or _CEL_; from that abbreviation came this framework's name: Solr Cell.+The names Solr Cell and `ExtractingRequestHandler` are usedinterchangeably for this feature.== Key Solr Cell Concepts@@ -37,7 +39,8 @@ See http://tika.apache.org/{ivy-tika-version}/formats.html for the file types suSolr responds to Tika's SAX events to create one or more text fields from the content.Tika exposes document metadata as well (apart from the XHTML).* Tika produces metadata such as Title, Subject, and Author according to specifications such as the DublinCore.-The metadata available is highly dependent on the file types and what they in turn contain. Some of the general metadata created is described in the section <<Metadata Created by Tika>> below.+The metadata available is highly dependent on the file types and what they in turn contain.+Some of the general metadata created is described in the section <<Metadata Created by Tika>> below.Solr Cell supplies some metadata of its own too.* Solr Cell concatenates text from the internal XHTML into a `content` field.You can configure which elements should be included/ignored, and which should map to another field.@@ -93,18 +96,22 @@ Once Solr is started, you can use curl to send a sample PDF included with Solr vcurl 'http://localhost:8983/solr/gettingstarted/update/extract?literal.id=doc1&commit=true' -F "myfile=@example/exampledocs/solr-word.pdf"-----The URL above calls the `ExtractingRequestHandler`, uploads the file `solr-word.pdf`, and assigns it the unique ID `doc1`. Here's a closer look at the components of this command:+The URL above calls the `ExtractingRequestHandler`, uploads the file `solr-word.pdf`, and assigns it the unique ID `doc1`.+Here's a closer look at the components of this command:* The `literal.id=doc1` parameter provides a unique ID for the document being indexed.Without this, the ID would be set to the absolute path to the file.+There are alternatives to this, such as mapping a metadata field to the ID, generating a new UUID, or generating an ID from a signature (hash) of the content.-* The `commit=true parameter` causes Solr to perform a commit after indexing the document, making it immediately searchable. For optimum performance when loading many documents, don't call the commit command until you are done.+* The `commit=true parameter` causes Solr to perform a commit after indexing the document, making it immediately searchable.+For optimum performance when loading many documents, don't call the commit command until you are done.-* The `-F` flag instructs curl to POST data using the Content-Type `multipart/form-data` and supports the uploading of binary files. The `@` symbol instructs curl to upload the attached file.+* The `-F` flag instructs curl to POST data using the Content-Type `multipart/form-data` and supports the uploading of binary files.+The `@` symbol instructs curl to upload the attached file.-* The argument `myfile=@example/exampledocs/solr-word.pdf` uploads the sample file. Note this includes the path, so if you upload a different file, always be sure to include either the relative or absolute path to the file.+* The argument `myfile=@example/exampledocs/solr-word.pdf` uploads the sample file.+Note this includes the path, so if you upload a different file, always be sure to include either the relative or absolute path to the file.You can also use `bin/post` to do the same thing:@@ -113,7 +120,8 @@ You can also use `bin/post` to do the same thing:bin/post -c gettingstarted example/exampledocs/solr-word.pdf -params "literal.id=doc1"-----Now you can execute a query and find that document with a request like `\http://localhost:8983/solr/gettingstarted/select?q=pdf`. The document will look something like this:+Now you can execute a query and find that document with a request like `\http://localhost:8983/solr/gettingstarted/select?q=pdf`.+The document will look something like this:image::images/indexing-with-tika/sample-pdf-query.png[float="right",width=50%,pdfwidth=60%]@@ -153,7 +161,10 @@ These parameters can be set for each indexing request (as request parameters), othe request handler generally by defining them in `solrconfig.xml`, as described in <<Configuring the ExtractingRequestHandler in solrconfig.xml>>.`capture`::-Captures XHTML elements with the specified name for a supplementary addition to the Solr document. This parameter can be useful for copying chunks of the XHTML into a separate field. For instance, it could be used to grab paragraphs (`<p>`) and index them into a separate field. Note that content is still also captured into the `content` field.+Captures XHTML elements with the specified name for a supplementary addition to the Solr document.+This parameter can be useful for copying chunks of the XHTML into a separate field.+For instance, it could be used to grab paragraphs (`<p>`) and index them into a separate field.+Note that content is still also captured into the `content` field.+Example: `capture=p` (in a request) or `<str name="capture">p</str>` (in `solrconfig.xml`)+@@ -162,7 +173,8 @@ Output: `"p": {"This is a paragraph from my document."}`This parameter can also be used with the `fmap._source_field_` parameter to map content from attributes to a new field.`captureAttr`::-Indexes attributes of the Tika XHTML elements into separate fields, named after the element. If set to `true`, when extracting from HTML, Tika can return the href attributes in `<a>` tags as fields named "`a`".+Indexes attributes of the Tika XHTML elements into separate fields, named after the element.+If set to `true`, when extracting from HTML, Tika can return the href attributes in `<a>` tags as fields named "`a`".+Example: `captureAttr=true`+@@ -179,12 +191,17 @@ A default field to use if the `uprefix` parameter is not specified and a field cExample: `defaultField=\_text_``extractOnly`::-Default is `false`. If `true`, returns the extracted content from Tika without indexing the document. This returns the extracted XHTML as a string in the response. When viewing on a screen, it may be useful to set the `extractFormat` parameter for a response format other than XML to aid in viewing the embedded XHTML tags.+Default is `false`.+If `true`, returns the extracted content from Tika without indexing the document.+This returns the extracted XHTML as a string in the response.+When viewing on a screen, it may be useful to set the `extractFormat` parameter for a response format other than XML to aid in viewing the embedded XHTML tags.+Example: `extractOnly=true``extractFormat`::-The default is `xml`, but the other option is `text`. Controls the serialization format of the extract content. The `xml` format is actually XHTML, the same format that results from passing the `-x` command to the Tika command line application, while the text format is like that produced by Tika's `-t` command.+The default is `xml`, but the other option is `text`.+Controls the serialization format of the extract content.+The `xml` format is actually XHTML, the same format that results from passing the `-x` command to the Tika command line application, while the text format is like that produced by Tika's `-t` command.+This parameter is valid only if `extractOnly` is set to true.+@@ -193,17 +210,20 @@ Example: `extractFormat=text`Output: For an example output (in XML), see https://cwiki.apache.org/confluence/display/solr/TikaExtractOnlyExampleOutput`fmap._source_field_`::-Maps (moves) one field name to another. The `source_field` must be a field in incoming documents, and the value is the Solr field to map to.+Maps (moves) one field name to another.+The `source_field` must be a field in incoming documents, and the value is the Solr field to map to.+Example: `fmap.content=text` causes the data in the `content` field generated by Tika to be moved to the Solr's `text` field.`ignoreTikaException`::-If `true`, exceptions found during processing will be skipped. Any metadata available, however, will be indexed.+If `true`, exceptions found during processing will be skipped.+Any metadata available, however, will be indexed.+Example: `ignoreTikaException=true``literal._fieldname_`::-Populates a field with the name supplied with the specified value for each document. The data can be multivalued if the field is multivalued.+Populates a field with the name supplied with the specified value for each document.+The data can be multivalued if the field is multivalued.+Example: `literal.doc_status=published`+@@ -213,7 +233,8 @@ Output: `"doc_status": "published"`If `true` (the default), literal field values will override other values with the same field name.+If `false`, literal values defined with `literal._fieldname_` will be appended to data already in the fields extracted-from Tika. When setting `literalsOverride` to `false`, the field must be multivalued.+from Tika.+When setting `literalsOverride` to `false`, the field must be multivalued.+Example: `literalsOverride=false`@@ -225,43 +246,51 @@ Example: `lowernames=true`Output: Assuming input of "Content-Type", the result in documents would be a field `content_type``multipartUploadLimitInKB`::-Defines the size in kilobytes of documents to allow. The default is `2048` (2Mb).+Defines the size in kilobytes of documents to allow.+The default is `2048` (2Mb).If you have very large documents, you should increase this or they will be rejected.+Example: `multipartUploadLimitInKB=2048000``parseContext.config`::If a Tika parser being used allows parameters, you can pass them to Tika by creating a parser configuration file and-pointing Solr to it. See the section <<Parser-Specific Properties>> for more information about how to use this parameter.+pointing Solr to it.+See the section <<Parser-Specific Properties>> for more information about how to use this parameter.+Example: `parseContext.config=pdf-config.xml``passwordsFile`::-Defines a file path and name for a file of file name to password mappings. See the section+Defines a file path and name for a file of file name to password mappings.+See the section<<Indexing Encrypted Documents>> for more information about using a password file.+Example: `passwordsFile=/path/to/passwords.txt``resource.name`::-Specifies the name of the file to index. This is optional, but Tika can use it as a hint for detecting a file's MIME type.+Specifies the name of the file to index.+This is optional, but Tika can use it as a hint for detecting a file's MIME type.+Example: `resource.name=mydoc.doc``resource.password`::-Defines a password to use for a password-protected PDF or OOXML file. See the section <<Indexing Encrypted Documents>>+Defines a password to use for a password-protected PDF or OOXML file.+See the section <<Indexing Encrypted Documents>>for more information about using this parameter.+Example: `resource.password=secret``tika.config`::-Defines a file path and name to a custom Tika configuration file. This is only required if you have customized your Tika implementation.+Defines a file path and name to a custom Tika configuration file.+This is only required if you have customized your Tika implementation.+Example: `tika.config=/path/to/tika.config``uprefix`::-Prefixes all fields _that are undefined in the schema_ with the given prefix. This is very useful when combined with dynamic field definitions.+Prefixes all fields _that are undefined in the schema_ with the given prefix.+This is very useful when combined with dynamic field definitions.+-Example: `uprefix=ignored_` would add `ignored_` as a prefix to all unknown fields. In this case, you could additionally define a rule in the Schema to not index these fields:+Example: `uprefix=ignored_` would add `ignored_` as a prefix to all unknown fields.+In this case, you could additionally define a rule in the Schema to not index these fields:+`<dynamicField name="ignored_*" type="ignored" />`@@ -284,7 +313,8 @@ You will need to configure your `solrconfig.xml` to find the `ExtractingRequestH<lib dir="${solr.install.dir:../../..}/dist/" regex="solr-cell-\d.*\.jar" />-----You can then configure the `ExtractingRequestHandler` in `solrconfig.xml`. The following is the default+You can then configure the `ExtractingRequestHandler` in `solrconfig.xml`.+The following is the defaultconfiguration found in Solr's `_default` configset, which you can modify as needed:[source,xml]@@ -309,7 +339,8 @@ that parse numbers and dates and do other manipulations on the metadata fields gIn Solr's default configsets, <<schemaless-mode.adoc#,"schemaless">> (aka data driven, or field guessing) mode is enabled, which does a variety of such processing already.If you instead explicitly define the fields for your schema, you can selectively specify the desired URPs.-An easy way to specify this is to configure the parameter `processor` (under `defaults`) to `uuid,remove-blank,field-name-mutating,parse-boolean,parse-long,parse-double,parse-date`. For example:+An easy way to specify this is to configure the parameter `processor` (under `defaults`) to `uuid,remove-blank,field-name-mutating,parse-boolean,parse-long,parse-double,parse-date`.+For example:[source,xml]----@@ -324,7 +355,9 @@ An easy way to specify this is to configure the parameter `processor` (under `de</requestHandler>-----The above suggested list was taken from the list of URPs that run as a part of schemaless mode and provide much of its functionality. However, one major part of the schemaless functionality is missing from the suggested list, `add-unknown-fields-to-the-schema`, which is the part that adds fields to the schema. So you can use the other URPs without worrying about unexpected field additions.+The above suggested list was taken from the list of URPs that run as a part of schemaless mode and provide much of its functionality.+However, one major part of the schemaless functionality is missing from the suggested list, `add-unknown-fields-to-the-schema`, which is the part that adds fields to the schema.+So you can use the other URPs without worrying about unexpected field additions.======= Parser-Specific Properties@@ -332,7 +365,8 @@ The above suggested list was taken from the list of URPs that run as a part of sParsers used by Tika may have specific properties to govern how data is extracted.These can be passed through Solr for special parsing situations.-For instance, when using the Tika library from a Java program, the `PDFParserConfig` class has a method `setSortByPosition(boolean)` that can extract vertically oriented text. To access that method via configuration with the `ExtractingRequestHandler`, one can add the `parseContext.config` property to `solrconfig.xml` and then set properties in Tika's `PDFParserConfig` as in the example below.+For instance, when using the Tika library from a Java program, the `PDFParserConfig` class has a method `setSortByPosition(boolean)` that can extract vertically oriented text.+To access that method via configuration with the `ExtractingRequestHandler`, one can add the `parseContext.config` property to `solrconfig.xml` and then set properties in Tika's `PDFParserConfig` as in the example below.[source,xml]----@@ -351,7 +385,9 @@ Consult the Tika Java API documentation for configuration parameters that can beThe ExtractingRequestHandler will decrypt encrypted files and index their content if you supply a password in either `resource.password` on the request, or in a `passwordsFile` file.-In the case of `passwordsFile`, the file supplied must be formatted so there is one line per rule. Each rule contains a file name regular expression, followed by "=", then the password in clear-text. Because the passwords are in clear-text, the file should have strict access restrictions.+In the case of `passwordsFile`, the file supplied must be formatted so there is one line per rule.+Each rule contains a file name regular expression, followed by "=", then the password in clear-text.+Because the passwords are in clear-text, the file should have strict access restrictions.[source,plain]----@@ -367,20 +403,26 @@ For a multi-core configuration, you can specify `sharedLib='lib'` in the `<solr/=== Extending the ExtractingRequestHandler-If you want to supply your own `ContentHandler` for Solr to use, you can extend the `ExtractingRequestHandler` and override the `createFactory()` method. This factory is responsible for constructing the `SolrContentHandler` that interacts with Tika, and allows literals to override Tika-parsed values. Set the parameter `literalsOverride`, which normally defaults to `true`, to `false` to append Tika-parsed values to literal values.+If you want to supply your own `ContentHandler` for Solr to use, you can extend the `ExtractingRequestHandler` and override the `createFactory()` method.+This factory is responsible for constructing the `SolrContentHandler` that interacts with Tika, and allows literals to override Tika-parsed values.+Set the parameter `literalsOverride`, which normally defaults to `true`, to `false` to append Tika-parsed values to literal values.== Solr Cell Internals=== Metadata Created by Tika-As mentioned before, Tika produces metadata about the document. Metadata describes different aspects of a document, such as the author's name, the number of pages, the file size, and so on. The metadata produced depends on the type of document submitted. For instance, PDFs have different metadata than Word documents do.+As mentioned before, Tika produces metadata about the document.+Metadata describes different aspects of a document, such as the author's name, the number of pages, the file size, and so on.+The metadata produced depends on the type of document submitted.+For instance, PDFs have different metadata than Word documents do.=== Metadata Added by SolrIn addition to the metadata added by Tika's parsers, Solr adds the following metadata:`stream_name`::-The name of the Content Stream as uploaded to Solr. Depending on how the file is uploaded, this may or may not be set.+The name of the Content Stream as uploaded to Solr.+Depending on how the file is uploaded, this may or may not be set.`stream_source_info`::Any source info about the stream.@@ -398,7 +440,8 @@ set for these metadata elements on your content.Here is the order in which the Solr Cell framework processes its input:-. Tika generates fields or passes them in as literals specified by `literal.<fieldname>=<value>`. If `literalsOverride=false`, literals will be appended as multi-value to the Tika-generated field.+. Tika generates fields or passes them in as literals specified by `literal.<fieldname>=<value>`.+If `literalsOverride=false`, literals will be appended as multi-value to the Tika-generated field.. If `lowernames=true`, Tika maps fields to lowercase.. Tika applies the mapping rules specified by `fmap.__source__=__target__` parameters.. If `uprefix` is specified, any unknown field names are prefixed with that value, else if `defaultField` is specified, any unknown fields are copied to the default field.@@ -437,7 +480,8 @@ bin/post -c gettingstarted -params "literal.id=doc5&captureAttr=true&defaultFiel=== Extracting Data without Indexing-Solr allows you to extract data without indexing. You might want to do this if you're using Solr solely as an extraction server or if you're interested in testing Solr extraction.+Solr allows you to extract data without indexing.+You might want to do this if you're using Solr solely as an extraction server or if you're interested in testing Solr extraction.The example below sets the `extractOnly=true` parameter to extract data without indexing it.@@ -486,6 +530,9 @@ public class SolrCellRequestDemo {This operation streams the file `my-file.pdf` into the Solr index for `my_collection`.-The sample code above calls the extract command, but you can easily substitute other commands that are supported by Solr Cell. The key class to use is the `ContentStreamUpdateRequest`, which makes sure the ContentStreams are set properly. SolrJ takes care of the rest.+The sample code above calls the extract command, but you can easily substitute other commands that are supported by Solr Cell.+The key class to use is the `ContentStreamUpdateRequest`, which makes sure the ContentStreams are set properly.+SolrJ takes care of the rest.-Note that the `ContentStreamUpdateRequest` is not just specific to Solr Cell. You can send CSV to the CSV Update handler and to any other Request Handler that works with Content Streams for updates.+Note that the `ContentStreamUpdateRequest` is not just specific to Solr Cell.+You can send CSV to the CSV Update handler and to any other Request Handler that works with Content Streams for updates.diff --git a/solr/solr-ref-guide/src/indexing-with-update-handlers.adoc b/solr/solr-ref-guide/src/indexing-with-update-handlers.adocindex 73d88c88ca5..f5e85a67446 100644--- a/solr/solr-ref-guide/src/indexing-with-update-handlers.adoc+++ b/solr/solr-ref-guide/src/indexing-with-update-handlers.adoc@@ -21,7 +21,8 @@ Update handlers are request handlers designed to add, delete and update documentIn addition to having plugins for importing rich documents <<indexing-with-tika.adoc#,Indexing with Solr Cell and Apache Tika>>, Solr natively supports indexing structured documents in XML, CSV, and JSON.The recommended way to configure and use request handlers is with path based names that map to paths in the request URL.-However, request handlers can also be specified with the `qt` (query type) parameter if the <<requestdispatcher.adoc#,`requestDispatcher`>> is appropriately configured. It is possible to access the same handler using more than one name, which can be useful if you wish to specify different sets of default options.+However, request handlers can also be specified with the `qt` (query type) parameter if the <<requestdispatcher.adoc#,`requestDispatcher`>> is appropriately configured.+It is possible to access the same handler using more than one name, which can be useful if you wish to specify different sets of default options.A single unified update request handler supports XML, CSV, JSON, and javabin update requests, delegating to the appropriate `ContentStreamLoader` based on the `Content-Type` of the <<content-streams.adoc#,ContentStream>>.@@ -80,9 +81,11 @@ The add command supports some optional attributes which may be specified.Add the document within the specified number of milliseconds.`overwrite`::-Default is `true`. Indicates if the unique key constraints should be checked to overwrite previous versions of the same document (see below).+Default is `true`.+Indicates if the unique key constraints should be checked to overwrite previous versions of the same document (see below).-If the document schema defines a unique key, then by default an `/update` operation to add a document will overwrite (i.e., replace) any document in the index with the same unique key. If no unique key has been defined, indexing performance is somewhat faster, as no check has to be made for an existing documents to replace.+If the document schema defines a unique key, then by default an `/update` operation to add a document will overwrite (i.e., replace) any document in the index with the same unique key.+If no unique key has been defined, indexing performance is somewhat faster, as no check has to be made for an existing documents to replace.If you have a unique key field, but you feel confident that you can safely bypass the uniqueness check (e.g., you build your indexes in batch, and your indexing code guarantees it never adds the same document more than once) you can specify the `overwrite="false"` option when adding your documents.@@ -90,24 +93,34 @@ If you have a unique key field, but you feel confident that you can safely bypas==== Commit and Optimize During Updates-The `<commit>` operation writes all documents loaded since the last commit to one or more segment files on the disk. Before a commit has been issued, newly indexed content is not visible to searches. The commit operation opens a new searcher, and triggers any event listeners that have been configured.+The `<commit>` operation writes all documents loaded since the last commit to one or more segment files on the disk.+Before a commit has been issued, newly indexed content is not visible to searches.+The commit operation opens a new searcher, and triggers any event listeners that have been configured.Commits may be issued explicitly with a `<commit/>` message, and can also be triggered from `<autocommit>` parameters in `solrconfig.xml`.-The `<optimize>` operation requests Solr to merge internal data structures. For a large index, optimization will take some time to complete, but by merging many small segment files into larger segments, search performance may improve. If you are using Solr's replication mechanism to distribute searches across many systems, be aware that after an optimize, a complete index will need to be transferred.+The `<optimize>` operation requests Solr to merge internal data structures.+For a large index, optimization will take some time to complete, but by merging many small segment files into larger segments, search performance may improve.+If you are using Solr's replication mechanism to distribute searches across many systems, be aware that after an optimize, a complete index will need to be transferred.-WARNING: You should only consider using optimize on static indexes, i.e., indexes that can be optimized as part of the regular update process (say once-a-day updates). Applications requiring NRT functionality should not use optimize.+WARNING: You should only consider using optimize on static indexes, i.e., indexes that can be optimized as part of the regular update process (say once-a-day updates).+Applications requiring NRT functionality should not use optimize.The `<commit>` and `<optimize>` elements accept these optional attributes:`waitSearcher`::-Default is `true`. Blocks until a new searcher is opened and registered as the main query searcher, making the changes visible.+Default is `true`.+Blocks until a new searcher is opened and registered as the main query searcher, making the changes visible.-`expungeDeletes`:: (commit only) Default is `false`. Merges segments that have more than 10% deleted docs, expunging the deleted documents in the process. Resulting segments will respect `maxMergedSegmentMB`.+`expungeDeletes`:: (commit only) Default is `false`.+Merges segments that have more than 10% deleted docs, expunging the deleted documents in the process.+Resulting segments will respect `maxMergedSegmentMB`.+WARNING: expungeDeletes is "less expensive" than optimize, but the same warnings apply.-`maxSegments`:: (optimize only) Default is unlimited, resulting segments respect the `maxMergedSegmentMB` setting. Makes a best effort attempt to merge the segments down to no more than this number of segments but does not guarantee that the goal will be achieved. Unless there is tangible evidence that optimizing to a small number of segments is beneficial, this parameter should be omitted and the default behavior accepted.+`maxSegments`:: (optimize only) Default is unlimited, resulting segments respect the `maxMergedSegmentMB` setting.+Makes a best effort attempt to merge the segments down to no more than this number of segments but does not guarantee that the goal will be achieved.+Unless there is tangible evidence that optimizing to a small number of segments is beneficial, this parameter should be omitted and the default behavior accepted.Here are examples of `<commit>` and `<optimize>` using optional attributes:@@ -123,7 +136,8 @@ Here are examples of `<commit>` and `<optimize>` using optional attributes:Documents can be deleted from the index in two ways."Delete by ID" deletes the document with the specified ID, and can be used only if a UniqueID field has been defined in the schema.It doesn't work for child/nested docs.-"Delete by Query" deletes all documents matching a specified query, although `commitWithin` is ignored for a Delete by Query. A single delete message can contain multiple delete operations.+"Delete by Query" deletes all documents matching a specified query, although `commitWithin` is ignored for a Delete by Query.+A single delete message can contain multiple delete operations.[source,xml]----@@ -138,13 +152,16 @@ It doesn't work for child/nested docs.[IMPORTANT]====-When using the Join query parser in a Delete By Query, you should use the `score` parameter with a value of "none" to avoid a `ClassCastException`. See the section on the <<other-parsers.adoc#,Join Query Parser>> for more details on the `score` parameter.+When using the Join query parser in a Delete By Query, you should use the `score` parameter with a value of "none" to avoid a `ClassCastException`.+See the section on the <<other-parsers.adoc#,Join Query Parser>> for more details on the `score` parameter.======== Rollback Operations-The rollback command rolls back all add and deletes made to the index since the last commit. It neither calls any event listeners nor creates a new searcher. Its syntax is simple: `<rollback/>`.+The rollback command rolls back all add and deletes made to the index since the last commit.+It neither calls any event listeners nor creates a new searcher.+Its syntax is simple: `<rollback/>`.==== Grouping Operations@@ -168,7 +185,8 @@ You can post several commands in a single XML file by grouping them with the sur=== Using curl to Perform Updates-You can use the `curl` utility to perform any of the above commands, using its `--data-binary` option to append the XML message to the `curl` command, and generating a HTTP POST request. For example:+You can use the `curl` utility to perform any of the above commands, using its `--data-binary` option to append the XML message to the `curl` command, and generating a HTTP POST request.+For example:[source,bash]----@@ -201,7 +219,8 @@ This alternative `curl` command performs equivalent operations but with minimalcurl http://localhost:8983/solr/my_collection/update -H "Content-Type: text/xml" -T "myfile.xml" -X POST-----Short requests can also be sent using a HTTP GET command, if enabled in <<requestdispatcher.adoc#requestparsers-element,`requestParsers`>> element of `solrconfig.xml`, URL-encoding the request, as in the following. Note the escaping of "<" and ">":+Short requests can also be sent using a HTTP GET command, if enabled in <<requestdispatcher.adoc#requestparsers-element,`requestParsers`>> element of `solrconfig.xml`, URL-encoding the request, as in the following.+Note the escaping of "<" and ">":[source,bash]----@@ -230,7 +249,8 @@ Learn more about adding the `dist/solr-scripting-*.jar` file into Solr's <<libs.=== tr Parameter-The XSLT Update Request Handler accepts one parameter: the `tr` parameter, which identifies the XML transformation to use. The transformation must be found in the Solr `conf/xslt` directory.+The XSLT Update Request Handler accepts one parameter: the `tr` parameter, which identifies the XML transformation to use.+The transformation must be found in the Solr `conf/xslt` directory.=== XSLT Configuration@@ -249,7 +269,8 @@ The example below, from the `sample_techproducts_configs` <<config-sets.adoc#,co</requestHandler>-----A value of 5 for `xsltCacheLifetimeSeconds` is good for development, to see XSLT changes quickly. For production you probably want a much higher value.+A value of 5 for `xsltCacheLifetimeSeconds` is good for development, to see XSLT changes quickly.+For production you probably want a much higher value.=== XSLT Update Example@@ -295,7 +316,8 @@ Here is the `sample_techproducts_configs/conf/xslt/updateXml.xsl` XSL file for c</xsl:stylesheet>-----This stylesheet transforms Solr's XML search result format into Solr's Update XML syntax. One example usage would be to copy a Solr 1.3 index (which does not have CSV response writer) into a format which can be indexed into another Solr file (provided that all fields are stored):+This stylesheet transforms Solr's XML search result format into Solr's Update XML syntax.+One example usage would be to copy a Solr 1.3 index (which does not have CSV response writer) into a format which can be indexed into another Solr file (provided that all fields are stored):[source,bash]----@@ -308,7 +330,8 @@ NOTE: You can see the opposite export/import cycle using the `tr` parameter in== JSON Formatted Index Updates-Solr can accept JSON that conforms to a defined structure, or can accept arbitrary JSON-formatted documents. If sending arbitrarily formatted JSON, there are some additional parameters that need to be sent with the update request, described below in the section <<transforming-and-indexing-custom-json.adoc#,Transforming and Indexing Custom JSON>>.+Solr can accept JSON that conforms to a defined structure, or can accept arbitrary JSON-formatted documents.+If sending arbitrarily formatted JSON, there are some additional parameters that need to be sent with the update request, described below in the section <<transforming-and-indexing-custom-json.adoc#,Transforming and Indexing Custom JSON>>.=== Solr-Style JSON@@ -316,7 +339,8 @@ JSON formatted update requests may be sent to Solr's `/update` handler using `CoJSON formatted updates can take 3 basic forms, described in depth below:-* <<Adding a Single JSON Document,A single document to add>>, expressed as a top level JSON Object. To differentiate this from a set of commands, the `json.command=false` request parameter is required.+* <<Adding a Single JSON Document,A single document to add>>, expressed as a top level JSON Object.+To differentiate this from a set of commands, the `json.command=false` request parameter is required.* <<Adding Multiple JSON Documents,A list of documents to add>>, expressed as a top level JSON Array containing a JSON Object per document.* <<Sending JSON Update Commands,A sequence of update commands>>, expressed as a top level JSON Object (aka: Map).@@ -361,7 +385,8 @@ curl 'http://localhost:8983/solr/techproducts/update?commit=true' --data-binary==== Sending JSON Update Commands-In general, the JSON update syntax supports all of the update commands that the XML update handler supports, through a straightforward mapping. Multiple commands, adding and deleting documents, may be contained in one message:+In general, the JSON update syntax supports all of the update commands that the XML update handler supports, through a straightforward mapping.+Multiple commands, adding and deleting documents, may be contained in one message:[source,bash,subs="verbatim,callouts"]----@@ -400,7 +425,9 @@ curl -X POST -H 'Content-Type: application/json' 'http://localhost:8983/solr/my_As with other update handlers, parameters such as `commit`, `commitWithin`, `optimize`, and `overwrite` may be specified in the URL instead of in the body of the message.-The JSON update format allows for a simple delete-by-id. The value of a `delete` can be an array which contains a list of zero or more specific document id's (not a range) to be deleted. For example, a single document:+The JSON update format allows for a simple delete-by-id.+The value of a `delete` can be an array which contains a list of zero or more specific document id's (not a range) to be deleted.+For example, a single document:[source,json]----@@ -447,7 +474,8 @@ The `/update/json` path may be useful for clients sending in JSON formatted upda=== Custom JSON Documents-Solr can support custom JSON. This is covered in the section <<transforming-and-indexing-custom-json.adoc#,Transforming and Indexing Custom JSON>>.+Solr can support custom JSON.+This is covered in the section <<transforming-and-indexing-custom-json.adoc#,Transforming and Indexing Custom JSON>>.== CSV Formatted Index Updates@@ -473,75 +501,97 @@ Character used as field separator; default is ",". This parameter is global; forExample: `separator=%09``trim`::-If `true`, remove leading and trailing whitespace from values. The default is `false`. This parameter can be either global or per-field.+If `true`, remove leading and trailing whitespace from values.+The default is `false`.+This parameter can be either global or per-field.+Examples: `f.isbn.trim=true` or `trim=false``header`::-Set to `true` if first line of input contains field names. These will be used if the `fieldnames` parameter is absent. This parameter is global.+Set to `true` if first line of input contains field names.+These will be used if the `fieldnames` parameter is absent.+This parameter is global.`fieldnames`::-Comma-separated list of field names to use when adding documents. This parameter is global.+Comma-separated list of field names to use when adding documents.+This parameter is global.+Example: `fieldnames=isbn,price,title``literal._field_name_`::-A literal value for a specified field name. This parameter is global.+A literal value for a specified field name.+This parameter is global.+Example: `literal.color=red``skip`::-Comma separated list of field names to skip. This parameter is global.+Comma separated list of field names to skip.+This parameter is global.+Example: `skip=uninteresting,shoesize``skipLines`::-Number of lines to discard in the input stream before the CSV data starts, including the header, if present. Default=`0`. This parameter is global.+Number of lines to discard in the input stream before the CSV data starts, including the header, if present.+Default=`0`.+This parameter is global.+Example: `skipLines=5`-`encapsulator`:: The character optionally used to surround values to preserve characters such as the CSV separator or whitespace. This standard CSV format handles the encapsulator itself appearing in an encapsulated value by doubling the encapsulator.+`encapsulator`:: The character optionally used to surround values to preserve characters such as the CSV separator or whitespace.+This standard CSV format handles the encapsulator itself appearing in an encapsulated value by doubling the encapsulator.+This parameter is global; for per-field usage, see `split`.+Example: `encapsulator="`-`escape`:: The character used for escaping CSV separators or other reserved characters. If an escape is specified, the encapsulator is not used unless also explicitly specified since most formats use either encapsulation or escaping, not both. |g |+`escape`:: The character used for escaping CSV separators or other reserved characte+If an escape is specified, the encapsulator is not used unless also explicitly specified since most formats use either encapsulation or escaping, not both.+Example: `escape=\``keepEmpty`::-Keep and index zero length (empty) fields. The default is `false`. This parameter can be global or per-field.+Keep and index zero length (empty) fields.+The default is `false`.+This parameter can be global or per-field.+Example: `f.price.keepEmpty=true`-`map`:: Map one value to another. Format is value:replacement (which can be empty). This parameter can be global or per-field.+`map`:: Map one value to another.+Format is value:replacement (which can be empty).+This parameter can be global or per-field.+Example: `map=left:right` or `f.subject.map=history:bunk``split`::-If `true`, split a field into multiple values by a separate parser. This parameter is used on a per-field basis.+If `true`, split a field into multiple values by a separate parser.+This parameter is used on a per-field basis.`overwrite`::-If `true` (the default), check for and overwrite duplicate documents, based on the uniqueKey field declared in the Solr schema. If you know the documents you are indexing do not contain any duplicates then you may see a considerable speed up setting this to `false`.+If `true` (the default), check for and overwrite duplicate documents, based on the uniqueKey field declared in the Solr schema.+If you know the documents you are indexing do not contain any duplicates then you may see a considerable speed up setting this to `false`.+This parameter is global.`commit`::-Issues a commit after the data has been ingested. This parameter is global.+Issues a commit after the data has been ingested.+This parameter is global.`commitWithin`::-Add the document within the specified number of milliseconds. This parameter is global.+Add the document within the specified number of milliseconds.+This parameter is global.+Example: `commitWithin=10000``rowid`::-Map the `rowid` (line number) to a field specified by the value of the parameter, for instance if your CSV doesn't have a unique key and you want to use the row id as such. This parameter is global.+Map the `rowid` (line number) to a field specified by the value of the parameter, for instance if your CSV doesn't have a unique key and you want to use the row id as such.+This parameter is global.+Example: `rowid=id``rowidOffset`::-Add the given offset (as an integer) to the `rowid` before adding it to the document. Default is `0`. This parameter is global.+Add the given offset (as an integer) to the `rowid` before adding it to the document.+Default is `0`.+This parameter is global.+Example: `rowidOffset=10`diff --git a/solr/solr-ref-guide/src/indexupgrader-tool.adoc b/solr/solr-ref-guide/src/indexupgrader-tool.adocindex e74ac032b47..b9b8f61ff3f 100644--- a/solr/solr-ref-guide/src/indexupgrader-tool.adoc+++ b/solr/solr-ref-guide/src/indexupgrader-tool.adoc@@ -31,18 +31,23 @@ If you are currently using a release two or more major versions older, such as mThe IndexUpgraderTool performs a forceMerge (optimize) down to one segment, which may be undesirable.====-In a Solr distribution, the Lucene files are located in `./server/solr-webapp/webapp/WEB-INF/lib`. You will need to include the `lucene-core-<version>.jar` and `lucene-backwards-codecs-<version>.jar` on the classpath when running the tool.+In a Solr distribution, the Lucene files are located in `./server/solr-webapp/webapp/WEB-INF/lib`.+You will need to include the `lucene-core-<version>.jar` and `lucene-backwards-codecs-<version>.jar` on the classpath when running the tool.[source,bash,subs="attributes"]----java -cp lucene-core-{solr-docs-version}.0.jar:lucene-backward-codecs-{solr-docs-version}.0.jar org.apache.lucene.index.IndexUpgrader [-delete-prior-commits] [-verbose] /path/to/index-----This tool keeps only the last commit in an index. For this reason, if the incoming index has more than one commit, the tool refuses to run by default. Specify `-delete-prior-commits` to override this, allowing the tool to delete all but the last commit.+This tool keeps only the last commit in an index.+For this reason, if the incoming index has more than one commit, the tool refuses to run by default.+Specify `-delete-prior-commits` to override this, allowing the tool to delete all but the last commit.-Upgrading large indexes may take a long time. As a rule of thumb, the upgrade processes about 1 GB per minute.+Upgrading large indexes may take a long time.+As a rule of thumb, the upgrade processes about 1 GB per minute.[WARNING]====-This tool may reorder documents if the index was partially upgraded before execution (e.g., documents were added). If your application relies on monotonicity of document IDs (i.e., the order in which the documents were added to the index is preserved), do a full optimize instead.+This tool may reorder documents if the index was partially upgraded before execution (e.g., documents were added).+If your application relies on monotonicity of document IDs (i.e., the order in which the documents were added to the index is preserved), do a full optimize instead.====diff --git a/solr/solr-ref-guide/src/initparams.adoc b/solr/solr-ref-guide/src/initparams.adocindex fee57c5126f..d2143473279 100644--- a/solr/solr-ref-guide/src/initparams.adoc+++ b/solr/solr-ref-guide/src/initparams.adoc@@ -45,7 +45,8 @@ For example, here is one of the `<initParams>` sections defined by default in thThis sets the default search field ("df") to be "_text_" for all of the request handlers named in the path section.If we later want to change the `/query` request handler to search a different field by default, we could override the `<initParams>` by defining the parameter in the `<requestHandler>` section for `/query`.-The syntax and semantics are similar to that of a `<requestHandler>`. The following are the attributes:+The syntax and semantics are similar to that of a `<requestHandler>`.+The following are the attributes:`path`::A comma-separated list of paths which will use the parameters.diff --git a/solr/solr-ref-guide/src/installing-solr.adoc b/solr/solr-ref-guide/src/installing-solr.adocindex 09d56c22d84..b069502388f 100644--- a/solr/solr-ref-guide/src/installing-solr.adoc+++ b/solr/solr-ref-guide/src/installing-solr.adoc@@ -23,19 +23,23 @@ Please be sure to review the <<system-requirements.adoc#,System Requirements>> b== Available Solr Packages-Solr is available from the Solr website. Download the latest release https://solr.apache.org/downloads.html.+Solr is available from the Solr website.+Download the latest release https://solr.apache.org/downloads.html.There are three separate packages:* `solr-{solr-docs-version}.0.tgz` for Linux/Unix/OSX systems* `solr-{solr-docs-version}.0.zip` for Microsoft Windows systems-* `solr-{solr-docs-version}.0-src.tgz` the package Solr source code. This is useful if you want to develop on Solr without using the official Git repository.+* `solr-{solr-docs-version}.0-src.tgz` the package Solr source code.+This is useful if you want to develop on Solr without using the official Git repository.== Preparing for Installation-When getting started with Solr, all you need to do is extract the Solr distribution archive to a directory of your choosing. This will suffice as an initial development environment, but take care not to overtax this "toy" installation before setting up your true development and production environments.+When getting started with Solr, all you need to do is extract the Solr distribution archive to a directory of your choosing.+This will suffice as an initial development environment, but take care not to overtax this "toy" installation before setting up your true development and production environments.-When you've progressed past initial evaluation of Solr, you'll want to take care to plan your implementation. You may need to reinstall Solr on another server or make a clustered SolrCloud environment.+When you've progressed past initial evaluation of Solr, you'll want to take care to plan your implementation.+You may need to reinstall Solr on another server or make a clustered SolrCloud environment.When you're ready to setup Solr for a production environment, please refer to the instructions provided on the <<taking-solr-to-production.adoc#,Taking Solr to Production>> page.@@ -44,10 +48,13 @@ When you're ready to setup Solr for a production environment, please refer to th====How to size your Solr installation is a complex question that relies on a number of factors, including the number and structure of documents, how many fields you intend to store, the number of users, etc.-It's highly recommended that you spend a bit of time thinking about the factors that will impact hardware sizing for your Solr implementation. A very good blog post that discusses the issues to consider is https://lucidworks.com/2012/07/23/sizing-hardware-in-the-abstract-why-we-dont-have-a-definitive-answer/[Sizing Hardware in the Abstract: Why We Don't have a Definitive Answer].+It's highly recommended that you spend a bit of time thinking about the factors that will impact hardware sizing for your Solr implementation.+A very good blog post that discusses the issues to consider is https://lucidworks.com/2012/07/23/sizing-hardware-in-the-abstract-why-we-dont-have-a-definitive-answer/[Sizing Hardware in the Abstract: Why We Don't have a Definitive Answer].====-One thing to note when planning your installation is that a hard limit exists in Lucene for the number of documents in a single index: approximately 2.14 billion documents (2,147,483,647 to be exact). In practice, it is highly unlikely that such a large number of documents would fit and perform well in a single index, and you will likely need to distribute your index across a cluster before you ever approach this number. If you know you will exceed this number of documents in total before you've even started indexing, it's best to plan your installation with <<cluster-types.adoc#solrcloud-mode,SolrCloud>> as part of your design from the start.+One thing to note when planning your installation is that a hard limit exists in Lucene for the number of documents in a single index: approximately 2.14 billion documents (2,147,483,647 to be exact).+In practice, it is highly unlikely that such a large number of documents would fit and perform well in a single index, and you will likely need to distribute your index across a cluster before you ever approach this number.+If you know you will exceed this number of documents in total before you've even started indexing, it's best to plan your installation with <<cluster-types.adoc#solrcloud-mode,SolrCloud>> as part of your design from the start.== Package Installation@@ -68,15 +75,20 @@ After installing Solr, you'll see the following directories and files within thebin/::This directory includes several important scripts that will make using Solr easier.-solr and solr.cmd::: This is <<solr-control-script-reference.adoc#,Solr's Control Script>>, also known as `bin/solr` (*nix) / `bin/solr.cmd` (Windows). This script is the preferred tool to start and stop Solr. You can also create collections or cores, configure authentication, and work with configuration files when running in SolrCloud mode.+solr and solr.cmd::: This is <<solr-control-script-reference.adoc#,Solr's Control Script>>, also known as `bin/solr` (*nix) / `bin/solr.cmd` (Windows).+This script is the preferred tool to start and stop Solr.+You can also create collections or cores, configure authentication, and work with configuration files when running in SolrCloud mode.post::: The <<post-tool.adoc#,PostTool>>, which provides a simple command line interface for POSTing content to Solr.solr.in.sh and solr.in.cmd:::-These are property files for *nix and Windows systems, respectively. System-level properties for Java, Jetty, and Solr are configured here. Many of these settings can be overridden when using `bin/solr` / `bin/solr.cmd`, but this allows you to set all the properties in one place.+These are property files for *nix and Windows systems, respectively.+System-level properties for Java, Jetty, and Solr are configured here.+Many of these settings can be overridden when using `bin/solr` / `bin/solr.cmd`, but this allows you to set all the properties in one place.install_solr_services.sh:::-This script is used on *nix systems to install Solr as a service. It is described in more detail in the section <<taking-solr-to-production.adoc#,Taking Solr to Production>>.+This script is used on *nix systems to install Solr as a service.+It is described in more detail in the section <<taking-solr-to-production.adoc#,Taking Solr to Production>>.contrib/::Solr's `contrib` directory includes add-on plugins for specialized features of Solr.@@ -88,36 +100,44 @@ docs/::The `docs` directory includes a link to online Javadocs for Solr.example/::-The `example` directory includes several types of examples that demonstrate various Solr capabilities. See the section <<Solr Examples>> below for more details on what is in this directory.+The `example` directory includes several types of examples that demonstrate various Solr capabilities.+See the section <<Solr Examples>> below for more details on what is in this directory.licenses/::The `licenses` directory includes all of the licenses for 3rd party libraries used by Solr.server/::-This directory is where the heart of the Solr application resides. A README in this directory provides a detailed overview, but here are some highlights:+This directory is where the heart of the Solr application resides.+A README in this directory provides a detailed overview, but here are some highlights:* Solr's Admin UI (`server/solr-webapp`)* Jetty libraries (`server/lib`)-* Log files (`server/logs`) and log configurations (`server/resources`). See the section <<configuring-logging.adoc#,Configuring Logging>> for more details on how to customize Solr's default logging.+* Log files (`server/logs`) and log configurations (`server/resources`).+See the section <<configuring-logging.adoc#,Configuring Logging>> for more details on how to customize Solr's default logging.* Sample configsets (`server/solr/configsets`)== Solr Examples-Solr includes a number of example documents and configurations to use when getting started. If you ran through the <<solr-tutorial.adoc#,Solr Tutorial>>, you have already interacted with some of these files.+Solr includes a number of example documents and configurations to use when getting started.+If you ran through the <<solr-tutorial.adoc#,Solr Tutorial>>, you have already interacted with some of these files.Here are the examples included with Solr:exampledocs::-This is a small set of simple CSV, XML, and JSON files that can be used with `bin/post` when first getting started with Solr. For more information about using `bin/post` with these files, see <<post-tool.adoc#,Post Tool>>.+This is a small set of simple CSV, XML, and JSON files that can be used with `bin/post` when first getting started with Solr.+For more information about using `bin/post` with these files, see <<post-tool.adoc#,Post Tool>>.files::-The `files` directory provides a basic search UI for documents such as Word or PDF that you may have stored locally. See the README there for details on how to use this example.+The `files` directory provides a basic search UI for documents such as Word or PDF that you may have stored locally.+See the README there for details on how to use this example.films::-The `films` directory includes a robust set of data about movies in three formats: CSV, XML, and JSON. See the README there for details on how to use this dataset.+The `films` directory includes a robust set of data about movies in three formats: CSV, XML, and JSON.+See the README there for details on how to use this dataset.== Starting Solr-Solr includes a command line interface tool called `bin/solr` (Linux/MacOS) or `bin\solr.cmd` (Windows). This tool allows you to start and stop Solr, create cores and collections, configure authentication, and check the status of your system.+Solr includes a command line interface tool called `bin/solr` (Linux/MacOS) or `bin\solr.cmd` (Windows).+This tool allows you to start and stop Solr, create cores and collections, configure authentication, and check the status of your system.To use it to start Solr you can simply enter:@@ -141,17 +161,21 @@ TIP: All of the options for the Solr CLI are described in the section <<solr-con=== Start Solr with a Specific Bundled Example-Solr also provides a number of useful examples to help you learn about key features. You can launch the examples using the `-e` flag. For instance, to launch the "techproducts" example, you would do:+Solr also provides a number of useful examples to help you learn about key features.+You can launch the examples using the `-e` flag.+For instance, to launch the "techproducts" example, you would do:[source,bash]----bin/solr -e techproducts-----Currently, the available examples you can run are: techproducts, schemaless, and cloud. See the section <<solr-control-script-reference.adoc#running-with-example-configurations,Running with Example Configurations>> for details on each example.+Currently, the available examples you can run are: techproducts, schemaless, and cloud.+See the section <<solr-control-script-reference.adoc#running-with-example-configurations,Running with Example Configurations>> for details on each example..Getting Started with SolrCloud-NOTE: Running the `cloud` example starts Solr in <<cluster-types.adoc#solrcloud-mode,SolrCloud>> mode. For more information on starting Solr in SolrCloud mode, see the section <<tutorial-solrcloud.adoc#,Getting Started with SolrCloud>>.+NOTE: Running the `cloud` example starts Solr in <<cluster-types.adoc#solrcloud-mode,SolrCloud>> mode.+For more information on starting Solr in SolrCloud mode, see the section <<tutorial-solrcloud.adoc#,Getting Started with SolrCloud>>.=== Check if Solr is Running@@ -164,18 +188,21 @@ bin/solr statusThis will search for running Solr instances on your computer and then gather basic information about them, such as the version and memory usage.-That's it! Solr is running. If you need convincing, use a Web browser to see the Admin Console.+That's it! Solr is running.+If you need convincing, use a Web browser to see the Admin Console.`\http://localhost:8983/solr/`.The Solr Admin interface.image::images/installing-solr/SolrAdminDashboard.png[Solr's Admin UI,pdfwidth=75%]-If Solr is not running, your browser will complain that it cannot connect to the server. Check your port number and try again.+If Solr is not running, your browser will complain that it cannot connect to the server.+Check your port number and try again.=== Create a Core-If you did not start Solr with an example configuration, you would need to create a core in order to be able to index and search. You can do so by running:+If you did not start Solr with an example configuration, you would need to create a core in order to be able to index and search.+You can do so by running:[source,bash]----diff --git a/solr/solr-ref-guide/src/jmx-with-solr.adoc b/solr/solr-ref-guide/src/jmx-with-solr.adocindex 252659d7415..695948a2ba3 100644--- a/solr/solr-ref-guide/src/jmx-with-solr.adoc+++ b/solr/solr-ref-guide/src/jmx-with-solr.adoc@@ -28,20 +28,24 @@ If you are unfamiliar with JMX, you may find the following overview useful: httJMX support is configured by defining a metrics reporter, as described in the section the section <<metrics-reporting.adoc#jmx-reporter,JMX Reporter>>.-If you have an existing MBean server running in Solr's JVM, or if you start Solr with the system property `-Dcom.sun.management.jmxremote`, Solr will automatically identify its location on startup even if you have not defined a reporter explicitly in `solr.xml`. You can also define the location of the MBean server with parameters defined in the reporter definition.+If you have an existing MBean server running in Solr's JVM, or if you start Solr with the system property `-Dcom.sun.management.jmxremote`, Solr will automatically identify its location on startup even if you have not defined a reporter explicitly in `solr.xml`.+You can also define the location of the MBean server with parameters defined in the reporter definition.== Configuring MBean Servers-Versions of Solr prior to 7.0 defined JMX support in `solrconfig.xml`. This has been changed to the metrics reporter configuration defined above.+Versions of Solr prior to 7.0 defined JMX support in `solrconfig.xml`.+This has been changed to the metrics reporter configuration defined above.Parameters for the reporter configuration allow defining the location or address of an existing MBean server.-An MBean server can be started at the time of Solr's startup by passing the system parameter `-Dcom.sun.management.jmxremote`. See Oracle's documentation for additional settings available to start and control an MBean server at http://docs.oracle.com/javase/8/docs/technotes/guides/management/agent.html.+An MBean server can be started at the time of Solr's startup by passing the system parameter `-Dcom.sun.management.jmxremote`.+See Oracle's documentation for additional settings available to start and control an MBean server at http://docs.oracle.com/javase/8/docs/technotes/guides/management/agent.html.=== Configuring a Remote Connection to Solr JMXIf you need to attach a JMX-enabled Java profiling tool, such as JConsole or VisualVM, to a remote Solr server, then you need to enable remote JMX access when starting the Solr server.Simply change the `ENABLE_REMOTE_JMX_OPTS` property in the `solr.in.sh` or `solr.in.cmd` (for Windows) file to `true`.-You’ll also need to choose a port for the JMX RMI connector to bind to, such as 18983. For example, if your Solr include script sets:+You’ll also need to choose a port for the JMX RMI connector to bind to, such as 18983.+For example, if your Solr include script sets:[source,bash]----@@ -49,7 +53,8 @@ ENABLE_REMOTE_JMX_OPTS=trueRMI_PORT=18983-----The JMX RMI connector will allow Java profiling tools to attach to port 18983. When enabled, the following properties are passed to the JVM when starting Solr:+The JMX RMI connector will allow Java profiling tools to attach to port 18983.+When enabled, the following properties are passed to the JVM when starting Solr:[source,plain]----diff --git a/solr/solr-ref-guide/src/join-query-parser.adoc b/solr/solr-ref-guide/src/join-query-parser.adocindex b1536ca297b..83d94c29948 100644--- a/solr/solr-ref-guide/src/join-query-parser.adoc+++ b/solr/solr-ref-guide/src/join-query-parser.adoc@@ -97,7 +97,8 @@ This method must be used if score information is required, and should also be co.dvWithScore and single value numerics[WARNING]====-The `dvWithScore` method doesn't support single value numeric fields. Users migrating from versions prior to 7.0 are encouraged to change field types to string and rebuild indexes during migration.+The `dvWithScore` method doesn't support single value numeric fields.+Users migrating from versions prior to 7.0 are encouraged to change field types to string and rebuild indexes during migration.====+`topLevelDV` can only be used when `to` and `from` fields have docValues data, and does not currently support numeric fields.@@ -109,9 +110,11 @@ Consider this method when the "from" query matches a large number of documents a== Joining Across Single Shard Collections-You can also specify a `fromIndex` parameter to join with a field from another core or a single shard collection. If running in SolrCloud mode, then the collection specified in the `fromIndex` parameter must have a single shard and a replica on all Solr nodes where the collection you're joining to has a replica.+You can also specify a `fromIndex` parameter to join with a field from another core or a single shard collection.+If running in SolrCloud mode, then the collection specified in the `fromIndex` parameter must have a single shard and a replica on all Solr nodes where the collection you're joining to has a replica.-Let's consider an example where you want to use a Solr join query to filter movies by directors that have won an Oscar. Specifically, imagine we have two collections with the following fields:+Let's consider an example where you want to use a Solr join query to filter movies by directors that have won an Oscar.+Specifically, imagine we have two collections with the following fields:*movies*: id, title, director_id, ...@@ -124,9 +127,12 @@ To filter movies by directors that have won an Oscar using a Solr join on the *mfq={!join from=id fromIndex=movie_directors to=director_id}has_oscar:true-----Notice that the query criteria of the filter (`has_oscar:true`) is based on a field in the collection specified using `fromIndex`. Keep in mind that you cannot return fields from the `fromIndex` collection using join queries, you can only use the fields for filtering results in the "to" collection (movies).+Notice that the query criteria of the filter (`has_oscar:true`) is based on a field in the collection specified using `fromIndex`.+Keep in mind that you cannot return fields from the `fromIndex` collection using join queries, you can only use the fields for filtering results in the "to" collection (movies).-Next, let's understand how these collections need to be deployed in your cluster. Imagine the *movies* collection is deployed to a four node SolrCloud cluster and has two shards with a replication factor of two. Specifically, the *movies* collection has replicas on the following four nodes:+Next, let's understand how these collections need to be deployed in your cluster.+Imagine the *movies* collection is deployed to a four node SolrCloud cluster and has two shards with a replication factor of two.+Specifically, the *movies* collection has replicas on the following four nodes:node 1: movies_shard1_replica1@@ -136,7 +142,8 @@ node 3: movies_shard2_replica1node 4: movies_shard2_replica2-To use the *movie_directors* collection in Solr join queries with the *movies* collection, it needs to have a replica on each of the four nodes. In other words, *movie_directors* must have one shard and replication factor of four:+To use the *movie_directors* collection in Solr join queries with the *movies* collection, it needs to have a replica on each of the four nodes.+In other words, *movie_directors* must have one shard and replication factor of four:node 1: movie_directors_shard1_replica1@@ -146,7 +153,10 @@ node 3: movie_directors_shard1_replica3node 4: movie_directors_shard1_replica4-At query time, the `JoinQParser` will access the local replica of the *movie_directors* collection to perform the join. If a local replica is not available or active, then the query will fail. At this point, it should be clear that since you're limited to a single shard and the data must be replicated across all nodes where it is needed, this approach works better with smaller data sets where there is a one-to-many relationship between the from collection and the to collection. Moreover, if you add a replica to the to collection, then you also need to add a replica for the from collection.+At query time, the `JoinQParser` will access the local replica of the *movie_directors* collection to perform the join.+If a local replica is not available or active, then the query will fail.+At this point, it should be clear that since you're limited to a single shard and the data must be replicated across all nodes where it is needed, this approach works better with smaller data sets where there is a one-to-many relationship between the from collection and the to collection.+Moreover, if you add a replica to the to collection, then you also need to add a replica for the from collection.For more information, Erick Erickson has written a blog post about join performance titled https://lucidworks.com/2012/06/20/solr-and-joins/[Solr and Joins].@@ -180,10 +190,13 @@ The remote Solr collection does not have any specific sharding requirements.The cross collection join has some configuration options that can be specified in `solrconfig.xml`.`routerField`::-If the documents are routed to shards using the CompositeID router by the join field, then that field name should be specified in the configuration here. This will allow the parser to optimize the resulting HashRange query.+If the documents are routed to shards using the CompositeID router by the join field, then that field name should be specified in the configuration here.+ This will allow the parser to optimize the resulting HashRange query.`solrUrl`::-If specified, this array of strings specifies the white listed Solr URLs that you can pass to the solrUrl query parameter. Without this configuration the solrUrl parameter cannot be used. This restriction is necessary to prevent an attacker from using Solr to explore the network.+If specified, this array of strings specifies the white listed Solr URLs that you can pass to the solrUrl query parameter.+Without this configuration the solrUrl parameter cannot be used.+This restriction is necessary to prevent an attacker from using Solr to explore the network.[source,xml]----@@ -201,10 +214,15 @@ If specified, this array of strings specifies the white listed Solr URLs that yoThe name of the external Solr collection to be queried to retrieve the set of join key values (required).`zkHost`::-The connection string to be used to connect to ZooKeeper. `zkHost` and `solrUrl` are both optional parameters, and at most one of them should be specified. If neither `zkHost` nor `solrUrl` are specified, the local ZooKeeper cluster will be used. (optional).+The connection string to be used to connect to ZooKeeper.+`zkHost` and `solrUrl` are both optional parameters, and at most one of them should be specified.+If neither `zkHost` nor `solrUrl` are specified, the local ZooKeeper cluster will be used.+(optional).`solrUrl`::-The URL of the external Solr node to be queried. Must be a character for character exact match of a whitelisted url. (optional, disabled by default for security).+The URL of the external Solr node to be queried.+Must be a character for character exact match of a whitelisted url.+(optional, disabled by default for security).`from`::The join key field name in the external collection (required).@@ -213,7 +231,8 @@ The join key field name in the external collection (required).The join key field name in the local collection.`v`::-The query substituted in as a local param. This is the query string that will match documents in the remote collection.+The query substituted in as a local param.+This is the query string that will match documents in the remote collection.`routed`::If `true`, the cross collection join query will use each shard's hash range to determine the set of join keys to retrieve for that shard.diff --git a/solr/solr-ref-guide/src/json-facet-api.adoc b/solr/solr-ref-guide/src/json-facet-api.adocindex 1154502c9fe..bf5b63a6b84 100644--- a/solr/solr-ref-guide/src/json-facet-api.adoc+++ b/solr/solr-ref-guide/src/json-facet-api.adoc@@ -21,7 +21,8 @@== Facet & Analytics Module-The JSON Faceting module exposes similar functionality to Solr's traditional faceting module but with a stronger emphasis on usability. It has several benefits over traditional faceting:+The JSON Faceting module exposes similar functionality to Solr's traditional faceting module but with a stronger emphasis on usability.+It has several benefits over traditional faceting:* easier programmatic construction of complex or nested facets* the nesting and structure offered by JSON makes facets easier to read and understand than the flat namespace of the traditional faceting API.@@ -73,7 +74,8 @@ include::{example-source-dir}JsonRequestApiTest.java[tag=solrj-json-simple-terms====---The response below shows us that 32 documents match the default root domain. and 12 documents have `cat:electronics`, 4 documents have `cat:currency`, etc.+The response below shows us that 32 documents match the default root domain.+Twelve documents have `cat:electronics`, 4 documents have `cat:currency`, etc.[source,java]----@@ -97,7 +99,9 @@ The response below shows us that 32 documents match the default root domain. and=== Stat Facet Example-Stat (also called `aggregation` or `analytic`) facets are useful for displaying information derived from query results, in addition to those results themselves. For example, stat facets can be used to provide context to users on an e-commerce site looking for memory. The example below computes the average price (and other statistics) and would allow a user to gauge whether the memory stick in their cart is a good price.+Stat (also called `aggregation` or `analytic`) facets are useful for displaying information derived from query results, in addition to those results themselves.+For example, stat facets can be used to provide context to users on an e-commerce site looking for memory.+The example below computes the average price (and other statistics) and would allow a user to gauge whether the memory stick in their cart is a good price.[.dynamic-tabs]--@@ -372,7 +376,8 @@ The output from the range facet above would look a bit like:==== Range Facet Parameters-Range facet parameter names and semantics largely mirror facet.range query-parameter style faceting. For example "start" here corresponds to "facet.range.start" in a facet.range command.+Range facet parameter names and semantics largely mirror facet.range query-parameter style faceting.+For example "start" here corresponds to "facet.range.start" in a facet.range command.[width="100%",cols="10%,90%",options="header",]|===@@ -411,7 +416,8 @@ Refer <<Arbitrary Range>>==== Arbitrary Range-An arbitrary range consists of from and to values over which range bucket is computed. This range can be specified in two syntax.+An arbitrary range consists of from and to values over which range bucket is computed.+This range can be specified in two syntax.[width="100%",cols="10%,90%",options="header",]|===@@ -442,7 +448,8 @@ For example, For range `(5,10]` 5 is excluded and 10 is included===== include with ranges-`include` parameter is ignored when `ranges` is specified but there are ways to achieve same behavior with `ranges`. `lower`, `upper`, `outer`, `edge` all can be achieved using combination of `inclusive_to` and `inclusive_from`.+`include` parameter is ignored when `ranges` is specified but there are ways to achieve same behavior with `ranges`.+`lower`, `upper`, `outer`, `edge` all can be achieved using combination of `inclusive_to` and `inclusive_from`.Range facet with `ranges`@@ -491,7 +498,9 @@ The output from the range facet above would look a bit like:}-----NOTE: When `range` is specified, its value in the request is used as key in the response. In the other case, key is generated using `from`, `to`, `inclusive_to` and `inclusive_from`. Currently, custom `key` is not supported.+NOTE: When `range` is specified, its value in the request is used as key in the response.+In the other case, key is generated using `from`, `to`, `inclusive_to` and `inclusive_from`.+Currently, custom `key` is not supported.=== Heatmap Facet@@ -559,7 +568,8 @@ And the facet response will look like:=== Stat Facet Functions-Unlike all the facets discussed so far, Aggregation functions (also called *facet functions*, *analytic functions*, or *metrics*) do not partition data into buckets. Instead, they calculate something over all the documents in the domain.+Unlike all the facets discussed so far, Aggregation functions (also called *facet functions*, *analytic functions*, or *metrics*) do not partition data into buckets.+Instead, they calculate something over all the documents in the domain.[width="100%",cols="10%,30%,60%",options="header",]|===@@ -578,7 +588,7 @@ Unlike all the facets discussed so far, Aggregation functions (also called *face|sumsq |`sumsq(rent)` |sum of squares of field or function|variance |`variance(rent)` |variance of numeric field or function|stddev |`stddev(rent)` |standard deviation of field or function-|relatedness |`relatedness('popularity:[100 TO *]','inStock:true')`|A function for computing a relatedness score of the documents in the domain to a Foreground set, relative to a Background set (both defined as queries). This is primarily for use when building <<json-facet-api.adoc#relatedness-and-semantic-knowledge-graphs,Semantic Knowledge Graphs>>.+|relatedness |`relatedness('popularity:[100 TO *]','inStock:true')`|A function for computing a relatedness score of the documents in the domain to a Foreground set, relative to a Background set (both defined as queries). This is primarily for use when building <<json-facet-api.adoc#relatedness-and-semantic-knowledge-graphs,Semantic Knowledge Graphs>>.|===Numeric aggregation functions such as `avg` can be on any numeric field, or on a <<function-queries.adoc#,nested function>> of multiple numeric fields such as `avg(div(popularity,price))`.@@ -617,7 +627,8 @@ include::{example-source-dir}JsonRequestApiTest.java[tag=solrj-json-metrics-face====---An expanded form allows for <<local-params.adoc#,local params>> to be specified. These may be used explicitly by some specialized aggregations such as `<<json-facet-api.adoc#relatedness-options,relatedness()>>`, but can also be used as parameter references to make aggregation expressions more readable, with out needing to use (global) request parameters:+An expanded form allows for <<local-params.adoc#,local params>> to be specified.+These may be used explicitly by some specialized aggregations such as `<<json-facet-api.adoc#relatedness-options,relatedness()>>`, but can also be used as parameter references to make aggregation expressions more readable, with out needing to use (global) request parameters:[.dynamic-tabs]--@@ -659,9 +670,11 @@ include::{example-source-dir}JsonRequestApiTest.java[tag=solrj-json-metrics-face== Nested Facets-Nested facets, or **sub-facets**, allow one to nest facet commands under any facet command that partitions the domain into buckets (i.e., `terms`, `range`, `query`). These sub-facets are then evaluated against the domains defined by the set of all documents in each bucket of their parent.+Nested facets, or **sub-facets**, allow one to nest facet commands under any facet command that partitions the domain into buckets (i.e., `terms`, `range`, `query`).+These sub-facets are then evaluated against the domains defined by the set of all documents in each bucket of their parent.-The syntax is identical to top-level facets - just add a `facet` command to the facet command block of the parent facet. Technically, every facet command is actually a sub-facet since we start off with a single facet bucket with a domain defined by the main query and filters.+The syntax is identical to top-level facets - just add a `facet` command to the facet command block of the parent facet.+Technically, every facet command is actually a sub-facet since we start off with a single facet bucket with a domain defined by the main query and filters.=== Nested Facet Example@@ -699,7 +712,9 @@ include::{example-source-dir}JsonRequestApiTest.java[tag=solrj-json-simple-terms====---The response for the facet above will show the top category and the number of documents that falls into each category bucket. Nested facets can be used to gather additional information about each bucket of documents. For example, using the nested facet below, we can find the top categories as well as who the leading manufacturer is in each category:+The response for the facet above will show the top category and the number of documents that falls into each category bucket.+Nested facets can be used to gather additional information about each bucket of documents.+For example, using the nested facet below, we can find the top categories as well as who the leading manufacturer is in each category:[.dynamic-tabs]--@@ -766,7 +781,8 @@ And the response will look something like:=== Sorting Facets By Nested Functions-The default sort for a field or terms facet is by bucket count descending. We can optionally `sort` ascending or descending by any facet function that appears in each bucket.+The default sort for a field or terms facet is by bucket count descending.+We can optionally `sort` ascending or descending by any facet function that appears in each bucket.[.dynamic-tabs]@@ -805,7 +821,9 @@ include::{example-source-dir}JsonRequestApiTest.java[tag=solrj-json-nested-cat-f====---In some situations the desired `sort` may be an aggregation function that is very costly to compute for every bucket. A `prelim_sort` option can be used to specify an approximation of the `sort`, for initially ranking the buckets to determine the top candidates (based on the `limit` and `overrequest`). Only after the top candidate buckets have been refined, will the actual `sort` be used.+In some situations the desired `sort` may be an aggregation function that is very costly to compute for every bucket.+A `prelim_sort` option can be used to specify an approximation of the `sort`, for initially ranking the buckets to determine the top candidates (based on the `limit` and `overrequest`).+Only after the top candidate buckets have been refined, will the actual `sort` be used.[source,java]----@@ -834,11 +852,15 @@ As discussed above, facets compute buckets or statistics based on their "domain"* By default, top-level facets use the set of all documents matching the main query as their domain.* Nested "sub-facets" are computed for every bucket of their parent facet, using a domain containing all documents in that bucket.-In addition to this default behavior, domains can be also be widened, narrowed, or changed entirely. The JSON Faceting API supports modifying domains through its `domain` property. This is discussed in more detail <<json-faceting-domain-changes.adoc#,here>>+In addition to this default behavior, domains can be also be widened, narrowed, or changed entirely.+The JSON Faceting API supports modifying domains through its `domain` property.+This is discussed in more detail <<json-faceting-domain-changes.adoc#,here>>== Special Stat Facet Functions-Most stat facet functions (`avg`, `sumsq`, etc.) allow users to perform math computations on groups of documents. A few functions are more involved though, and deserve an explanation of their own. These are described in more detail in the sections below.+Most stat facet functions (`avg`, `sumsq`, etc.) allow users to perform math computations on groups of documents.+A few functions are more involved though, and deserve an explanation of their own.+These are described in more detail in the sections below.=== uniqueBlock() and Block Join Counts@@ -903,25 +925,34 @@ The `relatedness(...)` stat function allows for sets of documents to be scored r[quote, Grainger et al., 'https://arxiv.org/abs/1609.00464[The Semantic Knowledge Graph]']____-At its heart, the Semantic Knowledge Graph leverages an inverted index, along with a complementary uninverted index, to represent nodes (terms) and edges (the documents within intersecting postings lists for multiple terms/nodes). This provides a layer of indirection between each pair of nodes and their corresponding edge, enabling edges to materialize dynamically from underlying corpus statistics. As a result, any combination of nodes can have edges to any other nodes materialize and be scored to reveal latent relationships between the nodes.+At its heart, the Semantic Knowledge Graph leverages an inverted index, along with a complementary uninverted index, to represent nodes (terms) and edges (the documents within intersecting postings lists for multiple terms/nodes).+This provides a layer of indirection between each pair of nodes and their corresponding edge, enabling edges to materialize dynamically from underlying corpus statistics.+As a result, any combination of nodes can have edges to any other nodes materialize and be scored to reveal latent relationships between the nodes.____The `relatedness(...)` function is used to "score" these relationships, relative to "Foreground" and "Background" sets of documents, specified in the function params as queries.Unlike most aggregation functions, the `relatedness(...)` function is aware of whether and how it's used in <<nested-facets,Nested Facets>>. It evaluates the query defining the current bucket _independently_ from its parent/ancestor buckets, and intersects those documents with a "Foreground Set" defined by the foreground query _combined with the ancestor buckets_. The result is then compared to a similar intersection done against the "Background Set" (defined exclusively by background query) to see if there is a positive, or negative, correlation between the current bucket and the Foreground Set, relative to the Background Set.-NOTE: The semantics of `relatedness(...)` in an `allBuckets` context is currently undefined. Accordingly, although the `relatedness(...)` stat may be specified for a facet request that also specifies `allBuckets:true`, the `allBuckets` bucket itself will not include a relatedness calculation.+NOTE: The semantics of `relatedness(...)` in an `allBuckets` context is currently undefined.+Accordingly, although the `relatedness(...)` stat may be specified for a facet request that also specifies `allBuckets:true`, the `allBuckets` bucket itself will not include a relatedness calculation.-NOTE: While it's very common to define the Background Set as `\*:*`, or some other super-set of the Foreground Query, it is not strictly required. The `relatedness(...)` function can be used to compare the statistical relatedness of sets of documents to orthogonal foreground/background queries.+NOTE: While it's very common to define the Background Set as `\*:*`, or some other super-set of the Foreground Query, it is not strictly required.+The `relatedness(...)` function can be used to compare the statistical relatedness of sets of documents to orthogonal foreground/background queries.[[relatedness-options]]==== relatedness() OptionsWhen using the extended `type:func` syntax for specifying a `relatedness()` aggregation, an optional `min_popularity` (float) option can be used to specify a lower bound on the `foreground_popularity` and `background_popularity` values, that must be met in order for the `relatedness` score to be valid -- If this `min_popularity` is not met, then the `relatedness` score will be `-Infinity`.-The default implementation for calculating `relatedness()` domain correlation depends on the type of facet being calculated. Generic domain correlation is calculated per-term, by selectively retrieving a DocSet for each bucket-associated query (consulting the `filterCache`) and calculating DocSet intersections with "foreground" and "background" sets. For term facets (especially over high-cardinality fields) this approach can lead to `filterCache` thrashing; accordingly, `relatedness()` over term facets defaults where possible to an approach that collects facet counts directly over all multiple domains in a single sweep (never touching the `filterCache`). It is possible to explicitly control this "single sweep" collection by setting the extended `type:func` syntax `sweep_collection` option to `true` (the default) or `false` (to disable sweep collection).+The default implementation for calculating `relatedness()` domain correlation depends on the type of facet being calculated.+Generic domain correlation is calculated per-term, by selectively retrieving a DocSet for each bucket-associated query (consulting the `filterCache`) and calculating DocSet intersections with "foreground" and "background" sets.+For term facets (especially over high-cardinality fields) this approach can lead to `filterCache` thrashing; accordingly, `relatedness()` over term facets defaults where possible to an approach that collects facet counts directly over all multiple domains in a single sweep (never touching the `filterCache`).+It is possible to explicitly control this "single sweep" collection by setting the extended `type:func` syntax `sweep_collection` option to `true` (the default) or `false` (to disable sweep collection).-NOTE: Disabling sweep collection for `relatedness()` stats over low-cardinality fields may yield a performance benefit, provided the `filterCache` is sufficiently large to accommodate an entry for each value in the associated field without inducing thrashing for anticipated use patterns. A reasonable heuristic is that fields of cardinality less than 1,000 _may_ benefit from disabling sweep. This heuristic is _not_ used to determine default behavior, particularly because non-sweep collection can so easily induce `filterCache` thrashing, with system-wide detrimental effects.+NOTE: Disabling sweep collection for `relatedness()` stats over low-cardinality fields may yield a performance benefit, provided the `filterCache` is sufficiently large to accommodate an entry for each value in the associated field without inducing thrashing for anticipated use patterns.+A reasonable heuristic is that fields of cardinality less than 1,000 _may_ benefit from disabling sweep.+This heuristic is _not_ used to determine default behavior, particularly because non-sweep collection can so easily induce `filterCache` thrashing, with system-wide detrimental effects.[source,json]----@@ -935,7 +966,8 @@ This can be particularly useful when using a descending sorting on `relatedness([TIP]====-When sorting on `relatedness(...)` requests can be processed much more quickly by adding a `prelim_sort: "count desc"` option. Increasing the `overrequest` can help improve the accuracy of the top buckets.+When sorting on `relatedness(...)` requests can be processed much more quickly by adding a `prelim_sort: "count desc"` option.+Increasing the `overrequest` can help improve the accuracy of the top buckets.======== Semantic Knowledge Graph Examplediff --git a/solr/solr-ref-guide/src/json-faceting-domain-changes.adoc b/solr/solr-ref-guide/src/json-faceting-domain-changes.adocindex 8c9fd2763d6..d50bda8c648 100644--- a/solr/solr-ref-guide/src/json-faceting-domain-changes.adoc+++ b/solr/solr-ref-guide/src/json-faceting-domain-changes.adoc@@ -18,7 +18,9 @@// specific language governing permissions and limitations// under the License.-Facet computation operates on a "domain" of documents. By default, this domain consists of the documents matched by the main query. For sub-facets, the domain consists of all documents placed in their bucket by the parent facet.+Facet computation operates on a "domain" of documents.+By default, this domain consists of the documents matched by the main query.+For sub-facets, the domain consists of all documents placed in their bucket by the parent facet.Users can also override the "domain" of a facet that partitions data, using an explicit `domain` attribute whose value is a JSON object that can support various options for restricting, expanding, or completely changing the original domain before the buckets are computed for the associated facet.@@ -32,7 +34,8 @@ A `\*:*` query facet with a `domain` change can be used to group multiple sub-fa== Adding Domain Filters-The simplest example of a domain change is to specify an additional filter which will be applied to the existing domain. This can be done via the `filter` keyword in the `domain` block of the facet.+The simplest example of a domain change is to specify an additional filter which will be applied to the existing domain.+This can be done via the `filter` keyword in the `domain` block of the facet.[.dynamic-tabs]--@@ -69,10 +72,12 @@ include::{example-source-dir}JsonRequestApiTest.java[tag=solrj-json-facet-filter====---The value of `filter` can be a single query to treat as a filter, or a JSON list of filter queries. Each query can be:+The value of `filter` can be a single query to treat as a filter, or a JSON list of filter queries.+Each query can be:* a string containing a query in Solr query syntax.-* a reference to a request parameter containing Solr query syntax, of the form: `{param: <request_param_name>}`. It's possible to refer to one or multiple queries in DSL syntax defined under <<json-query-dsl.adoc#additional-queries,queries>> key in JSON Request API.+* a reference to a request parameter containing Solr query syntax, of the form: `{param: <request_param_name>}`.+It's possible to refer to one or multiple queries in DSL syntax defined under <<json-query-dsl.adoc#additional-queries,queries>> key in JSON Request API.The referred parameter might have 0 (absent) or many values.** When no values are specified, no filter is applied and no error is thrown.** When many values are specified, each value is parsed and used as filters in conjunction.@@ -120,7 +125,8 @@ When a `filter` option is combined with other `domain` changing options, the filDomains can also be expanded by using the `excludeTags` keyword to discard or ignore particular tagged query filters.-This is used in the example below to show the top two manufacturers matching a search. The search results match the filter `manu_id_s:apple`, but the computed facet discards this filter and operates a domain widened by discarding the `manu_id_s` filter.+This is used in the example below to show the top two manufacturers matching a search.+The search results match the filter `manu_id_s:apple`, but the computed facet discards this filter and operates a domain widened by discarding the `manu_id_s` filter.[.dynamic-tabs]--@@ -165,7 +171,9 @@ See also the section on <<faceting.adoc#tagging-and-excluding-filters,multi-sele== Arbitrary Domain Query-A `query` domain can be specified when you wish to compute a facet against an arbitrary set of documents, regardless of the original domain. The most common use case would be to compute a top level facet against a specific subset of the collection, regardless of the main query. But it can also be useful on nested facets when building <<json-facet-api.adoc#relatedness-and-semantic-knowledge-graphs,Semantic Knowledge Graphs>>.+A `query` domain can be specified when you wish to compute a facet against an arbitrary set of documents, regardless of the original domain.+The most common use case would be to compute a top level facet against a specific subset of the collection, regardless of the main query.+But it can also be useful on nested facets when building <<json-facet-api.adoc#relatedness-and-semantic-knowledge-graphs,Semantic Knowledge Graphs>>.Example:@@ -202,7 +210,8 @@ include::{example-source-dir}JsonRequestApiTest.java[tag=solrj-json-facet-query-====---The value of `query` can be a single query, or a JSON list of queries. Each query can be:+The value of `query` can be a single query, or a JSON list of queries.+Each query can be:* a string containing a query in Solr query syntax.* a reference to a request parameter containing Solr query syntax, of the form: `{param: <request_param_name>}`.@@ -216,7 +225,10 @@ NOTE: While a `query` domain can be combined with an additional domain `filter`,When a collection contains <<indexing-nested-documents.adoc#, Nested Documents>>, the `blockChildren` or `blockParent` domain options can be used to transform an existing domain containing one type of document, into a domain containing the documents with the specified relationship (child or parent of) to the documents from the original domain.-Both of these options work similarly to the corresponding <<block-join-query-parser.adoc#,Block Join Query Parsers>> by taking in a single String query that exclusively matches all parent documents in the collection. If `blockParent` is used, then the resulting domain will contain all parent documents of the children from the original domain. If `blockChildren` is used, then the resulting domain will contain all child documents of the parents from the original domain. Quite often facets over child documents needs to be counted in parent documents, this can be done by `uniqueBlock(\_root_)` as described in <<json-facet-api#uniqueblock-and-block-join-counts, Block Join Facet Counts>>.+Both of these options work similarly to the corresponding <<block-join-query-parser.adoc#,Block Join Query Parsers>> by taking in a single String query that exclusively matches all parent documents in the collection.+If `blockParent` is used, then the resulting domain will contain all parent documents of the children from the original domain.+If `blockChildren` is used, then the resulting domain will contain all child documents of the parents from the original domain.+Quite often facets over child documents needs to be counted in parent documents, this can be done by `uniqueBlock(\_root_)` as described in <<json-facet-api#uniqueblock-and-block-join-counts, Block Join Facet Counts>>.[source,json,subs="verbatim,callouts"]]----@@ -235,7 +247,8 @@ Both of these options work similarly to the corresponding <<block-join-query-par----<1> This example assumes we parent documents corresponding to Products, with child documents corresponding to individual SKUs with unique colors, and that our original query was against SKU documents.<2> The `colors` facet will be computed against all of the original SKU documents matching our search.-<3> For each bucket in the `colors` facet, the set of all matching SKU documents will be transformed into the set of corresponding parent Product documents. The resulting `brands` sub-facet will count how many Product documents (that have SKUs with the associated color) exist for each Brand.+<3> For each bucket in the `colors` facet, the set of all matching SKU documents will be transformed into the set of corresponding parent Product documents.+The resulting `brands` sub-facet will count how many Product documents (that have SKUs with the associated color) exist for each Brand.== Join Query Domain Changes@@ -268,7 +281,9 @@ Example:-----`join` domain changes support an optional `method` parameter, which allows users to specify which join implementation they would like to use in this domain transform. Solr offers several join implementations, each with different performance characteristics. For more information on these implementations and their tradeoffs, see the `method` parameter documentation <<join-query-parser.adoc#parameters,here>>. Join domain changes support all `method` values except `crossCollection`.+`join` domain changes support an optional `method` parameter, which allows users to specify which join implementation they would like to use in this domain transform.+Solr offers several join implementations, each with different performance characteristics.+For more information on these implementations and their tradeoffs, see the `method` parameter documentation <<join-query-parser.adoc#parameters,here>>. Join domain changes support all `method` values except `crossCollection`.== Graph Traversal Domain Changesdiff --git a/solr/solr-ref-guide/src/json-query-dsl.adoc b/solr/solr-ref-guide/src/json-query-dsl.adocindex 81b90d89415..33d0fdf7c5f 100644--- a/solr/solr-ref-guide/src/json-query-dsl.adoc+++ b/solr/solr-ref-guide/src/json-query-dsl.adoc@@ -23,18 +23,22 @@ Queries and filters provided in JSON requests can be specified using a rich, pow== Query DSL StructureThe JSON Request API accepts query values in three different formats:-* A valid <<standard-query-parser.adoc#,query string>> that uses the default `deftype` (`lucene`, in most cases). e.g., `title:solr`.+* A valid <<standard-query-parser.adoc#,query string>> that uses the default `deftype` (`lucene`, in most cases), e.g., `title:solr`.-* A valid <<local-params.adoc#,local params query string>> that specifies its `deftype` explicitly. e.g., `{!dismax qf=title}solr`.+* A valid <<local-params.adoc#,local params query string>> that specifies its `deftype` explicitly, e.g., `{!dismax qf=title}solr`.-* A valid JSON object with the name of the query parser and any relevant parameters. e.g., `{ "lucene": {"df":"title", "query":"solr"}}`.-** The top level "query" JSON block generally only has a single property representing the name of the query parser to use. The value for the query parser property is a child block containing any relevant parameters as JSON properties. The whole structure is analogous to a "local-params" query string. The query itself (often represented in local params using the name `v`) is specified with the key `query` instead.+* A valid JSON object with the name of the query parser and any relevant parameters, e.g., `{ "lucene": {"df":"title", "query":"solr"}}`.+** The top level "query" JSON block generally only has a single property representing the name of the query parser to use.+The value for the query parser property is a child block containing any relevant parameters as JSON properties.+The whole structure is analogous to a "local-params" query string.+The query itself (often represented in local params using the name `v`) is specified with the key `query` instead.All of these syntaxes can be used to specify queries for either the JSON Request API's `query` or `filter` properties.=== Query DSL Examples-The examples below show how to use each of the syntaxes discussed above to represent a query. Each snippet represents the same basic search: the term `iPod` in a field called `name`:+The examples below show how to use each of the syntaxes discussed above to represent a query.+Each snippet represents the same basic search: the term `iPod` in a field called `name`:. Using the standard query API, with a simple query string+@@ -139,7 +143,9 @@ include::{example-source-dir}JsonRequestApiTest.java[tag=solrj-ipod-query-dsl-3]--== Nested Queries-Many of Solr's query parsers allow queries to be nested within one another. When these are used, requests using the standard query API quickly become hard to write, read, and understand. These sorts of queries are often much easier to work with in the JSON Request API.+Many of Solr's query parsers allow queries to be nested within one another.+When these are used, requests using the standard query API quickly become hard to write, read, and understand.+These sorts of queries are often much easier to work with in the JSON Request API.=== Nested Boost Query ExampleAs an example, consider the three requests below, which wrap a simple query (the term `iPod` in the field `name`) within a boost query:@@ -235,7 +241,8 @@ include::{example-source-dir}JsonRequestApiTest.java[tag=solrj-ipod-query-booste=== Nested Boolean Query ExampleQuery nesting is commonly seen when combining multiple query clauses together using pseudo-boolean logic with the <<other-parsers.adoc#boolean-query-parser,BoolQParser>>.-The example below shows how the `BoolQParser` can be used to create powerful nested queries. In this example, a user searches for results with `iPod` in the field `name` which are _not_ in the bottom half of the `popularity` rankings.+The example below shows how the `BoolQParser` can be used to create powerful nested queries.+In this example, a user searches for results with `iPod` in the field `name` which are _not_ in the bottom half of the `popularity` rankings.[.dynamic-tabs]--@@ -388,12 +395,19 @@ curl -X POST http://localhost:8983/solr/techproducts/query -d '}'-----Overall this example doesn't make much sense, but just demonstrates the syntax. This feature is useful in <<json-faceting-domain-changes.adoc#adding-domain-filters,filtering domain>> in JSON Facet API <<json-facet-api.adoc#changing-the-domain,domain changes>>. Note that these declarations add request parameters underneath, so using same names with other parameters might cause unexpected behavior.+Overall this example doesn't make much sense, but just demonstrates the syntax.+This feature is useful in <<json-faceting-domain-changes.adoc#adding-domain-filters,filtering domain>> in JSON Facet API <<json-facet-api.adoc#changing-the-domain,domain changes>>. Note that these declarations add request parameters underneath, so using same names with other parameters might cause unexpected behavior.== Tagging in JSON Query DSL-Query and filter clauses can also be individually "tagged". Tags serve as handles for query clauses, allowing them to be referenced from elsewhere in the request. This is most commonly used by the filter-exclusion functionality offered by both <<faceting.adoc#tagging-and-excluding-filters,traditional>> and <<json-faceting-domain-changes.adoc#filter-exclusions,JSON>> faceting.+Query and filter clauses can also be individually "tagged". Tags serve as handles for query clauses, allowing them to be referenced from elsewhere in the request.+This is most commonly used by the filter-exclusion functionality offered by both <<faceting.adoc#tagging-and-excluding-filters,traditional>> and <<json-faceting-domain-changes.adoc#filter-exclusions,JSON>> faceting.-Queries and filters are tagged by wrapping them in a surrounding JSON object. The name of the tag is specified as a JSON key, with the query string (or object) becoming the value associated with that key. Tag name properties are prefixed with a hash, and may include multiple tags, separated by commas. For example: `{"\#title,tag2,tag3":"title:solr"}`. Note that unlike the rest of the JSON request API which uses lax JSON parsing rules, tags must be surrounded by double-quotes because of the leading `#` character. The example below creates two tagged clauses: `titleTag` and `inStockTag`.+Queries and filters are tagged by wrapping them in a surrounding JSON object.+The name of the tag is specified as a JSON key, with the query string (or object) becoming the value associated with that key.+Tag name properties are prefixed with a hash, and may include multiple tags, separated by commas.+For example: `{"\#title,tag2,tag3":"title:solr"}`.+Note that unlike the rest of the JSON request API which uses lax JSON parsing rules, tags must be surrounded by double-quotes because of the leading `#` character.+The example below creates two tagged clauses: `titleTag` and `inStockTag`.[.dynamic-tabs]--@@ -427,4 +441,5 @@ include::{example-source-dir}JsonRequestApiTest.java[tag=solrj-tagged-query]====---Note that the tags created in the example above have no impact in how the search is executed. Tags will not affect a query unless they are referenced by some other part of the request that uses them.+Note that the tags created in the example above have no impact in how the search is executed.+Tags will not affect a query unless they are referenced by some other part of the request that uses them.diff --git a/solr/solr-ref-guide/src/jvm-settings.adoc b/solr/solr-ref-guide/src/jvm-settings.adocindex f0a7e6602bc..9b49a63a47c 100644--- a/solr/solr-ref-guide/src/jvm-settings.adoc+++ b/solr/solr-ref-guide/src/jvm-settings.adoc@@ -18,32 +18,52 @@Optimizing the JVM can be a key factor in getting the most from your Solr installation.-Configuring your JVM is a complex topic and a full discussion is beyond the scope of this document. Luckily, most modern JVMs are quite good at making the best use of available resources with default settings. The following sections contain a few tips that may be helpful when the defaults are not optimal for your situation.+Configuring your JVM is a complex topic and a full discussion is beyond the scope of this document.+Luckily, most modern JVMs are quite good at making the best use of available resources with default settings.+The following sections contain a few tips that may be helpful when the defaults are not optimal for your situation.For more general information about improving Solr performance, see https://cwiki.apache.org/confluence/display/solr/SolrPerformanceFactors[Solr Performance Factors] in the Solr Wiki.== Choosing Memory Heap Settings-The most important JVM configuration settings control the heap allocated to the JVM: `-Xms`, which sets the initial size of the JVM's memory heap, and `-Xmx`, which sets the maximum size of the heap. Setting these two options to the same value is a common practice.+The most important JVM configuration settings control the heap allocated to the JVM: `-Xms`, which sets the initial size of the JVM's memory heap, and `-Xmx`, which sets the maximum size of the heap.+Setting these two options to the same value is a common practice.-Heap size is critical and unfortunately there is no "one size fits all" solution, you must test with your data and your application. The best way to determine the correct size is to analyze the garbage collection (GC) logs located in your logs directory. There are various tools that help analyze these logs and, in particular, show the amount of memory used after GC has completed (http://www.tagtraum.com/gcviewer.html[GCViewer] and https://gceasy.io/[GCEasy] are two). Also you can attach jconsole (distributed with most Java runtimes) to check memory consumption as Solr is running. This will show the absolute minimum amount of memory required; adding 25-50% "headroom" is a reasonable starting point.+Heap size is critical and unfortunately there is no "one size fits all" solution, you must test with your data and your application.+The best way to determine the correct size is to analyze the garbage collection (GC) logs located in your logs directory.+There are various tools that help analyze these logs and, in particular, show the amount of memory used after GC has completed (http://www.tagtraum.com/gcviewer.html[GCViewer] and https://gceasy.io/[GCEasy] are two).+Also you can attach jconsole (distributed with most Java runtimes) to check memory consumption as Solr is running.+This will show the absolute minimum amount of memory required; adding 25-50% "headroom" is a reasonable starting point.There are several points to keep in mind:-* Running Solr with too little "headroom" allocated for the heap can cause excessive resources to be consumed by continual GC. Thus the 25-50% recommendation above.-* Lucene/Solr makes extensive use of MMapDirectory, which uses RAM _not_ reserved for the JVM for most of the Lucene index. Therefore, as much memory as possible should be left for the operating system to use for this purpose.-* The heap allocated should be as small as possible while maintaining good performance. 8-16Gb is quite common, and larger heaps are sometimes used. When heaps grow to larger sizes, it is imperative to test extensively before going to production.+* Running Solr with too little "headroom" allocated for the heap can cause excessive resources to be consumed by continual GC.+Thus the 25-50% recommendation above.+* Lucene/Solr makes extensive use of MMapDirectory, which uses RAM _not_ reserved for the JVM for most of the Lucene index.+Therefore, as much memory as possible should be left for the operating system to use for this purpose.+* The heap allocated should be as small as possible while maintaining good performance.+8-16Gb is quite common, and larger heaps are sometimes used.+When heaps grow to larger sizes, it is imperative to test extensively before going to production.* The G1GC garbage collector is currently preferred when using a JVM that supports it (Java 9 and later).-* Modern hardware can be configured with hundreds of gigabytes of physical RAM and many CPUs. It is often better in these cases to run multiple JVMs, each with a limited amount of memory allocated to their heaps. One way to achieve this is to run Solr as a https://hub.docker.com/_/solr?tab=tags[Docker container].+* Modern hardware can be configured with hundreds of gigabytes of physical RAM and many CPUs.+It is often better in these cases to run multiple JVMs, each with a limited amount of memory allocated to their heaps.+One way to achieve this is to run Solr as a https://hub.docker.com/_/solr?tab=tags[Docker container].* It's good practice to periodically re-analyze the GC logs and/or monitor with <<metrics-reporting#metrics-reporting,Metrics Reporting>> to see if the memory usage has changed due to changes in your application, number of documents, etc.-* On *nix systems, Solr will run with "OOM killer script" (see `solr/bin/oom_solr.sh`). This will forcefully stop Solr when the heap is exhausted rather than continue in an indeterminate state. You can additionally request a heap dump on OOM through the values in `solr.in.sh`-* All current (Java 11) garbage collectors can hit "stop the world" collections, which suspend the JVM until completed. If, through monitoring, these garbage collections are frequent and greater than your application can tolerate, additional tuning should be considered. "Stop the world" pauses greater than 5 seconds are rarely acceptable, and having them be less than 1 second is desirable.+* On *nix systems, Solr will run with "OOM killer script" (see `solr/bin/oom_solr.sh`).+This will forcefully stop Solr when the heap is exhausted rather than continue in an indeterminate state.+You can additionally request a heap dump on OOM through the values in `solr.in.sh`+* All current (Java 11) garbage collectors can hit "stop the world" collections, which suspend the JVM until completed.+If, through monitoring, these garbage collections are frequent and greater than your application can tolerate, additional tuning should be considered.+"Stop the world" pauses greater than 5 seconds are rarely acceptable, and having them be less than 1 second is desirable.Consult your JVM vendor's documentation for specifics in your particular case, the recommendations above are intended as starting points.== Use the Server HotSpot VM-If you are using Sun's JVM, add the `-server` command-line option when you start Solr. This tells the JVM that it should optimize for a long running, server process. If the Java runtime on your system is a JRE, rather than a full JDK distribution (including `javac` and other development tools), then it is possible that it may not support the `-server` JVM option. Test this by running `java -help` and look for `-server` as an available option in the displayed usage message.+If you are using Sun's JVM, add the `-server` command-line option when you start Solr.+This tells the JVM that it should optimize for a long running, server process.+If the Java runtime on your system is a JRE, rather than a full JDK distribution (including `javac` and other development tools), then it is possible that it may not support the `-server` JVM option.+Test this by running `java -help` and look for `-server` as an available option in the displayed usage message.== Checking JVM Settingsdiff --git a/solr/solr-ref-guide/src/jwt-authentication-plugin.adoc b/solr/solr-ref-guide/src/jwt-authentication-plugin.adocindex 8b6bc8eded8..9fa29b317e9 100644--- a/solr/solr-ref-guide/src/jwt-authentication-plugin.adoc+++ b/solr/solr-ref-guide/src/jwt-authentication-plugin.adoc@@ -39,7 +39,7 @@ The simplest possible `security.json` for registering the plugin without configuThe plugin will by default require a valid JWT token for all traffic.-If the `blockUnknown` property is set to `false` as in the above example, it is possible to start configuring the plugin using unauthenticated REST API calls, which is further described in section <<editing-jwt-authentication-plugin-configuration,Editing JWT Authentication Plugin Configuration>>.+If the `blockUnknown` property is set to `false` as in the above example, it is possible to start configuring the plugin using unauthenticated REST API calls, which is further described in section <<Editing JWT Authentication Plugin Configuration>>.== Configuration Parameters@@ -66,7 +66,9 @@ issuers ; List of issuers (Identity providers) to support. See sec=== Issuer Configuration-This plugin supports one or more token issuers (IdPs). Issuers are configured as a list of JSON objects under the `issuers` configuration key. The first issuer in the list is the "Primary Issuer", which is the one used for logging in to the Admin UI.+This plugin supports one or more token issuers (IdPs).+Issuers are configured as a list of JSON objects under the `issuers` configuration key.+The first issuer in the list is the "Primary Issuer", which is the one used for logging in to the Admin UI.[%header,format=csv,separator=;,cols="25%,50%,25%"]|===@@ -180,9 +182,14 @@ However, in development, it may be useful to use regular HTTP URLs, and bypass tTo support this you can set the environment variable `-Dsolr.auth.jwt.allowOutboundHttp=true` at startup.=== Trusting the IdP server-All communication with the Oauth2 server (IdP) is done over HTTPS. By default, Java's built-in TrustStore is used. However, by configuring one of the options `trustedCertsFile` or `trustedCerts`, the plugin will *instead* trust the set of certificates provided, not any certificate signed by a root CA. This is both more secure and also lets you trust self-signed certificates. It also has the benefit of working even if Solr is not started in SSL mode.--Please configure either the `trustedCerts` or `trustedCertsFile` option. Configuring both will cause an error.+All communication with the Oauth2 server (IdP) is done over HTTPS.+By default, Java's built-in TrustStore is used.+However, by configuring one of the options `trustedCertsFile` or `trustedCerts`, the plugin will *instead* trust the set of certificates provided, not any certificate signed by a root CA.+This is both more secure and also lets you trust self-signed certificates.+It also has the benefit of working even if Solr is not started in SSL mode.++Please configure either the `trustedCerts` or `trustedCertsFile` option.+Configuring both will cause an error.== Editing JWT Authentication Plugin Configurationdiff --git a/solr/solr-ref-guide/src/kerberos-authentication-plugin.adoc b/solr/solr-ref-guide/src/kerberos-authentication-plugin.adocindex a955368c52c..a1164192b82 100644--- a/solr/solr-ref-guide/src/kerberos-authentication-plugin.adoc+++ b/solr/solr-ref-guide/src/kerberos-authentication-plugin.adoc@@ -101,7 +101,8 @@ We'll walk through each of these steps below.[IMPORTANT]====To use host names instead of IP addresses, use the `SOLR_HOST` configuration in `bin/solr.in.sh` or pass a `-Dhost=<hostname>` system parameter during Solr startup.-This guide uses IP addresses. If you specify a hostname, replace all the IP addresses in the guide with the Solr hostname as appropriate.+This guide uses IP addresses.+If you specify a hostname, replace all the IP addresses in the guide with the Solr hostname as appropriate.======= Get Service Principals and Keytabs@@ -445,8 +446,10 @@ NOTE: If you have defined `ZK_HOST` in `solr.in.sh`/`solr.in.cmd` (see <<zookeep=== Test the Configuration-. Do a `kinit` with your username. For example, `kinit \[email protected]`.-. Try to access Solr using `curl`. You should get a successful response.+. Do a `kinit` with your username.+For example, `kinit \[email protected]`.+. Try to access Solr using `curl`.+You should get a successful response.+[source,bash]----diff --git a/solr/solr-ref-guide/src/language-analysis.adoc b/solr/solr-ref-guide/src/language-analysis.adocindex ab8f7c06c9d..71f38cf1227 100644--- a/solr/solr-ref-guide/src/language-analysis.adoc+++ b/solr/solr-ref-guide/src/language-analysis.adoc@@ -234,7 +234,8 @@ Unicode Collation in Solr is fast, because all the work is done at index time.Rather than specifying an analyzer within `<fieldtype ... class="solr.TextField">`, the `solr.CollationField` and `solr.ICUCollationField` field type classes provide this functionality.`solr.ICUCollationField`, which is backed by http://site.icu-project.org[the ICU4J library], provides more flexible configuration, has more locales, is significantly faster, and requires less memory and less index space, since its keys are smaller than those produced by the JDK implementation that backs `solr.CollationField`.-To use `solr.ICUCollationField`, you must add additional .jars to Solr's classpath (as described in the section <<solr-plugins.adoc#installing-plugins,Solr Plugins>>). See `solr/contrib/analysis-extras/README.md` for instructions on which jars you need to add.+To use `solr.ICUCollationField`, you must add additional .jars to Solr's classpath (as described in the section <<solr-plugins.adoc#installing-plugins,Solr Plugins>>).+See `solr/contrib/analysis-extras/README.md` for instructions on which jars you need to add.`solr.ICUCollationField` and `solr.CollationField` fields can be created in two ways:@@ -248,28 +249,36 @@ Using a System collator:`locale`:: (required) http://www.rfc-editor.org/rfc/rfc3066.txt[RFC 3066] locale ID.See http://demo.icu-project.org/icu-bin/locexp[the ICU locale explorer] for a list of supported locales.-`strength`:: Valid values are `primary`, `secondary`, `tertiary`, `quaternary`, or `identical`. See http://userguide.icu-project.org/collation/concepts#TOC-Comparison-Levels[Comparison Levels in ICU Collation Concepts] for more information.+`strength`:: Valid values are `primary`, `secondary`, `tertiary`, `quaternary`, or `identical`.+See http://userguide.icu-project.org/collation/concepts#TOC-Comparison-Levels[Comparison Levels in ICU Collation Concepts] for more information.-`decomposition`:: Valid values are `no` or `canonical`. See http://userguide.icu-project.org/collation/concepts#TOC-Normalization[Normalization in ICU Collation Concepts] for more information.+`decomposition`:: Valid values are `no` or `canonical`.+See http://userguide.icu-project.org/collation/concepts#TOC-Normalization[Normalization in ICU Collation Concepts] for more information.Using a Tailored ruleset:`custom`:: (required) Path to a UTF-8 text file containing rules supported by the ICU http://icu-project.org/apiref/icu4j/com/ibm/icu/text/RuleBasedCollator.html[`RuleBasedCollator`]-`strength`:: Valid values are `primary`, `secondary`, `tertiary`, `quaternary`, or `identical`. See http://userguide.icu-project.org/collation/concepts#TOC-Comparison-Levels[Comparison Levels in ICU Collation Concepts] for more information.+`strength`:: Valid values are `primary`, `secondary`, `tertiary`, `quaternary`, or `identical`.+See http://userguide.icu-project.org/collation/concepts#TOC-Comparison-Levels[Comparison Levels in ICU Collation Concepts] for more information.-`decomposition`:: Valid values are `no` or `canonical`. See http://userguide.icu-project.org/collation/concepts#TOC-Normalization[Normalization in ICU Collation Concepts] for more information.+`decomposition`:: Valid values are `no` or `canonical`.+See http://userguide.icu-project.org/collation/concepts#TOC-Normalization[Normalization in ICU Collation Concepts] for more information.Expert options:-`alternate`:: Valid values are `shifted` or `non-ignorable`. Can be used to ignore punctuation/whitespace.+`alternate`:: Valid values are `shifted` or `non-ignorable`.+Can be used to ignore punctuation/whitespace.`caseLevel`:: (true/false) If true, in combination with `strength="primary"`, accents are ignored but case is taken into account.-The default is `false`. See http://userguide.icu-project.org/collation/concepts#TOC-CaseLevel[CaseLevel in ICU Collation Concepts] for more information.+The default is `false`.+See http://userguide.icu-project.org/collation/concepts#TOC-CaseLevel[CaseLevel in ICU Collation Concepts] for more information.-`caseFirst`:: Valid values are `lower` or `upper`. Useful to control which is sorted first when case is not ignored.+`caseFirst`:: Valid values are `lower` or `upper`.+Useful to control which is sorted first when case is not ignored.-`numeric`:: (true/false) If true, digits are sorted according to numeric value, e.g., foobar-9 sorts before foobar-10. The default is false.+`numeric`:: (true/false) If true, digits are sorted according to numeric value, e.g., foobar-9 sorts before foobar-10.+The default is false.`variableTop`:: Single character or contraction.Controls what is variable for `alternate`.@@ -327,7 +336,8 @@ q=*:*&fl=city&sort=city_sort+asc=== Sorting Text for Multiple Languages-There are two approaches to supporting multiple languages: if there is a small list of languages you wish to support, consider defining collated fields for each language and using `copyField`. However, adding a large number of sort fields can increase disk and indexing costs.+There are two approaches to supporting multiple languages: if there is a small list of languages you wish to support, consider defining collated fields for each language and using `copyField`.+However, adding a large number of sort fields can increase disk and indexing costs.An alternative approach is to use the Unicode `default` collator.The Unicode `default` or `ROOT` locale has rules that are designed to work well for most languages.@@ -346,7 +356,8 @@ This Unicode default sort is still significantly more advanced than the standardYou can define your own set of sorting rules.It's easiest to take existing rules that are close to what you want and customize them.-In the example below, we create a custom rule set for German called DIN 5007-2. This rule set treats umlauts in German differently: it treats ö as equivalent to oe, ä as equivalent to ae, and ü as equivalent to ue.+In the example below, we create a custom rule set for German called DIN 5007-2.+This rule set treats umlauts in German differently: it treats ö as equivalent to oe, ä as equivalent to ae, and ü as equivalent to ue.For more information, see the http://icu-project.org/apiref/icu4j/com/ibm/icu/text/RuleBasedCollator.html[ICU RuleBasedCollator javadocs].This example shows how to create a custom rule set for `solr.ICUCollationField` and dump it to a file:@@ -398,17 +409,21 @@ Using a System collator (see http://www.oracle.com/technetwork/java/javase/java8`variant`:: Vendor or browser-specific code-`strength`:: Valid values are `primary`, `secondary`, `tertiary` or `identical`. See {java-javadocs}java/text/Collator.html[Java Collator javadocs] for more information.+`strength`:: Valid values are `primary`, `secondary`, `tertiary` or `identical`.+See {java-javadocs}java/text/Collator.html[Java Collator javadocs] for more information.-`decomposition`:: Valid values are `no`, `canonical`, or `full`. See {java-javadocs}java/text/Collator.html[Java Collator javadocs] for more information.+`decomposition`:: Valid values are `no`, `canonical`, or `full`.+See {java-javadocs}java/text/Collator.html[Java Collator javadocs] for more information.Using a Tailored ruleset:`custom`:: (required) Path to a UTF-8 text file containing rules supported by the {java-javadocs}java/text/RuleBasedCollator.html[`JDK RuleBasedCollator`]-`strength`:: Valid values are `primary`, `secondary`, `tertiary` or `identical`. See {java-javadocs}java/text/Collator.html[Java Collator javadocs] for more information.+`strength`:: Valid values are `primary`, `secondary`, `tertiary` or `identical`.+See {java-javadocs}java/text/Collator.html[Java Collator javadocs] for more information.-`decomposition`:: Valid values are `no`, `canonical`, or `full`. See {java-javadocs}java/text/Collator.html[Java Collator javadocs] for more information.+`decomposition`:: Valid values are `no`, `canonical`, or `full`.+See {java-javadocs}java/text/Collator.html[Java Collator javadocs] for more information..A `solr.CollationField` example:[source,xml]@@ -517,7 +532,8 @@ In addition to these analysis components, Solr also provides an update request pNOTE: The <<OpenNLP Tokenizer>> must be used with all other OpenNLP analysis components, for two reasons: first, the OpenNLP Tokenizer detects and marks the sentence boundaries required by all the OpenNLP filters; and second, since the pre-trained OpenNLP models used by these filters were trained using the corresponding language-specific sentence-detection/tokenization models, the same tokenization, using the same models, must be used at runtime for optimal performance.-To use the OpenNLP components, you must add additional .jars to Solr's classpath (as described in the section <<solr-plugins.adoc#installing-plugins,Solr Plugins>>). See `solr/contrib/analysis-extras/README.md` for instructions on which jars you need to add.+To use the OpenNLP components, you must add additional .jars to Solr's classpath (as described in the section <<solr-plugins.adoc#installing-plugins,Solr Plugins>>).+See `solr/contrib/analysis-extras/README.md` for instructions on which jars you need to add.=== OpenNLP Tokenizer@@ -951,7 +967,8 @@ They use the Lucene classes `org.apache.lucene.analysis.bn.BengaliNormalizationF=== Brazilian PortugueseThis is a Java filter written specifically for stemming the Brazilian dialect of the Portuguese language.-It uses the Lucene class `org.apache.lucene.analysis.br.BrazilianStemmer`. Although that stemmer can be configured to use a list of protected words (which should not be stemmed), this factory does not accept any arguments to specify such a list.+It uses the Lucene class `org.apache.lucene.analysis.br.BrazilianStemmer`.+Although that stemmer can be configured to use a list of protected words (which should not be stemmed), this factory does not accept any arguments to specify such a list.*Factory class:* `solr.BrazilianStemFilterFactory`@@ -1031,7 +1048,8 @@ Solr includes a light stemmer for Bulgarian, following http://members.unine.ch/j=== Catalan-Solr can stem Catalan using the Snowball Porter Stemmer with an argument of `language="Catalan"`. Solr includes a set of contractions for Catalan, which can be stripped using `solr.ElisionFilterFactory`.+Solr can stem Catalan using the Snowball Porter Stemmer with an argument of `language="Catalan"`.+Solr includes a set of contractions for Catalan, which can be stripped using `solr.ElisionFilterFactory`.*Factory class:* `solr.SnowballPorterFilterFactory`@@ -1083,7 +1101,8 @@ Solr can stem Catalan using the Snowball Porter Stemmer with an argument of `lanThe default configuration of the <<tokenizers.adoc#icu-tokenizer,ICU Tokenizer>> is suitable for Traditional Chinese text.It follows the Word Break rules from the Unicode Text Segmentation algorithm for non-Chinese text, and uses a dictionary to segment Chinese words.-To use this tokenizer, you must add additional .jars to Solr's classpath (as described in the section <<solr-plugins.adoc#installing-plugins,Solr Plugins>>). See the `solr/contrib/analysis-extras/README.md` for information on which jars you need to add.+To use this tokenizer, you must add additional .jars to Solr's classpath (as described in the section <<solr-plugins.adoc#installing-plugins,Solr Plugins>>).+See the `solr/contrib/analysis-extras/README.md` for information on which jars you need to add.<<tokenizers.adoc#standard-tokenizer,Standard Tokenizer>> can also be used to tokenize Traditional Chinese text.Following the Word Break rules from the Unicode Text Segmentation algorithm, it produces one token per Chinese character.@@ -1135,7 +1154,8 @@ When combined with <<CJK Bigram Filter>>, overlapping bigrams of Chinese charactForms bigrams (overlapping 2-character sequences) of CJK characters that are generated from <<tokenizers.adoc#standard-tokenizer,Standard Tokenizer>> or <<tokenizers.adoc#icu-tokenizer,ICU Tokenizer>>.-By default, all CJK characters produce bigrams, but finer grained control is available by specifying orthographic type arguments `han`, `hiragana`, `katakana`, and `hangul`. When set to `false`, characters of the corresponding type will be passed through as unigrams, and will not be included in any bigrams.+By default, all CJK characters produce bigrams, but finer grained control is available by specifying orthographic type arguments `han`, `hiragana`, `katakana`, and `hangul`.+When set to `false`, characters of the corresponding type will be passed through as unigrams, and will not be included in any bigrams.When a CJK character has no adjacent characters to form a bigram, it is output in unigram form.If you want to always output both unigrams and bigrams, set the `outputUnigrams` argument to `true`.@@ -1164,11 +1184,13 @@ See the example under <<Traditional Chinese>>.=== Simplified ChineseFor Simplified Chinese, Solr provides support for Chinese sentence and word segmentation with the <<HMM Chinese Tokenizer>>. This component includes a large dictionary and segments Chinese text into words with the Hidden Markov Model.-To use this tokenizer, you must add additional .jars to Solr's classpath (as described in the section <<solr-plugins.adoc#installing-plugins,Solr Plugins>>). See the `solr/contrib/analysis-extras/README.md` for information on which jars you need to add.+To use this tokenizer, you must add additional .jars to Solr's classpath (as described in the section <<solr-plugins.adoc#installing-plugins,Solr Plugins>>).+See the `solr/contrib/analysis-extras/README.md` for information on which jars you need to add.The default configuration of the <<tokenizers.adoc#icu-tokenizer,ICU Tokenizer>> is also suitable for Simplified Chinese text.It follows the Word Break rules from the Unicode Text Segmentation algorithm for non-Chinese text, and uses a dictionary to segment Chinese words.-To use this tokenizer, you must add additional .jars to Solr's classpath (as described in the section <<solr-plugins.adoc#installing-plugins,Solr Plugins>>). See the `solr/contrib/analysis-extras/README.md` for information on which jars you need to add.+To use this tokenizer, you must add additional .jars to Solr's classpath (as described in the section <<solr-plugins.adoc#installing-plugins,Solr Plugins>>).+See the `solr/contrib/analysis-extras/README.md` for information on which jars you need to add.Also useful for Chinese analysis:@@ -1225,7 +1247,8 @@ Also useful for Chinese analysis:For Simplified Chinese, Solr provides support for Chinese sentence and word segmentation with the `solr.HMMChineseTokenizerFactory` in the `analysis-extras` contrib module.This component includes a large dictionary and segments Chinese text into words with the Hidden Markov Model.-To use this tokenizer, you must add additional .jars to Solr's classpath (as described in the section <<solr-plugins.adoc#installing-plugins,Solr Plugins>>). See `solr/contrib/analysis-extras/README.md` for instructions on which jars you need to add.+To use this tokenizer, you must add additional .jars to Solr's classpath (as described in the section <<solr-plugins.adoc#installing-plugins,Solr Plugins>>).+See `solr/contrib/analysis-extras/README.md` for instructions on which jars you need to add.*Factory class:* `solr.HMMChineseTokenizerFactory`@@ -1479,7 +1502,8 @@ This filter can be useful for languages such as French, Catalan, Italian, and Ir*Arguments:*`articles`:: The pathname of a file that contains a list of articles, one per line, to be stripped.-Articles are words such as "le", which are commonly abbreviated, such as in _l'avion_ (the plane). This file should include the abbreviated form, which precedes the apostrophe.+Articles are words such as "le", which are commonly abbreviated, such as in _l'avion_ (the plane).+This file should include the abbreviated form, which precedes the apostrophe.In this case, simply "_l_". If no `articles` attribute is specified, a default set of French articles is used.`ignoreCase`:: (boolean) If true, the filter ignores the case of words when comparing them to the common word file.@@ -1525,7 +1549,8 @@ Defaults to `false`==== French Light Stem Filter-Solr includes three stemmers for French: one in the `solr.SnowballPorterFilterFactory`, a lighter stemmer called `solr.FrenchLightStemFilterFactory`, and an even less aggressive stemmer called `solr.FrenchMinimalStemFilterFactory`. Lucene includes an example stopword list.+Solr includes three stemmers for French: one in the `solr.SnowballPorterFilterFactory`, a lighter stemmer called `solr.FrenchLightStemFilterFactory`, and an even less aggressive stemmer called `solr.FrenchMinimalStemFilterFactory`.+Lucene includes an example stopword list.*Factory classes:* `solr.FrenchLightStemFilterFactory`, `solr.FrenchMinimalStemFilterFactory`@@ -1608,7 +1633,8 @@ Solr includes a stemmer for Galician following http://bvg.udc.es/recursos_lingua=== German-Solr includes four stemmers for German: one in the `solr.SnowballPorterFilterFactory language="German"`, a stemmer called `solr.GermanStemFilterFactory`, a lighter stemmer called `solr.GermanLightStemFilterFactory`, and an even less aggressive stemmer called `solr.GermanMinimalStemFilterFactory`. Lucene includes an example stopword list.+Solr includes four stemmers for German: one in the `solr.SnowballPorterFilterFactory language="German"`, a stemmer called `solr.GermanStemFilterFactory`, a lighter stemmer called `solr.GermanLightStemFilterFactory`, and an even less aggressive stemmer called `solr.GermanMinimalStemFilterFactory`.+Lucene includes an example stopword list.*Factory classes:* `solr.GermanStemFilterFactory`, `solr.LightGermanStemFilterFactory`, `solr.MinimalGermanStemFilterFactory`@@ -1674,7 +1700,8 @@ This filter converts uppercase letters in the Greek character set to the equival[IMPORTANT]====-Use of custom charsets is no longer supported as of Solr 3.1. If you need to index text in these encodings, please use Java's character set conversion facilities (InputStreamReader, etc.) during I/O, so that Lucene can analyze this text as Unicode instead.+Use of custom charsets is no longer supported as of Solr 3.1.+If you need to index text in these encodings, please use Java's character set conversion facilities (InputStreamReader, etc.) during I/O, so that Lucene can analyze this text as Unicode instead.====*Example:*@@ -1791,7 +1818,8 @@ Solr includes support for stemming Indonesian (Bahasa Indonesia) following http:=== Italian-Solr includes two stemmers for Italian: one in the `solr.SnowballPorterFilterFactory language="Italian"`, and a lighter stemmer called `solr.ItalianLightStemFilterFactory`. Lucene includes an example stopword list.+Solr includes two stemmers for Italian: one in the `solr.SnowballPorterFilterFactory language="Italian"`, and a lighter stemmer called `solr.ItalianLightStemFilterFactory`.+Lucene includes an example stopword list.*Factory class:* `solr.ItalianStemFilterFactory`@@ -1839,7 +1867,8 @@ Solr includes two stemmers for Italian: one in the `solr.SnowballPorterFilterFac=== Irish-Solr can stem Irish using the Snowball Porter Stemmer with an argument of `language="Irish"`. Solr includes `solr.IrishLowerCaseFilterFactory`, which can handle Irish-specific constructs.+Solr can stem Irish using the Snowball Porter Stemmer with an argument of `language="Irish"`.+Solr includes `solr.IrishLowerCaseFilterFactory`, which can handle Irish-specific constructs.Solr also includes a set of contractions for Irish which can be stripped using `solr.ElisionFilterFactory`.*Factory class:* `solr.SnowballPorterFilterFactory`@@ -1946,7 +1975,8 @@ See `lang/userdict_ja.txt` for a sample user dictionary file.==== Japanese Base Form Filter-Replaces original terms' text with the corresponding base form (lemma). (`JapaneseTokenizer` annotates each term with its base form.)+Replaces original terms' text with the corresponding base form (lemma).+(`JapaneseTokenizer` annotates each term with its base form.)*Factory class:* `JapaneseBaseFormFilterFactory`@@ -2037,7 +2067,8 @@ Example:=== Hebrew, Lao, Myanmar, KhmerLucene provides support, in addition to UAX#29 word break rules, for Hebrew's use of the double and single quote characters, and for segmenting Lao, Myanmar, and Khmer into syllables with the `solr.ICUTokenizerFactory` in the `analysis-extras` contrib module.-To use this tokenizer, you must add additional .jars to Solr's classpath (as described in the section <<solr-plugins.adoc#installing-plugins,Solr Plugins>>). See `solr/contrib/analysis-extras/README.md` for instructions on which jars you need to add.+To use this tokenizer, you must add additional .jars to Solr's classpath (as described in the section <<solr-plugins.adoc#installing-plugins,Solr Plugins>>).+See `solr/contrib/analysis-extras/README.md` for instructions on which jars you need to add.See <<tokenizers.adoc#icu-tokenizer,the ICUTokenizer>> for more information.@@ -2091,7 +2122,8 @@ Solr includes support for stemming Latvian, and Lucene includes an example stopw=== Norwegian-Solr includes two classes for stemming Norwegian, `NorwegianLightStemFilterFactory` and `NorwegianMinimalStemFilterFactory`. Lucene includes an example stopword list.+Solr includes two classes for stemming Norwegian, `NorwegianLightStemFilterFactory` and `NorwegianMinimalStemFilterFactory`.+Lucene includes an example stopword list.Another option is to use the Snowball Porter Stemmer with an argument of language="Norwegian".@@ -2294,7 +2326,8 @@ Solr includes support for normalizing Persian, and Lucene includes an example stSolr provides support for Polish stemming with the `solr.StempelPolishStemFilterFactory`, and `solr.MorphologikFilterFactory` for lemmatization, in the `contrib/analysis-extras` module.The `solr.StempelPolishStemFilterFactory` component includes an algorithmic stemmer with tables for Polish.-To use either of these filters, you must add additional .jars to Solr's classpath (as described in the section <<solr-plugins.adoc#installing-plugins,Solr Plugins>>). See `solr/contrib/analysis-extras/README.md` for instructions on which jars you need to add.+To use either of these filters, you must add additional .jars to Solr's classpath (as described in the section <<solr-plugins.adoc#installing-plugins,Solr Plugins>>).+See `solr/contrib/analysis-extras/README.md` for instructions on which jars you need to add.*Factory class:* `solr.StempelPolishStemFilterFactory` and `solr.MorfologikFilterFactory`@@ -2356,7 +2389,8 @@ If the dictionary attribute is not provided, the Polish dictionary is loaded and=== Portuguese-Solr includes four stemmers for Portuguese: one in the `solr.SnowballPorterFilterFactory`, an alternative stemmer called `solr.PortugueseStemFilterFactory`, a lighter stemmer called `solr.PortugueseLightStemFilterFactory`, and an even less aggressive stemmer called `solr.PortugueseMinimalStemFilterFactory`. Lucene includes an example stopword list.+Solr includes four stemmers for Portuguese: one in the `solr.SnowballPorterFilterFactory`, an alternative stemmer called `solr.PortugueseStemFilterFactory`, a lighter stemmer called `solr.PortugueseLightStemFilterFactory`, and an even less aggressive stemmer called `solr.PortugueseMinimalStemFilterFactory`.+Lucene includes an example stopword list.*Factory classes:* `solr.PortugueseStemFilterFactory`, `solr.PortugueseLightStemFilterFactory`, `solr.PortugueseMinimalStemFilterFactory`@@ -2460,7 +2494,8 @@ Solr can stem Romanian using the Snowball Porter Stemmer with an argument of `la==== Russian Stem Filter-Solr includes two stemmers for Russian: one in the `solr.SnowballPorterFilterFactory language="Russian"`, and a lighter stemmer called `solr.RussianLightStemFilterFactory`. Lucene includes an example stopword list.+Solr includes two stemmers for Russian: one in the `solr.SnowballPorterFilterFactory language="Russian"`, and a lighter stemmer called `solr.RussianLightStemFilterFactory`.+Lucene includes an example stopword list.*Factory class:* `solr.RussianLightStemFilterFactory`@@ -2635,7 +2670,8 @@ Valid values are:=== Spanish-Solr includes two stemmers for Spanish: one in the `solr.SnowballPorterFilterFactory language="Spanish"`, and a lighter stemmer called `solr.SpanishLightStemFilterFactory`. Lucene includes an example stopword list.+Solr includes two stemmers for Spanish: one in the `solr.SnowballPorterFilterFactory language="Spanish"`, and a lighter stemmer called `solr.SpanishLightStemFilterFactory`.+Lucene includes an example stopword list.*Factory class:* `solr.SpanishStemFilterFactory`@@ -2682,7 +2718,8 @@ Solr includes two stemmers for Spanish: one in the `solr.SnowballPorterFilterFac==== Swedish Stem Filter-Solr includes two stemmers for Swedish: one in the `solr.SnowballPorterFilterFactory language="Swedish"`, and a lighter stemmer called `solr.SwedishLightStemFilterFactory`. Lucene includes an example stopword list.+Solr includes two stemmers for Swedish: one in the `solr.SnowballPorterFilterFactory language="Swedish"`, and a lighter stemmer called `solr.SwedishLightStemFilterFactory`.+Lucene includes an example stopword list.Also relevant are the <<Scandinavian,Scandinavian normalization filters>>.@@ -2821,7 +2858,8 @@ Solr includes support for stemming Turkish with the `solr.SnowballPorterFilterFa=== UkrainianSolr provides support for Ukrainian lemmatization with the `solr.MorphologikFilterFactory`, in the `contrib/analysis-extras` module.-To use this filter, you must add additional .jars to Solr's classpath (as described in the section <<solr-plugins.adoc#installing-plugins,Solr Plugins>>). See `solr/contrib/analysis-extras/README.md` for instructions on which jars you need to add.+To use this filter, you must add additional .jars to Solr's classpath (as described in the section <<solr-plugins.adoc#installing-plugins,Solr Plugins>>).+See `solr/contrib/analysis-extras/README.md` for instructions on which jars you need to add.Lucene also includes an example Ukrainian stopword list, in the `lucene-analyzers-morfologik` jar.diff --git a/solr/solr-ref-guide/src/language-detection.adoc b/solr/solr-ref-guide/src/language-detection.adocindex 3383ebde479..70ba97def08 100644--- a/solr/solr-ref-guide/src/language-detection.adoc+++ b/solr/solr-ref-guide/src/language-detection.adoc@@ -33,7 +33,8 @@ For more information about language analysis in Solr, see <<language-analysis.ad== Configuring Language Detection-You can configure the `langid` UpdateRequestProcessor in `solrconfig.xml`. Both implementations take the same parameters, which are described in the following section.+You can configure the `langid` UpdateRequestProcessor in `solrconfig.xml`.+Both implementations take the same parameters, which are described in the following section.At a minimum, you must specify the fields for language identification and a field for the resulting language code.=== Configuring Tika Language Detection@@ -100,7 +101,8 @@ As previously mentioned, both implementations of the `langid` UpdateRequestProceWhen `true`, the default, enables language detection.`langid.fl`::-A comma- or space-delimited list of fields to be processed by `langid`. This parameter is required.+A comma- or space-delimited list of fields to be processed by `langid`.+This parameter is required.`langid.langField`::Specifies the field for the returned language code.@@ -136,7 +138,8 @@ Use this in combination with `langid.map` to ensure that you only index document`langid.map`::Enables field name mapping.-If `true`, Solr will map field names for all fields listed in `langid.fl`. The default is `false`.+If `true`, Solr will map field names for all fields listed in `langid.fl`.+The default is `false`.`langid.map.fl`::A comma-separated list of fields for `langid.map` that is different than the fields specified in `langid.fl`.@@ -171,7 +174,8 @@ A list defined with this parameter will override any configuration set with `lanBy default, fields are mapped as <field>_<language>. To change this pattern, you can specify a Java regular expression in this parameter.`langid.map.replace`::-By default, fields are mapped as `<field>_<language>`. To change this pattern, you can specify a Java replace in this parameter.+By default, fields are mapped as `<field>_<language>`.+To change this pattern, you can specify a Java replace in this parameter.`langid.enforceSchema`::If `false`, the `langid` processor does not validate field names against your schema.diff --git a/solr/solr-ref-guide/src/learning-to-rank.adoc b/solr/solr-ref-guide/src/learning-to-rank.adocindex 378698595ef..baf5a0ac746 100644--- a/solr/solr-ref-guide/src/learning-to-rank.adoc+++ b/solr/solr-ref-guide/src/learning-to-rank.adoc@@ -18,21 +18,25 @@With the *Learning To Rank* (or *LTR* for short) contrib module you can configure and run machine learned ranking models in Solr.-The module also supports feature extraction inside Solr. The only thing you need to do outside Solr is train your own ranking model.+The module also supports feature extraction inside Solr.+The only thing you need to do outside Solr is train your own ranking model.== Learning to Rank Concepts=== Re-Ranking-Re-Ranking allows you to run a simple query for matching documents and then re-rank the top N documents using the scores from a different, more complex query. This page describes the use of *LTR* complex queries, information on other rank queries included in the Solr distribution can be found on the <<query-re-ranking.adoc#,Query Re-Ranking>> page.+Re-Ranking allows you to run a simple query for matching documents and then re-rank the top N documents using the scores from a different, more complex query.+This page describes the use of *LTR* complex queries, information on other rank queries included in the Solr distribution can be found on the <<query-re-ranking.adoc#,Query Re-Ranking>> page.=== Learning To Rank Models-In information retrieval systems, https://en.wikipedia.org/wiki/Learning_to_rank[Learning to Rank] is used to re-rank the top N retrieved documents using trained machine learning models. The hope is that such sophisticated models can make more nuanced ranking decisions than standard ranking functions like https://en.wikipedia.org/wiki/Tf%E2%80%93idf[TF-IDF] or https://en.wikipedia.org/wiki/Okapi_BM25[BM25].+In information retrieval systems, https://en.wikipedia.org/wiki/Learning_to_rank[Learning to Rank] is used to re-rank the top N retrieved documents using trained machine learning models.+The hope is that such sophisticated models can make more nuanced ranking decisions than standard ranking functions like https://en.wikipedia.org/wiki/Tf%E2%80%93idf[TF-IDF] or https://en.wikipedia.org/wiki/Okapi_BM25[BM25].==== Ranking Model-A ranking model computes the scores used to rerank documents. Irrespective of any particular algorithm or implementation, a ranking model's computation can use three types of inputs:+A ranking model computes the scores used to rerank documents.+Irrespective of any particular algorithm or implementation, a ranking model's computation can use three types of inputs:* parameters that represent the scoring algorithm* features that represent the document being scored@@ -47,17 +51,21 @@ Interleaving is an approach to Online Search Quality evaluation that allows to c==== Feature-A feature is a value, a number, that represents some quantity or quality of the document being scored or of the query for which documents are being scored. For example documents often have a 'recency' quality and 'number of past purchases' might be a quantity that is passed to Solr as part of the search query.+A feature is a value, a number, that represents some quantity or quality of the document being scored or of the query for which documents are being scored.+For example documents often have a 'recency' quality and 'number of past purchases' might be a quantity that is passed to Solr as part of the search query.==== Normalizer-Some ranking models expect features on a particular scale. A normalizer can be used to translate arbitrary feature values into normalized values e.g., on a 0..1 or 0..100 scale.+Some ranking models expect features on a particular scale.+A normalizer can be used to translate arbitrary feature values into normalized values e.g., on a 0..1 or 0..100 scale.=== Training Models==== Feature Engineering-The LTR contrib module includes several feature classes as well as support for custom features. Each feature class's javadocs contain an example to illustrate use of that class. The process of https://en.wikipedia.org/wiki/Feature_engineering[feature engineering] itself is then entirely up to your domain expertise and creativity.+The LTR contrib module includes several feature classes as well as support for custom features.+Each feature class's javadocs contain an example to illustrate use of that class.+The process of https://en.wikipedia.org/wiki/Feature_engineering[feature engineering] itself is then entirely up to your domain expertise and creativity.[cols=",,,",options="header",]|===@@ -87,7 +95,10 @@ The ltr contrib module includes a <<document-transformers.adoc#,[features>> tran==== Feature Selection and Model Training-Feature selection and model training take place offline and outside Solr. The ltr contrib module supports two generalized forms of models as well as custom models. Each model class's javadocs contain an example to illustrate configuration of that class. In the form of JSON files your trained model or models (e.g., different models for different customer geographies) can then be directly uploaded into Solr using provided REST APIs.+Feature selection and model training take place offline and outside Solr.+The ltr contrib module supports two generalized forms of models as well as custom models.+Each model class's javadocs contain an example to illustrate configuration of that class.+In the form of JSON files your trained model or models (e.g., different models for different customer geographies) can then be directly uploaded into Solr using provided REST APIs.[cols=",,",options="header",]|===@@ -461,7 +472,8 @@ Read more about model evolution in the <<LTR Lifecycle>> section of this page.=== Training Example-Example training data and a demo `train_and_upload_demo_model.py` script can be found in the `solr/contrib/ltr/example` folder in the https://gitbox.apache.org/repos/asf?p=lucene-solr.git;a=tree;f=solr/contrib/ltr/example[Apache lucene-solr Git repository] (mirrored on https://github.com/apache/lucene-solr/tree/releases/lucene-solr/{solr-docs-version}.0/solr/contrib/ltr/example[github.com]). This example folder is not shipped in the Solr binary release.+Example training data and a demo `train_and_upload_demo_model.py` script can be found in the `solr/contrib/ltr/example` folder in the https://gitbox.apache.org/repos/asf?p=lucene-solr.git;a=tree;f=solr/contrib/ltr/example[Apache lucene-solr Git repository] (mirrored on https://github.com/apache/lucene-solr/tree/releases/lucene-solr/{solr-docs-version}.0/solr/contrib/ltr/example[github.com]).+This example folder is not shipped in the Solr binary release.== Installation of LTR@@ -473,7 +485,8 @@ Learning-To-Rank is a contrib module and therefore its plugins must be configure=== Minimum Requirements-* Include the required contrib JARs. Note that by default paths are relative to the Solr core so they may need adjustments to your configuration, or an explicit specification of the `$solr.install.dir`.+* Include the required contrib JARs.+Note that by default paths are relative to the Solr core so they may need adjustments to your configuration, or an explicit specification of the `$solr.install.dir`.+[source,xml]----@@ -519,7 +532,8 @@ Learning-To-Rank is a contrib module and therefore its plugins must be configure==== LTRThreadModule-A thread module can be configured for the query parser and/or the transformer to parallelize the creation of feature weights. For details, please refer to the {solr-javadocs}/contrib/ltr/org/apache/solr/ltr/LTRThreadModule.html[LTRThreadModule] javadocs.+A thread module can be configured for the query parser and/or the transformer to parallelize the creation of feature weights.+For details, please refer to the {solr-javadocs}/contrib/ltr/org/apache/solr/ltr/LTRThreadModule.html[LTRThreadModule] javadocs.==== Feature Vector Customization@@ -543,7 +557,8 @@ How does Solr Learning-To-Rank work under the hood?::Please refer to the `ltr` {solr-javadocs}/contrib/ltr/org/apache/solr/ltr/package-summary.html[javadocs] for an implementation overview.How could I write additional models and/or features?::-Contributions for further models, features, normalizers and interleaving algorithms are welcome. Related links:+Contributions for further models, features, normalizers and interleaving algorithms are welcome.+Related links:+* {solr-javadocs}/contrib/ltr/org/apache/solr/ltr/model/LTRScoringModel.html[LTRScoringModel javadocs]* {solr-javadocs}/contrib/ltr/org/apache/solr/ltr/feature/Feature.html[Feature javadocs]@@ -612,7 +627,8 @@ curl -XDELETE 'http://localhost:8983/solr/techproducts/schema/feature-store/curr==== Using Large Models-With SolrCloud, large models may fail to upload due to the limitation of ZooKeeper's buffer. In this case, `DefaultWrapperModel` may help you to separate the model definition from uploaded file.+With SolrCloud, large models may fail to upload due to the limitation of ZooKeeper's buffer.+In this case, `DefaultWrapperModel` may help you to separate the model definition from uploaded file.Assuming that you consider to use a large model placed at `/path/to/models/myModel.json` through `DefaultWrapperModel`.diff --git a/solr/solr-ref-guide/src/loading.adoc b/solr/solr-ref-guide/src/loading.adocindex 86bfa267531..6f3454a8f73 100644--- a/solr/solr-ref-guide/src/loading.adoc+++ b/solr/solr-ref-guide/src/loading.adoc@@ -22,7 +22,8 @@ These functions are designed to cut down the time spent on data preparation and== Reading FilesThe `cat` function can be used to read files under the *userfiles* directory in-`$SOLR_HOME`. The `cat` function takes two parameters.+`$SOLR_HOME`.+The `cat` function takes two parameters.The first parameter is a comma-delimited list of paths.If the path list contains directories, `cat` will crawl all the files in the directory and sub-directories.@@ -81,12 +82,10 @@ When this expression is sent to the `/stream` handler it responds with:== Parsing CSV and TSV Files-The `parseCSV` and `parseTSV` functions wrap the `cat` function and parse CSV-(comma separated values) and TSV (tab separated values). Both of these functions-expect a CSV or TSV header record at the beginning of each file.+The `parseCSV` and `parseTSV` functions wrap the `cat` function and parse CSV (comma separated values) and TSV (tab separated values).+Both of these functions expect a CSV or TSV header record at the beginning of each file.-Both `parseCSV` and `parseTSV` emit tuples with the header values mapped to their-corresponding values in each line.+Both `parseCSV` and `parseTSV` emit tuples with the header values mapped to their corresponding values in each line.[source,text]@@ -144,28 +143,24 @@ When this expression is sent to the `/stream` handler it responds with:== Visualizing-Once that data has been parsed into tuples with `parseCSV` or `parseTSV` it can be-visualized using Zeppelin-Solr.+Once that data has been parsed into tuples with `parseCSV` or `parseTSV` it can be visualized using Zeppelin-Solr.The example below shows the output of the `parseCSV` function visualized as a table.image::images/math-expressions/csvtable.png[]Columns from the table can then be visualized using one of Apache Zeppelin's-visualizations. The example below shows a scatter plot of the `petal_length` and `petal_width`-grouped by `species`.+visualizations.+The example below shows a scatter plot of the `petal_length` and `petal_width` grouped by `species`.image::images/math-expressions/csv.png[]== Selecting Fields and Field Types-The `select` function can be used to select specific fields from-the CSV file and map them to new field names for indexing.+The `select` function can be used to select specific fields from the CSV file and map them to new field names for indexing.-Fields in the CSV file can be mapped to field names with-dynamic field suffixes. This approach allows for fine grain-control over schema field types without having to make any-changes to schema files.+Fields in the CSV file can be mapped to field names with dynamic field suffixes.+This approach allows for fine grain control over schema field types without having to make any changes to schema files.Below is an example of selecting fields and mapping themto specific field types.@@ -192,16 +187,15 @@ The section below shows some useful transformations that can be applied while an=== Unique IDsBoth `parseCSV` and `parseTSV` emit an *id* field if one is not present in the data already.-The *id* field is a concatenation of the file path and the line number. This is a-convenient way to ensure that records have consistent ids if an id+The *id* field is a concatenation of the file path and the line number.+This is a convenient way to ensure that records have consistent ids if an idis not present in the file.You can also map any fields in the file to the id field using the `select` function.The `concat` function can be used to concatenate two or more fields in the file-to create an id. Or the `uuid` function can be used to create a random unique id. If-the `uuid` function is used the data cannot be reloaded without first deleting-the data, as the `uuid` function does not produce the same id for each document-on subsequent loads.+to create an id.+Or the `uuid` function can be used to create a random unique id.+If the `uuid` function is used the data cannot be reloaded without first deleting the data, as the `uuid` function does not produce the same id for each document on subsequent loads.Below is an example using the `concat` function to create a new id.@@ -213,9 +207,8 @@ image::images/math-expressions/selectuuid.png[]=== Record Numbers-The `recNum` function can be used inside of a `select` function to add a record number-to each tuple. The record number is useful for tracking location in the result set-and can be used for filtering strategies such as skipping, paging and striding described in+The `recNum` function can be used inside of a `select` function to add a record number to each tuple.+The record number is useful for tracking location in the result set and can be used for filtering strategies such as skipping, paging and striding described inthe <<Filtering Results>> section below.The example below shows the syntax of the `recNum` function:@@ -260,8 +253,8 @@ When this expression is sent to the `/stream` handler it responds with:Then we can use the `dateTime` function to format the datetime andmap it to a Solr date field.-The `dateTime` function takes three parameters. The field in the data-with the date string, a template to parse the date using a Java https://docs.oracle.com/javase/9/docs/api/java/text/SimpleDateFormat.html[`SimpleDateFormat` template],+The `dateTime` function takes three parameters.+The field in the data with the date string, a template to parse the date using a Java https://docs.oracle.com/javase/9/docs/api/java/text/SimpleDateFormat.html[`SimpleDateFormat` template],and an optional time zone.If the time zone is not present the time zone defaults to GMT time unless@@ -299,8 +292,7 @@ When this expression is sent to the `/stream` handler it responds with:=== String Manipulation-The `upper`, `lower`, `split`, `valueAt`, `trim`, and `concat` functions can be used to manipulate-strings inside of the `select` function.+The `upper`, `lower`, `split`, `valueAt`, `trim`, and `concat` functions can be used to manipulate strings inside of the `select` function.The example below shows the `upper` function used to upper case the *species*field.@@ -308,8 +300,8 @@ field.image::images/math-expressions/selectupper.png[]The example below shows the `split` function which splits a field on-a delimiter. This can be used to create multi-value fields from fields-with an internal delimiter.+a delimiter.+This can be used to create multi-value fields from fields with an internal delimiter.The example below demonstrates this with a direct call tothe `/stream` handler:@@ -360,8 +352,7 @@ When this expression is sent to the `/stream` handler it responds with:}]}}-----The `valueAt` function can be used to select a specific index from-a split array.+The `valueAt` function can be used to select a specific index from a split array.image::images/math-expressions/valueat.png[]@@ -401,22 +392,21 @@ image::images/math-expressions/paging.png[]==== StridingThe `eq` and nested `mod` function can be used to stride through the data at specific-record number intervals. This allows for a sample to be taken at different intervals in the data-in a systematic way.+record number interval+This allows for a sample to be taken at different intervals in the data in a systematic way.image::images/math-expressions/striding.png[]==== Regex Matching-The `matches` function can be used to test if a field in the record matches a specific-regular expression. This provides a powerful *grep* like capability over the record set.+The `matches` function can be used to test if a field in the record matches a specific regular expression.+This provides a powerful *grep* like capability over the record set.image::images/math-expressions/matches.png[]=== Handling Nulls-In most cases nulls do not need to be handled directly unless there is specific logic needed-to handle nulls during the load.+In most cases nulls do not need to be handled directly unless there is specific logic needed to handle nulls during the load.The `select` function does not output fields that contain a null value.This means as nulls are encountered in the data the fields are not included in the tuples.@@ -465,19 +455,15 @@ function can be used to expand the list of tokens to a stream of tuples.There are a number of interesting use cases for the `analyze` function:* Previewing the output of different analyzers before indexing.-* Annotating documents with NLP generated tokens (entity extraction, noun phrases etc...)-before the documents reach the indexing pipeline.-This removes heavy NLP processing from the servers that may also be handling queries. It also allows-more compute resources to be applied to the NLP indexing then is available on the search cluster.-* Using the `cartesianProduct` function the analyzed tokens can be indexed as individual documents which allows-analyzed tokens to be searched and analyzed with Solr's aggregation and graph expressions.-* Also using `cartesianProduct` the analyzed tokens can be aggregated, analyzed and visualized using-streaming expressions directly before indexing occurs.+* Annotating documents with NLP generated tokens (entity extraction, noun phrases etc...) before the documents reach the indexing pipeline.+This removes heavy NLP processing from the servers that may also be handling queries.+It also allows more compute resources to be applied to the NLP indexing then is available on the search cluster.+* Using the `cartesianProduct` function the analyzed tokens can be indexed as individual documents which allows analyzed tokens to be searched and analyzed with Solr's aggregation and graph expressions.+* Also using `cartesianProduct` the analyzed tokens can be aggregated, analyzed and visualized using streaming expressions directly before indexing occurs.-Below is an example of the `analyze` function being applied to the *Resolution.Description*-field in the tuples. The *\_text_* fields analyzer is used to analyze the text and the-analyzed tokens are added to the documents in the *token_ss* field.+Below is an example of the `analyze` function being applied to the *Resolution.Description* field in the tuples.+The *\_text_* fields analyzer is used to analyze the text and the analyzed tokens are added to the documents in the *token_ss* field.[source,text]----@@ -533,9 +519,9 @@ When this expression is sent to the `/stream` handler it responds with:}-----The example below shows the `cartesianProduct` function expanding the analyzed terms in the `term_s` field into-their own documents. Notice that the other fields from the document are maintained with each term. This allows each term-to be indexed in a separate document so the relationships between terms and the other fields can be explored through+The example below shows the `cartesianProduct` function expanding the analyzed terms in the `term_s` field into their own documents.+Notice that the other fields from the document are maintained with each term.+This allows each term to be indexed in a separate document so the relationships between terms and the other fields can be explored throughgraph expressions or aggregations.diff --git a/solr/solr-ref-guide/src/logs.adoc b/solr/solr-ref-guide/src/logs.adocindex 807cc058609..b09d0b13ac3 100644--- a/solr/solr-ref-guide/src/logs.adoc+++ b/solr/solr-ref-guide/src/logs.adoc@@ -24,8 +24,7 @@ See the <<math-start.adoc#,Getting Started>> chapter to learn how to get started== Loading-The out-of-the-box Solr log format can be loaded into a Solr index using the `bin/postlogs` command line tool-located in the `bin/` directory of the Solr distribution.+The out-of-the-box Solr log format can be loaded into a Solr index using the `bin/postlogs` command line tool located in the `bin/` directory of the Solr distribution.NOTE: If working from the source distribution thedistribution must first be built before `postlogs` can be run.@@ -51,32 +50,25 @@ The example above will index all the log files under `/var/logs/solrlogs` to theLog exploration is often the first step in log analytics and visualization.-When working with unfamiliar installations exploration can be used to understand which collections are-covered in the logs, what shards and cores are in those collections and the types of operations being-performed on those collections.+When working with unfamiliar installations exploration can be used to understand which collections are covered in the logs, what shards and cores are in those collections and the types of operations being performed on those collections.-Even with familiar Solr installations exploration is still extremely-important while troubleshooting because it will often turn up surprises such as unknown errors or-unexpected admin or indexing operations.+Even with familiar Solr installations exploration is still extremely important while troubleshooting because it will often turn up surprises such as unknown errors or unexpected admin or indexing operations.=== Sampling-The first step in exploration is to take a random sample from the `logs` collection-with the `random` function.+The first step in exploration is to take a random sample from the `logs` collection with the `random` function.-In the example below the `random` function is run with one-parameter which is the name of the collection to sample.+In the example below the `random` function is run with one parameter which is the name of the collection to sample.image::images/math-expressions/logs-sample.png[]-The sample contains 500 random records with the their full field list. By looking-at this sample we can quickly learn about the *fields* available in the `logs` collection.+The sample contains 500 random records with the their full field list.+By looking at this sample we can quickly learn about the *fields* available in the `logs` collection.=== Time PeriodEach log record contains a time stamp in the `date_dt` field.-Its often useful to understand what time period the logs cover and how many log records have been-indexed.+Its often useful to understand what time period the logs cover and how many log records have been indexed.The `stats` function can be run to display this information.@@ -85,22 +77,18 @@ image::images/math-expressions/logs-dates.png[]=== Record Types-One of the key fields in the index is the `type_s` field which is the type of log-record.+One of the key fields in the index is the `type_s` field which is the type of log record.-The `facet` expression can be used to visualize the different types of log records and how many-records of each type are in the index.+The `facet` expression can be used to visualize the different types of log records and how many records of each type are in the index.image::images/math-expressions/logs-type.png[]=== Collections-Another important field is the `collection_s` field which is the collection that the-log record was generated from.+Another important field is the `collection_s` field which is the collection that the log record was generated from.-The `facet` expression can be used to visualize the different collections and how many log records-they generate.+The `facet` expression can be used to visualize the different collections and how many log records they generate.image::images/math-expressions/logs-collection.png[]@@ -114,11 +102,9 @@ image::images/math-expressions/logs-type-collection.png[]=== Time Series-The `timeseries` function can be used to visualize a time series for a specific time range-of the logs.+The `timeseries` function can be used to visualize a time series for a specific time range of the logs.-In the example below a time series is used to visualize the log record counts-at 15 second intervals.+In the example below a time series is used to visualize the log record counts at 15 second intervals.image::images/math-expressions/logs-time-series.png[]@@ -127,19 +113,16 @@ Then a burst of log activity occurs from minute 27 to minute 52.This is then followed by a large spike of log activity.-The example below breaks this down further by adding a query on the `type_s` field to only-visualize *query* activity in the log.+The example below breaks this down further by adding a query on the `type_s` field to only visualize *query* activity in the log.image::images/math-expressions/logs-time-series2.png[]-Notice the query activity accounts for more then half of the burst of log records between-21:27 and 21:52. But the query activity does not account for the large spike in-log activity that follows.+Notice the query activity accounts for more then half of the burst of log records between 21:27 and 21:52.+But the query activity does not account for the large spike in log activity that follows.-We can account for that spike by changing the search to include only *update*, *commit*,-and *deleteByQuery* records in the logs. We can also narrow by collection-so we know where these activities are taking place.+We can account for that spike by changing the search to include only *update*, *commit*, and *deleteByQuery* records in the logs.+We can also narrow by collection so we know where these activities are taking place.image::images/math-expressions/logs-time-series3.png[]@@ -150,10 +133,9 @@ better understanding of what's contained in the logs.== Query Counting-Distributed searches produce more than one log record for each query. There will be one *top level* log-record for-the top level distributed query and a *shard level* log record on one replica from each shard. There may also-be a set of *ids* queries to retrieve fields by id from the shards to complete the page of results.+Distributed searches produce more than one log record for each query.+There will be one *top level* log record for the top level distributed query and a *shard level* log record on one replica from each shard.+There may also be a set of *ids* queries to retrieve fields by id from the shards to complete the page of results.There are fields in the log index that can be used to differentiate between the three types of query records.@@ -170,16 +152,14 @@ image::images/math-expressions/query-top-level.png[]=== Shard Level Queries-To find all the shard level queries that are not IDs queries, adjust the query to limit results to logs with `distrib_s:false AND ids_s:false`-as follows:+To find all the shard level queries that are not IDs queries, adjust the query to limit results to logs with `distrib_s:false AND ids_s:false` as follows:image::images/math-expressions/query-shard-level.png[]=== ID Queries-To find all the *ids* queries, adjust the query to limit results to logs with `distrib_s:false AND ids_s:true`-as follows:+To find all the *ids* queries, adjust the query to limit results to logs with `distrib_s:false AND ids_s:true` as follows:image::images/math-expressions/query-ids.png[]@@ -197,8 +177,7 @@ There are number of powerful visualizations and statistical approaches for analyScatter plots can be used to visualize random samples of the `qtime_i`field.-The example below demonstrates a scatter plot of 500 random samples-from the `ptest1` collection of log records.+The example below demonstrates a scatter plot of 500 random samples from the `ptest1` collection of log records.In this example, `qtime_i` is plotted on the y-axis and the x-axis is simply a sequence to spread the query times out across the plot.@@ -211,8 +190,7 @@ From this scatter plot we can tell a number of important things about the query* The sample query times range from a low of 122 to a high of 643.* The mean appears to be just above 400 ms.-* The query times tend to cluster closer to the mean and become less frequent as they move away-from the mean.+* The query times tend to cluster closer to the mean and become less frequent as they move away from the mean.=== Highest QTime Scatter Plot@@ -234,8 +212,7 @@ From this plot we can see that the 500 highest query times start at 510ms and sl=== QTime Distribution-In this example a visualization is created which shows the-distribution of query times rounded to the nearest second.+In this example a visualization is created which shows the distribution of query times rounded to the nearest second.The example below starts by taking a random sample of 10000 log records with a `type_s`* of `query`.The results of the `random` function are assigned to the variable `a`.@@ -250,8 +227,7 @@ The result is set to variable `c`.The `round` function then rounds all elements of the query times vector to the nearest second.This means all query times less than 500ms will round to 0.-The `freqTable` function is then applied to the vector of query times rounded to-the nearest second.+The `freqTable` function is then applied to the vector of query times rounded to the nearest second.The resulting frequency table is shown in the visualization below.The x-axis is the number of seconds.@@ -274,11 +250,9 @@ Then a random sample of 10000 log records is drawn and set to variable `a`.The `col` function is then used to extract the `qtime_i` field from the sample results and this vector is set to variable `b`.The `percentile` function is then used to calculate the value at each percentile for the vector of query times.-The array of percentiles set to variable `p` tells the `percentile` function-which percentiles to calculate.+The array of percentiles set to variable `p` tells the `percentile` function which percentiles to calculate.-Then the `zplot` function is used to plot the *percentiles* on the x-axis and-the *query time* at each percentile on the y-axis.+Then the `zplot` function is used to plot the *percentiles* on the x-axis and the *query time* at each percentile on the y-axis.image::images/math-expressions/query-qq.png[]@@ -307,8 +281,7 @@ Therefore one slow node can be responsible for slow overall search time.The fields `core_s`, `replica_s` and `shard_s` are available in the log records.These fields allow average query time to be calculated by *core*, *replica* or *shard*.-The `core_s` field is particularly useful as its the most granular element and-the naming convention often includes the collection, shard and replica information.+The `core_s` field is particularly useful as its the most granular element and the naming convention often includes the collection, shard and replica information.The example below uses the `facet` function to calculate `avg(qtime_i)` by core.@@ -326,14 +299,13 @@ If query analysis shows that most queries are performing well but there are outlThe `q_s` and `q_t` fields both hold the value of the *q* parameter from Solr requests.The `q_s` field is a string field and the `q_t` field has been tokenized.-The `search` function can be used to return the top N slowest queries in the logs by sorting the results by `qtime_i desc`. the example-below demonstrates this:+The `search` function can be used to return the top N slowest queries in the logs by sorting the results by `qtime_i desc`.+The example below demonstrates this:image::images/math-expressions/slow-queries.png[]Once the queries have been retrieved they can be inspected and tried individually to determine if the query is consistently slow.-If the query is shown to be slow a plan to improve the query performance-can be devised.+If the query is shown to be slow a plan to improve the query performance can be devised.=== Commits@@ -378,7 +350,8 @@ in max `qtime_i`.== Errors-The log index will contain any error records found in the logs. Error records will have a `type_s` field value of `error`.+The log index will contain any error records found in the logs.+Error records will have a `type_s` field value of `error`.The example below searches for error records:diff --git a/solr/solr-ref-guide/src/luke-request-handler.adoc b/solr/solr-ref-guide/src/luke-request-handler.adocindex bf799ebe2ba..f3a2aa396a0 100644--- a/solr/solr-ref-guide/src/luke-request-handler.adoc+++ b/solr/solr-ref-guide/src/luke-request-handler.adoc@@ -17,12 +17,17 @@// under the License.The Luke Request Handler offers programmatic access to the information provided on the <<schema-browser-screen#schema-browser-screen,Schema Browser>> page of the Admin UI.-It is modeled after the Luke, the Lucene Index Browser by Andrzej Bialecki. It is an implicit handler, so you don't need to define it.+It is modeled after the Luke, the Lucene Index Browser by Andrzej Bialecki.+It is an implicit handler, so you don't need to define it.The Luke Request Handler accepts the following parameters:`show`::-The data about the index to include in the response. Options are `schema`, `index`, `doc`, `all`. `index` describes the high level details about the index. `schema` returns details about the `schema` plus the `index` data. `doc` works in conjunction with `docId` or `id` parameters and returns details about a specific document plus the `index` data.+The data about the index to include in the response.+Options are `schema`, `index`, `doc`, `all`.+* `index` describes the high level details about the index.+* `schema` returns details about the `schema` plus the `index` data.+* `doc` works in conjunction with `docId` or `id` parameters and returns details about a specific document plus the `index` data.`id`::Get a document using the uniqueKeyField specified in schema.xml.@@ -31,13 +36,16 @@ Get a document using the uniqueKeyField specified in schema.xml.Get a document using a Lucene documentID.`fl`::-Limit the returned values to a set of fields. This is useful if you want to increase the `numTerms` and don't want a massive response.+Limit the returned values to a set of fields.+This is useful if you want to increase the `numTerms` and don't want a massive response.`numTerms`::-How many top terms for each field. The default is 10.+How many top terms for each field.+The default is `10`.`includeIndexFieldFlags`::-Choose whether /luke should return the index-flags for each field. Fetching and returning the index-flags for each field in the index has non-zero cost, and can slow down requests to /luke.+Choose whether /luke should return the index-flags for each field.+Fetching and returning the index-flags for each field in the index has non-zero cost, and can slow down requests to /luke.== LukeRequestHandler Examplesdiff --git a/solr/solr-ref-guide/src/machine-learning.adoc b/solr/solr-ref-guide/src/machine-learning.adocindex 6ddb56cd139..c7952a63830 100644--- a/solr/solr-ref-guide/src/machine-learning.adoc+++ b/solr/solr-ref-guide/src/machine-learning.adoc@@ -33,8 +33,7 @@ There are six distance measure functions that return a function that performs th* `cosine`* `haversineMeters` (Geospatial distance measure)-The distance measure functions can be used with all machine learning functions-that support distance measures.+The distance measure functions can be used with all machine learning functions that support distance measures.Below is an example for computing Euclidean distance for two numeric arrays:@@ -94,53 +93,43 @@ When this expression is sent to the `/stream` handler it responds with:=== Distance Matrices-Distance matrices are powerful tools for visualizing the distance-between two or more-vectors.+Distance matrices are powerful tools for visualizing the distance between two or more vectors.-The `distance` function builds a distance matrix-if a matrix is passed as the parameter. The distance matrix is computed for the *columns*-of the matrix.+The `distance` function builds a distance matrix if a matrix is passed as the parameter.+The distance matrix is computed for the *columns* of the matrix.The example below demonstrates the power of distance matrices combined with 2 dimensional faceting.-In this example the `facet2D` function is used to generate a two dimensional facet aggregation-over the fields `complaint_type_s` and `zip_s` from the `nyc311` complaints database.+In this example the `facet2D` function is used to generate a two dimensional facet aggregation over the fields `complaint_type_s` and `zip_s` from the `nyc311` complaints database.The *top 20* complaint types and the *top 25* zip codes for each complaint type are aggregated.The result is a stream of tuples each containing the fields `complaint_type_s`, `zip_s` and the count for the pair.-The `pivot` function is then used to pivot the fields into a *matrix* with the `zip_s`-field as the *rows* and the `complaint_type_s` field as the *columns*. The `count(*)` field populates-the values in the cells of the matrix.+The `pivot` function is then used to pivot the fields into a *matrix* with the `zip_s` field as the *rows* and the `complaint_type_s` field as the *columns*.+The `count(*)` field populates the values in the cells of the matrix.-The `distance` function is then used to compute the distance matrix for the columns-of the matrix using `cosine` distance. This produces a distance matrix-that shows distance between complaint types based on the zip codes they appear in.+The `distance` function is then used to compute the distance matrix for the columns of the matrix using `cosine` distance.+This produces a distance matrix that shows distance between complaint types based on the zip codes they appear in.-Finally the `zplot` function is used to plot the distance matrix as a heat map. Notice that the-heat map has been configured so that the intensity of color increases as the distance between vectors-decreases.+Finally the `zplot` function is used to plot the distance matrix as a heat map.+Notice that the heat map has been configured so that the intensity of color increases as the distance between vectors decreases.image::images/math-expressions/distance.png[]-The heat map is interactive, so mousing over one of the cells pops up the values-for the cell.+The heat map is interactive, so mousing over one of the cells pops up the values for the cell.image::images/math-expressions/distanceview.png[]-Notice that HEAT/HOT WATER and UNSANITARY CONDITION complaints have a cosine distance of .1 (rounded to the nearest-tenth).+Notice that HEAT/HOT WATER and UNSANITARY CONDITION complaints have a cosine distance of .1 (rounded to the nearest tenth).== K-Nearest Neighbor (KNN)The `knn` function searches the rows of a matrix with a search vector and-returns a matrix of the k-nearest neighbors. This allows for secondary vector-searches over result sets.+returns a matrix of the k-nearest neighbors.+This allows for secondary vector searches over result sets.-The `knn` function supports changing of the distance measure by providing one of the following-distance measure functions:+The `knn` function supports changing of the distance measure by providing one of the following distance measure functions:* `euclidean` (Default)* `manhattan`@@ -150,18 +139,15 @@ distance measure functions:* `haversineMeters` (Geospatial distance measure)The example below shows how to perform a secondary search over an aggregation-result set. The goal of the example is to find zip codes in the nyc311 complaint-database that have similar complaint types to the zip code 10280.+result set.+The goal of the example is to find zip codes in the nyc311 complaint database that have similar complaint types to the zip code 10280.-The first step in the example is to use the `facet2D` function to perform a two-dimensional aggregation over the `zip_s` and `complaint_type_s` fields. In the example-the top 119 zip codes and top 5 complaint types for each zip code are calculated-for the borough of Manhattan. The result is a list of tuples each containing-the `zip_s`, `complaint_type_s` and the `count(*)` for the combination.+The first step in the example is to use the `facet2D` function to perform a two-dimensional aggregation over the `zip_s` and `complaint_type_s` fields.+In the example the top 119 zip codes and top 5 complaint types for each zip code are calculated for the borough of Manhattan.+The result is a list of tuples each containing the `zip_s`, `complaint_type_s` and the `count(*)` for the combination.The list of tuples is then *pivoted* into a matrix with the `pivot` function.-The `pivot` function in this example returns a matrix with rows of zip codes-and columns of complaint types.+The `pivot` function in this example returns a matrix with rows of zip codes and columns of complaint types.The `count(*)` field from the tuples populates the cells of the matrix.This matrix will be used as the secondary search matrix.@@ -174,126 +160,102 @@ The `rowAt` function is then used to return the vector at that *index* from theThis vector is the *search vector*.Now that we have a matrix and search vector we can use the `knn` function to perform the search.-In the example the `knn` function searches the matrix with the search vector with a K of 5, using-*cosine* distance. Cosine distance is useful for comparing sparse vectors which is the case in this-example. The `knn` function returns a matrix with the top 5 nearest neighbors to the search vector.+In the example the `knn` function searches the matrix with the search vector with a K of 5, using *cosine* distance.+Cosine distance is useful for comparing sparse vectors which is the case in this+example.+The `knn` function returns a matrix with the top 5 nearest neighbors to the search vector.The `knn` function populates the row and column labels of the return matrix andalso adds a vector of *distances* for each row as an attribute to the matrix.-In the example the `zplot` function extracts the row labels and-the distance vector with the `getRowLabels` and `getAttribute` functions.-The `topFeatures` function is used to extract-the top 5 column labels for each zip code vector, based on the counts for each-column. Then `zplot` outputs the data in a format that can be visualized in+In the example the `zplot` function extracts the row labels and the distance vector with the `getRowLabels` and `getAttribute` functions.+The `topFeatures` function is used to extract the top 5 column labels for each zip code vector, based on the counts for each column.+Then `zplot` outputs the data in a format that can be visualized ina table with Zeppelin-Solr.image::images/math-expressions/knn.png[]-The table above shows each zip code returned by the `knn` function along-with the list of complaints and distances. These are the zip codes that are most similar-to the 10280 zip code based on their top 5 complaint types.+The table above shows each zip code returned by the `knn` function along with the list of complaints and distances.+These are the zip codes that are most similar to the 10280 zip code based on their top 5 complaint types.== K-Nearest Neighbor RegressionK-nearest neighbor regression is a non-linear, bivariate and multivariate regression method.-KNN regression is a lazy learning-technique which means it does not fit a model to the training set in advance. Instead the-entire training set of observations and outcomes are held in memory and predictions are made-by averaging the outcomes of the k-nearest neighbors.+KNN regression is a lazy learning technique which means it does not fit a model to the training set in advance.+Instead the entire training set of observations and outcomes are held in memory and predictions are made by averaging the outcomes of the k-nearest neighbors.The `knnRegress` function is used to perform nearest neighbor regression.-=== 2D Non-Linear RegressionThe example below shows the *regression plot* for KNN regression applied to a 2D scatter plot.-In this example the `random` function is used to draw 500 random samples from the `logs` collection-containing two fields `filesize_d` and `eresponse_d`. The sample is then vectorized with the-`filesize_d` field stored in a vector assigned to variable *x* and the `eresponse_d` vector stored in-variable `y`. The `knnRegress` function is then applied with `20` as the nearest neighbor parameter,-which returns a KNN function which can be used to predict values.+In this example the `random` function is used to draw 500 random samples from the `logs` collection containing two fields `filesize_d` and `eresponse_d`.+The sample is then vectorized with the `filesize_d` field stored in a vector assigned to variable *x* and the `eresponse_d` vector stored in+variable `y`.+The `knnRegress` function is then applied with `20` as the nearest neighbor parameter, which returns a KNN function which can be used to predict values.The `predict` function is then called on the KNN function to predict values for the original `x` vector.Finally `zplot` is used to plot the original `x` and `y` vectors along with the predictions.image::images/math-expressions/knnRegress.png[]-Notice that the regression plot shows a non-linear relations ship between the `filesize_d`-field and the `eresponse_d` field. Also note that KNN regression-plots a non-linear curve through the scatter plot. The larger the size-of K (nearest neighbors), the smoother the line.+Notice that the regression plot shows a non-linear relations ship between the `filesize_d` field and the `eresponse_d` field.+Also note that KNN regression plots a non-linear curve through the scatter plot.+The larger the size of K (nearest neighbors), the smoother the line.=== Multivariate Non-Linear Regression-The `knnRegress` function is also a powerful and flexible tool for-multi-variate non-linear regression.+The `knnRegress` function is also a powerful and flexible tool for multi-variate non-linear regression.-In the example below a multi-variate regression is performed using-a database designed for analyzing and predicting wine quality. The-database contains nearly 1600 records with 9 predictors of wine quality:-pH, alcohol, fixed_acidity, sulphates, density, free_sulfur_dioxide,-volatile_acidity, citric_acid, residual_sugar. There is also a field-called quality assigned to each wine ranging-from 3 to 8.+In the example below a multi-variate regression is performed using a database designed for analyzing and predicting wine quality.+The database contains nearly 1600 records with 9 predictors of wine quality:+pH, alcohol, fixed_acidity, sulphates, density, free_sulfur_dioxide, volatile_acidity, citric_acid, residual_sugar.+There is also a field called quality assigned to each wine ranging from 3 to 8.-KNN regression can be used to predict wine quality for vectors containing-the predictor values.+KNN regression can be used to predict wine quality for vectors containing the predictor values.-In the example a search is performed on the `redwine` collection to-return all the rows in the database of observations. Then the quality field and-predictor fields are read into vectors and set to variables.+In the example a search is performed on the `redwine` collection to return all the rows in the database of observations.+Then the quality field and predictor fields are read into vectors and set to variables.-The predictor variables are added as rows to a matrix which is-transposed so each row in the matrix contains one observation with the 9-predictor values.+The predictor variables are added as rows to a matrix which is transposed so each row in the matrix contains one observation with the 9 predictor values.This is our observation matrix which is assigned to the variable `obs`.Then the `knnRegress` function regresses the observations with quality outcomes.-The value for K is set to 5 in the example, so the average quality of the 5-nearest neighbors will be used to calculate the quality.+The value for K is set to 5 in the example, so the average quality of the 5 nearest neighbors will be used to calculate the quality.-The `predict` function is then used to generate a vector of predictions-for the entire observation set. These predictions will be used to determine-how well the KNN regression performed over the observation data.+The `predict` function is then used to generate a vector of predictions for the entire observation set.+These predictions will be used to determine how well the KNN regression performed over the observation data.-The error, or *residuals*, for the regression are then calculated by-subtracting the *predicted* quality from the *observed* quality.-The `ebeSubtract` function is used to perform the element-by-element-subtraction between the two vectors.+The error, or *residuals*, for the regression are then calculated by subtracting the *predicted* quality from the *observed* quality.+The `ebeSubtract` function is used to perform the element-by-element subtraction between the two vectors.-Finally the `zplot` function formats the predictions and errors for-for the visualization of the *residual plot*.+Finally the `zplot` function formats the predictions and errors for the visualization of the *residual plot*.image::images/math-expressions/redwine1.png[]-The residual plot plots the *predicted* values on the x-axis and the *error* for the-prediction on the y-axis. The scatter plot shows how the errors-are distributed across the full range of predictions.+The residual plot plots the *predicted* values on the x-axis and the *error* for the prediction on the y-axis.+The scatter plot shows how the errors are distributed across the full range of predictions.-The residual plot can be interpreted to understand how the KNN regression performed on the-training data.+The residual plot can be interpreted to understand how the KNN regression performed on the training data.-* The plot shows the prediction error appears to be fairly evenly distributed-above and below zero. The density of the errors increases as it approaches zero. The-bubble size reflects the density of errors at the specific point in the plot.+* The plot shows the prediction error appears to be fairly evenly distributed above and below zero.+The density of the errors increases as it approaches zero.+The bubble size reflects the density of errors at the specific point in the plot.This provides an intuitive feel for the distribution of the model's error.* The plot also visualizes the variance of the error across the range of-predictions. This provides an intuitive understanding of whether the KNN predictions-will have similar error variance across the full range predictions.+predictions.+This provides an intuitive understanding of whether the KNN predictions will have similar error variance across the full range predictions.-The residuals can also be visualized using a histogram to better understand-the shape of the residuals distribution. The example below shows the same KNN-regression as above with a plot of the distribution of the errors.+The residuals can also be visualized using a histogram to better understand the shape of the residuals distribution.+The example below shows the same KNN regression as above with a plot of the distribution of the errors.-In the example the `zplot` function is used to plot the `empiricalDistribution`-function of the residuals, with an 11 bin histogram.+In the example the `zplot` function is used to plot the `empiricalDistribution` function of the residuals, with an 11 bin histogram.image::images/math-expressions/redwine2.png[]-Notice that the errors follow a bell curve centered close to 0. From this plot-we can see the probability of getting prediction errors between -1 and 1 is quite high.+Notice that the errors follow a bell curve centered close to 0.+From this plot we can see the probability of getting prediction errors between -1 and 1 is quite high.*Additional KNN Regression Parameters*@@ -319,9 +281,8 @@ Sample syntax:r=knnRegress(obs, quality, 5, robust="true"),-----. The `scale` named parameter can be used to scale the columns of the observations and search vectors-at prediction time. This can improve the performance of the KNN regression when the feature columns-are at different scales causing the distance calculations to be place too much weight on the larger columns.+. The `scale` named parameter can be used to scale the columns of the observations and search vectors at prediction time.+This can improve the performance of the KNN regression when the feature columns are at different scales causing the distance calculations to be place too much weight on the larger columns.+Sample syntax:+@@ -332,22 +293,20 @@ r=knnRegress(obs, quality, 5, scale="true"),== knnSearch-The `knnSearch` function returns the k-nearest neighbors-for a document based on text similarity.+The `knnSearch` function returns the k-nearest neighbors for a document based on text similarity.Under the covers the `knnSearch` function uses Solr's <<other-parsers.adoc#more-like-this-query-parser,More Like This>> query parser plugin.This capability uses the search engine's query, term statistics, scoring, and ranking capability to perform a fast, nearest neighbor search for similar documents over large distributed indexes.The results of this search can be used directly or provide *candidates* for machine learning operations such as a secondary KNN vector search.-The example below shows the `knnSearch` function on a movie reviews data set. The search returns the 50 documents most similar to a specific document ID (`83e9b5b0...`) based on the similarity of the `review_t` field.+The example below shows the `knnSearch` function on a movie reviews data set.+The search returns the 50 documents most similar to a specific document ID (`83e9b5b0...`) based on the similarity of the `review_t` field.The `mindf` and `maxdf` specify the minimum and maximum document frequency of the terms used to perform the search.These parameters can make the query faster by eliminating high frequency terms and also improves accuracy by removing noise terms from the search.image::images/math-expressions/knnSearch.png[]-NOTE: In this example the `select`-function is used to truncate the review in the output to 220 characters to make it easier-to read in a table.+NOTE: In this example the `select` function is used to truncate the review in the output to 220 characters to make it easier to read in a table.== DBSCAN@@ -363,25 +322,25 @@ DBSCAN uses two parameters to filter result sets to clusters of specific densityThe `zplot` function has direct support for plotting 2D clusters by using the `clusters` named parameter.-The example below uses DBSCAN clustering and cluster visualization to find-the *hot spots* on a map for rat sightings in the NYC 311 complaints database.+The example below uses DBSCAN clustering and cluster visualization to find the *hot spots* on a map for rat sightings in the NYC 311 complaints database.-In this example the `random` function draws a sample of records from the `nyc311` collection where-the complaint description matches "rat sighting" and latitude is populated in the record.+In this example the `random` function draws a sample of records from the `nyc311` collection where the complaint description matches "rat sighting" and latitude is populated in the record.The latitude and longitude fields are then vectorized and added as rows to a matrix.-The matrix is transposed so each row contains a single latitude, longitude-point.+The matrix is transposed so each row contains a single latitude, longitude point.The `dbscan` function is then used to cluster the latitude and longitude points.Notice that the `dbscan` function in the example has four parameters.* `obs` : The observation matrix of lat/lon points-* `eps` : The distance between points to be considered a cluster. 100 meters in the example.+* `eps` : The distance between points to be considered a cluster.+100 meters in the example.-* `min points`: The minimum points in a cluster for the cluster to be returned by the function. `5` in the example.+* `min points`: The minimum points in a cluster for the cluster to be returned by the function.+`5` in the example.* `distance measure`: An optional distance measure used to determine the-distance between points. The default is Euclidean distance.+distance between points.+The default is Euclidean distance.The example uses `haversineMeters` which returns the distance in meters which is much more meaningful for geospatial use cases.Finally, the `zplot` function is used to visualize the clusters on a map with Zeppelin-Solr.@@ -390,8 +349,8 @@ The map below has been zoomed to a specific area of Brooklyn with a high densityimage::images/math-expressions/dbscan1.png[]Notice in the visualization that only 1019 points were returned from the 5000 samples.-This is the power of the DBSCAN algorithm to filter records that don't match the criteria-of a cluster. The points that are plotted all belong to clearly defined clusters.+This is the power of the DBSCAN algorithm to filter records that don't match the criteria of a cluster.+The points that are plotted all belong to clearly defined clusters.The map visualization can be zoomed further to explore the locations of specific clusters.The example below shows a zoom into an area of dense clusters.@@ -402,74 +361,62 @@ image::images/math-expressions/dbscan2.png[]== K-Means ClusteringThe `kmeans` functions performs k-means clustering of the rows of a matrix.-Once the clustering has been completed there are a number of useful functions available-for examining and visualizing the clusters and centroids.+Once the clustering has been completed there are a number of useful functions available for examining and visualizing the clusters and centroids.=== Clustered Scatter Plot-In this example we'll again be clustering 2D lat/lon points of rat sightings. But unlike the DBSCAN example, k-means clustering-does not on its own-perform any noise reduction. So in order to reduce the noise a smaller random sample is selected from the data than was used-for the DBSCAN example.+In this example we'll again be clustering 2D lat/lon points of rat sightings.+But unlike the DBSCAN example, k-means clustering does not on its own perform any noise reduction.+So in order to reduce the noise a smaller random sample is selected from the data than was used for the DBSCAN example.We'll see that sampling itself is a powerful noise reduction tool which helps visualize the cluster density.-This is because there is a higher probability that samples will be drawn from higher density clusters and a lower-probability that samples will be drawn from lower density clusters.+This is because there is a higher probability that samples will be drawn from higher density clusters and a lower probability that samples will be drawn from lower density clusters.-In this example the `random` function draws a sample of 1500 records from the `nyc311` (complaints database) collection where-the complaint description matches "rat sighting" and latitude is populated in the record. The latitude and longitude fields-are then vectorized and added as rows to a matrix. The matrix is transposed so each row contains a single latitude, longitude-point. The `kmeans` function is then used to cluster the latitude and longitude points into 21 clusters.+In this example the `random` function draws a sample of 1500 records from the `nyc311` (complaints database) collection where the complaint description matches "rat sighting" and latitude is populated in the record.+The latitude and longitude fields are then vectorized and added as rows to a matrix.+The matrix is transposed so each row contains a single latitude, longitude point.+The `kmeans` function is then used to cluster the latitude and longitude points into 21 clusters.Finally, the `zplot` function is used to visualize the clusters as a scatter plot.image::images/math-expressions/2DCluster1.png[]-The scatter plot above shows each lat/lon point plotted on a Euclidean plain with longitude on the-x-axis and-latitude on the y-axis. The plot is dense enough so the outlines of the different boroughs are visible-if you know the boroughs of New York City.-+The scatter plot above shows each lat/lon point plotted on a Euclidean plain with longitude on the x-axis and latitude on the y-axis.+The plot is dense enough so the outlines of the different boroughs are visible if you know the boroughs of New York City.-Each cluster is shown in a different color. This plot provides interesting-insight into the densities of rat sightings throughout the five boroughs of New York City. For-example it highlights a cluster of dense sightings in Brooklyn at cluster1+Each cluster is shown in a different color.+This plot provides interesting insight into the densities of rat sightings throughout the five boroughs of New York City.+For example it highlights a cluster of dense sightings in Brooklyn at cluster1surrounded by less dense but still high activity clusters.=== Plotting the Centroids-The centroids of each cluster can then be plotted on a map to visualize the center of the-clusters. In the example below the centroids are extracted from the clusters using the `getCentroids`-function, which returns a matrix of the centroids.--The centroids matrix contains 2D lat/lon points. The `colAt` function can then be used-to extract the latitude and longitude columns by index from the matrix so they can be-plotted with `zplot`. A map visualization is used below to display the centroids.+The centroids of each cluster can then be plotted on a map to visualize the center of the clusters.+In the example below the centroids are extracted from the clusters using the `getCentroids` function, which returns a matrix of the centroids.+The centroids matrix contains 2D lat/lon points.+The `colAt` function can then be used to extract the latitude and longitude columns by index from the matrix so they can be plotted with `zplot`.+A map visualization is used below to display the centroids.image::images/math-expressions/centroidplot.png[]-The map can then be zoomed to get a closer look at the centroids in the high density areas shown-in the cluster scatter plot.+The map can then be zoomed to get a closer look at the centroids in the high density areas shown in the cluster scatter plot.image::images/math-expressions/centroidzoom.png[]=== Phrase Extraction-K-means clustering produces centroids or *prototype* vectors which can be used to represent-each cluster. In this example the key features of the centroids are extracted-to represent the key phrases for clusters of TF-IDF term vectors.+K-means clustering produces centroids or *prototype* vectors which can be used to represent each cluster.+In this example the key features of the centroids are extracted to represent the key phrases for clusters of TF-IDF term vectors.NOTE: The example below works with TF-IDF _term vectors_.-The section <<term-vectors.adoc#,Text Analysis and Term Vectors>> offers-a full explanation of this features.+The section <<term-vectors.adoc#,Text Analysis and Term Vectors>> offers a full explanation of this features.In the example the `search` function returns documents where the `review_t` field matches the phrase "star wars".-The `select` function is run over the result set and applies the `analyze` function-which uses the Lucene/Solr analyzer attached to the schema field `text_bigrams` to re-analyze the `review_t`-field. This analyzer returns bigrams which are then annotated to documents in a field called `terms`.+The `select` function is run over the result set and applies the `analyze` function which uses the Lucene/Solr analyzer attached to the schema field `text_bigrams` to re-analyze the `review_t` field.+This analyzer returns bigrams which are then annotated to documents in a field called `terms`.The `termVectors` function then creates TD-IDF term vectors from the bigrams stored in the `terms` field.The `kmeans` function is then used to cluster the bigram term vectors into 5 clusters.@@ -544,15 +491,12 @@ When this expression is sent to the `/stream` handler it responds with:== Multi K-Means Clustering-K-means clustering will produce different outcomes depending on-the initial placement of the centroids. K-means is fast enough-that multiple trials can be performed so that the best outcome can be selected.+K-means clustering will produce different outcomes depending on the initial placement of the centroids.+K-means is fast enough that multiple trials can be performed so that the best outcome can be selected.-The `multiKmeans` function runs the k-means clustering algorithm for a given number of trials and selects the-best result based on which trial produces the lowest intra-cluster variance.+The `multiKmeans` function runs the k-means clustering algorithm for a given number of trials and selects the best result based on which trial produces the lowest intra-cluster variance.-The example below is identical to the phrase extraction example except that it uses `multiKmeans` with 15 trials,-rather than a single trial of the `kmeans` function.+The example below is identical to the phrase extraction example except that it uses `multiKmeans` with 15 trials, rather than a single trial of the `kmeans` function.[source,text]----@@ -621,27 +565,23 @@ This expression returns the following response:== Fuzzy K-Means Clustering-The `fuzzyKmeans` function is a soft clustering algorithm which-allows vectors to be assigned to more then one cluster. The `fuzziness` parameter-is a value between `1` and `2` that determines how fuzzy to make the cluster assignment.+The `fuzzyKmeans` function is a soft clustering algorithm which allows vectors to be assigned to more then one cluster.+The `fuzziness` parameter is a value between `1` and `2` that determines how fuzzy to make the cluster assignment.-After the clustering has been performed the `getMembershipMatrix` function can be called-on the clustering result to return a matrix describing the probabilities-of cluster membership for each vector.+After the clustering has been performed the `getMembershipMatrix` function can be called on the clustering result to return a matrix describing the probabilities of cluster membership for each vector.This matrix can be used to understand relationships between clusters.In the example below `fuzzyKmeans` is used to cluster the movie reviews matching the phrase "star wars".-But instead of looking at the clusters or centroids, the `getMembershipMatrix` is used to return the-membership probabilities for each document. The membership matrix is comprised of a row for each-vector that was clustered. There is a column in the matrix for each cluster.+But instead of looking at the clusters or centroids, the `getMembershipMatrix` is used to return the membership probabilities for each document.+The membership matrix is comprised of a row for each vector that was clustered.+There is a column in the matrix for each cluster.The values in the matrix contain the probability that a specific vector belongs to a specific cluster.-In the example the `distance` function is then used to create a *distance matrix* from the columns of the-membership matrix. The distance matrix is then visualized with the `zplot` function as a heat map.+In the example the `distance` function is then used to create a *distance matrix* from the columns of the membership matrix.+The distance matrix is then visualized with the `zplot` function as a heat map.In the example `cluster1` and `cluster5` have the shortest distance between the clusters.-Further analysis of the features in both clusters can be performed to understand-the relationship between `cluster1` and `cluster5`.+Further analysis of the features in both clusters can be performed to understand the relationship between `cluster1` and `cluster5`.image::images/math-expressions/fuzzyk.png[]@@ -649,8 +589,7 @@ NOTE: The heat map has been configured to increase in color intensity as the dis== Feature Scaling-Before performing machine learning operations its often necessary to-scale the feature vectors so they can be compared at the same scale.+Before performing machine learning operations its often necessary to scale the feature vectors so they can be compared at the same scale.All the scaling functions below operate on vectors and matrices.When operating on a matrix the rows of the matrix are scaled.@@ -660,12 +599,10 @@ When operating on a matrix the rows of the matrix are scaled.The `minMaxScale` function scales a vector or matrix between a minimum and maximum value.By default it will scale between `0` and `1` if min/max values are not provided.-Below is a plot of a sine wave, with an amplitude of 1, before and-after it has been scaled between -5 and 5.+Below is a plot of a sine wave, with an amplitude of 1, before and after it has been scaled between -5 and 5.image::images/math-expressions/minmaxscale.png[]-Below is a simple example of min/max scaling of a matrix between 0 and 1.Notice that once brought into the same scale the vectors are the same.@@ -711,11 +648,9 @@ When this expression is sent to the `/stream` handler it responds with:=== Standardization-The `standardize` function scales a vector so that it has a-mean of 0 and a standard deviation of 1.+The `standardize` function scales a vector so that it has a mean of 0 and a standard deviation of 1.-Below is a plot of a sine wave, with an amplitude of 1, before and-after it has been standardized.+Below is a plot of a sine wave, with an amplitude of 1, before and after it has been standardized.image::images/math-expressions/standardize.png[]@@ -764,12 +699,12 @@ When this expression is sent to the `/stream` handler it responds with:=== Unit Vectors-The `unitize` function scales vectors to a magnitude of 1. A vector with a-magnitude of 1 is known as a unit vector. Unit vectors are preferred-when the vector math deals with vector direction rather than magnitude.+The `unitize` function scales vectors to a magnitude of 1.+A vector with a+magnitude of 1 is known as a unit vector.+Unit vectors are preferred when the vector math deals with vector direction rather than magnitude.-Below is a plot of a sine wave, with an amplitude of 1, before and-after it has been unitized.+Below is a plot of a sine wave, with an amplitude of 1, before and after it has been unitized.image::images/math-expressions/unitize.png[]diff --git a/solr/solr-ref-guide/src/major-changes-in-solr-8.adoc b/solr/solr-ref-guide/src/major-changes-in-solr-8.adocindex 53647f22154..e343e5e3892 100644--- a/solr/solr-ref-guide/src/major-changes-in-solr-8.adoc+++ b/solr/solr-ref-guide/src/major-changes-in-solr-8.adoc@@ -30,7 +30,8 @@ A thorough review of the list in <<Major Changes in Earlier 7.x Versions>>, belo=== Upgrade Prerequisites-*If using SolrCloud, you must be on Solr 7.3.0 or higher*. Solr's LeaderInRecovery (LIR) functionality <<Solr 7.3,changed significantly>> in Solr 7.3. While these changes were back-compatible for all subsequent 7.x releases, that compatibility has been removed in 8.0.+*If using SolrCloud, you must be on Solr 7.3.0 or higher*.+Solr's LeaderInRecovery (LIR) functionality <<Solr 7.3,changed significantly>> in Solr 7.3. While these changes were back-compatible for all subsequent 7.x releases, that compatibility has been removed in 8.0.In order to upgrade to Solr 8.x, all nodes of your cluster must be running Solr 7.3 or higher. If an upgrade is attempted with nodes running versions earlier than 7.3, documents could be lost.If you are not using Solr in SolrCloud mode (you run a user-managed cluster or a single-node installation), we expect you can upgrade to Solr 8 from any 7.x version without major issues.diff --git a/solr/solr-ref-guide/src/managed-resources.adoc b/solr/solr-ref-guide/src/managed-resources.adocindex f35f1608a25..17ee5314b51 100644--- a/solr/solr-ref-guide/src/managed-resources.adoc+++ b/solr/solr-ref-guide/src/managed-resources.adoc@@ -55,7 +55,8 @@ To begin, you need to define a field type that uses the <<filters.adoc#managed-sThere are two important things to notice about this field type definition:-<1> The filter implementation class is `solr.ManagedStopFilterFactory`. This is a special implementation of the <<filters.adoc#stop-filter,StopFilterFactory>> that uses a set of stop words that are managed from a REST API.+<1> The filter implementation class is `solr.ManagedStopFilterFactory`.+This is a special implementation of the <<filters.adoc#stop-filter,StopFilterFactory>> that uses a set of stop words that are managed from a REST API.<2> The `managed=”english”` attribute gives a name to the set of managed stop words, in this case indicating the stop words are for English text.@@ -105,8 +106,10 @@ Assuming you sent this request to Solr, the response body is a JSON document:The `sample_techproducts_configs` <<config-sets.adoc#,configset>> ships with a pre-built set of managed stop words, however you should only interact with this file using the API and not edit it directly.-One thing that should stand out to you in this response is that it contains a `managedList` of words as well as `initArgs`. This is an important concept in this framework -- managed resources typically have configuration and data.-For stop words, the only configuration parameter is a boolean that determines whether to ignore the case of tokens during stop word filtering (ignoreCase=true|false). The data is a list of words, which is represented as a JSON array named `managedList` in the response.+One thing that should stand out to you in this response is that it contains a `managedList` of words as well as `initArgs`.+This is an important concept in this framework -- managed resources typically have configuration and data.+For stop words, the only configuration parameter is a boolean that determines whether to ignore the case of tokens during stop word filtering (ignoreCase=true|false).+The data is a list of words, which is represented as a JSON array named `managedList` in the response.Now, let’s add a new word to the English stop word list using an HTTP PUT:@@ -212,7 +215,8 @@ curl -X PUT -H 'Content-type:application/json' --data-binary '["funny", "enterta----Note that the expansion is performed when processing the PUT request so the underlying persistent state is still a managed map.-Consequently, if after sending the previous PUT request, you did a GET for `/schema/analysis/synonyms/english/jocular`, then you would receive a list containing `["funny", "entertaining", "whimiscal"]`. Once you've created synonym mappings using a list, each term must be managed separately.+Consequently, if after sending the previous PUT request, you did a GET for `/schema/analysis/synonyms/english/jocular`, then you would receive a list containing `["funny", "entertaining", "whimiscal"]`.+Once you've created synonym mappings using a list, each term must be managed separately.Lastly, you can delete a mapping by sending a DELETE request to the managed endpoint.diff --git a/solr/solr-ref-guide/src/metrics-reporting.adoc b/solr/solr-ref-guide/src/metrics-reporting.adocindex a8c6dc6881d..33789e92cf1 100644--- a/solr/solr-ref-guide/src/metrics-reporting.adoc+++ b/solr/solr-ref-guide/src/metrics-reporting.adoc@@ -34,7 +34,8 @@ Some of these meters may be missing or empty for any number of valid reasons.In these cases, missing values of any type will be returned as `null` by default so empty values won't impact averages or histograms.This is configurable for several types of missing values; see the <<The <metrics> <missingValues> Element>> section below.-Each group of related metrics with unique names is managed in a *metric registry*. Solr maintains several such registries, each corresponding to a high-level group such as: `jvm`, `jetty`, `node`, and `core` (see <<Metric Registries>> below).+Each group of related metrics with unique names is managed in a *metric registry*.+Solr maintains several such registries, each corresponding to a high-level group such as: `jvm`, `jetty`, `node`, and `core` (see <<Metric Registries>> below).For each group (and/or for each registry) there can be several *reporters*, which are components responsible for communication of metrics from selected registries to external systems.Currently implemented reporters support emitting metrics via JMX, Ganglia, Graphite and SLF4J.@@ -171,7 +172,8 @@ The default is `1028`.* `alpha`, the decay parameter.The default is `0.015`.This is only valid for the `ExponentiallyDecayingReservoir`.-* `window`, the window size, in seconds, and only valid for the `SlidingTimeWindowReservoir`. The default is 300 (5 minutes).+* `window`, the window size, in seconds, and only valid for the `SlidingTimeWindowReservoir`.+The default is `300` (5 minutes).<timer>:: This element defines an implementation of a `Timer` supplier.The default implementation supports the `clock` and `reservoir` parameters described above.@@ -195,7 +197,8 @@ As an example of a section of `solr.xml` that defines some of these custom param=== The <metrics> <missingValues> ElementLong-lived metrics values are still reported when the underlying value is unavailable (e.g., "INDEX.sizeInBytes" when-IndexReader is closed). Short-lived transient metrics (such as cache entries) that are properties of complex gauges+IndexReader is closed).+Short-lived transient metrics (such as cache entries) that are properties of complex gauges(internally represented as `MetricsMap`) are simply skipped when not available, and neither their names nor valuesappear in registries (or in `/admin/metrics` reports).@@ -300,7 +303,8 @@ Default is no filtering, i.e., all metrics from the selected registry will be reReporters are instantiated for every group and registry that they were configured for, at the time when the respective components are initialized (e.g., on JVM startup or SolrCore load).-When reporters are created their configuration is validated (and e.g., necessary connections are established). Uncaught errors at this initialization stage cause the reporter to be discarded from the running configuration.+When reporters are created their configuration is validated (and e.g., necessary connections are established).+Uncaught errors at this initialization stage cause the reporter to be discarded from the running configuration.Reporters are closed when the corresponding component is being closed (e.g., on SolrCore close, or JVM shutdown) but metrics that they reported are still maintained in respective registries, as explained in the previous section.@@ -329,7 +333,8 @@ Note either `serviceUrl` or `agentId` can be specified but not both - if both arObject names created by this reporter are hierarchical, dot-separated but also properly structured to form corresponding hierarchies in e.g., JConsole.This hierarchy consists of the following elements in the top-down order:-* registry name (e.g., `solr.core.collection1.shard1.replica1`). Dot-separated registry names are also split into ObjectName hierarchy levels, so that metrics for this registry will be shown under `/solr/core/collection1/shard1/replica1` in JConsole, with each domain part being assigned to `dom1, dom2, ... domN` property.+* registry name (e.g., `solr.core.collection1.shard1.replica1`).+Dot-separated registry names are also split into ObjectName hierarchy levels, so that metrics for this registry will be shown under `/solr/core/collection1/shard1/replica1` in JConsole, with each domain part being assigned to `dom1, dom2, ... domN` property.* reporter name (the value of reporter's `name` attribute)* category, scope and name for request handlers* or additional `name1, name2, ... nameN` elements for metrics from other components.@@ -538,11 +543,13 @@ A few query parameters are available to limit your request to only certain metri`group`:: The metric group to retrieve.The default is `all` to retrieve all metrics for all groups.-Other possible values are: `jvm`, `jetty`, `node`, and `core`. More than one group can be specified in a request; multiple group names should be separated by a comma.+Other possible values are: `jvm`, `jetty`, `node`, and `core`.+More than one group can be specified in a request; multiple group names should be separated by a comma.`type`:: The type of metric to retrieve.The default is `all` to retrieve all metric types.-Other possible values are `counter`, `gauge`, `histogram`, `meter`, and `timer`. More than one type can be specified in a request; multiple types should be separated by a comma.+Other possible values are `counter`, `gauge`, `histogram`, `meter`, and `timer`.+More than one type can be specified in a request; multiple types should be separated by a comma.`prefix`:: The first characters of metric name that will filter the metrics returned to those starting with the provided string.It can be combined with `group` and/or `type` parameters.diff --git a/solr/solr-ref-guide/src/numerical-analysis.adoc b/solr/solr-ref-guide/src/numerical-analysis.adocindex de43acd9841..6c644b3c62f 100644--- a/solr/solr-ref-guide/src/numerical-analysis.adoc+++ b/solr/solr-ref-guide/src/numerical-analysis.adoc@@ -50,7 +50,9 @@ The visualization above first creates two arrays with x and y-axis points.Notice that the x-axis ranges from 0 to 9.Then the `akima`, `spline` and `lerp` functions are applied to the vectors to create three interpolation functions.-Then 500 hundred random samples are drawn from a uniform distribution between 0 and 3. These are the new zoomed in x-axis points, between 0 and 3. Notice that we are sampling a specific area of the curve.+Then 500 hundred random samples are drawn from a uniform distribution between 0 and 3.+These are the new zoomed in x-axis points, between 0 and 3.+Notice that we are sampling a specific area of the curve.Then the `predict` function is used to predict y-axis points for the sampled x-axis, for all three interpolation functions.Finally all three prediction vectors are plotted with the sampled x-axis points.@@ -101,7 +103,8 @@ The `derivative` function is then applied to the linear interpolation.image::images/math-expressions/derivative.png[]-Notice that the *miles_traveled* line has a slope of 10 until the 5th hour where it changes to a slope of 50. The *mph* line, which is the derivative, visualizes the *velocity* of the *miles_traveled* line.+Notice that the *miles_traveled* line has a slope of 10 until the 5th hour where it changes to a slope of 50.+The *mph* line, which is the derivative, visualizes the *velocity* of the *miles_traveled* line.Also notice that the derivative is calculated along straight lines showing immediate change from one point to the next.This is because linear interpolation (`lerp`) is used as the interpolation function.@@ -110,7 +113,8 @@ If the `spline` or `akima` functions had been used it would have produced a deri=== The Second Derivative (Acceleration)-While the first derivative represents velocity, the second derivative represents `acceleration`. The second the derivative is the derivative of the first derivative.+While the first derivative represents velocity, the second derivative represents `acceleration`.+The second the derivative is the derivative of the first derivative.The example below builds on the first example and adds the second derivative.Notice that the second derivative `d2` is taken by applying the derivative function to a linear interpolation of the first derivative.@@ -226,12 +230,14 @@ let(years=array(1998, 2000, 2002, 2004, 2006),----In this example a bicubic spline is used to interpolate a matrix of real estate data.-Each row of the matrix represent specific `years`. Each column of the matrix represents `floors` of the building.+Each row of the matrix represent specific `years`.+Each column of the matrix represents `floors` of the building.The grid of numbers is the average selling price of an apartment for each year and floor.For example in 2002 the average selling price for the 9th floor was `415000` (row 3, column 3).The `bicubicSpline` function is then used to interpolate the grid, and the `predict` function is used to predict a value for year 2003, floor 8.-Notice that the matrix does not include a data point for year 2003, floor 8. The `bicubicSpline` function creates that data point based on the surrounding data in the matrix:+Notice that the matrix does not include a data point for year 2003, floor 8.+The `bicubicSpline` function creates that data point based on the surrounding data in the matrix:[source,json]----diff --git a/solr/solr-ref-guide/src/package-manager-internals.adoc b/solr/solr-ref-guide/src/package-manager-internals.adocindex 19baa3d8ffa..9c2db2253ed 100644--- a/solr/solr-ref-guide/src/package-manager-internals.adoc+++ b/solr/solr-ref-guide/src/package-manager-internals.adoc@@ -16,7 +16,8 @@// specific language governing permissions and limitations// under the License.-The package manager (CLI) internally uses various Solr APIs to install, deploy and update packages. This document contains an overview of those APIs.+The package manager (CLI) internally uses various Solr APIs to install, deploy and update packages.+This document contains an overview of those APIs.== Salient Features@@ -30,12 +31,16 @@ The package manager (CLI) internally uses various Solr APIs to install, deploy a== Classloaders-At the heart of the system, we have classloader isolation. To achieve this, the system is simplified into two layered classloaders:-The root classloader which has all the jars from Solr classpath. This requires Solr node restart to change anything.-A set of named classloaders that inherit from the root classloader. The life cycles of the named classloaders are tied to the package configuration in ZooKeeper. As soon as the configuration is modified, the corresponding classloaders are reloaded and components are asked to reload.+At the heart of the system, we have classloader isolation.+To achieve this, the system is simplified into two layered classloaders: the root classloader which has all the jars from Solr classpath.+This requires Solr node restart to change anything.+A set of named classloaders that inherit from the root classloader.+The life cycles of the named classloaders are tied to the package configuration in ZooKeeper.+As soon as the configuration is modified, the corresponding classloaders are reloaded and components are asked to reload.== Package Loading Security-Packages are disabled by default. Start all your nodes with the system property `-Denable.packages=true` to use this feature.+Packages are disabled by default.+Start all your nodes with the system property `-Denable.packages=true` to use this feature.*Example*[source,bash]@@ -62,16 +67,19 @@ Package store is a distributed file store which can store arbitrary files in the* This is a fully replicated file system based repository.* It lives at <solr.home>/filestore on each Solr node.-* Every entry is a file + metadata. The metadata is named .<filename>.json.+* Every entry is a file + metadata.+The metadata is named .<filename>.json.* The metadata file contains the sha256, signatures of the file.* Users can’t create files starting with period (.).-* It is agnostic of content type of files. You may store jars as well as other files..+* It is agnostic of content type of files.+You may store jars as well as other files..=== How Does the Package Store Work?When a file is uploaded to the PackageStore, the following is true:* It’s saved to the local file system.-* It’s saved along with the metadata. The metadata file stores the sha512 and signatures of the jar files too.+* It’s saved along with the metadata.+The metadata file stores the sha512 and signatures of the jar files too.* Every live node in the cluster is asked to download it as well.=== Package Store APIs@@ -133,11 +141,15 @@ A Package have the following attributes:** `version`: The version string** `files`: An array of files from the package store-For every package/version in the packages definition, there is a unique `SolrResourceLoader` instance. This is a child of the `CoreContainer` resource loader.+For every package/version in the packages definition, there is a unique `SolrResourceLoader` instance.+This is a child of the `CoreContainer` resource loader.=== packages.json-The package configurations live in a file called `packages.json` in ZooKeeper. At any given moment we can have multiple versions of a given package in the package configuration. The system will always use the latest version. Versions are sorted by their numeric value and the highest is the latest.+The package configurations live in a file called `packages.json` in ZooKeeper.+At any given moment we can have multiple versions of a given package in the package configuration.+The system will always use the latest version.+Versions are sorted by their numeric value and the highest is the latest.For example:@@ -174,7 +186,8 @@ Use the `delete` command to delete the highest version and choose the next highe=== Using Multiple Versions in Parallel-We use `params.json` in the collection config to store a version of a package it uses. By default it is the `$LATEST`.+We use `params.json` in the collection config to store a version of a package it uses.+By default it is the `$LATEST`.[source, json]----@@ -186,7 +199,9 @@ We use `params.json` in the collection config to store a version of a package it----<1> For `mypkg`, use the version `0.1` irrespective of whether there is a newer version available or not.-<2> For `pkg2`, use the latest. This is optional. The default is `$LATEST`.+<2> For `pkg2`, use the latest.+This is optional.+The default is `$LATEST`.=== Workflow@@ -197,7 +212,8 @@ We use `params.json` in the collection config to store a version of a package it=== Using Packages in Plugins-Any class name can be prefixed with the package name, e.g., `mypkg:fully.qualified.ClassName` and Solr would use the latest version of the package to load the classes from. The plugins loaded from packages cannot depend on core level classes.+Any class name can be prefixed with the package name, e.g., `mypkg:fully.qualified.ClassName` and Solr would use the latest version of the package to load the classes from.+The plugins loaded from packages cannot depend on core level classes..Plugin declaration in `solrconfig.xml`[source, xml]@@ -235,7 +251,10 @@ curl http://localhost:8983/api/cluster/package?omitHeader=true"files":["/mypkg/1.0/myplugins.jar"]}]}}}-----. The package should be ready to use at this point. Next, register a plugin in your collection from the package. Note the `mypkg:` prefix applied to the `class` attribute. The same result can be achieved by editing your `solrconfig.xml` as well:+. The package should be ready to use at this point.+Next, register a plugin in your collection from the package.+Note the `mypkg:` prefix applied to the `class` attribute.+The same result can be achieved by editing your `solrconfig.xml` as well:+[source,bash]----@@ -283,7 +302,8 @@ $ curl http://localhost:8983/solr/gettingstarted/test?omitHeader=true"loader":"java.net.FactoryURLClassLoader"}-----. Update the version of our component. Get a new version of the jar, sign and upload it:+. Update the version of our component.+Get a new version of the jar, sign and upload it:+[source, bash]----@@ -373,7 +393,8 @@ Note that the `Version` value is `"2"`, which means the plugin is updated.=== How to Avoid Automatic Upgrade-The default version used in any collection is always the latest. However, setting a per-collection property in `params.json` ensures that the versions are always fixed irrespective of the new versions added.+The default version used in any collection is always the latest.+However, setting a per-collection property in `params.json` ensures that the versions are always fixed irrespective of the new versions added.[source,bash]----diff --git a/solr/solr-ref-guide/src/package-manager.adoc b/solr/solr-ref-guide/src/package-manager.adocindex 4d2f0bf0b44..4a290680c7a 100644--- a/solr/solr-ref-guide/src/package-manager.adoc+++ b/solr/solr-ref-guide/src/package-manager.adoc@@ -40,7 +40,8 @@ The package manager (CLI) allows you to:=== Enable the Package Manager-The package manager is disabled by default. To enable it, start all Solr nodes with the `-Denable.packages=true` parameter.+The package manager is disabled by default.+To enable it, start all Solr nodes with the `-Denable.packages=true` parameter.[source,bash]----@@ -48,11 +49,13 @@ $ bin/solr -c -Denable.packages=true----WARNING: There are security consequences to enabling the package manager.-If an unauthorized user gained access to the system, they would have write access to ZooKeeper and could install packages from untrusted sources. Always ensure you have secured Solr with firewalls and <<authentication-and-authorization-plugins.adoc#,authentication>> before enabling the package manager.+If an unauthorized user gained access to the system, they would have write access to ZooKeeper and could install packages from untrusted sources.+Always ensure you have secured Solr with firewalls and <<authentication-and-authorization-plugins.adoc#,authentication>> before enabling the package manager.=== Add Trusted Repositories-A _repository_ is a location hosting one or many packages. Often, this is a web service that serves meta-information about packages, the package artifacts for downloading, and a public key to validate the jar file signatures while installing.+A _repository_ is a location hosting one or many packages.+Often, this is a web service that serves meta-information about packages, the package artifacts for downloading, and a public key to validate the jar file signatures while installing.In order to install packages into Solr, one has to add a repository hosting the packages.@@ -61,7 +64,8 @@ In order to install packages into Solr, one has to add a repository hosting the$ bin/solr package add-repo <name-of-repo> <repo-url>-----NOTE: Do not add repositories that you don't trust or control. Only add repositories that are based on HTTPS and avoid repositories based on HTTP to safeguard against MITM attacks.+NOTE: Do not add repositories that you don't trust or control.+Only add repositories that are based on HTTPS and avoid repositories based on HTTP to safeguard against MITM attacks.=== Listing and Installing Packages@@ -124,7 +128,8 @@ For example, if a package named `mypackage` contains a request handler, we wouldThen use either the Collections API <<collection-management.adoc#reload,RELOAD command>> or the <<collections-core-admin.adoc#,Admin UI>> to reload the collection.-Next, set the package version that this collection is using. If the collection is named `collection1`, the package name is `mypackage`, and the installed version is `1.0.0`, the command would look like this:+Next, set the package version that this collection is using.+If the collection is named `collection1`, the package name is `mypackage`, and the installed version is `1.0.0`, the command would look like this:[source,bash]----@@ -153,7 +158,8 @@ Next, install the new version of the package from the repositories.$ bin/solr package install <package-name>:<version>-----Once you have installed the new version, you can selectively update each of your collections or the cluster level plugins. Assuming the old version is `1.0.0` of the package `mypackage`, and the new version is `2.0.0`, the command would be as follows:+Once you have installed the new version, you can selectively update each of your collections or the cluster level plugins.+Assuming the old version is `1.0.0` of the package `mypackage`, and the new version is `2.0.0`, the command would be as follows:[source,bash]----@@ -194,6 +200,10 @@ $ bin/solr package deploy <package-name> -cluster== Security-The `add-repo` step should only be executed using HTTPS enabled repository urls only so as to prevent against MITM attacks when Solr is fetching the public key for the repository. This `add-repo` step registers the public key of the trusted repository, and hence can only be executed using the package manager (CLI) having direct write access to the trusted store of the package store (a special location in the package store that cannot be written to using the package store APIs). Also, it is critical to protect ZooKeeper from unauthorized write access.+The `add-repo` step should only be executed using HTTPS enabled repository urls only so as to prevent against MITM attacks when Solr is fetching the public key for the repository.+This `add-repo` step registers the public key of the trusted repository, and hence can only be executed using the package manager (CLI) having direct write access to the trusted store of the package store (a special location in the package store that cannot be written to using the package store APIs).+Also, it is critical to protect ZooKeeper from unauthorized write access.-Also, keep in mind, that it is possible to install *any* package from a repository once it has been added. If you want to use some packages in production, a best practice is to setup your own repository and add that to Solr instead of adding a generic third-party repository that is beyond your administrative control. You might want to re-sign packages from a third-party repository using your own private keys and host them at your own repository.+Also, keep in mind, that it is possible to install *any* package from a repository once it has been added.+If you want to use some packages in production, a best practice is to setup your own repository and add that to Solr instead of adding a generic third-party repository that is beyond your administrative control.+You might want to re-sign packages from a third-party repository using your own private keys and host them at your own repository.diff --git a/solr/solr-ref-guide/src/pagination-of-results.adoc b/solr/solr-ref-guide/src/pagination-of-results.adocindex ca7db7bb9c1..5d35ddc3e26 100644--- a/solr/solr-ref-guide/src/pagination-of-results.adoc+++ b/solr/solr-ref-guide/src/pagination-of-results.adoc@@ -79,7 +79,8 @@ Pagination using `start` and `rows` not only require Solr to compute (and sort)While a request for `start=0&rows=1000000` may be obviously inefficient because it requires Solr to maintain & sort in memory a set of 1 million documents, likewise a request for `start=999000&rows=1000` is equally inefficient for the same reasons.Solr can't compute which matching document is the 999001st result in sorted order, without first determining what the first 999000 matching sorted results are.-If the index is distributed, which is common when running in SolrCloud mode, then 1 million documents are retrieved from *each shard*. For a ten shard index, ten million entries must be retrieved and sorted to figure out the 1000 documents that match those query parameters.+If the index is distributed, which is common when running in SolrCloud mode, then 1 million documents are retrieved from *each shard*.+For a ten shard index, ten million entries must be retrieved and sorted to figure out the 1000 documents that match those query parameters.== Fetching A Large Number of Sorted Results: Cursorsdiff --git a/solr/solr-ref-guide/src/partial-document-updates.adoc b/solr/solr-ref-guide/src/partial-document-updates.adocindex ae71bbdfeba..1b3b0b96232 100644--- a/solr/solr-ref-guide/src/partial-document-updates.adoc+++ b/solr/solr-ref-guide/src/partial-document-updates.adoc@@ -63,7 +63,8 @@ Must be specified as a single numeric value.=== Field Storage-The core functionality of atomically updating a document requires that all fields in your schema must be configured as stored (`stored="true"`) or docValues (`docValues="true"`) except for fields which are `<copyField/>` destinations, which must be configured as `stored="false"`. Atomic updates are applied to the document represented by the existing stored field values.+The core functionality of atomically updating a document requires that all fields in your schema must be configured as stored (`stored="true"`) or docValues (`docValues="true"`) except for fields which are `<copyField/>` destinations, which must be configured as `stored="false"`.+Atomic updates are applied to the document represented by the existing stored field values.All data in copyField destinations fields must originate from ONLY copyField sources.If `<copyField/>` destinations are configured as stored, then Solr will attempt to index both the current value of the field as well as an additional copy from any source fields.@@ -234,7 +235,8 @@ curl -X POST 'http://localhost:8983/solr/gettingstarted/update?commit=true' -H 'In-place updates are very similar to atomic updates; in some sense, this is a subset of atomic updates.In regular atomic updates, the entire document is reindexed internally during the application of the update.However, in this approach, only the fields to be updated are affected and the rest of the documents are not reindexed internally.-Hence, the efficiency of updating in-place is unaffected by the size of the documents that are updated (i.e., number of fields, size of fields, etc.). Apart from these internal differences in efficiency, there is no functional difference between atomic updates and in-place updates.+Hence, the efficiency of updating in-place is unaffected by the size of the documents that are updated (i.e., number of fields, size of fields, etc.).+Apart from these internal differences in efficiency, there is no functional difference between atomic updates and in-place updates.An atomic update operation is performed using this In-Place approach only when the fields to be updated meet these three conditions:@@ -246,7 +248,8 @@ To use in-place updates, add a modifier to the field that needs to be updated.The content can be updated or incrementally increased.`set`::-Set or replace the field value(s) with the specified value(s). May be specified as a single value.+Set or replace the field value(s) with the specified value(s).+May be specified as a single value.`inc`::Increments a numeric value by a specific amount.@@ -501,7 +504,8 @@ Users can not re-purpose that field and specify it as the `versionField` for use`DocBasedVersionConstraintsProcessorFactory` supports the following additional configuration parameters, which are all optional:`ignoreOldUpdates`::-A boolean option which defaults to `false`. If set to `true`, the update will be silently ignored (and return a status 200 to the client) instead of rejecting updates where the `versionField` is too low.+A boolean option which defaults to `false`.+If set to `true`, the update will be silently ignored (and return a status 200 to the client) instead of rejecting updates where the `versionField` is too low.`deleteVersionParam`::A String parameter that can be specified to indicate that this processor should also inspect Delete By Id commands.diff --git a/solr/solr-ref-guide/src/performance-statistics-reference.adoc b/solr/solr-ref-guide/src/performance-statistics-reference.adocindex 883319e310b..79c3578a184 100644--- a/solr/solr-ref-guide/src/performance-statistics-reference.adoc+++ b/solr/solr-ref-guide/src/performance-statistics-reference.adoc@@ -18,15 +18,19 @@This page explains some of the statistics that Solr exposes.-There are two approaches to retrieving metrics. First, you can use the <<metrics-reporting.adoc#metrics-api,Metrics API>>, or you can enable JMX and get metrics from the <<mbean-request-handler.adoc#,MBean Request Handler>> or via an external tool such as JConsole. The below descriptions focus on retrieving the metrics using the Metrics API, but the metric names are the same if using the MBean Request Handler or an external tool.+There are two approaches to retrieving metrics.+First, you can use the <<metrics-reporting.adoc#metrics-api,Metrics API>>, or you can enable JMX and get metrics from the <<mbean-request-handler.adoc#,MBean Request Handler>> or via an external tool such as JConsole.+The below descriptions focus on retrieving the metrics using the Metrics API, but the metric names are the same if using the MBean Request Handler or an external tool.-These statistics are per core. When you are running in SolrCloud mode these statistics would co-relate to the performance of an individual replica.+These statistics are per core.+When you are running in SolrCloud mode these statistics would co-relate to the performance of an individual replica.== Request Handler Statistics=== Update Request Handler-The update request handler is an endpoint to send data to Solr. We can see how many update requests are being fired, how fast is it performing, and other valuable information regarding requests.+The update request handler is an endpoint to send data to Solr.+We can see how many update requests are being fired, how fast is it performing, and other valuable information regarding requests.*Registry & Path:* `solr.<core>:UPDATE./update`@@ -34,7 +38,9 @@ You can request update request handler statistics with an API request such as `\=== Search Request Handler-Can be useful to measure and track number of search queries, response times, etc. If you are not using the “select” handler then the path needs to be changed appropriately. Similarly if you are using the “sql” handler or “export” handler, the realtime handler “get”, or any other handler similar statistics can be found for that as well.+Can be useful to measure and track number of search queries, response times, etc.+If you are not using the “select” handler then the path needs to be changed appropriately.+Similarly if you are using the “sql” handler or “export” handler, the realtime handler “get”, or any other handler similar statistics can be found for that as well.*Registry & Path*: `solr.<core>:QUERY./select`@@ -66,7 +72,9 @@ To get request times, specifically, you can send an API request such as:*Errors and Other Times*-Other types of data such as errors and timeouts are also provided. These are available under different metric names. For example:+Other types of data such as errors and timeouts are also provided.+These are available under different metric names.+For example:* `\http://localhost:8983/solr/admin/metrics?group=core&prefix=UPDATE./update.errors`* `\http://localhost:8983/solr/admin/metrics?group=core&prefix=QUERY./select.errors`@@ -94,17 +102,13 @@ The table below shows the metric names and attributes to request:*Distributed vs. Local Request Times*-Processing of a single distributed request in SolrCloud usually requires making several requests to-other nodes and other replicas. The common statistics listed above lump these timings together, even though-they are very different in nature, thus making it difficult to measure the latency of distributed and-local requests separately. Solr 8.4 introduced additional statistics that help to do this.+Processing of a single distributed request in SolrCloud usually requires making several requests to other nodes and other replicas.+The common statistics listed above lump these timings together, even though they are very different in nature, thus making it difficult to measure the latency of distributed and local requests separately.+Solr 8.4 introduced additional statistics that help to do this.-These metrics are structured the same as `requestTimes` and `totalTime` metrics above but they use-different full names, e.g., `QUERY./select.distrib.requestTimes` and `QUERY./select.local.requestTimes`.-The metrics under the `distrib` path correspond to the time it takes for a (potentially) distributed-request to complete all remote calls plus any local processing, and return the result to the caller.-The metrics under the `local` path correspond to the time it takes for a local call (non-distributed,-i.e., being processed only by the Solr core where the handler operates) to complete.+These metrics are structured the same as `requestTimes` and `totalTime` metrics above but they use different full names, e.g., `QUERY./select.distrib.requestTimes` and `QUERY./select.local.requestTimes`.+The metrics under the `distrib` path correspond to the time it takes for a (potentially) distributed request to complete all remote calls plus any local processing, and return the result to the caller.+The metrics under the `local` path correspond to the time it takes for a local call (non-distributed, i.e., being processed only by the Solr core where the handler operates) to complete.== Update Handler@@ -161,7 +165,8 @@ In addition to a count of rollbacks, mean, 1 minute, 5 minute, and 15 minute rat=== Document Cache-This cache holds Lucene Document objects (the stored fields for each document). Since Lucene internal document IDs are transient, this cache cannot be auto-warmed.+This cache holds Lucene Document objects (the stored fields for each document).+Since Lucene internal document IDs are transient, this cache cannot be auto-warmed.*Registry and Path:* `solr.<core>:CACHE.searcher.documentCache`diff --git a/solr/solr-ref-guide/src/ping.adoc b/solr/solr-ref-guide/src/ping.adocindex 38cff84b467..dd450d2f93c 100644--- a/solr/solr-ref-guide/src/ping.adoc+++ b/solr/solr-ref-guide/src/ping.adoc@@ -23,7 +23,8 @@ image::images/ping/ping.png[image,width=171,height=195]The search executed by a Ping is configured with the <<request-parameters-api.adoc#,Request Parameters API>>. See <<implicit-requesthandlers.adoc#,Implicit Request Handlers>> for the paramset to use for the `/admin/ping` endpoint.-The Ping option doesn't open a page, but the status of the request can be seen on the core overview page shown when clicking on a collection name. The length of time the request has taken is displayed next to the Ping option, in milliseconds.+The Ping option doesn't open a page, but the status of the request can be seen on the core overview page shown when clicking on a collection name.+The length of time the request has taken is displayed next to the Ping option, in milliseconds.== Ping API Examples@@ -67,7 +68,8 @@ This command will ping all replicas of the given collection name for a response:</response>-----Both API calls have the same output. A status=OK indicates that the nodes are responding.+Both API calls have the same output.+A status=OK indicates that the nodes are responding.*SolrJ Example with SolrPing*diff --git a/solr/solr-ref-guide/src/plugins-stats-screen.adoc b/solr/solr-ref-guide/src/plugins-stats-screen.adocindex 76c8b06fc4f..84ecf83564a 100644--- a/solr/solr-ref-guide/src/plugins-stats-screen.adoc+++ b/solr/solr-ref-guide/src/plugins-stats-screen.adoc@@ -16,11 +16,15 @@// specific language governing permissions and limitations// under the License.-The Plugins screen shows information and statistics about the status and performance of various plugins running in each Solr core. You can find information about the performance of the Solr caches, the state of Solr's searchers, and the configuration of Request Handlers and Search Components.+The Plugins screen shows information and statistics about the status and performance of various plugins running in each Solr core.+You can find information about the performance of the Solr caches, the state of Solr's searchers, and the configuration of Request Handlers and Search Components.-Choose an area of interest on the right, and then drill down into more specifics by clicking on one of the names that appear in the central part of the window. In this example, we've chosen to look at the Searcher stats, from the Core area:+Choose an area of interest on the right, and then drill down into more specifics by clicking on one of the names that appear in the central part of the window.+In this example, we've chosen to look at the Searcher stats, from the Core area:.Searcher Statisticsimage::images/plugins-stats-screen/plugin-searcher.png[image,width=462,height=250]-The display is a snapshot taken when the page is loaded. You can get updated status by choosing to either *Watch Changes* or *Refresh Values*. Watching the changes will highlight those areas that have changed, while refreshing the values will reload the page with updated information.+The display is a snapshot taken when the page is loaded.+You can get updated status by choosing to either *Watch Changes* or *Refresh Values*.+Watching the changes will highlight those areas that have changed, while refreshing the values will reload the page with updated information.diff --git a/solr/solr-ref-guide/src/post-tool.adoc b/solr/solr-ref-guide/src/post-tool.adocindex 0cda48dd6fb..d02facb9bbb 100644--- a/solr/solr-ref-guide/src/post-tool.adoc+++ b/solr/solr-ref-guide/src/post-tool.adoc@@ -18,7 +18,8 @@Solr includes a simple command line tool for POSTing various types of content to a Solr server.-The tool is `bin/post`. The bin/post tool is a Unix shell script; for Windows (non-Cygwin) usage, see the section <<Post Tool Windows Support>> below.+The tool is `bin/post`.+The bin/post tool is a Unix shell script; for Windows (non-Cygwin) usage, see the section <<Post Tool Windows Support>> below.NOTE: This tool is meant for use by new users exploring Solr's capabilities, and is not intended as a robust solution to be used for indexing documents into production systems.@@ -29,7 +30,9 @@ To run it, open a window and enter:bin/post -c gettingstarted example/films/films.json-----This will contact the server at `localhost:8983`. Specifying the `collection/core name` is *mandatory*. The `-help` (or simply `-h`) option will output information on its usage (i.e., `bin/post -help)`.+This will contact the server at `localhost:8983`.+Specifying the `collection/core name` is *mandatory*.+The `-help` (or simply `-h`) option will output information on its usage (i.e., `bin/post -help)`.== Using the bin/post Tool@@ -75,7 +78,8 @@ OPTIONS== Examples Using bin/post-There are several ways to use `bin/post`. This section presents several examples.+There are several ways to use `bin/post`.+This section presents several examples.=== Indexing XML@@ -116,7 +120,8 @@ Index a tab-separated file into `gettingstarted`:bin/post -c signals -params "separator=%09" -type text/csv data.tsv-----The content type (`-type`) parameter is required to treat the file as the proper type, otherwise it will be ignored and a WARNING logged as it does not know what type of content a .tsv file is. The <<indexing-with-update-handlers.adoc#csv-formatted-index-updates,CSV handler>> supports the `separator` parameter, and is passed through using the `-params` setting.+The content type (`-type`) parameter is required to treat the file as the proper type, otherwise it will be ignored and a WARNING logged as it does not know what type of content a .tsv file is.+The <<indexing-with-update-handlers.adoc#csv-formatted-index-updates,CSV handler>> supports the `separator` parameter, and is passed through using the `-params` setting.=== Indexing JSONdiff --git a/solr/solr-ref-guide/src/probability-distributions.adoc b/solr/solr-ref-guide/src/probability-distributions.adocindex 85e42475bc4..b82bbd57c69 100644--- a/solr/solr-ref-guide/src/probability-distributions.adoc+++ b/solr/solr-ref-guide/src/probability-distributions.adoc@@ -16,30 +16,25 @@// specific language governing permissions and limitations// under the License.-This section of the user guide covers the probability distribution-framework included in the math expressions library.+This section of the user guide covers the probability distribution framework included in the math expressions library.== Visualization-Probability distributions can be visualized with Zeppelin-Solr using the-`zplot` function with the `dist` parameter, which visualizes the-probability density function (PDF) of the distribution.+Probability distributions can be visualized with Zeppelin-Solr using the `zplot` function with the `dist` parameter, which visualizes the probability density function (PDF) of the distribution.Example visualizations are shown with each distribution below.=== Continuous Distributions-Continuous probability distributions work with continuous numbers (floating points). Below-are the supported continuous probability distributions.+Continuous probability distributions work with continuous numbers (floating points).+Below are the supported continuous probability distributions.==== empiricalDistribution-The `empiricalDistribution` function creates a continuous probability-distribution from actual data.+The `empiricalDistribution` function creates a continuous probability distribution from actual data.Empirical distributions can be used to conveniently visualize the probability density function of a random sample from a SolrCloud collection.-The example below shows the `zplot` function visualizing the probability-density of a random sample with a 32 bin histogram.+The example below shows the `zplot` function visualizing the probability density of a random sample with a 32 bin histogram.image::images/math-expressions/empirical.png[]@@ -52,22 +47,19 @@ image::images/math-expressions/dist.png[]==== logNormalDistribution-The visualization below shows a log normal distribution with a `shape` of .25 and `scale`-of 0.+The visualization below shows a log normal distribution with a `shape` of .25 and `scale` of 0.image::images/math-expressions/lognormal.png[]==== gammaDistribution-The visualization below shows a gamma distribution with a `shape` of 7.5 and `scale`-of 1.+The visualization below shows a gamma distribution with a `shape` of 7.5 and `scale` of 1.image::images/math-expressions/gamma.png[]==== betaDistribution-The visualization below shows a beta distribution with a `shape1` of 2 and `shape2`-of 2.+The visualization below shows a beta distribution with a `shape1` of 2 and `shape2` of 2.image::images/math-expressions/beta.png[]@@ -79,15 +71,13 @@ image::images/math-expressions/uniformr.png[]==== weibullDistribution-The visualization below shows a Weibull distribution with a `shape` of 5 and `scale`-of 1.+The visualization below shows a Weibull distribution with a `shape` of 5 and `scale` of 1.image::images/math-expressions/weibull.png[]==== triangularDistribution-The visualization below shows a triangular distribution with a low of 5 a mode of 10-and a high value of 20.+The visualization below shows a triangular distribution with a low of 5 a mode of 10 and a high value of 20.image::images/math-expressions/triangular.png[]@@ -97,27 +87,21 @@ The visualization below shows a constant distribution of 10.5.image::images/math-expressions/constant.png[]--=== Discrete Distributions-Discrete probability distributions work with discrete numbers (integers). Below are the-supported discrete probability distributions.+Discrete probability distributions work with discrete numbers (integers).+Below are the supported discrete probability distributions.==== enumeratedDistribution-The `enumeratedDistribution` function creates a discrete-distribution function-from an enumerated list of values and probabilities or-from a data set of discrete values+The `enumeratedDistribution` function creates a discrete distribution function+from an enumerated list of values and probabilities or from a data set of discrete values.-The visualization below shows an enumerated distribution created from a list of-discrete values and probabilities.+The visualization below shows an enumerated distribution created from a list of discrete values and probabilities.image::images/math-expressions/enum1.png[]-The visualization below shows an enumerated distribution generated from a search-result that has been transformed into a vector of discrete values.+The visualization below shows an enumerated distribution generated from a search result that has been transformed into a vector of discrete values.image::images/math-expressions/enum2.png[]@@ -130,8 +114,7 @@ image::images/math-expressions/poisson.png[]==== binomialDistribution-The visualization below shows a binomial distribution with a 100 trials and .15-probability of success.+The visualization below shows a binomial distribution with a 100 trials and .15 probability of success.image::images/math-expressions/binomial.png[]@@ -145,8 +128,7 @@ image::images/math-expressions/uniform.png[]==== geometricDistribution-The visualization below shows a geometric distribution probability of success of-.25.+The visualization below shows a geometric distribution probability of success of .25.image::images/math-expressions/geometric.png[]@@ -161,13 +143,9 @@ image::images/math-expressions/zipf.png[]=== Cumulative Probability-The `cumulativeProbability` function can be used with all-probability distributions to calculate the-cumulative probability of encountering a specific-random variable within a specific distribution.+The `cumulativeProbability` function can be used with all probability distributions to calculate the cumulative probability of encountering a specific random variable within a specific distribution.-Below is example of calculating the cumulative probability-of a random variable within a normal distribution.+Below is example of calculating the cumulative probability of a random variable within a normal distribution.[source,text]----@@ -175,10 +153,8 @@ let(a=normalDistribution(10, 5),b=cumulativeProbability(a, 12))-----In this example a normal distribution function is created-with a mean of 10 and a standard deviation of 5. Then-the cumulative probability of the value 12 is calculated for this-specific distribution.+In this example a normal distribution function is created with a mean of 10 and a standard deviation of 5.+Then the cumulative probability of the value 12 is calculated for this specific distribution.When this expression is sent to the `/stream` handler it responds with:@@ -201,13 +177,10 @@ When this expression is sent to the `/stream` handler it responds with:=== Probability-All probability distributions can calculate the probability-between a range of values.+All probability distributions can calculate the probability between a range of values.-In the following example an empirical distribution is created-from a sample of file sizes drawn from the logs collection.-Then the probability of a file size between the range of 40000-and 41000 is calculated to be 19%.+In the following example an empirical distribution is created from a sample of file sizes drawn from the logs collection.+Then the probability of a file size between the range of 40000 and 41000 is calculated to be 19%.[source,text]----@@ -238,17 +211,12 @@ When this expression is sent to the `/stream` handler it responds with:=== Discrete Probability-The `probability` function can be used with any discrete-distribution function to compute the probability of a-discrete value.+The `probability` function can be used with any discrete distribution function to compute the probability of a discrete value.-Below is an example which calculates the probability-of a discrete value within a Poisson distribution.+Below is an example which calculates the probability of a discrete value within a Poisson distribution.-In the example a Poisson distribution function is created-with a mean of `100`. Then the-probability of encountering a sample of the discrete value 101 is calculated for this-specific distribution.+In the example a Poisson distribution function is created with a mean of `100`.+Then the probability of encountering a sample of the discrete value 101 is calculated for this specific distribution.[source,text]----@@ -278,8 +246,8 @@ When this expression is sent to the `/stream` handler it responds with:=== Sampling-All probability distributions support sampling. The `sample`-function returns one or more random samples from a probability distribution.+All probability distributions support sampling.+The `sample` function returns one or more random samples from a probability distribution.Below is an example drawing a single sample from a normal distribution.@@ -308,70 +276,55 @@ When this expression is sent to the `/stream` handler it responds with:}-----The sample function can also return a vector of samples. Vectors of samples-can be visualized as scatter plots to gain an intuitive understanding-of the underlying distribution.+The sample function can also return a vector of samples.+Vectors of samples can be visualized as scatter plots to gain an intuitive understanding of the underlying distribution.-The first example shows the scatter plot of a normal distribution with-a mean of 0 and a standard deviation of 5.+The first example shows the scatter plot of a normal distribution with a mean of 0 and a standard deviation of 5.image::images/math-expressions/sample-scatter.png[]-The next example shows a scatter plot of the same distribution with-an ascending sort applied to the sample vector.+The next example shows a scatter plot of the same distribution with an ascending sort applied to the sample vector.image::images/math-expressions/sample-scatter1.png[]-The next example shows two different distributions overlaid-in the same scatter plot.+The next example shows two different distributions overlaid in the same scatter plot.image::images/math-expressions/sample-overlay.png[]----=== Multivariate Normal Distribution-The multivariate normal distribution is a generalization of the-univariate normal distribution to higher dimensions.+The multivariate normal distribution is a generalization of the univariate normal distribution to higher dimensions.-The multivariate normal distribution models two or more random-variables that are normally distributed. The relationship between the variables is defined by a covariance matrix.+The multivariate normal distribution models two or more random variables that are normally distributed.+The relationship between the variables is defined by a covariance matrix.==== Sampling-The `sample` function can be used to draw samples-from a multivariate normal distribution in much the same-way as a univariate normal distribution.+The `sample` function can be used to draw samples from a multivariate normal distribution in much the same way as a univariate normal distribution.-The difference is that each sample will be an array containing a sample-drawn from each of the underlying normal distributions.-If multiple samples are drawn, the `sample` function returns a matrix with a-sample in each row. Over the long term the columns of the sample-matrix will conform to the covariance matrix used to parametrize the-multivariate normal distribution.+The difference is that each sample will be an array containing a sample drawn from each of the underlying normal distributions.+If multiple samples are drawn, the `sample` function returns a matrix with a sample in each row.+Over the long term the columns of the sample matrix will conform to the covariance matrix used to parametrize the multivariate normal distribution.-The example below demonstrates how to initialize and draw samples-from a multivariate normal distribution.+The example below demonstrates how to initialize and draw samples from a multivariate normal distribution.-In this example 5000 random samples are selected from a collection of log records. Each sample contains-the fields `filesize_d` and `response_d`. The values of both fields conform to a normal distribution.+In this example 5000 random samples are selected from a collection of log records.+Each sample contains the fields `filesize_d` and `response_d`.+The values of both fields conform to a normal distribution.-Both fields are then vectorized. The `filesize_d` vector is stored in-variable `b` and the `response_d` variable is stored in variable `c`.+Both fields are then vectorized.+The `filesize_d` vector is stored in variable `b` and the `response_d` variable is stored in variable `c`.An array is created that contains the means of the two vectorized fields.-Then both vectors are added to a matrix which is transposed. This creates-an observation matrix where each row contains one observation of-`filesize_d` and `response_d`. A covariance matrix is then created from the columns of-the observation matrix with the `cov` function. The covariance matrix describes the covariance between+Then both vectors are added to a matrix which is transposed.+This creates an observation matrix where each row contains one observation of`filesize_d` and `response_d`.+A covariance matrix is then created from the columns of the observation matrix with the `cov` function.+The covariance matrix describes the covariance between `filesize_d` and `response_d`.-The `multivariateNormalDistribution` function is then called with the-array of means for the two fields and the covariance matrix. The model for the-multivariate normal distribution is assigned to variable `g`.+The `multivariateNormalDistribution` function is then called with the array of means for the two fields and the covariance matrix.+The model for the multivariate normal distribution is assigned to variable `g`.Finally five samples are drawn from the multivariate normal distribution.@@ -387,11 +340,10 @@ let(a=random(logs, q="*:*", rows="5000", fl="filesize_d, response_d"),h=sample(g, 5))-----The samples are returned as a matrix, with each row representing one sample. There are two-columns in the matrix. The first column contains samples for `filesize_d` and the second-column contains samples for `response_d`. Over the long term the covariance between-the columns will conform to the covariance matrix used to instantiate the-multivariate normal distribution.+The samples are returned as a matrix, with each row representing one sample.+There are two columns in the matrix.+The first column contains samples for `filesize_d` and the second column contains samples for `response_d`.+Over the long term the covariance between the columns will conform to the covariance matrix used to instantiate the multivariate normal distribution.[source,json]----diff --git a/solr/solr-ref-guide/src/query-re-ranking.adoc b/solr/solr-ref-guide/src/query-re-ranking.adocindex afc64e42659..dfc915f5a81 100644--- a/solr/solr-ref-guide/src/query-re-ranking.adoc+++ b/solr/solr-ref-guide/src/query-re-ranking.adoc@@ -47,7 +47,8 @@ This parameter is required.`reRankDocs`::The number of top N documents from the original query that should be re-ranked.-This number will be treated as a minimum, and may be increased internally automatically in order to rank enough documents to satisfy the query (i.e., start+rows). The default is `200`.+This number will be treated as a minimum, and may be increased internally automatically in order to rank enough documents to satisfy the query (i.e., start+rows).+The default is `200`.`reRankWeight`::A multiplicative factor that will be applied to the score from the reRankQuery for each of the top matching documents, before that score is added to the original score.diff --git a/solr/solr-ref-guide/src/query-screen.adoc b/solr/solr-ref-guide/src/query-screen.adocindex 510a7e0568c..ee27bc54399 100644--- a/solr/solr-ref-guide/src/query-screen.adoc+++ b/solr/solr-ref-guide/src/query-screen.adoc@@ -23,56 +23,79 @@ In the example in the screenshot, a query has been submitted, and the screen sho.JSON Results of a Queryimage::images/query-screen/query-top.png[image,height=400]-In this example, a query for `genre:Fantasy` was sent to a "films" collection. Defaults were used for all other options in the form, which are explained briefly in the table below, and covered in detail in later parts of this Guide.+In this example, a query for `genre:Fantasy` was sent to a "films" collection.+Defaults were used for all other options in the form, which are explained briefly in the table below, and covered in detail in later parts of this Guide.-The response is shown to the right of the form. Requests to Solr are simply HTTP requests, and the query submitted is shown in light type above the results; if you click on this it will open a new browser window with just this request and response (without the rest of the Solr Admin UI). The rest of the response is shown in JSON, which is the default output format.+The response is shown to the right of the form.+Requests to Solr are simply HTTP requests, and the query submitted is shown in light type above the results; if you click on this it will open a new browser window with just this request and response (without the rest of the Solr Admin UI).+The rest of the response is shown in JSON, which is the default output format.-The response has at least two sections, but may have several more depending on the options chosen. The two sections it always has are the `responseHeader` and the `response`. The `responseHeader` includes the status of the search (`status`), the processing time (`QTime`), and the parameters (`params`) that were used to process the query.+The response has at least two sections, but may have several more depending on the options chosen.+The two sections it always has are the `responseHeader` and the `response`.+The `responseHeader` includes the status of the search (`status`), the processing time (`QTime`), and the parameters (`params`) that were used to process the query.-The `response` includes the documents that matched the query, in `doc` sub-sections. The fields return depend on the parameters of the query (and the defaults of the request handler used). The number of results is also included in this section.+The `response` includes the documents that matched the query, in `doc` sub-sections.+The fields return depend on the parameters of the query (and the defaults of the request handler used).+The number of results is also included in this section.-This screen allows you to experiment with different query options, and inspect how your documents were indexed. The query parameters available on the form are some basic options that most users want to have available, but there are dozens more available which could be simply added to the basic request by hand (if opened in a browser). The following parameters are available:+This screen allows you to experiment with different query options, and inspect how your documents were indexed.+The query parameters available on the form are some basic options that most users want to have available, but there are dozens more available which could be simply added to the basic request by hand (if opened in a browser).+The following parameters are available:Request-handler (qt)::-Specifies the query handler for the request. If a query handler is not specified, Solr processes the response with the standard query handler.+Specifies the query handler for the request.+If a query handler is not specified, Solr processes the response with the standard query handler.q::-The query event. See <<query-guide.adoc#,Query Guide>> for an explanation of this parameter.+The query event.+See <<query-guide.adoc#,Query Guide>> for an explanation of this parameter.fq::-The filter queries. See <<common-query-parameters.adoc#,Common Query Parameters>> for more information on this parameter.+The filter queries.+See <<common-query-parameters.adoc#,Common Query Parameters>> for more information on this parameter.sort::Sorts the response to a query in either ascending or descending order based on the response's score or another specified characteristic.start, rows::-`start` is the offset into the query result starting at which documents should be returned. The default value is 0, meaning that the query should return results starting with the first document that matches. This field accepts the same syntax as the start query parameter, which is described in <<query-guide.adoc#,Query Guide>>. `rows` is the number of rows to return.+`start` is the offset into the query result starting at which documents should be returned.+The default value is 0, meaning that the query should return results starting with the first document that matches.+This field accepts the same syntax as the start query parameter, which is described in <<query-guide.adoc#,Query Guide>>. `rows` is the number of rows to return.fl::-Defines the fields to return for each document. You can explicitly list the stored fields, <<function-queries.adoc#,functions>>, and <<document-transformers.adoc#,doc transformers>> you want to have returned by separating them with either a comma or a space.+Defines the fields to return for each document.+You can explicitly list the stored fields, <<function-queries.adoc#,functions>>, and <<document-transformers.adoc#,doc transformers>> you want to have returned by separating them with either a comma or a space.wt::-Specifies the Response Writer to be used to format the query response. Defaults to JSON if not specified.+Specifies the Response Writer to be used to format the query response.+Defaults to JSON if not specified.indent::Click this button to request that the Response Writer use indentation to make the responses more readable.debugQuery::-Click this button to augment the query response with debugging information, including "explain info" for each document returned. This debugging information is intended to be intelligible to the administrator or programmer.+Click this button to augment the query response with debugging information, including "explain info" for each document returned.+This debugging information is intended to be intelligible to the administrator or programmer.dismax::-Click this button to enable the DisMax query parser. See <<dismax-query-parser.adoc#,The DisMax Query Parser>> for further information.+Click this button to enable the DisMax query parser.+See <<dismax-query-parser.adoc#,The DisMax Query Parser>> for further information.edismax::-Click this button to enable the Extended query parser. See <<edismax-query-parser.adoc#,The Extended DisMax Query Parser>> for further information.+Click this button to enable the Extended query parser.+See <<edismax-query-parser.adoc#,The Extended DisMax Query Parser>> for further information.-hl:: Click this button to enable highlighting in the query response. See <<highlighting.adoc#,Highlighting>> for more information.+hl:: Click this button to enable highlighting in the query response.+See <<highlighting.adoc#,Highlighting>> for more information.facet::-Enables faceting, the arrangement of search results into categories based on indexed terms. See <<faceting.adoc#,Faceting>> for more information.+Enables faceting, the arrangement of search results into categories based on indexed terms.+See <<faceting.adoc#,Faceting>> for more information.spatial::-Click to enable using location data for use in spatial or geospatial searches. See <<spatial-search.adoc#,Spatial Search>> for more information.+Click to enable using location data for use in spatial or geospatial searches.+See <<spatial-search.adoc#,Spatial Search>> for more information.spellcheck::-Click this button to enable the Spellchecker, which provides inline query suggestions based on other, similar, terms. See <<spell-checking.adoc#,Spell Checking>> for more information.+Click this button to enable the Spellchecker, which provides inline query suggestions based on other, similar, terms.+See <<spell-checking.adoc#,Spell Checking>> for more information.diff --git a/solr/solr-ref-guide/src/query-syntax-and-parsers.adoc b/solr/solr-ref-guide/src/query-syntax-and-parsers.adocindex b2cdc1e44bb..16a2ad23a13 100644--- a/solr/solr-ref-guide/src/query-syntax-and-parsers.adoc+++ b/solr/solr-ref-guide/src/query-syntax-and-parsers.adoc@@ -39,7 +39,8 @@ This section explains how to specify a query parser and describes the syntax andThere are some query parameters common to all Solr parsers; these are discussed in the section <<common-query-parameters.adoc#common-query-parameters,Common Query Parameters>>.-Query parsers are also called `QParserPlugins`. They are all subclasses of {solr-javadocs}/core/org/apache/solr/search/QParserPlugin.html[QParserPlugin]. If you have custom parsing needs, you may want to extend that class to create your own query parser.+Query parsers are also called `QParserPlugins`.+They are all subclasses of {solr-javadocs}/core/org/apache/solr/search/QParserPlugin.html[QParserPlugin]. If you have custom parsing needs, you may want to extend that class to create your own query parser.****// This tags the below list so it can be used in the parent page section listdiff --git a/solr/solr-ref-guide/src/rate-limiters.adoc b/solr/solr-ref-guide/src/rate-limiters.adocindex 708f1f8883f..05fd86e2b1a 100644--- a/solr/solr-ref-guide/src/rate-limiters.adoc+++ b/solr/solr-ref-guide/src/rate-limiters.adoc@@ -16,22 +16,25 @@// specific language governing permissions and limitations// under the License.-Solr allows rate limiting per request type. Each request type can be allocated a maximum allowed number of concurrent requests-that can be active. The default rate limiting is implemented for updates and searches.+Solr allows rate limiting per request type.+Each request type can be allocated a maximum allowed number of concurrent requests that can be active.+The default rate limiting is implemented for updates and searches.If a request exceeds the request quota, further incoming requests are rejected with HTTP error code 429 (Too Many Requests).-Note that rate limiting works at an instance (JVM) level, not at a core or collection level. Consider that when planning capacity.+Note that rate limiting works at an instance (JVM) level, not at a core or collection level.+Consider that when planning capacity.There is future work planned to have finer grained execution here (https://issues.apache.org/jira/browse/SOLR-14710[SOLR-14710]).== When To Use Rate Limiters-Rate limiters should be used when the user wishes to allocate a guaranteed capacity of the request threadpool to a specific-request type. Indexing and search requests are mostly competing with each other for CPU resources. This becomes especially-pronounced under high stress in production workloads. The current implementation has a query rate limiter which can free up-resources for indexing.+Rate limiters should be used when the user wishes to allocate a guaranteed capacity of the request threadpool to a specific request type.+Indexing and search requests are mostly competing with each other for CPU resources.+This becomes especially pronounced under high stress in production workloads.+The current implementation has a query rate limiter which can free up resources for indexing.== Rate Limiter Configurations-The default rate limiter is search rate limiter. Accordingly, it can be configured using the following command:+The default rate limiter is search rate limiter.+Accordingly, it can be configured using the following command:curl -X POST -H 'Content-type:application/json' -d '{"set-ratelimiter": {@@ -44,40 +47,40 @@ The default rate limiter is search rate limiter. Accordingly, it can be configur}' http://localhost:8983/api/cluster=== Enable Query Rate Limiter-Controls enabling of query rate limiter. Default value is `false`.+Controls enabling of query rate limiter.+Default value is `false`."enabled": true=== Maximum Number Of Concurrent Requests-Allows setting maximum concurrent search requests at a given point in time. Default value is number of cores * 3.+Allows setting maximum concurrent search requests at a given point in time.+Default value is number of cores * 3."allowedRequests":20=== Request Slot Allocation Wait Time-Wait time in ms for which a request will wait for a slot to be available when all slots are full,-before the request is put into the wait queue. This allows requests to have a chance to proceed if-the unavailability of the request slots for this rate limiter is a transient phenomenon. Default value-is -1, indicating no wait. 0 will represent the same -- no wait. Note that higher request allocation times-can lead to larger queue times and can potentially lead to longer wait times for queries.+Wait time in ms for which a request will wait for a slot to be available when all slots are full, before the request is put into the wait queue.+This allows requests to have a chance to proceed if the unavailability of the request slots for this rate limiter is a transient phenomenon.+Default value is -1, indicating no wait.+0 will represent the same -- no wait.+Note that higher request allocation times can lead to larger queue times and can potentially lead to longer wait times for queries."slotAcquisitionTimeoutInMS":70=== Slot Borrowing Enabled-If slot borrowing (described below) is enabled or not. Default value is false.+If slot borrowing (described below) is enabled or not.+Default value is false.-NOTE: This feature is experimental and can cause slots to be blocked if the-borrowing request is long lived.+NOTE: This feature is experimental and can cause slots to be blocked if the borrowing request is long lived."slotBorrowingEnabled":true,=== Guaranteed Slots-The number of guaranteed slots that the query rate limiter will reserve irrespective-of the load of query requests. This is used only if slot borrowing is enabled and acts-as the threshold beyond which query rate limiter will not allow other request types to-borrow slots from its quota. Default value is allowed number of concurrent requests / 2.+The number of guaranteed slots that the query rate limiter will reserve irrespective of the load of query requests.+This is used only if slot borrowing is enabled and acts as the threshold beyond which query rate limiter will not allow other request types to borrow slots from its quota.+Default value is allowed number of concurrent requests / 2.-NOTE: This feature is experimental and can cause slots to be blocked if the-borrowing request is long lived.+NOTE: This feature is experimental and can cause slots to be blocked if theborrowing request is long lived."guaranteedSlots":5,@@ -86,16 +89,13 @@ borrowing request is long lived.These are some of the things to keep in mind when using rate limiters.=== Over Subscribing-It is possible to define a size of quota for a request type which exceeds the size-of the available threadpool. Solr does not enforce rules on the size of a quota that-can be define for a request type. This is intentionally done to allow users full-control on their quota allocation. However, if the quota exceeds the available threadpool's-size, the standard queuing policies of the threadpool will kick in.+It is possible to define a size of quota for a request type which exceeds the size of the available threadpool.+Solr does not enforce rules on the size of a quota that can be define for a request type.+This is intentionally done to allow users full control on their quota allocation.+However, if the quota exceeds the available threadpool's size, the standard queuing policies of the threadpool will kick in.=== Slot Borrowing-If a quota does not have backlog but other quotas do, then the relatively less busier quota can-"borrow" slot from the busier quotas. This is done on a round robin basis today with a futuristic-pending task to make it a priority based model (https://issues.apache.org/jira/browse/SOLR-14709).+If a quota does not have backlog but other quotas do, then the relatively less busier quota can "borrow" slot from the busier quotas.+This is done on a round robin basis today with a futuristic pending task to make it a priority based model (https://issues.apache.org/jira/browse/SOLR-14709).-NOTE: This feature is experimental and gives no guarantee of borrowed slots being-returned in time.+NOTE: This feature is experimental and gives no guarantee of borrowed slots being returned in time.diff --git a/solr/solr-ref-guide/src/regression.adoc b/solr/solr-ref-guide/src/regression.adocindex 4e3dcad9cef..928c60d107f 100644--- a/solr/solr-ref-guide/src/regression.adoc+++ b/solr/solr-ref-guide/src/regression.adoc@@ -20,15 +20,13 @@ The math expressions library supports simple and multivariate linear regression.== Simple Linear Regression-The `regress` function is used to build a linear regression model-between two random variables. Sample observations are provided with two-numeric arrays. The first numeric array is the independent variable and-the second array is the dependent variable.+The `regress` function is used to build a linear regression model between two random variables.+Sample observations are provided with two numeric arrays.+The first numeric array is the independent variable and the second array is the dependent variable.-In the example below the `random` function selects 50000 random samples each containing-the fields `filesize_d` and `response_d`. The two fields are vectorized-and stored in variables `x` and `y`. Then the `regress` function performs a regression-analysis on the two numeric arrays.+In the example below the `random` function selects 50000 random samples each containing the fields `filesize_d` and `response_d`.+The two fields are vectorized and stored in variables `x` and `y`.+Then the `regress` function performs a regression analysis on the two numeric arrays.The `regress` function returns a single tuple with the results of the regressionanalysis.@@ -41,8 +39,8 @@ let(a=random(logs, q="*:*", rows="50000", fl="filesize_d, response_d"),r=regress(x, y))-----Note that in this regression analysis the value of `RSquared` is `.75`. This means that changes in-`filesize_d` explain 75% of the variability of the `response_d` variable:+Note that in this regression analysis the value of `RSquared` is `.75`.+This means that changes in `filesize_d` explain 75% of the variability of the `response_d` variable:[source,json]----@@ -79,11 +77,9 @@ image::images/math-expressions/diagnostics.png[]=== PredictionThe `predict` function uses the regression model to make predictions.-Using the example above the regression model can be used to predict the value-of `response_d` given a value for `filesize_d`.+Using the example above the regression model can be used to predict the value of `response_d` given a value for `filesize_d`.-In the example below the `predict` function uses the regression analysis to predict-the value of `response_d` for the `filesize_d` value of `40000`.+In the example below the `predict` function uses the regression analysis to predict the value of `response_d` for the `filesize_d` value of `40000`.[source,text]----@@ -113,11 +109,10 @@ When this expression is sent to the `/stream` handler it responds with:}-----The `predict` function can also make predictions for an array of values. In this-case it returns an array of predictions.+The `predict` function can also make predictions for an array of values.+In this case it returns an array of predictions.-In the example below the `predict` function uses the regression analysis to-predict values for each of the 5000 samples of `filesize_d` used to generate the model.+In the example below the `predict` function uses the regression analysis to predict values for each of the 5000 samples of `filesize_d` used to generate the model.In this case 5000 predictions are returned.[source,text]@@ -162,23 +157,19 @@ When this expression is sent to the `/stream` handler it responds with:=== Regression Plot-Using `zplot` and the Zeppelin-Solr interpreter we can visualize both the observations and the predictions in-the same scatter plot.-In the example below `zplot` is plotting the `filesize_d` observations on the-x-axis, the `response_d` observations on the y-axis and the predictions on the y1-axis.+Using `zplot` and the Zeppelin-Solr interpreter we can visualize both the observations and the predictions in the same scatter plot.+In the example below `zplot` is plotting the `filesize_d` observations on the x-axis, the `response_d` observations on the y-axis and the predictions on the y1-axis.image::images/math-expressions/linear.png[]=== Residuals-The difference between the observed value and the predicted value is known as the-residual. There isn't a specific function to calculate the residuals but vector-math can used to perform the calculation.+The difference between the observed value and the predicted value is known as the residual.+There isn't a specific function to calculate the residuals but vector math can used to perform the calculation.-In the example below the predictions are stored in variable `p`. The `ebeSubtract`-function is then used to subtract the predictions-from the actual `response_d` values stored in variable `y`. Variable `e` contains-the array of residuals.+In the example below the predictions are stored in variable `p`.+The `ebeSubtract` function is then used to subtract the predictions from the actual `response_d` values stored in variable `y`.+Variable `e` contains the array of residuals.[source,text]----@@ -225,13 +216,13 @@ When this expression is sent to the `/stream` handler it responds with:=== Residual Plot-Using `zplot` and Zeppelin-Solr we can visualize the residuals with-a residuals plot. The example residual plot below plots the predicted value on the-x-axis and the error of the prediction on the y-axis.+Using `zplot` and Zeppelin-Solr we can visualize the residuals with a residuals plot.+The example residual plot below plots the predicted value on the x-axis and the error of the prediction on the y-axis.image::images/math-expressions/residual-plot.png[]-The residual plot can be used to interpret reliability of the model. Three things to look for are:+The residual plot can be used to interpret reliability of the model.+Three things to look for are:. Do the residuals appear to be normally distributed with a mean of 0?This makes it easier to interpret the results of the model to determine if the distribution of the errors is acceptable for predictions.@@ -247,17 +238,16 @@ If the residuals are heteroscedastic it means that we can trust the models error== Multivariate Linear Regression-The `olsRegress` function performs a multivariate linear regression analysis. Multivariate linear-regression models the linear relationship between two or more independent variables and a dependent variable.+The `olsRegress` function performs a multivariate linear regression analysis.+Multivariate linear regression models the linear relationship between two or more independent variables and a dependent variable.-The example below extends the simple linear regression example by introducing a new independent variable-called `load_d`. The `load_d` variable is the load on the network while the file is being downloaded.+The example below extends the simple linear regression example by introducing a new independent variable called `load_d`.+The `load_d` variable is the load on the network while the file is being downloaded.-Notice that the two independent variables `filesize_d` and `load_d` are vectorized and stored-in the variables `b` and `c`. The variables `b` and `c` are then added as rows to a `matrix`. The matrix is-then transposed so that each row in the matrix represents one observation with `filesize_d` and `service_d`.-The `olsRegress` function then performs the multivariate regression analysis using the observation matrix as the-independent variables and the `response_d` values, stored in variable *`d`*, as the dependent variable.+Notice that the two independent variables `filesize_d` and `load_d` are vectorized and stored in the variables `b` and `c`.+The variables `b` and `c` are then added as rows to a `matrix`.+The matrix is then transposed so that each row in the matrix represents one observation with `filesize_d` and `service_d`.+The `olsRegress` function then performs the multivariate regression analysis using the observation matrix as the independent variables and the `response_d` values, stored in variable *`d`*, as the dependent variable.[source,text]----@@ -269,8 +259,8 @@ let(a=random(testapp, q="*:*", rows="30000", fl="filesize_d, load_d, response_d"r=olsRegress(m, z))-----Notice in the response that the `RSquared` of the regression analysis is `1`. This means that linear relationship between-`filesize_d` and `service_d` describe 100% of the variability of the `response_d` variable:+Notice in the response that the `RSquared` of the regression analysis is `1`.+This means that linear relationship between `filesize_d` and `service_d` describe 100% of the variability of the `response_d` variable:[source,json]----@@ -324,8 +314,8 @@ Notice in the response that the `RSquared` of the regression analysis is `1`. ThThe `predict` function can also be used to make predictions for multivariate linear regression.Below is an example of a single prediction using the multivariate linear regression model and a single observation.-The observation is an array that matches the structure of the observation matrix used to build the model. In this case-the first value represents a `filesize_d` of `40000` and the second value represents a `load_d` of `4`.+The observation is an array that matches the structure of the observation matrix used to build the model.+In this case the first value represents a `filesize_d` of `40000` and the second value represents a `load_d` of `4`.[source,text]----@@ -357,11 +347,10 @@ When this expression is sent to the `/stream` handler it responds with:}-----The `predict` function can also make predictions for more than one multivariate observation. In this scenario-an observation matrix used.+The `predict` function can also make predictions for more than one multivariate observation.+In this scenario an observation matrix used.-In the example below the observation matrix used to build the multivariate regression model-is passed to the `predict` function and it returns an array of predictions.+In the example below the observation matrix used to build the multivariate regression model is passed to the `predict` function and it returns an array of predictions.[source,text]----@@ -408,11 +397,10 @@ When this expression is sent to the `/stream` handler it responds with:=== Residuals-Once the predictions are generated the residuals can be calculated using the same approach used with-simple linear regression.+Once the predictions are generated the residuals can be calculated using the same approach used with simple linear regression.-Below is an example of the residuals calculation following a multivariate linear regression. In the example-the predictions stored variable `g` are subtracted from observed values stored in variable `d`.+Below is an example of the residuals calculation following a multivariate linear regression.+In the example the predictions stored variable `g` are subtracted from observed values stored in variable `d`.[source,text]----diff --git a/solr/solr-ref-guide/src/reindexing.adoc b/solr/solr-ref-guide/src/reindexing.adocindex ad99c22d986..55853823a17 100644--- a/solr/solr-ref-guide/src/reindexing.adoc+++ b/solr/solr-ref-guide/src/reindexing.adoc@@ -35,23 +35,34 @@ Reindexing is recommended during major upgrades, so in addition to covering what=== Schema Changes-With very few exceptions, changes to a collection's schema require reindexing. This is because many of the available options are only applied during the indexing process. Solr has no way to implement the desired change without reindexing the data.+With very few exceptions, changes to a collection's schema require reindexing.+This is because many of the available options are only applied during the indexing process.+Solr has no way to implement the desired change without reindexing the data.-To understand the general reason why reindexing is ever required, it's helpful to understand the relationship between Solr's schema and the underlying Lucene index. Lucene does not use a schema, schemas a Solr-only concept. When you change Solr's schema, the Lucene index is not modified in any way.+To understand the general reason why reindexing is ever required, it's helpful to understand the relationship between Solr's schema and the underlying Lucene index.+Lucene does not use a schema, schemas a Solr-only concept.+When you change Solr's schema, the Lucene index is not modified in any way.-This means that there are many types of schema changes that cannot be reflected in the index simply by modifying Solr's schema. This is different from most database models where schemas are used. When indexing, Solr's schema acts like a rulebook for indexing documents by telling Lucene how to interpret the data being sent. Once the documents are in Lucene, Solr's schema has no control over the underlying data structure.+This means that there are many types of schema changes that cannot be reflected in the index simply by modifying Solr's schema.+This is different from most database models where schemas are used.+When indexing, Solr's schema acts like a rulebook for indexing documents by telling Lucene how to interpret the data being sent.+Once the documents are in Lucene, Solr's schema has no control over the underlying data structure.-In addition, changing the schema `version` property is equivalent to changing field type properties. This type of change is usually only made during or because of a major upgrade.+In addition, changing the schema `version` property is equivalent to changing field type properties.+This type of change is usually only made during or because of a major upgrade.==== Changing Field and Field Type Properties-When you change your schema by adding fields, removing fields, or changing the field or field type definitions you generally do so with the intent that those changes alter how documents are searched. The full effects of those changes are not reflected in the corpus as a whole until all documents are reindexed.+When you change your schema by adding fields, removing fields, or changing the field or field type definitions you generally do so with the intent that those changes alter how documents are searched.+The full effects of those changes are not reflected in the corpus as a whole until all documents are reindexed.Changes to *any* field/field type property described in <<field-type-definitions-and-properties.adoc#field-type-properties,Field Type Properties>> must be reindexed in order for the change to be reflected in _all_ documents.[CAUTION]====-Changing field properties that affect indexing without reindexing is not recommended. This should only be attempted with a thorough understanding of the consequences. Negative impacts on the user may not be immediately apparent.+Changing field properties that affect indexing without reindexing is not recommended.+This should only be attempted with a thorough understanding of the consequences.+Negative impacts on the user may not be immediately apparent.====@@ -64,26 +75,37 @@ If separate analysis chains are defined for query and indexing events for a fielAny change to the index-time analysis chain requires reindexing in almost all cases.=== Solrconfig Changes-Identifying changes to solrconfig.xml that alter how data is ingested and thus require reindexing is less straightforward. The general rule is "anything that changes what gets stored in the index requires reindexing". Here are several known examples.+Identifying changes to solrconfig.xml that alter how data is ingested and thus require reindexing is less straightforward.+The general rule is "anything that changes what gets stored in the index requires reindexing". Here are several known examples.-The parameter `luceneMatchVersion` in solrconfig.xml controls the compatibility of Solr with Lucene. Since this parameter can change the rules for analysis behind the scenes, it's always recommended to reindex when changing it. Usually this is only changed in conjunction with a major upgrade.+The parameter `luceneMatchVersion` in solrconfig.xml controls the compatibility of Solr with Lucene.+Since this parameter can change the rules for analysis behind the scenes, it's always recommended to reindex when changing it.+Usually this is only changed in conjunction with a major upgrade.-If you make a change to Solr's <<update-request-processors.adoc#,Update Request Processors>>, it's generally because you want to change something about how _update requests_ (documents) are _processed_ (indexed). In this case, we recommend that you reindex your documents to implement the changes you've made just as if you had changed the schema.+If you make a change to Solr's <<update-request-processors.adoc#,Update Request Processors>>, it's generally because you want to change something about how _update requests_ (documents) are _processed_ (indexed).+In this case, we recommend that you reindex your documents to implement the changes you've made just as if you had changed the schema.Similarly, if you change the `codecFactory` parameter in `solrconfig.xml`, it is again strongly recommended that youplan to reindex your documents to avoid unintended behavior.== Upgrades-When upgrading between major versions (for example, from a 7.x release to 8.x), a best practice is to always reindex your data. The reason for this is that subtle changes may occur in default field type definitions or the underlying code.+When upgrading between major versions (for example, from a 7.x release to 8.x), a best practice is to always reindex your data.+The reason for this is that subtle changes may occur in default field type definitions or the underlying code.-Lucene works hard to insure one major version back-compatability, thus Solr 8x functions with indexes created with Solr 7x. However, given that this guarantee does _not_ apply to Solr X-2 (Solr 6x in this example) we still recommend completely reindexing when moving from Solr X-1 to Solr X.+Lucene works hard to insure one major version back-compatability, thus Solr 8x functions with indexes created with Solr 7x.+However, given that this guarantee does _not_ apply to Solr X-2 (Solr 6x in this example) we still recommend completely reindexing when moving from Solr X-1 to Solr X.[NOTE]-If you have *not* changed your schema as part of an upgrade from one minor release to another (such as, from 7.x to a later 7.x release), you can often skip reindexing your documents. However, when upgrading to a major release, you should plan to reindex your documents.+If you have *not* changed your schema as part of an upgrade from one minor release to another (such as, from 7.x to a later 7.x release), you can often skip reindexing your documents.+However, when upgrading to a major release, you should plan to reindex your documents.[NOTE]-You must always reindex your corpus when upgrading an index produced with a Solr version more than X-1 old. For instance, if you're upgrading to Solr 8x, an index ever used by Solr 6x must be deleted and re-ingested as outlined below. A marker is written identifying the version of Lucene used to ingest the first document. That marker is preserved in the index forever unless the index is entirely deleted. If Lucene finds a marker more than X-1 major versions old, it will refuse to open the index.+You must always reindex your corpus when upgrading an index produced with a Solr version more than X-1 old.+For instance, if you're upgrading to Solr 8x, an index ever used by Solr 6x must be deleted and re-ingested as outlined below.+A marker is written identifying the version of Lucene used to ingest the first document.+That marker is preserved in the index forever unless the index is entirely deleted.+If Lucene finds a marker more than X-1 major versions old, it will refuse to open the index.== Reindexing Strategies@@ -94,7 +116,8 @@ They allow you to recreate the Lucene index without having Lucene segments linge[CAUTION]====-A Lucene index is a _lossy abstraction designed for fast search_. Once a document is added to the index, the original data cannot be assumed to be available. Therefore it is not possible for Lucene to "fix up" existing documents to reflect changes to the schema, they must be indexed again.+A Lucene index is a _lossy abstraction designed for fast search_. Once a document is added to the index, the original data cannot be assumed to be available.+Therefore it is not possible for Lucene to "fix up" existing documents to reflect changes to the schema, they must be indexed again.There are a number of technical reasons that make re-ingesting all documents correctly without deleting the entire corpus first difficult and error-prone to code and maintain.@@ -112,7 +135,8 @@ curl -X POST -H 'Content-Type: application/json' --data-binary '{"delete":{"querIt's important to verify that *all* documents have been deleted, as that ensures the Lucene index segments have beendeleted as well.-To verify that there are no segments in your index, look in the data/index directory and confirm it has no segments files. Since the data directory can be customized, see the section <<index-location-format.adoc#specifying-a-location-for-index-data-with-the-datadir-parameter,Specifying a Location for Index Data with the dataDir Parameter>> for the location of your index files.+To verify that there are no segments in your index, look in the data/index directory and confirm it has no segments files.+Since the data directory can be customized, see the section <<index-location-format.adoc#specifying-a-location-for-index-data-with-the-datadir-parameter,Specifying a Location for Index Data with the dataDir Parameter>> for the location of your index files.Note you will need to verify the indexes have been removed in every shard and every replica on every node of a cluster.It is not sufficient to only query for the number of documents because you may have no documents but still have index@@ -129,12 +153,14 @@ Another approach is to use index to a new collection and use Solr's <<alias-manaThis option is only available for Solr installations running in SolrCloud mode.-With this approach, you will index your documents into a new collection that uses your changes and, once indexing and testing are complete, create an alias that points your front-end at the new collection. From that point, new queries and updates will be routed to the new collection seamlessly.+With this approach, you will index your documents into a new collection that uses your changes and, once indexing and testing are complete, create an alias that points your front-end at the new collection.+From that point, new queries and updates will be routed to the new collection seamlessly.Once the alias is in place and you are satisfied you no longer need the old data, you can delete the old collection with the Collections API <<collection-management.adoc#delete,DELETE command>>.[NOTE]-One advantage of this option is that if you you can switch back to the old collection if you discover problems our testing did not uncover. Of course this option can require more resources until the old collection can be deleted.+One advantage of this option is that if you you can switch back to the old collection if you discover problems our testing did not uncover.+Of course this option can require more resources until the old collection can be deleted.== Changes that Do Not Require Reindexdiff --git a/solr/solr-ref-guide/src/relevance.adoc b/solr/solr-ref-guide/src/relevance.adocindex aed2bfb7f01..b5912f8006a 100644--- a/solr/solr-ref-guide/src/relevance.adoc+++ b/solr/solr-ref-guide/src/relevance.adoc@@ -18,20 +18,33 @@*Relevance* is the degree to which a query response satisfies a user who is searching for information.-The relevance of a query response depends on the context in which the query was performed. A single search application may be used in different contexts by users with different needs and expectations. For example, a search engine of climate data might be used by a university researcher studying long-term climate trends, a farmer interested in calculating the likely date of the last frost of spring, a civil engineer interested in rainfall patterns and the frequency of floods, and a college student planning a vacation to a region and wondering what to pack. Because the motivations of these users vary, the relevance of any particular response to a query will vary as well.+The relevance of a query response depends on the context in which the query was performed.+A single search application may be used in different contexts by users with different needs and expectations.+For example, a search engine of climate data might be used by a university researcher studying long-term climate trends, a farmer interested in calculating the likely date of the last frost of spring, a civil engineer interested in rainfall patterns and the frequency of floods, and a college student planning a vacation to a region and wondering what to pack.+Because the motivations of these users vary, the relevance of any particular response to a query will vary as well.-How comprehensive should query responses be? Like relevance in general, the answer to this question depends on the context of a search. The cost of _not_ finding a particular document in response to a query is high in some contexts, such as a legal e-discovery search in response to a subpoena, and quite low in others, such as a search for a cake recipe on a Web site with dozens or hundreds of cake recipes. When configuring Solr, you should weigh comprehensiveness against other factors such as timeliness and ease-of-use.+How comprehensive should query responses be? Like relevance in general, the answer to this question depends on the context of a search.+The cost of _not_ finding a particular document in response to a query is high in some contexts, such as a legal e-discovery search in response to a subpoena, and quite low in others, such as a search for a cake recipe on a Web site with dozens or hundreds of cake recipes.+When configuring Solr, you should weigh comprehensiveness against other factors such as timeliness and ease-of-use.The e-discovery and recipe examples demonstrate the importance of two concepts related to relevance:* *Precision* is the percentage of documents in the returned results that are relevant.-* *Recall* is the percentage of relevant results returned out of all relevant results in the system. Obtaining perfect recall is trivial: simply return every document in the collection for every query.+* *Recall* is the percentage of relevant results returned out of all relevant results in the system.+Obtaining perfect recall is trivial: simply return every document in the collection for every query.-Returning to the examples above, it's important for an e-discovery search application to have 100% recall returning all the documents that are relevant to a subpoena. It's far less important that a recipe application offer this degree of precision, however. In some cases, returning too many results in casual contexts could overwhelm users. In some contexts, returning fewer results that have a higher likelihood of relevance may be the best approach.+Returning to the examples above, it's important for an e-discovery search application to have 100% recall returning all the documents that are relevant to a subpoena.+It's far less important that a recipe application offer this degree of precision, however.+In some cases, returning too many results in casual contexts could overwhelm users.+In some contexts, returning fewer results that have a higher likelihood of relevance may be the best approach.-Using the concepts of precision and recall, it's possible to quantify relevance across users and queries for a collection of documents. A perfect system would have 100% precision and 100% recall for every user and every query. In other words, it would retrieve all the relevant documents and nothing else. In practical terms, when talking about precision and recall in real systems, it is common to focus on precision and recall at a certain number of results, the most common (and useful) being ten results.+Using the concepts of precision and recall, it's possible to quantify relevance across users and queries for a collection of documents.+A perfect system would have 100% precision and 100% recall for every user and every query.+In other words, it would retrieve all the relevant documents and nothing else.+In practical terms, when talking about precision and recall in real systems, it is common to focus on precision and recall at a certain number of results, the most common (and useful) being ten results.-Through faceting, query filters, and other search components, a Solr application can be configured with the flexibility to help users fine-tune their searches in order to return the most relevant results for users. That is, Solr can be configured to balance precision and recall to meet the needs of a particular user community.+Through faceting, query filters, and other search components, a Solr application can be configured with the flexibility to help users fine-tune their searches in order to return the most relevant results for users.+That is, Solr can be configured to balance precision and recall to meet the needs of a particular user community.The configuration of a Solr application should take into account:@@ -40,6 +53,7 @@ The configuration of a Solr application should take into account:* any inherent relevance of documents (e.g., it might make sense to ensure that an official product description or FAQ is always returned near the top of the search results)* whether or not the age of documents matters significantly (in some contexts, the most recent documents might always be the most important)-Keeping all these factors in mind, it's often helpful in the planning stages of a Solr deployment to sketch out the types of responses you think the search application should return for sample queries. Once the application is up and running, you can employ a series of testing methodologies, such as focus groups, in-house testing, http://trec.nist.gov[TREC] tests, and A/B testing to fine tune the configuration of the application to best meet the needs of its users.+Keeping all these factors in mind, it's often helpful in the planning stages of a Solr deployment to sketch out the types of responses you think the search application should return for sample queries.+Once the application is up and running, you can employ a series of testing methodologies, such as focus groups, in-house testing, http://trec.nist.gov[TREC] tests, and A/B testing to fine tune the configuration of the application to best meet the needs of its users.For more information about relevance, see Grant Ingersoll's blog post https://lucidworks.com/post/debugging-search-application-relevance-issues/[Debugging Search Application Relevance Issues].diff --git a/solr/solr-ref-guide/src/replica-placement-plugins.adoc b/solr/solr-ref-guide/src/replica-placement-plugins.adocindex b7b7a43b16f..5299a61ef29 100644--- a/solr/solr-ref-guide/src/replica-placement-plugins.adoc+++ b/solr/solr-ref-guide/src/replica-placement-plugins.adoc@@ -19,50 +19,36 @@// under the License.== Introduction-When user wants to create a new collection or add a replica to one of the existing-collections Solr needs to first determine on what node to put the replica so that the-cluster resources are allocated in a well-balanced manner (according to some criteria).+When user wants to create a new collection or add a replica to one of the existing collections Solr needs to first determine on what node to put the replica so that the cluster resources are allocated in a well-balanced manner (according to some criteria).Replica placement plugin is the configurable component that determines these placements.-It can also enforce additional constraints on operations such as collection or replica removal-- for example if the plugin wants some collections to be always co-located on the same nodes,-or always present when some other collection is present.+It can also enforce additional constraints on operations such as collection or replica removal - for example if the plugin wants some collections to be always co-located on the same nodes, or always present when some other collection is present.-(In Solr 8.x this functionality was using either per-collection rules, or it could be configured-in the autoscaling framework).+(In Solr 8.x this functionality was using either per-collection rules, or it could be configured in the autoscaling framework).=== Plugin ConfigurationReplica placement plugin configurations are maintained using the `/cluster/plugin` API.-There can be only one cluster-wide plugin configuration at a time, and it uses a pre-defined-plugin name: `.placement-plugin`.+There can be only one cluster-wide plugin configuration at a time, and it uses a pre-defined plugin name: `.placement-plugin`.-There are several placement plugins included in the Solr distribution. Additional placement-plugins can be added - they have to implement a `PlacementPluginFactory` interface. The-configuration entry may also contain a `config` element if the plugin implements the-`ConfigurablePlugin` interface.+There are several placement plugins included in the Solr distribution.+Additional placement plugins can be added - they have to implement a `PlacementPluginFactory` interface.+The configuration entry may also contain a `config` element if the plugin implements the `ConfigurablePlugin` interface.=== Legacy Replica Placement-By default Solr 9.0 uses a legacy method for replica placements, which doesn't use a placement-plugin at all. This method is used whenever the placement plugin configuration is missing or-invalid.+By default Solr 9.0 uses a legacy method for replica placements, which doesn't use a placement plugin at all.+This method is used whenever the placement plugin configuration is missing or invalid.-Legacy placement simply assigns new replicas to live nodes in a round-robin fashion: first it-prepares a sorted list of nodes with the smallest number of existing replicas of the collection.-Then for each shard in the request it adds the replicas to consecutive nodes in this order,-wrapping around to the first node if the number of replicas is larger than the number of nodes.+Legacy placement simply assigns new replicas to live nodes in a round-robin fashion: first it prepares a sorted list of nodes with the smallest number of existing replicas of the collection.+Then for each shard in the request it adds the replicas to consecutive nodes in this order, wrapping around to the first node if the number of replicas is larger than the number of nodes.-This placement strategy doesn't ensure that no more than 1 replica of a shard is placed on the-same node. Also, the round-robin assignment only roughly approximates an even spread of replicas-across the nodes.+This placement strategy doesn't ensure that no more than 1 replica of a shard is placed on the same node.+Also, the round-robin assignment only roughly approximates an even spread of replicas across the nodes.=== Placement plugins included in Solr-The following placement plugins are available out-of-the-box in Solr 9.0 (however, as-described above the default configuration doesn't use any of them, instead it uses the legacy-replica placement method).+The following placement plugins are available out-of-the-box in Solr 9.0 (however, as described above the default configuration doesn't use any of them, instead it uses the legacy replica placement method).In order to use a plugin its configuration must be added using the `/cluster/plugin` API.-For example, in order to use the `AffinityPlacementFactory` plugin the following command-should be executed:+For example, in order to use the `AffinityPlacementFactory` plugin the following command should be executed:[source,bash]----@@ -93,26 +79,23 @@ There can be only one (or none) placement configuration defined.==== `RandomPlacementFactory`-This plugin creates random placements, while preventing two replicas of the same shard from being-placed on the same node. If there are too few nodes to satisfy these constraints an exception is-thrown, and the request is rejected.+This plugin creates random placements, while preventing two replicas of the same shard from being placed on the same node.+If there are too few nodes to satisfy these constraints an exception is thrown, and the request is rejected.Due to its simplistic algorithm this plugin should only be used in simple deployments.This plugin doesn't require any configuration.==== `MinimizeCoresPlacementFactory`-This plugin creates placements that minimize the number of cores per node across all live nodes,-while not placing two replicas of the same shard on the same node. If there are too few nodes-to satisfy these constraints an exception is thrown, and the request is rejected.+This plugin creates placements that minimize the number of cores per node across all live nodes, while not placing two replicas of the same shard on the same node.+If there are too few nodes to satisfy these constraints an exception is thrown, and the request is rejected.Due to its simplistic algorithm this plugin should only be used in simple deployments.This plugin doesn't require any configuration.==== `AffinityPlacementFactory`-This plugin implements replica placement algorithm that roughly replicates Solr 8.x autoscaling-configuration defined https://github.com/lucidworks/fusion-cloud-native/blob/master/policy.json#L16[here].+This plugin implements replica placement algorithm that roughly replicates Solr 8.x autoscaling configuration defined https://github.com/lucidworks/fusion-cloud-native/blob/master/policy.json#L16[here].The autoscaling specification in the configuration linked above aimed to do the following:@@ -125,58 +108,43 @@ The autoscaling specification in the configuration linked above aimed to do theIt also supports additional per-collection constraints:-* `withCollection` constraint enforces the placement of co-located collections' replicas on the-same nodes, and prevents deletions of collections and replicas that would break this constraint.-* `nodeType` constraint limits the nodes eligible for placement to only those that match one or-more of the specified node types.+* `withCollection` constraint enforces the placement of co-located collections' replicas on the same nodes, and prevents deletions of collections and replicas that would break this constraint.+* `nodeType` constraint limits the nodes eligible for placement to only those that match one or more of the specified node types.See below for more details on these constraints.Overall strategy of this plugin:-* The set of nodes in the cluster is obtained, and if the `withCollection` mapping is present- and applicable to the current collection then this candidate set is filtered so that only- eligible nodes remain according to this constraint.+* The set of nodes in the cluster is obtained, and if the `withCollection` mapping is present and applicable to the current collection then this candidate set is filtered so that only eligible nodes remain according to this constraint.* The resulting node set is transformed into 3 independent sets (that can overlap) of nodes accepting each of the three replica types (NRT, TLOG and PULL).* For each shard on which placing replicas is required and then for each replica type to place (starting with NRT, then TLOG then PULL):-** The set of candidates nodes corresponding to the replica type is used and from that set are removed nodes that already have a replica (of any type) for that shard+** The set of candidates nodes corresponding to the replica type is used and from that set are removed nodes that already have a replica (of any type) for that shard.** If there are not enough nodes, an error is thrown (this is checked further down during processing).** The number of (already existing) replicas of the current type on each Availability Zone is collected.** Separate the set of available nodes to as many subsets (possibly some are empty) as there are Availability Zones defined for the candidate nodes** In each AZ nodes subset, sort the nodes by increasing total number of cores count.** Iterate over the number of replicas to place (for the current replica type for the current shard):-*** Based on the number of replicas per AZ collected previously, pick the non-empty set of nodes having the lowest number of replicas. Then pick the first node in that set. That's the node the replica is placed one. Remove the node from the set of available nodes for the given AZ and increase the number of replicas placed on that AZ.+*** Based on the number of replicas per AZ collected previously, pick the non-empty set of nodes having the lowest number of replicas.+Then pick the first node in that set.+That's the node the replica is placed one.+Remove the node from the set of available nodes for the given AZ and increase the number of replicas placed on that AZ.** During this process, the number of cores on the nodes in general is tracked to take into account placement decisions so that not all shards decide to put their replicas on the same nodes (they might though if these are the less loaded nodes).-NOTE: At the moment the names of availability zone property and the name of the replica type-property are not configurable, and set respectively to `availability_zone` and `replica_type`.+NOTE: At the moment the names of availability zone property and the name of the replica type property are not configurable, and set respectively to `availability_zone` and `replica_type`.===== `withCollection` constraint-This plugin supports enforcing additional constraint named `withCollection`, which causes-replicas of two paired collections to be placed on the same nodes.--Users can define the collection pairs in the plugin configuration, in the `config/withCollection`-element, which is a JSON map where keys are the primary collection names, and the values are the-secondary collection names (currently only 1:1 mapping is supported - however, multiple primary-collections may use the same secondary collection, which effectively relaxes this to N:1 mapping).--Unlike previous versions of Solr, this plugin does NOT automatically place replicas of the-secondary collection - those replicas are assumed to be already in place, and it's the-responsibility of the user to already place them on the right nodes (most likely simply by-using this plugin to create the secondary collection first, with large enough replication-factor to ensure that the target node set is populated with secondary replicas).--When a request to compute placements is processed for the primary collection that has a-key in the `withCollection` map, the set of candidate nodes is first filtered to eliminate nodes-that don't contain the replicas of the secondary collection. Please note that this may-result in an empty set, and an exception - in this case the sufficient number of secondary-replicas needs to be created first.--The plugin preserves this co-location by rejecting delete operation of secondary collections-(or their replicas) if they are still in use on the nodes where primary replicas are located-- requests to do so will be rejected with errors. In order to delete a secondary collection-(or its replicas) from these nodes first the replicas of the primary collection must be-removed from the co-located nodes, or the configuration must be changed to remove the+This plugin supports enforcing additional constraint named `withCollection`, which causes replicas of two paired collections to be placed on the same nodes.++Users can define the collection pairs in the plugin configuration, in the `config/withCollection` element, which is a JSON map where keys are the primary collection names, and the values are the secondary collection names.+Currently only 1:1 mapping is supported - however, multiple primary collections may use the same secondary collection, which effectively relaxes this to N:1 mapping.++Unlike previous versions of Solr, this plugin does NOT automatically place replicas of the secondary collection - those replicas are assumed to be already in place, and it's the responsibility of the user to already place them on the right nodes (most likely simply by using this plugin to create the secondary collection first, with large enough replication factor to ensure that the target node set is populated with secondary replicas).++When a request to compute placements is processed for the primary collection that has a key in the `withCollection` map, the set of candidate nodes is first filtered to eliminate nodes that don't contain the replicas of the secondary collection.+Please note that this may result in an empty set, and an exception - in this case the sufficient number of secondary replicas needs to be created first.++The plugin preserves this co-location by rejecting delete operation of secondary collections (or their replicas) if they are still in use on the nodes where primary replicas are located - requests to do so will be rejected with errors.+In order to delete a secondary collection (or its replicas) from these nodes first the replicas of the primary collection must be removed from the co-located nodes, or the configuration must be changed to remove theco-location mapping for the primary collection.===== `nodeType` constraint@@ -186,29 +154,24 @@ co-location mapping for the primary collection.This plugin supports the following configuration parameters:`minimalFreeDiskGB`::-(optional, integer) if a node has strictly less GB of free disk than this value, the node is-excluded from assignment decisions. Set to 0 or less to disable. Default value is 10.+(optional, integer) if a node has strictly less GB of free disk than this value, the node is excluded from assignment decisions.+Set to 0 or less to disable.+Default value is 10.`prioritizedFreeDiskGB`::-(optional, integer) replica allocation will assign replicas to nodes with at least this number-of GB of free disk space regardless of the number of cores on these nodes rather than assigning-replicas to nodes with less than this amount of free disk space if that's an option (if that's-not an option, replicas can still be assigned to nodes with less than this amount of free space).+(optional, integer) replica allocation will assign replicas to nodes with at least this number of GB of free disk space regardless of the number of cores on these nodes rather than assigning replicas to nodes with less than this amount of free disk space if that's an option (if that's not an option, replicas can still be assigned to nodes with less than this amount of free space).Default value is 100.`withCollection`::-(optional, map) this property defines an additional constraint that primary collections (keys)-must be located on the same nodes as the secondary collections (values). The plugin will-assume that the secondary collection replicas are already in place and ignore candidate-nodes where they are not already present. Default value is none.+(optional, map) this property defines an additional constraint that primary collections (keys) must be located on the same nodes as the secondary collections (values).+The plugin will assume that the secondary collection replicas are already in place and ignore candidate nodes where they are not already present.+Default value is none.`nodeType`::(optional, map) this property defines an additional constraint that collections (keys)-must be located only on the nodes that are labeled with one or more of the matching-"node type" labels (values in the map are comma-separated labels). Nodes are labeled using the-`node_type` system property with the value being an arbitrary comma-separated list of labels.-Correspondingly, the plugin configuration can specify that a particular collection must be placed-only on the nodes that match at least one of the (comma-separated) labels defined here.+must be located only on the nodes that are labeled with one or more of the matching "node type" labels (values in the map are comma-separated labels).+Nodes are labeled using the `node_type` system property with the value being an arbitrary comma-separated list of labels.+Correspondingly, the plugin configuration can specify that a particular collection must be placed only on the nodes that match at least one of the (comma-separated) labels defined here.=== Example ConfigurationsThis is a simple configuration that uses default values:@@ -239,8 +202,7 @@ curl -X POST -H 'Content-type: application/json' -d '{----This configuration defines that collection `A_primary` must be co-located with-collection `Common_secondary`, and collection `B_primary` must be co-located also with the-collection `Common_secondary`:+collection `Common_secondary`, and collection `B_primary` must be co-located also with the collection `Common_secondary`:[source,bash]----@@ -258,10 +220,8 @@ curl -X POST -H 'Content-type: application/json' -d '{http://localhost:8983/api/cluster/plugin-----This configuration defines that collection `collection_A` must be placed only on the nodes with-the `node_type` system property containing either `searchNode` or `indexNode` (for example, a node-may be labeled as `-Dnode_type=searchNode,indexNode,uiNode,zkNode`). Similarly, the-collection `collection_B` must be placed only on the nodes that contain the `analyticsNode` label:+This configuration defines that collection `collection_A` must be placed only on the nodes with the `node_type` system property containing either `searchNode` or `indexNode` (for example, a node may be labeled as `-Dnode_type=searchNode,indexNode,uiNode,zkNode`).+Similarly, the collection `collection_B` must be placed only on the nodes that contain the `analyticsNode` label:[source,bash]----diff --git a/solr/solr-ref-guide/src/request-parameters-api.adoc b/solr/solr-ref-guide/src/request-parameters-api.adocindex 4600de35ce2..4c2daad0190 100644--- a/solr/solr-ref-guide/src/request-parameters-api.adoc+++ b/solr/solr-ref-guide/src/request-parameters-api.adoc@@ -149,7 +149,8 @@ See <<implicit-requesthandlers.adoc#,Implicit Request Handlers>> for the paramse=== Viewing Expanded Paramsets and Effective Parameters with RequestHandlers-To see the expanded paramset and the resulting effective parameters for a RequestHandler defined with `useParams`, use the `expandParams` request parameter. As an example, for the `/export` request handler:+To see the expanded paramset and the resulting effective parameters for a RequestHandler defined with `useParams`, use the `expandParams` request parameter.+As an example, for the `/export` request handler:[source,bash]----diff --git a/solr/solr-ref-guide/src/response-writers.adoc b/solr/solr-ref-guide/src/response-writers.adocindex c40bc267f37..717e0e6bd34 100644--- a/solr/solr-ref-guide/src/response-writers.adoc+++ b/solr/solr-ref-guide/src/response-writers.adoc@@ -149,7 +149,8 @@ The behavior of the XML Response Writer can be driven by the following query parThe `version` parameter determines the XML protocol used in the response.Clients are strongly encouraged to _always_ specify the protocol version, so as to ensure that the format of the response they receive does not change unexpectedly if the Solr server is upgraded and a new default format is introduced.+-The only currently supported version value is `2.2`. The format of the `responseHeader` changed to use the same `<lst>` structure as the rest of the response.+The only currently supported version value is `2.2`.+The format of the `responseHeader` changed to use the same `<lst>` structure as the rest of the response.+The default value is the latest supported.diff --git a/solr/solr-ref-guide/src/result-clustering.adoc b/solr/solr-ref-guide/src/result-clustering.adocindex 54bdd67ccaa..b070d37156a 100644--- a/solr/solr-ref-guide/src/result-clustering.adoc+++ b/solr/solr-ref-guide/src/result-clustering.adoc@@ -202,10 +202,13 @@ The output XML should include search hits and an array of automatically discoverA few clusters discovered for this query (`\*:*`), separate all search hits into various categories: DDR, iPod, Hard Drive, etc.Each cluster has a label and score that indicates the "goodness" of the cluster.The score is algorithm-specific and is meaningful only in relation to the scores of other clusters in the same set.-In other words, if cluster _A_ has a higher score than cluster _B_, cluster _A_ should be of better quality (have a better label and/or more coherent document set). Each cluster has an array of identifiers of documents belonging to it.+In other words, if cluster _A_ has a higher score than cluster _B_, cluster _A_ should be of better quality (have a better label and/or more coherent document set).+Each cluster has an array of identifiers of documents belonging to it.These identifiers correspond to the `uniqueKey` field declared in the schema.-Sometimes cluster labels may not make much sense (this depends on many factors -- text in clustered fields, number of documents, algorithm paramerters). Also, some documents may be left out and not be clustered at all; these will be assigned to the synthetic _Other Topics_ group, marked with the `other-topics` property set to `true` (see the XML dump above for an example). The score of the other topics group is zero.+Sometimes cluster labels may not make much sense (this depends on many factors -- text in clustered fields, number of documents, algorithm paramerters).+Also, some documents may be left out and not be clustered at all; these will be assigned to the synthetic _Other Topics_ group, marked with the `other-topics` property set to `true` (see the XML dump above for an example).+The score of the other topics group is zero.== Installation@@ -301,7 +304,8 @@ A commercial clustering algorithm `Lingo3G` plugs into the same extension point.How to choose the Clustering Algorithm?****-The question of which algorithm to choose depends on the amount of traffic, the expected result, and the input data (each algorithm will cluster the input slightly differently). There is no one answer which algorithm is "the best": Lingo3G provides hierarchical clusters, Lingo and STC provide flat clusters.+The question of which algorithm to choose depends on the amount of traffic, the expected result, and the input data (each algorithm will cluster the input slightly differently).+There is no one answer which algorithm is "the best": Lingo3G provides hierarchical clusters, Lingo and STC provide flat clusters.STC is faster than Lingo, but arguably produces less intuitive clusters, Lingo3G is the fastest algorithm but is not free or open source... Experiment and pick one that suits your needs.For a comparison of characteristics of these algorithms see the following links:@@ -316,7 +320,8 @@ in your Solr installation.`clustering.maxLabels`::Maximum number of returned cluster labels (if the algorithm returns more labels, the list will-be truncated). By default all labels are returned.+be truncated).+By default all labels are returned.`clustering.includeSubclusters`::If `true`, sub-clusters are included in the response for algorithms that support hierarchical@@ -334,15 +339,13 @@ This property is `null` by default and all resources are read from their respectIf this property is not empty, it resolves relative to Solr core's configuration directory.This parameter can be applied during Solr startup _only_, it can't be overriden per-request.-[.text-center]-🙪-There are more properties applying to engine configuration.We describe these in functional sections that follow.=== Full Field and Query-Context (Snippet) Clustering-The clustering algorithm can consume full content of fields or just the left and right context around query-matching regions (so-called _snippets_). Contrary to the intuition, using query contexts can increase the quality of clustering even if it feeds less data to the algorithm.+The clustering algorithm can consume full content of fields or just the left and right context around query-matching regions (so-called _snippets_).+Contrary to the intuition, using query contexts can increase the quality of clustering even if it feeds less data to the algorithm.This is typically caused by the fact that snippets are more focused around the phrases and terms surrounding the query and the algorithm has a better signal-to-noise ratio of data to work with.We recommend using query contexts when fields contain a lot of content (this would affect clustering performance).@@ -374,7 +377,8 @@ The provided language must be available and the clustering algorithm must supporName of the document field that stores the document's language.If the field does not exist for a document or the value is blank, the default language is used.-The list of supported languages can change dynamically (languages are loaded via external service provider extension) and may depend on the selected algorithm (algorithms can support a subset of languages for which resources are available). The clustering component will log all supported algorithm-language pairs at Solr startup, so you can inspect what's supported on your particular Solr instance.+The list of supported languages can change dynamically (languages are loaded via external service provider extension) and may depend on the selected algorithm (algorithms can support a subset of languages for which resources are available).+The clustering component will log all supported algorithm-language pairs at Solr startup, so you can inspect what's supported on your particular Solr instance.For example:[source,text]diff --git a/solr/solr-ref-guide/src/result-grouping.adoc b/solr/solr-ref-guide/src/result-grouping.adocindex b7924e1b180..4fe12f608a4 100644--- a/solr/solr-ref-guide/src/result-grouping.adoc+++ b/solr/solr-ref-guide/src/result-grouping.adoc@@ -122,7 +122,8 @@ The default value is `false`.`group.facet`::Determines whether to compute grouped facets for the field facets specified in facet.field parameters.Grouped facets are computed based on the first specified group.-As with normal field faceting, fields shouldn't be tokenized (otherwise counts are computed for each token). Grouped faceting supports single and multivalued fields.+As with normal field faceting, fields shouldn't be tokenized (otherwise counts are computed for each token).+Grouped faceting supports single and multivalued fields.Default is `false`.+WARNING: There can be a heavy performance cost to this option.diff --git a/solr/solr-ref-guide/src/scalar-math.adoc b/solr/solr-ref-guide/src/scalar-math.adocindex 696aa00de67..b219b14be96 100644--- a/solr/solr-ref-guide/src/scalar-math.adoc+++ b/solr/solr-ref-guide/src/scalar-math.adoc@@ -16,8 +16,8 @@// specific language governing permissions and limitations// under the License.-The most basic math expressions are scalar expressions. Scalar expressions-perform mathematical operations on numbers.+The most basic math expressions are scalar expressions.+Scalar expressions perform mathematical operations on numbers.For example the expression below adds two numbers together:@@ -26,8 +26,7 @@ For example the expression below adds two numbers together:add(1, 1)-----When this expression is sent to the `/stream` handler it-responds with:+When this expression is sent to the `/stream` handler it responds with:[source,json]----@@ -46,9 +45,8 @@ responds with:}-----Math expressions can be nested. For example in the expression-below the output of the `add` function is the second parameter-of the `pow` function:+Math expressions can be nested.+For example in the expression below the output of the `add` function is the second parameter of the `pow` function:[source,text]----@@ -76,8 +74,7 @@ This expression returns the following response:== Visualization-In the Zeppelin-Solr interpreter you can simply type in scalar math functions and the-result will be shown in a table format.+In the Zeppelin-Solr interpreter you can simply type in scalar math functions and the result will be shown in a table format.image::images/math-expressions/scalar.png[]@@ -87,20 +84,16 @@ image::images/math-expressions/num.png[]== Streaming Scalar Math-Scalar math expressions can also be applied to each tuple in a stream-through use of the `select` stream decorator. The `select` function wraps a-stream of tuples and selects fields to include in each tuple.-The `select` function can also use math expressions to compute-new values and add them to the outgoing tuples.+Scalar math expressions can also be applied to each tuple in a stream through use of the `select` stream decorator.+The `select` function wraps a stream of tuples and selects fields to include in each tuple.+The `select` function can also use math expressions to compute new values and add them to the outgoing tuples.-In the example below the `select` expression is wrapping a search-expression. The `select` function is selecting the `response_d` field-and computing a new field called `new_response` using the `mult` math-expression.+In the example below the `select` expression is wrapping a search expression.+The `select` function is selecting the `response_d` field and computing a new field called `new_response` using the `mult` math expression.The first parameter of the `mult` expression is the `response_d` field.-The second parameter is the scalar value 10. This multiplies the value-of the `response_d` field in each tuple by 10.+The second parameter is the scalar value 10.+This multiplies the value of the `response_d` field in each tuple by 10.[source,text]----diff --git a/solr/solr-ref-guide/src/schema-browser-screen.adoc b/solr/solr-ref-guide/src/schema-browser-screen.adocindex 69b6c4eb379..5ceab10f727 100644--- a/solr/solr-ref-guide/src/schema-browser-screen.adoc+++ b/solr/solr-ref-guide/src/schema-browser-screen.adoc@@ -18,20 +18,32 @@The Schema Browser screen lets you review schema data in a browser window.-If you have accessed this window from the Analysis screen, it will be opened to a specific field, dynamic field rule or field type. If there is nothing chosen, use the pull-down menu to choose the field or field type.+If you have accessed this window from the Analysis screen, it will be opened to a specific field, dynamic field rule or field type.+If there is nothing chosen, use the pull-down menu to choose the field or field type..Schema Browser Screenimage::images/schema-browser-screen/schema_browser_terms.png[image,height=400]-The screen provides a great deal of useful information about each particular field and fieldtype in the Schema, and provides a quick UI for adding fields or fieldtypes using the <<schema-api.adoc#,Schema API>> (if enabled). In the example above, we have chosen the `cat` field. On the left side of the main view window, we see the field name, that it is copied to the `\_text_` (because of a copyField rule) and that it use the `strings` fieldtype. Click on one of those field or fieldtype names, and you can see the corresponding definitions.+The screen provides a great deal of useful information about each particular field and fieldtype in the Schema, and provides a quick UI for adding fields or fieldtypes using the <<schema-api.adoc#,Schema API>> (if enabled).+In the example above, we have chosen the `cat` field.+On the left side of the main view window, we see the field name, that it is copied to the `\_text_` (because of a copyField rule) and that it uses the `strings` fieldtype.+Click on one of those field or fieldtype names, and you can see the corresponding definitions.-In the right part of the main view, we see the specific properties of how the `cat` field is defined – either explicitly or implicitly via its fieldtype, as well as how many documents have populated this field. Then we see the analyzer used for indexing and query processing. Click the icon to the left of either of those, and you'll see the definitions for the tokenizers and/or filters that are used. The output of these processes is the information you see when testing how content is handled for a particular field with the <<analysis-screen.adoc#,Analysis Screen>>.+In the right part of the main view, we see the specific properties of how the `cat` field is defined – either explicitly or implicitly via its fieldtype, as well as how many documents have populated this field.+Then we see the analyzer used for indexing and query processing.+Click the icon to the left of either of those, and you'll see the definitions for the tokenizers and/or filters that are used.+The output of these processes is the information you see when testing how content is handled for a particular field with the <<analysis-screen.adoc#,Analysis Screen>>.-Under the analyzer information is a button to *Load Term Info*. Clicking that button will show the top _N_ terms that are in a sample shard for that field, as well as a histogram showing the number of terms with various frequencies. Click on a term, and you will be taken to the <<query-screen.adoc#,Query Screen>> to see the results of a query of that term in that field. If you want to always see the term information for a field, choose *Autoload* and it will always appear when there are terms for a field. A histogram shows the number of terms with a given frequency in the field.+Under the analyzer information is a button to *Load Term Info*.+Clicking that button will show the top _N_ terms that are in a sample shard for that field, as well as a histogram showing the number of terms with various frequencies.+Click on a term, and you will be taken to the <<query-screen.adoc#,Query Screen>> to see the results of a query of that term in that field.+If you want to always see the term information for a field, choose *Autoload* and it will always appear when there are terms for a field.+A histogram shows the number of terms with a given frequency in the field.[IMPORTANT]====-Term Information is loaded from single arbitrarily selected core from the collection, to provide a representative sample for the collection. Full <<faceting.adoc#,Field Facet>> query results are needed to see precise term counts across the entire collection.+Term Information is loaded from single arbitrarily selected core from the collection, to provide a representative sample for the collection.+Full <<faceting.adoc#,Field Facet>> query results are needed to see precise term counts across the entire collection.====For programmatic access to the underlying information in this screen please reference the <<luke-request-handler.adoc#,Luke Request Handler>>diff --git a/solr/solr-ref-guide/src/schemaless-mode.adoc b/solr/solr-ref-guide/src/schemaless-mode.adocindex d62e9825ff0..4dcbd7f466c 100644--- a/solr/solr-ref-guide/src/schemaless-mode.adoc+++ b/solr/solr-ref-guide/src/schemaless-mode.adoc@@ -20,13 +20,15 @@ Schemaless Mode is a set of Solr features that, when used together, allow usersThese Solr features, all controlled via `solrconfig.xml`, are:-. Managed schema: Schema modifications are made at runtime through Solr APIs, which requires the use of a `schemaFactory` that supports these changes. See the section <<schema-factory.adoc#,Schema Factory Definition in SolrConfig>> for more details.+. Managed schema: Schema modifications are made at runtime through Solr APIs, which requires the use of a `schemaFactory` that supports these changes.+See the section <<schema-factory.adoc#,Schema Factory Definition in SolrConfig>> for more details.. Field value class guessing: Previously unseen fields are run through a cascading set of value-based parsers, which guess the Java class of field values - parsers for Boolean, Integer, Long, Float, Double, and Date are currently available.. Automatic schema field addition, based on field value class(es): Previously unseen fields are added to the schema, based on field value Java classes, which are mapped to schema field types - see <<field-types.adoc#,Field Types>>.== Using the Schemaless Example-The three features of schemaless mode are pre-configured in the `_default` <<config-sets.adoc#,configset>> in the Solr distribution. To start an example instance of Solr using these configs, run the following command:+The three features of schemaless mode are pre-configured in the `_default` <<config-sets.adoc#,configset>> in the Solr distribution.+To start an example instance of Solr using these configs, run the following command:[source,bash]----@@ -66,7 +68,9 @@ You can use the `/schema/fields` <<schema-api.adoc#,Schema API>> to confirm this== Configuring Schemaless Mode-As described above, there are three configuration elements that need to be in place to use Solr in schemaless mode. In the `_default` configset included with Solr these are already configured. If, however, you would like to implement schemaless on your own, you should make the following changes.+As described above, there are three configuration elements that need to be in place to use Solr in schemaless mode.+In the `_default` configset included with Solr these are already configured.+If, however, you would like to implement schemaless on your own, you should make the following changes.=== Enable Managed Schema@@ -86,7 +90,8 @@ You can configure the `ManagedIndexSchemaFactory` (and control the resource fileIn Solr, an <<update-request-processors.adoc#,UpdateRequestProcessorChain>> defines a chain of plugins that are applied to documents before or while they are indexed.-The field guessing aspect of Solr's schemaless mode uses a specially-defined UpdateRequestProcessorChain that allows Solr to guess field types. You can also define the default field type classes to use.+The field guessing aspect of Solr's schemaless mode uses a specially-defined UpdateRequestProcessorChain that allows Solr to guess field types.+You can also define the default field type classes to use.To start, you should define it as follows (see the javadoc links below for update processor factory documentation):@@ -151,17 +156,34 @@ To start, you should define it as follows (see the javadoc links below for updat</updateRequestProcessorChain>-----There are many things defined in this chain. Let's step through a few of them.--<1> First, we're using the FieldNameMutatingUpdateProcessorFactory to lower-case all field names. Note that this and every following `<processor>` element include a `name`. These names will be used in the final chain definition at the end of this example.-<2> Next we add several update request processors to parse different field types. Note the ParseDateFieldUpdateProcessorFactory includes a long list of possible date formations that would be parsed into valid Solr dates. If you have a custom date, you could add it to this list (see the link to the Javadocs below to get information on how).-<3> Once the fields have been parsed, we define the field types that will be assigned to those fields. You can modify any of these that you would like to change.-<4> In this definition, if the parsing step decides the incoming data in a field is a string, we will put this into a field in Solr with the field type `text_general`. This field type by default allows Solr to query on this field.-<5> After we've added the `text_general` field, we have also defined a copy field rule that will copy all data from the new `text_general` field to a field with the same name suffixed with `_str`. This is done by Solr's dynamic fields feature. By defining the target of the copy field rule as a dynamic field in this way, you can control the field type used in your schema. The default selection allows Solr to facet, highlight, and sort on these fields.-<6> This is another example of a mapping rule. In this case we define that when either of the `Long` or `Integer` field parsers identify a field, they should both map their fields to the `plongs` field type.-<7> Finally, we add a chain definition that calls the list of plugins. These plugins are each called by the names we gave to them when we defined them. We can also add other processors to the chain, as shown here. Note we have also given the entire chain a `name` ("add-unknown-fields-to-the-schema"). We'll use this name in the next section to specify that our update request handler should use this chain definition.--CAUTION: This chain definition will make a number of copy field rules for string fields to be created from corresponding text fields. If your data causes you to end up with a lot of copy field rules, indexing may be slowed down noticeably, and your index size will be larger. To control for these issues, it's recommended that you review the copy field rules that are created, and remove any which you do not need for faceting, sorting, highlighting, etc.+There are many things defined in this chain.+Let's step through a few of them.++<1> First, we're using the FieldNameMutatingUpdateProcessorFactory to lower-case all field names.+Note that this and every following `<processor>` element include a `name`.+These names will be used in the final chain definition at the end of this example.+<2> Next we add several update request processors to parse different field types.+Note the ParseDateFieldUpdateProcessorFactory includes a long list of possible date formations that would be parsed into valid Solr dates.+If you have a custom date, you could add it to this list (see the link to the Javadocs below to get information on how).+<3> Once the fields have been parsed, we define the field types that will be assigned to those fields.+You can modify any of these that you would like to change.+<4> In this definition, if the parsing step decides the incoming data in a field is a string, we will put this into a field in Solr with the field type `text_general`.+This field type by default allows Solr to query on this field.+<5> After we've added the `text_general` field, we have also defined a copy field rule that will copy all data from the new `text_general` field to a field with the same name suffixed with `_str`.+This is done by Solr's dynamic fields feature.+By defining the target of the copy field rule as a dynamic field in this way, you can control the field type used in your schema.+The default selection allows Solr to facet, highlight, and sort on these fields.+<6> This is another example of a mapping rule.+In this case we define that when either of the `Long` or `Integer` field parsers identify a field, they should both map their fields to the `plongs` field type.+<7> Finally, we add a chain definition that calls the list of plugins.+These plugins are each called by the names we gave to them when we defined them.+We can also add other processors to the chain, as shown here.+Note we have also given the entire chain a `name` ("add-unknown-fields-to-the-schema").+We'll use this name in the next section to specify that our update request handler should use this chain definition.++CAUTION: This chain definition will make a number of copy field rules for string fields to be created from corresponding text fields.+If your data causes you to end up with a lot of copy field rules, indexing may be slowed down noticeably, and your index size will be larger.+To control for these issues, it's recommended that you review the copy field rules that are created, and remove any which you do not need for faceting, sorting, highlighting, etc.If you're interested in more information about the classes used in this chain, here are links to the Javadocs for update processor factories mentioned above:@@ -178,7 +200,8 @@ If you're interested in more information about the classes used in this chain, hOnce the UpdateRequestProcessorChain has been defined, you must instruct your UpdateRequestHandlers to use it when working with index updates (i.e., adding, removing, replacing documents).-There are two ways to do this. The update chain shown above has a `default=true` attribute which will use it for any update handler.+There are two ways to do this.+The update chain shown above has a `default=true` attribute which will use it for any update handler.An alternative, more explicit way is to use <<initparams.adoc#,InitParams>> to set the defaults on all `/update` request handlers:@@ -195,7 +218,8 @@ IMPORTANT: After all of these changes have been made, Solr should be restarted o=== Disabling Automatic Field Guessing-Automatic field creation can be disabled with the `update.autoCreateFields` property. To do this, you can use <<solr-control-script-reference.adoc#set-or-unset-configuration-properties,`bin/solr config`>> with a command such as:+Automatic field creation can be disabled with the `update.autoCreateFields` property.+To do this, you can use <<solr-control-script-reference.adoc#set-or-unset-configuration-properties,`bin/solr config`>> with a command such as:[source,bash]bin/solr config -c mycollection -p 8983 -action set-user-property -property update.autoCreateFields -value false@@ -287,7 +311,9 @@ Internally, the Schema API and the Schemaless Update Processors both use the samAlso, if you do not need the `*_str` version of a text field, you can simply remove the `copyField` definition from the auto-generated schema and it will not be re-added since the original field is now defined.====-Once a field has been added to the schema, its field type is fixed. As a consequence, adding documents with field value(s) that conflict with the previously guessed field type will fail. For example, after adding the above document, the "```Sold```" field has the fieldType `plongs`, but the document below has a non-integral decimal value in this field:+Once a field has been added to the schema, its field type is fixed.+As a consequence, adding documents with field value(s) that conflict with the previously guessed field type will fail.+For example, after adding the above document, the "```Sold```" field has the fieldType `plongs`, but the document below has a non-integral decimal value in this field:[source,bash]----diff --git a/solr/solr-ref-guide/src/script-update-processor.adoc b/solr/solr-ref-guide/src/script-update-processor.adocindex db86bd29ab2..a2ca37c800d 100644--- a/solr/solr-ref-guide/src/script-update-processor.adoc+++ b/solr/solr-ref-guide/src/script-update-processor.adoc@@ -16,27 +16,24 @@// specific language governing permissions and limitations// under the License.-The {solr-javadocs}/contrib/scripting/org/apache/solr/scripting/update/ScriptUpdateProcessorFactory.html[ScriptUpdateProcessorFactory] allows Java scripting engines to be used-during Solr document update processing, allowing dramatic flexibility in-expressing custom document processing logic before being indexed. It has hooks to the-commit, delete, rollback, etc indexing actions, however add is the most common usage.+The {solr-javadocs}/contrib/scripting/org/apache/solr/scripting/update/ScriptUpdateProcessorFactory.html[ScriptUpdateProcessorFactory] allows Java scripting engines to be used during Solr document update processing, allowing dramatic flexibility in expressing custom document processing logic before being indexed.++It has hooks to the commit, delete, rollback, etc indexing actions, however add is the most common usage.It is implemented as an UpdateProcessor to be placed in an UpdateChain.TIP: This used to be known as the _StatelessScriptingUpdateProcessor_ and was renamed to clarify the key aspect of this update processor is it enables scripting.-The script can be written in any scripting language supported by your JVM (such-as JavaScript), and executed dynamically so no pre-compilation is necessary.+The script can be written in any scripting language supported by your JVM (such as JavaScript), and executed dynamically so no pre-compilation is necessary.-WARNING: Being able to run a script of your choice as part of the indexing pipeline is a really powerful tool, that I sometimes call the-_Get out of jail free_ card because you can solve some problems this way that you can't in any other way. However, you are introducing some-potential security vulnerabilities.+WARNING: Being able to run a script of your choice as part of the indexing pipeline is a really powerful tool, that I sometimes call the _Get out of jail free_ card because you can solve some problems this way that you can't in any other way.+However, you are introducing some potential security vulnerabilities.== Installing the ScriptingUpdateProcessor and Scripting EnginesThe scripting update processor lives in the contrib module `/contrib/scripting`, and you need to explicitly add it to your Solr setup.-Java 11 and previous versions come with a JavaScript engine called Nashorn, but Java 12 will require you to add your own JavaScript engine. Other supported scripting engines like-JRuby, Jython, Groovy, all require you to add JAR files.+Java 11 and previous versions come with a JavaScript engine called Nashorn, but Java 12 will require you to add your own JavaScript engine.+Other supported scripting engines like JRuby, Jython, Groovy, all require you to add JAR files.Learn more about adding the `dist/solr-scripting-*.jar` file, and any other needed JAR files (depending on your scripting engine) into Solr's <<libs.adoc#lib-directories,Lib Directories>>.@@ -64,19 +61,19 @@ However, it is also possible to skip this level and configure the parameters dirBelow follows a list of each configuration parameters and their meaning:`script`::-The script file name. The script file must be placed in the `conf/ directory.+The script file name.+The script file must be placed in the `conf/ directory.There can be one or more "script" parameters specified; multiple scripts are executed in the order specified.`engine`::-Optionally specifies the scripting engine to use. This is only needed if the extension-of the script file is not a standard mapping to the scripting engine. For example, if your-script file was coded in JavaScript but the file name was called `update-script.foo`,-use "javascript" as the engine name.+Optionally specifies the scripting engine to use.+This is only needed if the extension of the script file is not a standard mapping to the scripting engine.+For example, if your script file was coded in JavaScript but the file name was called `update-script.foo`, use "javascript" as the engine name.`params`::-Optional parameters that are passed into the script execution context. This is-specified as a named list (`<lst>`) structure with nested typed parameters. If-specified, the script context will get a "params" object, otherwise there will be no "params" object available.+Optional parameters that are passed into the script execution context.+This is specified as a named list (`<lst>`) structure with nested typed parameters.+If specified, the script context will get a "params" object, otherwise there will be no "params" object available.== Script Execution Context@@ -84,7 +81,8 @@ specified, the script context will get a "params" object, otherwise there will bEvery script has some variables provided to it.`logger`::-Logger (org.slf4j.Logger) instance. This is useful for logging information from the script.+Logger (org.slf4j.Logger) instance.+This is useful for logging information from the script.`req`::{solr-javadocs}/core/org/apache/solr/request/SolrQueryRequest.html[SolrQueryRequest] instance.@@ -97,13 +95,10 @@ The "params" object, if any specified, from the configuration.== Examples-The `processAdd()` and the other script methods can return false to skip further-processing of the document. All methods must be defined, though generally the-`processAdd()` method is where the action is.+The `processAdd()` and the other script methods can return false to skip further processing of the document.+All methods must be defined, though generally the `processAdd()` method is where the action is.-Here's a URL that works with the techproducts example setup demonstrating specifying-the "script" update chain: `http://localhost:8983/solr/techproducts/update?commit=true&stream.contentType=text/csv&fieldnames=id,description&stream.body=1,foo&update.chain=script`-which logs the following:+Here's a URL that works with the techproducts example setup demonstrating specifying the "script" update chain: `http://localhost:8983/solr/techproducts/update?commit=true&stream.contentType=text/csv&fieldnames=id,description&stream.body=1,foo&update.chain=script` which logs the following:[source,text]----@@ -217,9 +212,8 @@ The following in JRuby does not work as expected for some reason, though it does=== Groovy-Add JARs from a Groovy distro's `lib/` directory to Solr. All JARs from-Groovy's distro probably aren't required, but more than just the main `groovy.jar`-file is needed (at least when this was tested using Groovy 2.0.6)+Add JARs from a Groovy distro's `lib/` directory to Solr.+All JARs from Groovy's distro probably aren't required, but more than just the main `groovy.jar` file is needed (at least when this was tested using Groovy 2.0.6)[source,groovy]----diff --git a/solr/solr-ref-guide/src/search-sample.adoc b/solr/solr-ref-guide/src/search-sample.adocindex 7834e1d763a..2abd03e2123 100644--- a/solr/solr-ref-guide/src/search-sample.adoc+++ b/solr/solr-ref-guide/src/search-sample.adoc@@ -16,21 +16,17 @@// specific language governing permissions and limitations// under the License.-Data is the indispensable factor in statistical analysis. This section-provides an overview of the key functions for retrieving data for-visualization and statistical analysis: searching, sampling-and aggregation.+Data is the indispensable factor in statistical analysis.+This section provides an overview of the key functions for retrieving data for visualization and statistical analysis: searching, sampling and aggregation.== Searching=== Exploring-The `search` function can be used to search a SolrCloud collection and return a-result set.+The `search` function can be used to search a SolrCloud collection and return a result set.Below is an example of the most basic `search` function called from the Zeppelin-Solr interpreter.-Zeppelin-Solr sends the `seach(logs)` call to the `/stream` handler and displays the results-in *table* format.+Zeppelin-Solr sends the `seach(logs)` call to the `/stream` handler and displays the results in *table* format.In the example the `search` function is passed only the name of the collection being searched.This returns a result set of 10 records with all fields.@@ -43,41 +39,34 @@ image::images/math-expressions/search1.png[]Once the format of the records is known, parameters can be added to the `*search*` function to begin analyzing the data.In the example below a search query, field list, rows and sort have been added to the `search` function.-Now the search is limited to records within a specific time range and returns-a maximum result set of 750 records sorted by `tdate_dt` ascending.+Now the search is limited to records within a specific time range and returns a maximum result set of 750 records sorted by `tdate_dt` ascending.We have also limited the result set to three specific fields.image::images/math-expressions/search-sort.png[]-Once the data is loaded into the table we can switch to a scatter plot and plot the `filesize_d` column-on the *x-axis* and the `response_d` column on the *y-axis*.+Once the data is loaded into the table we can switch to a scatter plot and plot the `filesize_d` column on the *x-axis* and the `response_d` column on the *y-axis*.image::images/math-expressions/search-sort-plot.png[]-This allows us to quickly visualize the relationship between two variables-selected from a very specific slice of the index.+This allows us to quickly visualize the relationship between two variables selected from a very specific slice of the index.=== Scoring-The `search` function will score and rank documents when a query is performed on-a text field. The example below shows an example of the scoring and ranking of results.+The `search` function will score and rank documents when a query is performed on a text field.+The example below shows an example of the scoring and ranking of results.image::images/math-expressions/scoring.png[]== SamplingThe `random` function returns a random sample from a distributed search result set.-This allows for fast visualization, statistical analysis, and modeling of-samples that can be used to infer information about the larger result set.+This allows for fast visualization, statistical analysis, and modeling of samples that can be used to infer information about the larger result set.The visualization examples below use small random samples, but Solr's random sampling provides sub-second response times on sample sizes of over 200,000.These larger samples can be used to build reliable statistical models that describe large data sets (billions of documents) with sub-second performance.-The examples below demonstrate univariate and bivariate scatter-plots of random samples.-Statistical modeling with random samples-is covered in the <<statistics.adoc#,Statistics>>, <<probability-distributions.adoc#,Probability>>, <<regression.adoc#,Linear Regression>>, <<curve-fitting.adoc#,Curve Fitting>>,-and <<machine-learning.adoc#,Machine Learning>> sections.+The examples below demonstrate univariate and bivariate scatter plots of random samples.+Statistical modeling with random samples is covered in the <<statistics.adoc#,Statistics>>, <<probability-distributions.adoc#,Probability>>, <<regression.adoc#,Linear Regression>>, <<curve-fitting.adoc#,Curve Fitting>>, and <<machine-learning.adoc#,Machine Learning>> sections.=== Univariate Scatter Plots@@ -87,31 +76,25 @@ When called with no other parameters the `random` function returns a random sampWhen called without the field list parameter (`fl`) the `random` function also generates a sequence, 0-499 in this case, which can be used for plotting the x-axis.This sequence is returned in a field called `x`.-The visualization below shows a scatter plot with the `filesize_d` field-plotted on the y-axis and the `x` sequence plotted on the x-axis.-The effect of this is to spread the `filesize_d` samples across the length-of the plot so they can be more easily studied.+The visualization below shows a scatter plot with the `filesize_d` field plotted on the y-axis and the `x` sequence plotted on the x-axis.+The effect of this is to spread the `filesize_d` samples across the length of the plot so they can be more easily studied.-By studying the scatter plot we can learn a number of things about the-distribution of the `filesize_d` variable:+By studying the scatter plot we can learn a number of things about the distribution of the `filesize_d` variable:* The sample set ranges from 34,875 to 45,902.* The highest density appears to be at about 40,000.-* The sample seems to have a balanced number of observations above and below-40,000. Based on this the *mean* and *mode* would appear to be around 40,000.-* The number of observations tapers off to a small number of outliers on-the and low and high end of the sample.+* The sample seems to have a balanced number of observations above and below 40,000.+Based on this the *mean* and *mode* would appear to be around 40,000.+* The number of observations tapers off to a small number of outliers on the and low and high end of the sample.-This sample can be re-run multiple times to see if the samples-produce similar plots.+This sample can be re-run multiple times to see if the samples produce similar plots.image::images/math-expressions/univariate.png[]=== Bivariate Scatter PlotsIn the next example parameters have been added to the `random` function.-The field list (`fl`) now specifies two fields to be-returned with each sample: `filesize_d` and `response_d`.+The field list (`fl`) now specifies two fields to be returned with each sample: `filesize_d` and `response_d`.The `q` and `rows` parameters are the same as the defaults but are included as an example of how to set these parameters.By plotting `filesize_d` on the x-axis and `response_d` on the y-axis we can begin to study the relationship between the two variables.@@ -121,14 +104,14 @@ By studying the scatter plot we can learn the following:* As `filesize_d` rises, `response_d` tends to rise.* This relationship appears to be linear, as a straight line put through the data could be used to model the relationship.* The points appear to cluster more densely along a straight line through the middle and become less dense as they move away from the line.-* The variance of the data at each `filesize_d` point seems fairly consistent. This means a predictive model would have consistent error across the range of predictions.+* The variance of the data at each `filesize_d` point seems fairly consistent.+This means a predictive model would have consistent error across the range of predictions.image::images/math-expressions/bivariate.png[]== Aggregation-Aggregations are a powerful statistical tool for summarizing large data sets and-surfacing patterns, trends, and correlations within the data.+Aggregations are a powerful statistical tool for summarizing large data sets and surfacing patterns, trends, and correlations within the data.Aggregations are also a powerful tool for visualization and provide data sets for further statistical analysis.=== stats@@ -150,13 +133,10 @@ image::images/math-expressions/stats.png[]=== facet-The `facet` function performs single and multi-dimension-aggregations that behave in a similar manner to SQL group by aggregations.-Under the covers the `facet` function pushes down the aggregations to Solr's-<<json-facet-api.adoc#,JSON Facet API>> for fast distributed execution.+The `facet` function performs single and multi-dimension aggregations that behave in a similar manner to SQL group by aggregations.+Under the covers the `facet` function pushes down the aggregations to Solr's <<json-facet-api.adoc#,JSON Facet API>> for fast distributed execution.-The example below performs a single dimension aggregation from the-nyc311 (NYC complaints) dataset.+The example below performs a single dimension aggregation from the nyc311 (NYC complaints) dataset.The aggregation returns the top five *complaint types* by *count* for records with a status of *Pending*.The results are displayed with Zeppelin-Solr in a table.@@ -176,23 +156,19 @@ The example below shows the multi-dimension aggregation visualized as a groupedimage::images/math-expressions/facetviz2.png[]-The `facet` function supports any combination of the following aggregate functions: count(*), sum, avg, min,-max.+The `facet` function supports any combination of the following aggregate functions: count(*), sum, avg, min, max.=== facet2D-The `facet2D` function performs two dimensional aggregations that can be-visualized as heat maps or pivoted into matrices and operated on by machine learning functions.+The `facet2D` function performs two dimensional aggregations that can be visualized as heat maps or pivoted into matrices and operated on by machine learning functions.-`facet2D` has different syntax and behavior then a two dimensional `facet` function which-does not control the number of unique facets of each dimension. The `facet2D` function-has the `dimensions` parameter which controls the number of unique facets-for the *x* and *y* dimensions.+`facet2D` has different syntax and behavior then a two dimensional `facet` function which does not control the number of unique facets of each dimension.+The `facet2D` function has the `dimensions` parameter which controls the number of unique facets for the *x* and *y* dimensions.-The example below visualizes the output of the `facet2D` function. In the example `facet2D`-returns the top 5 boroughs and the top 5 complaint types for each borough. The output is-then visualized as a heatmap.+The example below visualizes the output of the `facet2D` function.+In the example `facet2D` returns the top 5 boroughs and the top 5 complaint types for each borough.+The output is then visualized as a heatmap.image::images/math-expressions/facet2D.png[]@@ -200,12 +176,10 @@ The `facet2D` function supports one of the following aggregate functions: `count=== timeseries-The `timeseries` function performs fast, distributed time-series aggregation leveraging Solr's builtin faceting and date math capabilities.+The `timeseries` function performs fast, distributed time series aggregation leveraging Solr's builtin faceting and date math capabilities.-The example below performs a monthly time series aggregation over a collection of-daily stock price data. In this example the average monthly closing price is-calculated for the stock ticker *amzn* between a specific date range.+The example below performs a monthly time series aggregation over a collection of daily stock price data.+In this example the average monthly closing price is calculated for the stock ticker *amzn* between a specific date range.The output of the `timeseries` function is then visualized with a line chart.@@ -226,11 +200,8 @@ The foreground and background counts are global for the collection.The `significantTerms` function can often provide insights that cannot be gleaned from other types of aggregations.The example below illustrates the difference between the `facet` function and the `significantTerms` function.-In the first example the `facet` function aggregates the top 5 complaint types-in Brooklyn.-This returns the five most common complaint types in Brooklyn, but-its not clear that these terms appear more frequently in Brooklyn then-then the other boroughs.+In the first example the `facet` function aggregates the top 5 complaint types in Brooklyn.+This returns the five most common complaint types in Brooklyn, but it's not clear that these terms appear more frequently in Brooklyn then then the other boroughs.image::images/math-expressions/significantTermsCompare.png[]@@ -241,13 +212,11 @@ This shows that Elder Abuse complaints have a much higher occurrence rate in Broimage::images/math-expressions/significantTerms2.png[]-The final example shows a visualization of the `significantTerms` from a-text field containing movie reviews. The result shows the-significant terms that appear in movie reviews that have the phrase "sci-fi".+The final example shows a visualization of the `significantTerms` from a text field containing movie reviews.+The result shows the significant terms that appear in movie reviews that have the phrase "sci-fi".-The results are visualized using a bubble chart with the *foreground* count on-plotted on the x-axis and the *background* count on the y-axis. Each term is-shown in a bubble sized by the *score*.+The results are visualized using a bubble chart with the *foreground* count on plotted on the x-axis and the *background* count on the y-axis.+Each term is shown in a bubble sized by the *score*.image::images/math-expressions/sterms.png[]@@ -255,21 +224,18 @@ image::images/math-expressions/sterms.png[]The `nodes` function performs aggregations of nodes during a breadth first search of a graph.This function is covered in detail in the section <<graph-traversal.adoc#,Graph Traversal>>.-In this example the focus will be on finding correlated nodes in a time series-graph using the `nodes` expressions.+In this example the focus will be on finding correlated nodes in a time series graph using the `nodes` expressions.The example below finds stock tickers whose daily movements tend to be correlated with the ticker *jpm* (JP Morgan).-The inner `search` expression finds records between a specific date range-where the ticker symbol is *jpm* and the `change_d` field (daily change in stock price) is greater then .25.+The inner `search` expression finds records between a specific date range where the ticker symbol is *jpm* and the `change_d` field (daily change in stock price) is greater then .25.This search returns all fields in the index including the `yearMonthDay_s` which is the string representation of the year, month, and day of the matching records.-The `nodes` function wraps the `search` function and operates over its results. The `walk` parameter maps a field from the search results to a field in the index.+The `nodes` function wraps the `search` function and operates over its results.+The `walk` parameter maps a field from the search results to a field in the index.In this case the `yearMonthDay_s` is mapped back to the `yearMonthDay_s` field in the same index.-This will find records that have same `yearMonthDay_s` field value returned-by the initial search, and will return records for all tickers on those days.-A filter query is applied to the search to filter the search to rows that have a `change_d`-greater the .25.+This will find records that have same `yearMonthDay_s` field value returned by the initial search, and will return records for all tickers on those days.+A filter query is applied to the search to filter the search to rows that have a `change_d` greater the .25.This will find all records on the matching days that have a daily change greater then .25.The `gather` parameter tells the nodes expression to gather the `ticker_s` symbols during the breadth first search.@@ -279,8 +245,7 @@ This will count the number of times each ticker appears in the breadth first seaFinally the `top` function selects the top 5 tickers by count and returns them.The result below shows the ticker symbols in the `nodes` field and the counts for each node.-Notice *jpm* is first, which shows how many days *jpm* had a change greater then .25 in this time-period.+Notice *jpm* is first, which shows how many days *jpm* had a change greater then .25 in this time period.The next set of ticker symbols (*mtb*, *slvb*, *gs* and *pnc*) are the symbols with highest number of days with a change greater then .25 on the same days that *jpm* had a change greater then .25.image::images/math-expressions/nodestab.png[]diff --git a/solr/solr-ref-guide/src/shard-management.adoc b/solr/solr-ref-guide/src/shard-management.adocindex d8f7e0f9091..f6e31a704ac 100644--- a/solr/solr-ref-guide/src/shard-management.adoc+++ b/solr/solr-ref-guide/src/shard-management.adoc@@ -17,7 +17,8 @@// specific language governing permissions and limitations// under the License.-In SolrCloud, a shard is a logical partition of a collection. This partition stores part of the entire index for a collection.+In SolrCloud, a shard is a logical partition of a collection.+This partition stores part of the entire index for a collection.The number of shards you have helps to determine how many documents a single collection can contain in total, and also impacts search performance.@@ -87,11 +88,18 @@ curl -X POST http://localhost:8983/api/collections/techproducts/shards -H 'Conte====---Splitting a shard will take an existing shard and break it into two pieces which are written to disk as two (new) shards. The original shard will continue to contain the same data as-is but it will start re-routing requests to the new shards. The new shards will have as many replicas as the original shard. A soft commit is automatically issued after splitting a shard so that documents are made visible on sub-shards. An explicit commit (hard or soft) is not necessary after a split operation because the index is automatically persisted to disk during the split operation.+Splitting a shard will take an existing shard and break it into two pieces which are written to disk as two (new) shards.+The original shard will continue to contain the same data as-is but it will start re-routing requests to the new shards.+The new shards will have as many replicas as the original shard.+A soft commit is automatically issued after splitting a shard so that documents are made visible on sub-shards.+An explicit commit (hard or soft) is not necessary after a split operation because the index is automatically persisted to disk during the split operation.-This command allows for seamless splitting and requires no downtime. A shard being split will continue to accept query and indexing requests and will automatically start routing requests to the new shards once this operation is complete. This command can only be used for SolrCloud collections created with `numShards` parameter, meaning collections which rely on Solr's hash-based routing mechanism.+This command allows for seamless splitting and requires no downtime.+A shard being split will continue to accept query and indexing requests and will automatically start routing requests to the new shards once this operation is complete.+This command can only be used for SolrCloud collections created with `numShards` parameter, meaning collections which rely on Solr's hash-based routing mechanism.-The split is performed by dividing the original shard's hash range into two equal partitions and dividing up the documents in the original shard according to the new sub-ranges. Two parameters discussed below, `ranges` and `split.key` provide further control over how the split occurs.+The split is performed by dividing the original shard's hash range into two equal partitions and dividing up the documents in the original shard according to the new sub-ranges.+Two parameters discussed below, `ranges` and `split.key` provide further control over how the split occurs.The newly created shards will have as many replicas as the parent shard, of the same replica types.@@ -99,53 +107,58 @@ When using `splitMethod=rewrite` (default) you must ensure that the node runningAlso, the first replicas of resulting sub-shards will always be placed on the shard leader node.-Shard splitting can be a long running process. In order to avoid timeouts, you should run this as an <<collections-api.adoc#asynchronous-calls,asynchronous call>>.+Shard splitting can be a long running process.+In order to avoid timeouts, you should run this as an <<collections-api.adoc#asynchronous-calls,asynchronous call>>.=== SPLITSHARD Parameters`collection`::-The name of the collection that includes the shard to be split. This parameter is required.+The name of the collection that includes the shard to be split.+This parameter is required.`shard`::-The name of the shard to be split. This parameter is required when `split.key` is not specified.+The name of the shard to be split.+This parameter is required when `split.key` is not specified.`ranges`::A comma-separated list of hash ranges in hexadecimal, such as `ranges=0-1f4,1f5-3e8,3e9-5dc`.+-This parameter can be used to divide the original shard's hash range into arbitrary hash range intervals specified in hexadecimal. For example, if the original hash range is `0-1500` then adding the parameter: `ranges=0-1f4,1f5-3e8,3e9-5dc` will divide the original shard into three shards with hash range `0-500`, `501-1000`, and `1001-1500` respectively.+This parameter can be used to divide the original shard's hash range into arbitrary hash range intervals specified in hexadecimal.+For example, if the original hash range is `0-1500` then adding the parameter: `ranges=0-1f4,1f5-3e8,3e9-5dc` will divide the original shard into three shards with hash range `0-500`, `501-1000`, and `1001-1500` respectively.`split.key`::The key to use for splitting the index.+-This parameter can be used to split a shard using a route key such that all documents of the specified route key end up in a single dedicated sub-shard. Providing the `shard` parameter is not required in this case because the route key is enough to figure out the right shard. A route key which spans more than one shard is not supported.+This parameter can be used to split a shard using a route key such that all documents of the specified route key end up in a single dedicated sub-shard.+Providing the `shard` parameter is not required in this case because the route key is enough to figure out the right shard.+A route key which spans more than one shard is not supported.+-For example, suppose `split.key=A!` hashes to the range `12-15` and belongs to shard 'shard1' with range `0-20`. Splitting by this route key would yield three sub-shards with ranges `0-11`, `12-15` and `16-20`. Note that the sub-shard with the hash range of the route key may also contain documents for other route keys whose hash ranges overlap.+For example, suppose `split.key=A!` hashes to the range `12-15` and belongs to shard 'shard1' with range `0-20`.+Splitting by this route key would yield three sub-shards with ranges `0-11`, `12-15` and `16-20`.+Note that the sub-shard with the hash range of the route key may also contain documents for other route keys whose hash ranges overlap.`numSubShards`::-The number of sub-shards to split the parent shard into. Allowed values for this are in the range of `2`-`8` and defaults to `2`.+The number of sub-shards to split the parent shard into.+Allowed values for this are in the range of `2`-`8` and defaults to `2`.+This parameter can only be used when `ranges` or `split.key` are not specified.`splitMethod`::Currently two methods of shard splitting are supported:-* `splitMethod=rewrite` (default) after selecting documents to retain in each partition this method creates sub-indexes from-scratch, which is a lengthy CPU- and I/O-intensive process but results in optimally-sized sub-indexes that don't contain-any data from documents not belonging to each partition.-* `splitMethod=link` uses file system-level hard links for creating copies of the original index files and then only modifies the-file that contains the list of deleted documents in each partition. This method is many times quicker and lighter on resources than the-`rewrite` method but the resulting sub-indexes are still as large as the original index because they still contain data from documents not-belonging to the partition. This slows down the replication process and consumes more disk space on replica nodes (the multiple hard-linked-copies don't occupy additional disk space on the leader node, unless hard-linking is not supported).+* `splitMethod=rewrite` (default) after selecting documents to retain in each partition this method creates sub-indexes from scratch, which is a lengthy CPU- and I/O-intensive process but results in optimally-sized sub-indexes that don't contain any data from documents not belonging to each partition.+* `splitMethod=link` uses file system-level hard links for creating copies of the original index files and then only modifies the file that contains the list of deleted documents in each partition.+This method is many times quicker and lighter on resources than the `rewrite` method but the resulting sub-indexes are still as large as the original index because they still contain data from documents not belonging to the partition.+This slows down the replication process and consumes more disk space on replica nodes (the multiple hard-linked copies don't occupy additional disk space on the leader node, unless hard-linking is not supported).`splitFuzz`::-A float value (default is 0.0f, must be smaller than 0.5f) that allows to vary the sub-shard ranges-by this percentage of total shard range, odd shards being larger and even shards being smaller.+A float value (default is 0.0f, must be smaller than 0.5f) that allows to vary the sub-shard ranges by this percentage of total shard range, odd shards being larger and even shards being smaller.`property._name_=_value_`::Set core property _name_ to _value_. See the section <<core-discovery.adoc#,Core Discovery>> for details on supported properties and values.`waitForFinalState`::-If `true`, the request will complete only when all affected replicas become active. The default is `false`, which means that the API will return the status of the single action, which may be before the new replica is online and active.+If `true`, the request will complete only when all affected replicas become active.+The default is `false`, which means that the API will return the status of the single action, which may be before the new replica is online and active.`timing`::If `true` then each stage of processing will be timed and a `timing` section will be included in response.@@ -184,12 +197,14 @@ Current implementation details and limitations:=== SPLITSHARD Response-The output will include the status of the request and the new shard names, which will use the original shard as their basis, adding an underscore and a number. For example, "shard1" will become "shard1_0" and "shard1_1". If the status is anything other than "success", an error message will explain why the request failed.+The output will include the status of the request and the new shard names, which will use the original shard as their basis, adding an underscore and a number.+For example, "shard1" will become "shard1_0" and "shard1_1". If the status is anything other than "success", an error message will explain why the request failed.[[createshard]]== CREATESHARD: Create a Shard-Shards can only created with this API for collections that use the 'implicit' router (i.e., when the collection was created, `router.name=implicit`). A new shard with a name can be created for an existing 'implicit' collection.+Shards can only created with this API for collections that use the 'implicit' router (i.e., when the collection was created, `router.name=implicit`).+A new shard with a name can be created for an existing 'implicit' collection.Use SPLITSHARD for collections created with the 'compositeId' router (`router.key=compositeId`).@@ -249,18 +264,22 @@ curl -X POST http://localhost:8983/api/collections/techproducts/shards -H 'Conte====---The default values for `replicationFactor` or `nrtReplicas`, `tlogReplicas`, `pullReplicas` from the collection is used to determine the number of replicas to be created for the new shard. This can be customized by explicitly passing the corresponding parameters to the request.+The default values for `replicationFactor` or `nrtReplicas`, `tlogReplicas`, `pullReplicas` from the collection is used to determine the number of replicas to be created for the new shard.+This can be customized by explicitly passing the corresponding parameters to the request.=== CREATESHARD Parameters`collection`::-The name of the collection that includes the shard to be split. This parameter is required.+The name of the collection that includes the shard to be split.+This parameter is required.`shard`::-The name of the shard to be created. This parameter is required.+The name of the shard to be created.+This parameter is required.`createNodeSet`::-Allows defining the nodes to spread the new collection across. If not provided, the CREATESHARD operation will create shard-replica spread across all live Solr nodes.+Allows defining the nodes to spread the new collection across.+If not provided, the CREATESHARD operation will create shard-replica spread across all live Solr nodes.+The format is a comma-separated list of node_names, such as `localhost:8983_solr,localhost:8984_solr,localhost:8985_solr`.@@ -277,19 +296,22 @@ The number of `pull` replicas that should be created for the new shard (optionalSet core property _name_ to _value_. See the section <<core-discovery.adoc#,Core Discovery>> for details on supported properties and values.`waitForFinalState`::-If `true`, the request will complete only when all affected replicas become active. The default is `false`, which means that the API will return the status of the single action, which may be before the new replica is online and active.+If `true`, the request will complete only when all affected replicas become active.+The default is `false`, which means that the API will return the status of the single action, which may be before the new replica is online and active.`async`::Request ID to track this action which will be <<collections-api.adoc#asynchronous-calls,processed asynchronously>>.=== CREATESHARD Response-The output will include the status of the request. If the status is anything other than "success", an error message will explain why the request failed.+The output will include the status of the request.+If the status is anything other than "success", an error message will explain why the request failed.[[deleteshard]]== DELETESHARD: Delete a Shard-Deleting a shard will unload all replicas of the shard, remove them from the collection's `state.json`, and (by default) delete the instanceDir and dataDir for each replica. It will only remove shards that are inactive, or which have no range given for custom sharding.+Deleting a shard will unload all replicas of the shard, remove them from the collection's `state.json`, and (by default) delete the instanceDir and dataDir for each replica.+It will only remove shards that are inactive, or which have no range given for custom sharding.[.dynamic-tabs]@@ -326,26 +348,32 @@ curl -X DELETE http://localhost:8983/api/collections/techproducts/shards/shard1?=== DELETESHARD Parameters`collection`::-The name of the collection that includes the shard to be deleted. This parameter is required.+The name of the collection that includes the shard to be deleted.+This parameter is required.`shard`::-The name of the shard to be deleted. This parameter is required.+The name of the shard to be deleted.+This parameter is required.`deleteInstanceDir`::-By default Solr will delete the entire instanceDir of each replica that is deleted. Set this to `false` to prevent the instance directory from being deleted.+By default Solr will delete the entire instanceDir of each replica that is deleted.+Set this to `false` to prevent the instance directory from being deleted.`deleteDataDir`::-By default Solr will delete the dataDir of each replica that is deleted. Set this to `false` to prevent the data directory from being deleted.+By default Solr will delete the dataDir of each replica that is deleted.+Set this to `false` to prevent the data directory from being deleted.`deleteIndex`::-By default Solr will delete the index of each replica that is deleted. Set this to `false` to prevent the index directory from being deleted.+By default Solr will delete the index of each replica that is deleted.+Set this to `false` to prevent the index directory from being deleted.`async`::Request ID to track this action which will be <<collections-api.adoc#asynchronous-calls,processed asynchronously>>.=== DELETESHARD Response-The output will include the status of the request. If the status is anything other than "success", an error message will explain why the request failed.+The output will include the status of the request.+If the status is anything other than "success", an error message will explain why the request failed.[[forceleader]]== FORCELEADER: Force Shard Leader@@ -409,9 +437,12 @@ curl -X POST http://localhost:8983/api/collections/techproducts/shards/shard1 -H=== FORCELEADER Parameters`collection`::-The name of the collection. This parameter is required.+The name of the collection.+This parameter is required.`shard`::-The name of the shard where leader election should occur. This parameter is required.+The name of the shard where leader election should occur.+This parameter is required.-WARNING: This is an expert level command, and should be invoked only when regular leader election is not working. This may potentially lead to loss of data in the event that the new leader doesn't have certain updates, possibly recent ones, which were acknowledged by the old leader before going down.+WARNING: This is an expert level command, and should be invoked only when regular leader election is not working.+This may potentially lead to loss of data in the event that the new leader doesn't have certain updates, possibly recent ones, which were acknowledged by the old leader before going down.diff --git a/solr/solr-ref-guide/src/simulations.adoc b/solr/solr-ref-guide/src/simulations.adocindex b468a5aee11..e402a0a00ad 100644--- a/solr/solr-ref-guide/src/simulations.adoc+++ b/solr/solr-ref-guide/src/simulations.adoc@@ -17,53 +17,46 @@// under the License.Monte Carlo simulations are commonly used to model the behavior of-stochastic (random) systems. This section of the user guide covers-the basics of performing Monte Carlo simulations with Math Expressions.+stochastic (random) systems.+This section of the user guide covers the basics of performing Monte Carlo simulations with Math Expressions.== Random Time SeriesThe daily movement of stock prices is often described as a "random walk".But what does that really mean, and how is this different than a random time series?-The examples below will use Monte Carlo simulations to explore both "random walks"-and random time series.+The examples below will use Monte Carlo simulations to explore both "random walks" and random time series.-A useful first step in understanding the difference is to visualize-daily stock returns, calculated as closing price minus opening price, as a time series.+A useful first step in understanding the difference is to visualize daily stock returns, calculated as closing price minus opening price, as a time series.-The example below uses the `search` function to return 1000 days of daily stock-returns for the ticker *CVX* (Chevron). The `change_d` field, which is the-change in price for the day, is then plotted as a time series.+The example below uses the `search` function to return 1000 days of daily stock returns for the ticker *CVX* (Chevron).+The `change_d` field, which is the change in price for the day, is then plotted as a time series.image::images/math-expressions/randomwalk1.png[]Notice that the time series of daily price changes moves randomly above and-below zero. Some days the stock is up, some days its down, but there-does not seem to be a noticeable pattern or any dependency between steps. This is a hint-that this is a *random time series*.+below zero.+Some days the stock is up, some days its down, but there does not seem to be a noticeable pattern or any dependency between steps.+This is a hint that this is a *random time series*.=== AutocorrelationAutocorrelation measures the degree to which a signal is correlated with itself.- Autocorrelation can be used to determine-if a vector contains a signal or if there is dependency between values in a time series. If there is no-signal and no dependency between values in the time series then the time series is random.+Autocorrelation can be used to determine if a vector contains a signal or if there is dependency between values in a time series.+If there is no signal and no dependency between values in the time series then the time series is random.It's useful to plot the autocorrelation of the `change_d` vector to confirm that it is indeed random.In the example below the search results are set to a variable and then the `change_d` field is vectorized and stored in variable `b`.-Then the `conv` (convolution) function is used to autocorrelate-the `change_d` vector.+Then the `conv` (convolution) function is used to autocorrelate the `change_d` vector.Notice that the `conv` function is simply "convolving" the `change_d` vectorwith a reversed copy of itself.This is the technique for performing autocorrelation using convolution.-The <<dsp.adoc#,Signal Processing>> section-of the user guide covers both convolution and autocorrelation in detail.+The <<dsp.adoc#,Signal Processing>> section of the user guide covers both convolution and autocorrelation in detail.In this section we'll just discuss the plot.The plot shows the intensity of correlation that is calculated as the `change_d` vector is slid across itself by the `conv` function.Notice in the plot there is long period of low intensity correlation that appears to be random.-Then in the center a peak of high intensity correlation where the vectors-are directly lined up.+Then in the center a peak of high intensity correlation where the vectors are directly lined up.This is followed by another long period of low intensity correlation.This is the autocorrelation plot of pure noise.@@ -75,8 +68,7 @@ image::images/math-expressions/randomwalk2.png[]The random daily changes in stock prices cannot be predicted, but they can be modeled with a probability distribution.To model the time series we'll start by visualizing the distribution of the `change_d` vector.-In the example below the `change_d` vector is plotted using the `empiricalDistribution` function to create an 11 bin-histogram of the data.+In the example below the `change_d` vector is plotted using the `empiricalDistribution` function to create an 11 bin histogram of the data.Notice that the distribution appears to be normally distributed.Daily stock price changes do tend to be normally distributed although *CVX* was chosen specifically for this example because of this characteristic.@@ -115,34 +107,27 @@ The example below uses the `monteCarlo` function to simulate a distribution forIn the example a `normalDistribution` is created from the *mean* and *standard deviation* of the `change_d` vector.The `monteCarlo` function then draws 100 samples from the normal distribution to represent 100 days of stock returns and sets the vector of samples to the variable `d`.-The `add` function then calculates the total return-from the 100 day sample.+The `add` function then calculates the total return from the 100 day sample.The output of the `add` function is collected by the `monteCarlo` function.This is repeated 50000 times, with each run drawing a different set of samples from the normal distribution.-The result of the simulation is set to variable `s`, which contains-the total returns from the 50000 runs.+The result of the simulation is set to variable `s`, which contains the total returns from the 50000 runs.The `empiricalDistribution` function is then used to visualize the output of the simulation as a 50 bin histogram.-The distribution visualizes the probability of the different total-returns from 100 days of stock returns for ticker *CVX*.+The distribution visualizes the probability of the different total returns from 100 days of stock returns for ticker *CVX*.image::images/math-expressions/randomwalk5.png[]-The `probability` and `cumulativeProbability` functions can then used to-learn more about the `empiricalDistribution`.+The `probability` and `cumulativeProbability` functions can then used to learn more about the `empiricalDistribution`.For example the `probability` function can be used to calculate the probability of a non-negative return from 100 days of stock returns.-The example below uses the `probability` function to return the probability of a-return between the range of 0 and 40 from the `empiricalDistribution`-of the simulation.+The example below uses the `probability` function to return the probability of a return between the range of 0 and 40 from the `empiricalDistribution` of the simulation.image::images/math-expressions/randomwalk5.1.png[]=== Random Walk-The `monteCarlo` function can also be used to model a random walk of-daily stock prices from the `normalDistribution` of daily stock returns.+The `monteCarlo` function can also be used to model a random walk of daily stock prices from the `normalDistribution` of daily stock returns.A random walk is a time series where each step is calculated by adding a random sample to the previous step.This creates a time series where each value is dependent on the previous value, which simulates the autocorrelation of stock prices.@@ -153,28 +138,22 @@ The example iterates 1000 times to create a random walk with 1000 steps.image::images/math-expressions/randomwalk6.png[]-Notice the autocorrelation in the daily stock prices caused by the dependency-between steps produces a very different plot then the-random daily change in stock price.+Notice the autocorrelation in the daily stock prices caused by the dependency between steps produces a very different plot then the random daily change in stock price.== Multivariate Normal Distribution-The `multiVariateNormalDistribution` function can be used to model and simulate-two or more normally distributed variables.+The `multiVariateNormalDistribution` function can be used to model and simulate two or more normally distributed variables.It also incorporates the *correlation* between variables into the model which allows for the study of how correlation effects the possible outcomes.-In the examples below a simulation of the total daily returns of two-stocks is explored.+In the examples below a simulation of the total daily returns of two stocks is explored.The *ALL* ticker (*Allstate*) is used along with the *CVX* ticker (*Chevron*) from the previous examples.=== Correlation and Covariance-The multivariate simulations show the effect of correlation on possible-outcomes.+The multivariate simulations show the effect of correlation on possible outcomes.Before getting started with actual simulations it's useful to first understand the correlation and covariance between the Allstate and Chevron stock returns.-The example below runs two searches to retrieve the daily stock returns-for all Allstate and Chevron.+The example below runs two searches to retrieve the daily stock returns for all Allstate and Chevron.The `change_d` vectors from both returns are read into variables (`all` and `cvx`) and Pearson's correlation is calculated for the two vectors with the `corr` function.image::images/math-expressions/corrsim1.png[]@@ -187,12 +166,8 @@ image::images/math-expressions/corrsim2.png[]=== Covariance Matrix-A covariance matrix is actually whats needed by the-`multiVariateNormalDistribution` as it contains both the variance of the-two stock return vectors and the covariance between the two-vectors.-The `cov` function will compute the covariance matrix for the-the columns of a matrix.+A covariance matrix is actually whats needed by the `multiVariateNormalDistribution` as it contains both the variance of the two stock return vectors and the covariance between the two vectors.+The `cov` function will compute the covariance matrix for the the columns of a matrix.The example below demonstrates how to compute the covariance matrix by adding the `all` and `cvx` vectors as rows to a matrix.The matrix is then transposed with the `transpose` function so that the `all` vector is the first column and the `cvx` vector is the second column.@@ -201,9 +176,7 @@ The `cov` function then computes the covariance matrix for the columns of the maimage::images/math-expressions/corrsim3.png[]-The covariance matrix is a square matrix which contains the-variance of each vector and the covariance between the-vectors as follows:+The covariance matrix is a square matrix which contains the variance of each vector and the covariance between the vectors as follows:[source,text]----@@ -214,16 +187,14 @@ cvx [0.13106056985285258, 0.7409729840230235]=== Multivariate Simulation-The example below demonstrates a Monte Carlo simulation with two stock tickers using the-`multiVariateNormalDistribution`.+The example below demonstrates a Monte Carlo simulation with two stock tickers using the `multiVariateNormalDistribution`.-In the example, result sets with the `change_d` field for both stock tickers, `all` (Allstate) and `cvx` (Chevron),-are retrieved and read into vectors.+In the example, result sets with the `change_d` field for both stock tickers, `all` (Allstate) and `cvx` (Chevron), are retrieved and read into vectors.-A matrix is then created from the two vectors and is transposed so-the matrix contains two columns, one with the `all` vector and one with the `cvx` vector.+A matrix is then created from the two vectors and is transposed so the matrix contains two columns, one with the `all` vector and one with the `cvx` vector.-Then the `multiVariateNormalDistribution` is created with two parameters. The first parameter is an array of `mean` values.+Then the `multiVariateNormalDistribution` is created with two parameters.+The first parameter is an array of `mean` values.In this case the means for the `all` vector and the `cvx` vector.The second parameter is the covariance matrix which was created from the 2-column matrix of the two vectors.@@ -245,7 +216,6 @@ image::images/math-expressions/mnorm.png[]The covariance matrix can be changed to study the effect on the simulation.The example below demonstrates this by providing a hard coded covariance matrix with a higher covariance value for the two vectors.This results is a simulated outcome distribution with a higher standard deviation or larger spread from the mean.-This measures the degree that higher correlation produces higher volatility-in the random walk.+This measures the degree that higher correlation produces higher volatility in the random walk.image::images/math-expressions/mnorm2.png[]diff --git a/solr/solr-ref-guide/src/solr-admin-ui.adoc b/solr/solr-ref-guide/src/solr-admin-ui.adocindex a02ec19e69c..d8c0bf186b4 100644--- a/solr/solr-ref-guide/src/solr-admin-ui.adoc+++ b/solr/solr-ref-guide/src/solr-admin-ui.adoc@@ -43,7 +43,8 @@ Under the covers, the Solr Admin UI uses the same HTTP APIs available to all cli[TIP]====-The path to the Solr Admin UI given above is `\http://hostname:port/solr`, which redirects to `\http://hostname:port/solr/\#/`. A convenience redirect is also supported, so simply accessing the Admin UI at `\http://hostname:port/` will also redirect to `\http://hostname:port/solr/#/`.+The path to the Solr Admin UI given above is `\http://hostname:port/solr`, which redirects to `\http://hostname:port/solr/\#/`.+A convenience redirect is also supported, so simply accessing the Admin UI at `\http://hostname:port/` will also redirect to `\http://hostname:port/solr/#/`.======= Login Screendiff --git a/solr/solr-ref-guide/src/solr-control-script-reference.adoc b/solr/solr-ref-guide/src/solr-control-script-reference.adocindex c9be84a53bd..64c9608dbb0 100644--- a/solr/solr-ref-guide/src/solr-control-script-reference.adoc+++ b/solr/solr-ref-guide/src/solr-control-script-reference.adoc@@ -438,7 +438,8 @@ Name of the core or collection to create (required).*Example*: `bin/solr create -c mycollection``-d <confdir>`::-The configuration directory. This defaults to `_default`.+The configuration directory.+This defaults to `_default`.+See the section <<Configuration Directories and SolrCloud>> below for more details about this option when running in SolrCloud mode.+@@ -497,7 +498,8 @@ You can override the name given to the configuration directory in ZooKeeper by uFor instance, the command `bin/solr create -c logs -d _default -n basic` will upload the `server/solr/configsets/_default/conf` directory to ZooKeeper as `/configs/basic`.Notice that we used the `-d` option to specify a different configuration than the default.-Solr provides several built-in configurations under `server/solr/configsets`. However you can also provide the path to your own configuration directory using the `-d` option.+Solr provides several built-in configurations under `server/solr/configsets`.+However you can also provide the path to your own configuration directory using the `-d` option.For instance, the command `bin/solr create -c mycoll -d /tmp/myconfigs`, will upload `/tmp/myconfigs` into ZooKeeper under `/configs/mycoll`.To reiterate, the configuration directory is named after the collection unless you override it using the `-n` option.@@ -569,7 +571,8 @@ For more information on Basic Authentication support specifically, see the sectiThe `bin/solr auth enable` command makes several changes to enable Basic Authentication:-* Creates a `security.json` file and uploads it to ZooKeeper. The `security.json` file will look similar to:+* Creates a `security.json` file and uploads it to ZooKeeper.+The `security.json` file will look similar to:+[source,json]----@@ -605,7 +608,8 @@ The command takes the following parameters:`-credentials`::The username and password in the format of `username:password` of the initial user.+-If you prefer not to pass the username and password as an argument to the script, you can choose the `-prompt` option. Either `-credentials` or `-prompt` *must* be specified.+If you prefer not to pass the username and password as an argument to the script, you can choose the `-prompt` option.+Either `-credentials` or `-prompt` *must* be specified.`-prompt`::If prompt is preferred, pass *true* as a parameter to request the script to prompt the user to enter a username and password.@@ -638,7 +642,8 @@ You can disable Basic Authentication with `bin/solr auth disable`.If the `-updateIncludeFileOnly` option is set to *true*, then only the settings in `bin/solr.in.sh` or `bin\solr.in.cmd` will be updated, and `security.json` will not be removed.-If the `-updateIncludeFileOnly` option is set to *false*, then the settings in `bin/solr.in.sh` or `bin\solr.in.cmd` will be updated, and `security.json` will be removed. However, the `basicAuth.conf` file is not removed with either option.+If the `-updateIncludeFileOnly` option is set to *false*, then the settings in `bin/solr.in.sh` or `bin\solr.in.cmd` will be updated, and `security.json` will be removed.+However, the `basicAuth.conf` file is not removed with either option.== Set or Unset Configuration Properties@@ -722,7 +727,8 @@ All parameters below are required.Name of the configuration set in ZooKeeper.This command will upload the configuration set to the "configs" ZooKeeper node giving it the name specified.+-You can see all uploaded configuration sets in the Admin UI via the Cloud screens. Choose Cloud -> Tree -> configs to see them.+You can see all uploaded configuration sets in the Admin UI via the Cloud screens.+Choose Cloud -> Tree -> configs to see them.+If a pre-existing configuration set is specified, it will be overwritten in ZooKeeper.+@@ -768,7 +774,8 @@ Use the `zk downconfig` command to download a configuration set from ZooKeeper tAll parameters listed below are required.`-n <name>`::-Name of the configset in ZooKeeper to download. The Admin UI Cloud -> Tree -> configs node lists all available configuration sets.+Name of the configset in ZooKeeper to download.+The Admin UI Cloud -> Tree -> configs node lists all available configuration sets.+*Example*: `-n myconfig`diff --git a/solr/solr-ref-guide/src/solr-glossary.adoc b/solr/solr-ref-guide/src/solr-glossary.adocindex 100dccb7eef..319ca0b5a48 100644--- a/solr/solr-ref-guide/src/solr-glossary.adoc+++ b/solr/solr-ref-guide/src/solr-glossary.adoc@@ -184,7 +184,9 @@ See also <<solrclouddef,SolrCloud>>.Umbrella term for a suite of functionality in Solr which allows managing a <<cluster,Cluster>> of Solr <<node,Nodes>> for scalability, fault tolerance, and high availability.[[schema]]<<solr-schema.adoc#,Solr Schema (managed-schema or schema.xml)>>::-The Solr index Schema defines the fields to be indexed and the type for the field (text, integers, etc.) By default schema data can be "managed" at run time using the <<schema-api.adoc#,Schema API>> and is typically kept in a file named `managed-schema` which Solr modifies as needed, but a collection may be configured to use a static Schema, which is only loaded on startup from a human edited configuration file - typically named `schema.xml`. See <<schema-factory.adoc#,Schema Factory Definition in SolrConfig>> for details.+The Solr index Schema defines the fields to be indexed and the type for the field (text, integers, etc.).+By default schema data can be "managed" at run time using the <<schema-api.adoc#,Schema API>> and is typically kept in a file named `managed-schema` which Solr modifies as needed, but a collection may be configured to use a static Schema, which is only loaded on startup from a human edited configuration file - typically named `schema.xml`.+See <<schema-factory.adoc#,Schema Factory Definition in SolrConfig>> for details.[[solrconfig]]<<configuring-solrconfig-xml.adoc#,SolrConfig (solrconfig.xml)>>::diff --git a/solr/solr-ref-guide/src/solr-in-docker.adoc b/solr/solr-ref-guide/src/solr-in-docker.adocindex a248ce822a8..d5393f874e8 100644--- a/solr/solr-ref-guide/src/solr-in-docker.adoc+++ b/solr/solr-ref-guide/src/solr-in-docker.adoc@@ -36,8 +36,8 @@ mkdir solrdatadocker run -d -v "$PWD/solrdata:/var/solr" -p 8983:8983 --name my_solr solr solr-precreate gettingstarted-----Then with a web browser go to `+http://localhost:8983/+` to see the Admin Console (adjust the hostname for your docker host).-In the web UI if you click on "Core Admin" you should now see the "gettingstarted" core.+Then with a web browser go to `+http://localhost:8983/+` to see Solr's Admin UI (adjust the hostname for your docker host).+In the UI, click on "Core Admin" and should now see the "gettingstarted" core.Next load some of the example data that is included in the container:@@ -46,7 +46,10 @@ Next load some of the example data that is included in the container:docker exec -it my_solr post -c gettingstarted example/exampledocs/manufacturers.xml-----In the UI, find the "Core selector" popup menu and select the "gettingstarted" core, then select the "Query" menu item. This gives you a default search for `*:*` which returns all docs. Hit the "Execute Query" button, and you should see a few docs with data. Congratulations!+In the UI, find the "Core selector" popup menu and select the "gettingstarted" core, then select the "Query" menu item.+This gives you a default search for `*:*` which returns all docs.+Hit the "Execute Query" button, and you should see a few docs with data.+Congratulations!=== Docker-Compose@@ -93,13 +96,16 @@ The container contains an installation of Solr, as installed by the <<taking-solThis stores the Solr distribution in `/opt/solr`, and configures Solr to use `/var/solr` to store data and logs, using the `/etc/default/solr` file for configuration.If you want to persist the data, mount a volume or directory on `/var/solr`.Solr expects some files and directories in `/var/solr`; if you use your own directory or volume you can either pre-populate them, or let Solr docker copy them for you.-If you want to use custom configuration, mount it in the appropriate place. See below for examples.+If you want to use custom configuration, mount it in the appropriate place.+See below for examples.The Solr docker distribution adds scripts in `/opt/solr/docker/scripts` to make it easier to use under Docker, for example to create cores on container startup.=== Creating Cores-When Solr runs in standalone mode, you create "cores" to store data. On a non-Docker Solr, you would run the server in the background, then use the <<solr-control-script-reference.adoc#,Solr control script>> to create cores and load data. With Solr docker you have various options.+When Solr runs in standalone mode, you create "cores" to store data.+On a non-Docker Solr, you would run the server in the background, then use the <<solr-control-script-reference.adoc#,Solr control script>> to create cores and load data.+With Solr docker you have various options.The first is exactly the same: start Solr running in a container, then execute the control script manually in the same container:@@ -118,16 +124,20 @@ docker run -d -p 8983:8983 --name my_solr solr solr-precreate gettingstarted----The `solr-precreate` command takes an optional extra argument to specify a configset directory below `/opt/solr/server/solr/configsets/`.-This allows you to specify your own config. See https://github.com/docker-solr/docker-solr-examples/tree/master/custom-configset[this example].+This allows you to specify your own config.+See https://github.com/docker-solr/docker-solr-examples/tree/master/custom-configset[this example].-The third option is to use the `solr-create` command. This runs a Solr in the background in the container, then uses the Solr control script to create the core, then stops the Solr server and restarts it in the foreground. This method is less popular because the double Solr run can be confusing.+The third option is to use the `solr-create` command.+This runs a Solr in the background in the container, then uses the Solr control script to create the core, then stops the Solr server and restarts it in the foreground.+This method is less popular because the double Solr run can be confusing.[source,bash]----docker run -d -p 8983:8983 --name my_solr solr solr-create -c gettingstarted-----Finally, you can run your own command-line and specify what to do, and even invoke mounted scripts. For example:+Finally, you can run your own command-line and specify what to do, and even invoke mounted scripts.+For example:[source,bash]----@@ -213,16 +223,21 @@ It has various commented-out values, which you can override when running the condocker run -d -p 8983:8983 -e SOLR_HEAP=800m solr-----You can also mount your own config file. Do no modify the values that are set at the end of the file.+You can also mount your own config file.+Do no modify the values that are set at the end of the file.=== Extending the Image-The Solr docker image has an extension mechanism. At run time, before starting Solr, the container will execute scripts-in the `/docker-entrypoint-initdb.d/` directory. You can add your own scripts there either by using mounted volumes-or by using a custom Dockerfile. These scripts can for example copy a core directory with pre-loaded data for continuous+The Solr docker image has an extension mechanism.+At run time, before starting Solr, the container will execute scripts+in the `/docker-entrypoint-initdb.d/` directory.+You can add your own scripts there either by using mounted volumes+or by using a custom Dockerfile.+These scripts can for example copy a core directory with pre-loaded data for continuousintegration testing, or modify the Solr configuration.-Here is a simple example. With a `custom.sh` script like:+Here is a simple example.+With a `custom.sh` script like:[source,bash]----@@ -245,7 +260,8 @@ Starting Solr on port 8983 from /opt/solr/server----With this extension mechanism it can be useful to see the shell commands that are being executed by the `docker-entrypoint.sh`-script in the docker log. To do that, set an environment variable using Docker's `-e VERBOSE=yes`.+script in the docker log.+To do that, set an environment variable using Docker's `-e VERBOSE=yes`.Instead of using this mechanism, you can of course create your own script that does setup and then call `solr-foreground`, mount that script into the container, and execute it as a command when running the container.@@ -253,7 +269,8 @@ Other ways of extending the image are to create custom Docker images that inheri=== Debugging with jattach-The `jcmd`, `jmap` `jstack` tools can be useful for debugging Solr inside the container. These tools are not included with the JRE, but this image includes the https://github.com/apangin/jattach[jattach] utility which lets you do much of the same.+The `jcmd`, `jmap` `jstack` tools can be useful for debugging Solr inside the container.+These tools are not included with the JRE, but this image includes the https://github.com/apangin/jattach[jattach] utility which lets you do much of the same.....Usage: jattach <pid> <cmd> [args ...]@@ -284,16 +301,24 @@ jattach 10 jcmd GC.heap_infoIn Solr 8, the Solr Docker image switched from just extracting the Solr tar, to using the <<taking-solr-to-production.adoc#service-installation-script,service installation script>>. This was done for various reasons: to bring it in line with the recommendations by the Solr Ref Guide and to make it easier to mount volumes.-This is a backwards incompatible change, and means that if you're upgrading from an older version, you will most likely need to make some changes. If you don't want to upgrade at this time, specify `solr:7` as your container image. If you use `solr:8` you will use the new style. If you use just `solr` then you risk being tripped up by backwards incompatible changes; always specify at least a major version.+This is a backwards incompatible change, and means that if you're upgrading from an older version, you will most likely need to make some changes.+If you don't want to upgrade at this time, specify `solr:7` as your container image.+If you use `solr:8` you will use the new style.+If you use just `solr` then you risk being tripped up by backwards incompatible changes; always specify at least a major version.Changes:-* The Solr data is now stored in `/var/solr/data` rather than `/opt/solr/server/solr`. The `/opt/solr/server/solr/mycores` no longer exists-* The custom `SOLR_HOME` can no longer be used, because various scripts depend on the new locations. Consequently, `INIT_SOLR_HOME` is also no longer supported.+* The Solr data is now stored in `/var/solr/data` rather than `/opt/solr/server/solr`.+The `/opt/solr/server/solr/mycores` no longer exists.+* The custom `SOLR_HOME` can no longer be used, because various scripts depend on the new locations.+Consequently, `INIT_SOLR_HOME` is also no longer supported.== Running under tini-The Solr docker image runs Solr under https://github.com/krallin/tini[tini], to make signal handling work better; in particular, this allows you to `kill -9` the JVM. If you run `docker run --init`, or use `init: true` in `docker-compose.yml`, or have added `--init` to `dockerd`, docker will start its `tini` and docker-solr will notice it is not PID 1, and just `exec` Solr. If you do not run with `--init`, then the docker entrypoint script detects that it is running as PID 1, and will start the `tini` present in the docker-solr image, and run Solr under that. If you really do not want to run `tini`, and just run Solr as PID 1 instead, then you can set the `TINI=no` environment variable.+The Solr docker image runs Solr under https://github.com/krallin/tini[tini], to make signal handling work better; in particular, this allows you to `kill -9` the JVM.+If you run `docker run --init`, or use `init: true` in `docker-compose.yml`, or have added `--init` to `dockerd`, docker will start its `tini` and docker-solr will notice it is not PID 1, and just `exec` Solr.+If you do not run with `--init`, then the docker entrypoint script detects that it is running as PID 1, and will start the `tini` present in the docker-solr image, and run Solr under that.+If you really do not want to run `tini`, and just run Solr as PID 1 instead, then you can set the `TINI=no` environment variable.== Out of Memory Handling@@ -304,4 +329,6 @@ If you specify `OOM=script`, Solr docker will add `-XX:OnOutOfMemoryError=/opt/d== History-The Docker-Solr project was started in 2015 by https://github.com/makuk66[Martijn Koster] in the https://github.com/docker-solr/docker-solr[docker-solr] repository. In 2019 maintainership and copyright was transferred to the Apache Lucene/Solr project, and in 2020 the project was migrated to live within the Solr project. Many thanks to Martijn for all your contributions over the years!+The Docker-Solr project was started in 2015 by https://github.com/makuk66[Martijn Koster] in the https://github.com/docker-solr/docker-solr[docker-solr] repository.+In 2019 maintainership and copyright was transferred to the Apache Lucene/Solr project, and in 2020 the project was migrated to live within the Solr project.+Many thanks to Martijn for all your contributions over the years!diff --git a/solr/solr-ref-guide/src/solr-on-hdfs.adoc b/solr/solr-ref-guide/src/solr-on-hdfs.adocindex 9f31235efd9..2be7d447afb 100644--- a/solr/solr-ref-guide/src/solr-on-hdfs.adoc+++ b/solr/solr-ref-guide/src/solr-on-hdfs.adoc@@ -16,7 +16,8 @@// specific language governing permissions and limitations// under the License.-WARNING: Storing indexes in HDFS is deprecated and may be be removed in 9.0. This functionality may be moved to a 3rd-party plugin in the future.+WARNING: Storing indexes in HDFS is deprecated and may be be removed in 9.0.+This functionality may be moved to a 3rd-party plugin in the future.Solr has support for writing and reading its index and transaction log files to the HDFS distributed filesystem.diff --git a/solr/solr-ref-guide/src/solr-plugins.adoc b/solr/solr-ref-guide/src/solr-plugins.adocindex 084dabd6950..dda54b37941 100644--- a/solr/solr-ref-guide/src/solr-plugins.adoc+++ b/solr/solr-ref-guide/src/solr-plugins.adoc@@ -35,9 +35,12 @@ One resource is the Solr Wiki documentation on plugins at https://cwiki.apache.oThere are essentially two types of plugins in Solr:-* Collection level plugins. These are registered on individual collections, either by hand-editing the `solrconfig.xml` or schema files for the collection's configset or by using the <<config-api.adoc#,config API>> or <<schema-api.adoc#,schema API>>. Examples of these are query parsers, request handlers, update request processors, value source parsers, response writers etc.+* Collection level plugins.+These are registered on individual collections, either by hand-editing the `solrconfig.xml` or schema files for the collection's configset or by using the <<config-api.adoc#,config API>> or <<schema-api.adoc#,schema API>>. Examples of these are query parsers, request handlers, update request processors, value source parsers, response writers etc.-* Cluster level (or Core Container level) plugins. These are plugins that are installed at a cluster level and every Solr node has one instance each of these plugins. Examples of these are <<authentication-and-authorization-plugins.adoc#,authentication and authorization plugins>>, <<metrics-reporting.adoc#reporters,metrics reporters>>, https://issues.apache.org/jira/browse/SOLR-14404[cluster level request handlers] etc.+* Cluster level (or Core Container level) plugins.+These are plugins that are installed at a cluster level and every Solr node has one instance each of these plugins.+Examples of these are <<authentication-and-authorization-plugins.adoc#,authentication and authorization plugins>>, <<metrics-reporting.adoc#reporters,metrics reporters>>, https://issues.apache.org/jira/browse/SOLR-14404[cluster level request handlers] etc.== Installing Plugins ==diff --git a/solr/solr-ref-guide/src/solr-tutorial.adoc b/solr/solr-ref-guide/src/solr-tutorial.adocindex 4ebbe7ce8e0..3f8c81eebc8 100644--- a/solr/solr-ref-guide/src/solr-tutorial.adoc+++ b/solr/solr-ref-guide/src/solr-tutorial.adoc@@ -47,7 +47,8 @@ For best results, please run the browser showing this tutorial and the Solr serv== Unpack Solr-Begin by unzipping the Solr release and changing your working directory to the subdirectory where Solr was installed. For example, with a shell in UNIX, Cygwin, or MacOS:+Begin by unzipping the Solr release and changing your working directory to the subdirectory where Solr was installed.+For example, with a shell in UNIX, Cygwin, or MacOS:[source,bash,subs="verbatim,attributes+"]----diff --git a/solr/solr-ref-guide/src/solr-upgrade-notes.adoc b/solr/solr-ref-guide/src/solr-upgrade-notes.adocindex 8863a7bd683..2b1bf505d82 100644--- a/solr/solr-ref-guide/src/solr-upgrade-notes.adoc+++ b/solr/solr-ref-guide/src/solr-upgrade-notes.adoc@@ -24,11 +24,13 @@The following notes describe changes to Solr in recent releases that you should be aware of before upgrading.These notes highlight the biggest changes that may impact the largest number of-implementations. It is not a comprehensive list of all changes to Solr in any release.+implementations.+It is not a comprehensive list of all changes to Solr in any release.When planning your Solr upgrade, consider the customizations you have made toyour system and review the {solr-javadocs}/changes/Changes.html[`CHANGES.txt`]-file found in your Solr package. That file includes all the changes and updates+file found in your Solr package.+That file includes all the changes and updatesthat may effect your existing implementation.Detailed steps for upgrading a Solr cluster are in the section <<upgrading-a-solr-cluster.adoc#,Upgrading a Solr Cluster>>.@@ -69,7 +71,8 @@ New Collections API commands for backups:* LISTBACKUP: Lists information about each backup stored at the specified repository location.See <<collection-management.adoc#listbackup,List Backups>> for more details.-* DELETEBACKUP: Deletes specified backups from the repository. See <<collection-management.adoc#deletebackup,Delete Backups>> for more details.+* DELETEBACKUP: Deletes specified backups from the repository.+See <<collection-management.adoc#deletebackup,Delete Backups>> for more details.A new option for backup repository is also available in 8.9, which is to use Google Cloud Storage (GCS).This is a contrib (located in `contrib/gcs-repository`).@@ -79,7 +82,8 @@ The Solr community is working to add support for S3 buckets in the near future.*Nested Docs*Child Doc Transformer's `childFilter` parameter no longer applies query syntax-escaping because it's inconsistent with the rest of Solr and was limiting. This refers to `[child childFilter='field:value']`.+escaping because it's inconsistent with the rest of Solr and was limiting.+ This refers to `[child childFilter='field:value']`.There was no escaping here prior to 8.0 either.*Collapse and Expand*@@ -88,7 +92,8 @@ There was no escaping here prior to 8.0 either.+See <<collapse-and-expand-results.adoc#block-collapsing,Block Collapsing>> for details.-* Expand Null Groups: A new parameter `expand.nullGroup` allows an expanded group to be returned containing document with no value in the expanded field. See <<collapse-and-expand-results.adoc#expand-component,Expand Component>> for details.+* Expand Null Groups: A new parameter `expand.nullGroup` allows an expanded group to be returned containing document with no value in the expanded field.+See <<collapse-and-expand-results.adoc#expand-component,Expand Component>> for details.*In-Place Updates*@@ -126,7 +131,8 @@ continue to define it as it's useful for other things.*Removed Contribs*-* The search results clustering contrib (Carrot2) has been removed from 8.x Solr due to lack of Java 1.8 compatibility in the dependency that provides online clustering of search results. The contrib will be re-introduced in Solr 9.0.+* The search results clustering contrib (Carrot2) has been removed from 8.x Solr due to lack of Java 1.8 compatibility in the dependency that provides online clustering of search results.+The contrib will be re-introduced in Solr 9.0.*Learning to Rank*@@ -156,13 +162,17 @@ https://issues.apache.org/jira/browse/SOLR-14934[SOLR-14934] has more technical*base_url removed from stored state*-As of Solr 8.8.0, the `base_url` property was removed from the stored state for replicas (SOLR-12182). If you're able to upgrade SolrJ to 8.8.x+As of Solr 8.8.0, the `base_url` property was removed from the stored state for replicas (SOLR-12182).+If you're able to upgrade SolrJ to 8.8.xfor all of your client applications, then you can set `-Dsolr.storeBaseUrl=false` (introduced in Solr 8.8.1) to better align the stored state-in ZooKeeper with future versions of Solr. However, if you are not able to upgrade SolrJ to 8.8.x for all client applications,+in ZooKeeper with future versions of Solr.+However, if you are not able to upgrade SolrJ to 8.8.x for all client applications,then leave the default `-Dsolr.storeBaseUrl=true` so that Solr will continue to store the `base_url` in ZooKeeper.-You may also see some NPE in collection state updates during a rolling upgrade to 8.8.0 from a previous version of Solr. After upgrading all nodes in your cluster-to 8.8.0, collections should fully recover. Trigger another rolling restart if there are any replicas that do not recover after the upgrade to re-elect leaders.+You may also see some NPE in collection state updates during a rolling upgrade to 8.8.0 from a previous version of Solr.+After upgrading all nodes in your cluster+to 8.8.0, collections should fully recover.+Trigger another rolling restart if there are any replicas that do not recover after the upgrade to re-elect leaders.=== Solr 8.7@@ -179,12 +189,14 @@ If you are already on 8.6.1 or higher, you can ignore these instructions.*Deprecations** The autoscaling framework is now formally deprecated and will be removed in Solr 9.0.-The Solr community is working on pluggable API to replace this functionality, with the goal for it to be ready by the time 9.0 is released. Deprecations include: autoscaling policy, triggers, `withCollection` support, simulation framework, autoscaling suggestions tab in the UI, `autoAddReplicas` and `UTILIZENODE` command.+The Solr community is working on pluggable API to replace this functionality, with the goal for it to be ready by the time 9.0 is released.+Deprecations include: autoscaling policy, triggers, `withCollection` support, simulation framework, autoscaling suggestions tab in the UI, `autoAddReplicas` and `UTILIZENODE` command.* Similarly, rule-based replica placement strategy has been deprecated and will be replacedin Solr 9.0 by APIs for replica placement and cluster events, with plugin-based implementations.-* Support for detecting spinning disks has been removed in LUCENE-9576. Corresponding+* Support for detecting spinning disks has been removed in LUCENE-9576.+Corresponding`spins` metrics in Solr still exist but now they always return `false` and will be removed in Solr 9.0.*User-Managed Cluster Terminology Updated*@@ -256,7 +268,8 @@ Replace `localhost:8983` with your Solr endpoint.[source,text]curl -X POST -H 'Content-type:application/json' -d '{set-cluster-policy : [], set-cluster-preferences : []}' http://localhost:8983/api/cluster/autoscaling+-This information is only relevant for users upgrading from 8.6.0. If upgrading from an earlier version to 8.6.1+, this warning can be ignored.+This information is only relevant for users upgrading from 8.6.0.+If upgrading from an earlier version to 8.6.1+, this warning can be ignored.=== Solr 8.6@@ -359,7 +372,8 @@ __Note: an index incompatibility warning was retroactively added below to 8.4 fo*Considerations for a SolrCloud Upgrade*-Solr 8.5 introduces a change in the format used for the elements in the Overseer queues and maps (see https://issues.apache.org/jira/browse/SOLR-14095[SOLR-14095] for technical discussion of the change). This queue is used internally by the Overseer to reliably handle+Solr 8.5 introduces a change in the format used for the elements in the Overseer queues and maps (see https://issues.apache.org/jira/browse/SOLR-14095[SOLR-14095] for technical discussion of the change).+This queue is used internally by the Overseer to reliably handleoperations, to communicate operation results between the Overseer and the coordinator node, and by the REQUESTSTATUS API for displaying information about async Collection operations.This change won’t require you to change any client-side code you should see no differences on the client side.@@ -389,17 +403,21 @@ if you use async operations.If you use Collection API operations and can’t pause them during the upgrade:-. Start 8.5 nodes with the system property: `-Dsolr.useUnsafeOverseerResponse=deserialization`. Ensure the+. Start 8.5 nodes with the system property: `-Dsolr.useUnsafeOverseerResponse=deserialization`.+Ensure theOverseer node is upgraded last.. Once all nodes are in 8.5 and once you don’t need to read old status anymore, restart again removing thesystem property.If you prefer to keep the old (but insecure) serialization strategy, you can start your nodes using the system-property: `-Dsolr.useUnsafeOverseerResponse=true`. Keep in mind that this will be removed in future version of Solr.+property: `-Dsolr.useUnsafeOverseerResponse=true`.+Keep in mind that this will be removed in future version of Solr.*Security Manager*-Solr now has the ability to run with a Java security manager enabled. To enable this, set the property `SOLR_SECURITY_MANAGER_ENABLED=true` in `solr.in.sh` or `solr.in.cmd`. Note that if you are using HDFS to store indexes, you cannot enable the security manager.+Solr now has the ability to run with a Java security manager enabled.+To enable this, set the property `SOLR_SECURITY_MANAGER_ENABLED=true` in `solr.in.sh` or `solr.in.cmd`.+Note that if you are using HDFS to store indexes, you cannot enable the security manager.In Solr 9.0, this will be the default.@@ -407,7 +425,9 @@ In Solr 9.0, this will be the default.*Block/Allow Specific IPs*-Solr has two new parameters to allow you to restrict access to Solr using IP addresses. Use `SOLR_IP_WHITELIST` to configure a whitelist, and `SOLR_IP_BLACKLIST` to configure a blacklist. These properties are defined in `solr.in.sh` or `solr.in.cmd`.+Solr has two new parameters to allow you to restrict access to Solr using IP addresses.+Use `SOLR_IP_WHITELIST` to configure a whitelist, and `SOLR_IP_BLACKLIST` to configure a blacklist.+These properties are defined in `solr.in.sh` or `solr.in.cmd`.See also the section <<securing-solr.adoc#ip-access-control,Enable IP Access Control>>.@@ -419,7 +439,8 @@ More information about this is available in the section <<json-faceting-domain-c*Caching with the Boolean Query Parser*-By default, the <<other-parsers.adoc#boolean-query-parser,Boolean Query Parser>> caches queries in Solr's filterCache. It's now possible to disable this with the local param `cache=false`.+By default, the <<other-parsers.adoc#boolean-query-parser,Boolean Query Parser>> caches queries in Solr's filterCache.+It's now possible to disable this with the local param `cache=false`.*Indexing Log Files*@@ -456,7 +477,8 @@ There is now improved documentation to navigate this trade-off choice.*Package Management System*-Version 8.4 introduces a package management system to Solr. The goals of the+Version 8.4 introduces a package management system to Solr.+The goals of thesystem are to allow hot (live) deployment of plugins, provide packagingguidelines for plugins, and standardize Solr's approach by following familiarconcepts used in other package management systems.@@ -465,7 +487,8 @@ The system is designed to eventually replace use of the `<lib ../>` directive,the Blob Store, and other methods of deploying plugins and custom componentsto Solr.-The system is currently considered experimental, so use with caution. It must+The system is currently considered experimental, so use with caution.+It mustbe enabled with a system parameter passed at start up before it can be used.For details, please see the section <<package-manager.adoc#,Package Management>>.@@ -478,7 +501,8 @@ The follow mix of changes were all made with the intention of making Solr more s* The `solrconfig.xml` file in Solr's `_default` configset has been trimmed ofthe following previously pre-configured items:+-** All `<lib .../>` directives. This means that Solr Cell (aka Tika), Learning+** All `<lib .../>` directives.+This means that Solr Cell (aka Tika), Learningto Rank, Clustering (with Carrot2), language identification, and Velocity (forthe `/browse` sample search interface) are no longer enabled out of the box.** The `/browse`, `/tvrh`, and `/update/extract` request handlers.@@ -498,7 +522,8 @@ or custom Jars.+When upgrading to 8.4, if you are using untrusted configsets that contain `<lib ../>`directives, their corresponding collections will not load (they will cease to-work). You have a few options in this case:+work).+You have a few options in this case:** You can secure your Solr instance with <<authentication-and-authorization-plugins.adoc#,authentication>>and re-upload the configset (using the `bin/solr zk upconfig ...`@@ -510,16 +535,19 @@ See the section <<configsets-api.adoc#configsets-upload,Upload a Configset>>for more details about trusted vs. untrusted configsets.* Our default Jetty configuration has been updated to now set a-Content-Security-Policy (CSP) by default. See `./server/etc/jetty.xml` for+Content-Security-Policy (CSP) by default.+See `./server/etc/jetty.xml` fordetails about how it is configured.+-As a result of this change, any custom HTML served by Solr's HTTP server that contains inline Javascript will no longer execute in modern browsers. The options for you are:+As a result of this change, any custom HTML served by Solr's HTTP server that contains inline Javascript will no longer execute in modern browsers.+The options for you are:** Change your JavaScript code to not run inline any longer;** Edit `jetty.xml` to remove CSP (creating weaker security protection);** Remove/alter the headers with a reverse proxy.-* Solr's Blob Store and runtime libs functionality are now deprecated and are planned to be removed from Solr in version 9.0. It has been replaced with the new package management system.+* Solr's Blob Store and runtime libs functionality are now deprecated and are planned to be removed from Solr in version 9.0.+It has been replaced with the new package management system.* The Velocity response writer is also now deprecated and is planned to be removed from Solr in version 9.0.@@ -541,7 +569,8 @@ See <<solrj.adoc#cloud-request-routing,Cloud Request Routing>> and* `QueryResponse.getExplainMap()` type has changed from `Map<String, String>` to `Map<String, Object>` in order to support structured explanations.+-This change is expected to be mostly back-compatible. Compiled third-party+This change is expected to be mostly back-compatible.+Compiled third-partycomponents will work the same due to type erasure, but source code changes maybe required.@@ -586,14 +615,16 @@ Implementations using `jwkUrl` will continue to work as normal, but users*Caches*-* Solr has a new cache implementation, `CaffeineCache`, which is now recommended over other caches. This cache is expected to generally provide most users lower memory footprint, higher hit ratio, and better multi-threaded performance.+* Solr has a new cache implementation, `CaffeineCache`, which is now recommended over other caches.+This cache is expected to generally provide most users lower memory footprint, higher hit ratio, and better multi-threaded performance.+Since caching has a direct impact on the performance of your Solrimplementation, before switching to any new cache implementation inproduction, take care to test for your environment and traffic patterns soyou fully understand the ramifications of the change.-* A new parameter, `maxIdleTime`, allows automatic eviction of cache items that have not been used in the defined amount of time. This allows the cache to release some memory and should aid those who want or need to fine-tune their caches.+* A new parameter, `maxIdleTime`, allows automatic eviction of cache items that have not been used in the defined amount of time.+This allows the cache to release some memory and should aid those who want or need to fine-tune their caches.See the section <<caches-warming.adoc#,Caches and Query Warming>> for more details about these and other cache options and parameters.@@ -615,14 +646,16 @@ At a minimum, `ruok`, `conf`, and `mntr` must be enabled, but other commands canSee the section <<zookeeper-ensemble.adoc#configuration-for-a-zookeeper-ensemble,Configuration for a ZooKeeper Ensemble>> for details.[WARNING]-Until 8.3, https://issues.apache.org/jira/browse/SOLR-13672[SOLR-13672] causes the ZK Status screen in the Admin UI to not be able to report status. This only impacts the UI, ZooKeeper still operates correctly.+Until 8.3, https://issues.apache.org/jira/browse/SOLR-13672[SOLR-13672] causes the ZK Status screen in the Admin UI to not be able to report status.+This only impacts the UI, ZooKeeper still operates correctly.*Routed Aliases** Routed aliases now use collection properties to identify collections that belong to the alias; prior to 8.2, these aliases used core properties.+This is backward-compatible and aliases created with prior versions will- continue to work. However, new collections will no longer add the+ continue to work.+However, new collections will no longer add the`routedAliasName` property to the `core.properties` file so any external codedepending on this location will need to be updated.@@ -633,7 +666,8 @@ Collections created with older versions will continue to work.*Distributed Tracing Support*-This release adds support for tracing requests in Solr. Please review the section <<distributed-tracing.adoc#,Distributed Tracing>> for details on how to configure this feature.+This release adds support for tracing requests in Solr.+Please review the section <<distributed-tracing.adoc#,Distributed Tracing>> for details on how to configure this feature.=== Solr 8.1@@ -645,7 +679,9 @@ When upgrading to 8.1.x, users should be aware of the following major changes fr* The behavior of the `maxBooleanClauses` parameter has changed to reduce the risk of exponential query expansion when dealing with pathological query strings.+-A default upper limit of 1024 clauses is now enforced at the node level. This was the default prior to 7.0, and it can be overridden with a new global parameter in `solr.xml`. This limit will be enforced for all queries whether explicitly defined by the user (or client), or created by Solr and Lucene internals.+A default upper limit of 1024 clauses is now enforced at the node level.+This was the default prior to 7.0, and it can be overridden with a new global parameter in `solr.xml`.+This limit will be enforced for all queries whether explicitly defined by the user (or client), or created by Solr and Lucene internals.+An identical parameter is available in `solrconfig.xml` for limiting the size of queries explicitly defined by the user (or client), but this per-collection limit will still be restricted by the global limit set in `solr.xml`.+@@ -655,39 +691,56 @@ For more information about the new parameter, see the section <<configuring-solr*Security*-* JSON Web Tokens (JWT) are now supported for authentication. These allow Solr to assert a user is already authenticated via an external identity provider, such as an OpenID Connect-enabled IdP. For more information, see the section <<jwt-authentication-plugin.adoc#,JWT Authentication Plugin>>.+* JSON Web Tokens (JWT) are now supported for authentication.+These allow Solr to assert a user is already authenticated via an external identity provider, such as an OpenID Connect-enabled IdP.+For more information, see the section <<jwt-authentication-plugin.adoc#,JWT Authentication Plugin>>.-* A new security plugin for audit logging has been added. A default class `SolrLogAuditLoggerPlugin` is available and configurable in `security.json`. The base class is also extendable for adding custom audit plugins if needed. See the section <<audit-logging.adoc#,Audit Logging>> for more information.+* A new security plugin for audit logging has been added.+A default class `SolrLogAuditLoggerPlugin` is available and configurable in `security.json`.+The base class is also extendable for adding custom audit plugins if needed.+See the section <<audit-logging.adoc#,Audit Logging>> for more information.*Collections API** The output of the REQUESTSTATUS command in the Collections API will now include internal asynchronous requests (if any) in the "success" or "failed" keys.-* The CREATE command will now return the appropriate status code (4xx, 5xx, etc.) when the command has failed. Previously, it always returned `0`, even in failure.+* The CREATE command will now return the appropriate status code (4xx, 5xx, etc.) when the command has failed.+Previously, it always returned `0`, even in failure.-* The MODIFYCOLLECTION command now accepts an attribute to set a collection as read-only. This can be used to block a collection from receiving any updates while still allowing queries to be served. See the section <<collection-management.adoc#modifycollection,MODIFYCOLLECTION>> for details on how to use it.+* The MODIFYCOLLECTION command now accepts an attribute to set a collection as read-only.+This can be used to block a collection from receiving any updates while still allowing queries to be served.+See the section <<collection-management.adoc#modifycollection,MODIFYCOLLECTION>> for details on how to use it.-* A new command RENAME allows renaming a collection by setting up a one-to-one alias using the new name. For more information, see the section <<collection-management.adoc#rename,RENAME>>.+* A new command RENAME allows renaming a collection by setting up a one-to-one alias using the new name.+For more information, see the section <<collection-management.adoc#rename,RENAME>>.-* A new command REINDEXCOLLECTION allows indexing existing stored fields from a source collection into a new collection. For more information, please see the section <<collection-management.adoc#reindexcollection,REINDEXCOLLECTION>>.+* A new command REINDEXCOLLECTION allows indexing existing stored fields from a source collection into a new collection.+For more information, please see the section <<collection-management.adoc#reindexcollection,REINDEXCOLLECTION>>.*Logging*-* The default Log4j2 logging mode has been changed from synchronous to asynchronous. This will improve logging throughput and reduce system contention at the cost of a _slight_ chance that some logging messages may be missed in the event of abnormal Solr termination.+* The default Log4j2 logging mode has been changed from synchronous to asynchronous.+This will improve logging throughput and reduce system contention at the cost of a _slight_ chance that some logging messages may be missed in the event of abnormal Solr termination.+If even this slight risk is unacceptable, the Log4j configuration file found in `server/resources/log4j2.xml` has the synchronous logging configuration in a commented section and can be edited to re-enable synchronous logging.*Metrics*-* The SolrGangliaReporter has been removed from Solr. The metrics library used by Solr, Dropwizard Metrics, was updated to version 4, and Ganglia support was removed from it due to a dependency on the LGPL license.+* The SolrGangliaReporter has been removed from Solr.+The metrics library used by Solr, Dropwizard Metrics, was updated to version 4, and Ganglia support was removed from it due to a dependency on the LGPL license.*Browse UI (Velocity)*-* Velocity and Velocity Tools were both upgraded as part of this release. Velocity upgraded from 1.7 to 2.0. Please see https://velocity.apache.org/engine/2.0/upgrading.html about upgrading. Velocity Tools upgraded from 2.0 to 3.0. For more details, please see https://velocity.apache.org/tools/3.0/upgrading.html for details about the upgrade.+* Velocity and Velocity Tools were both upgraded as part of this release.+Velocity upgraded from 1.7 to 2.0.+Please see https://velocity.apache.org/engine/2.0/upgrading.html about upgrading.+Velocity Tools upgraded from 2.0 to 3.0.+For more details, please see https://velocity.apache.org/tools/3.0/upgrading.html for details about the upgrade.*Default Garbage Collector (GC)*-* Solr's default GC has been changed from CMS to G1. If you prefer to use CMS or any other GC method, you can modify the `GC_TUNE` section of `solr.in.sh` (*nix) or `solr.in.cmd` (Windows).+* Solr's default GC has been changed from CMS to G1.+If you prefer to use CMS or any other GC method, you can modify the `GC_TUNE` section of `solr.in.sh` (*nix) or `solr.in.cmd` (Windows).== Upgrading from 7.x Releases@@ -702,6 +755,7 @@ If you run in SolrCloud mode, you must be on Solr version 7.3 or higher in orderUsers upgrading from versions of Solr prior to 7.x are strongly encouraged to consult {solr-javadocs}/changes/Changes.html[`CHANGES.txt`] for the details of _all_ changes since the version they are upgrading from.-The upgrade from Solr 6.x to Solr 7.0 introduced several *major* changes that you should be aware of before upgrading. Please do a thorough review of the section <<major-changes-in-solr-7.adoc#,Major Changes in Solr 7>> before starting your upgrade.+The upgrade from Solr 6.x to Solr 7.0 introduced several *major* changes that you should be aware of before upgrading.+Please do a thorough review of the section <<major-changes-in-solr-7.adoc#,Major Changes in Solr 7>> before starting your upgrade.A summary of the significant changes between Solr 5.x and Solr 6.0 is in the section <<major-changes-in-solr-6.adoc#,Major Changes in Solr 6>>.diff --git a/solr/solr-ref-guide/src/solrcloud-distributed-requests.adoc b/solr/solr-ref-guide/src/solrcloud-distributed-requests.adocindex 5755cab846b..9a95f115667 100644--- a/solr/solr-ref-guide/src/solrcloud-distributed-requests.adoc+++ b/solr/solr-ref-guide/src/solrcloud-distributed-requests.adoc@@ -170,7 +170,8 @@ http://localhost:8983/solr/gettingstarted/select?q=*:*&shards=shard1,localhost:7Solr allows you to pass an optional string parameter named `shards.preference` to indicate that a distributed query should sort the available replicas in the given order of precedence within each shard.-The syntax is: `shards.preference=_property_:__value__`. The order of the properties and the values are significant: the first one is the primary sort, the second is secondary, etc.+The syntax is: `shards.preference=_property_:__value__`.+The order of the properties and the values are significant: the first one is the primary sort, the second is secondary, etc.IMPORTANT: `shards.preference` is supported for single shard scenarios only when using the SolrJ clients.Queries that do not use the SolrJ client cannot use `shards.preference` in single shard collections.@@ -178,25 +179,31 @@ Queries that do not use the SolrJ client cannot use `shards.preference` in singlThe properties that can be specified are as follows:`replica.type`::-One or more replica types that are preferred. Any combination of `PULL`, `TLOG` and `NRT` is allowed.+One or more replica types that are preferred.+Any combination of `PULL`, `TLOG` and `NRT` is allowed.`replica.location`::One or more replica locations that are preferred.+-A location starts with `http://hostname:port`. Matching is done for the given string as a prefix, so it's possible to e.g., leave out the port.+A location starts with `http://hostname:port`.+Matching is done for the given string as a prefix, so it's possible to e.g., leave out the port.+-A special value `local` may be used to denote any local replica running on the same Solr instance as the one handling the query. This is useful when a query requests many fields or large fields to be returned per document because it avoids moving large amounts of data over the network when it is available locally. In addition, this feature can be useful for minimizing the impact of a problematic replica with degraded performance, as it reduces the likelihood that the degraded replica will be hit by other healthy replicas.+A special value `local` may be used to denote any local replica running on the same Solr instance as the one handling the query.+This is useful when a query requests many fields or large fields to be returned per document because it avoids moving large amounts of data over the network when it is available locally.+In addition, this feature can be useful for minimizing the impact of a problematic replica with degraded performance, as it reduces the likelihood that the degraded replica will be hit by other healthy replicas.+The value of `replica.location:local` diminishes as the number of shards (that have no locally-available replicas) in a collection increases because the query controller will have to direct the query to non-local replicas for most of the shards.+In other words, this feature is mostly useful for optimizing queries directed towards collections with a small number of shards and many replicas.+-Also, this option should only be used if you are load balancing requests across all nodes that host replicas for the collection you are querying, as Solr's `CloudSolrClient` will do. If not load-balancing, this feature can introduce a hotspot in the cluster since queries won't be evenly distributed across the cluster.+Also, this option should only be used if you are load balancing requests across all nodes that host replicas for the collection you are querying, as Solr's `CloudSolrClient` will do.+If not load-balancing, this feature can introduce a hotspot in the cluster since queries won't be evenly distributed across the cluster.`replica.base`::Applied after sorting by inherent replica attributes, this property defines a fallback ordering among sets of preference-equivalent replicas; if specified, only one value may be specified for this property, and it must be specified last.+-`random`, the default, randomly shuffles replicas for each request. This distributes requests evenly, but can result in sub-optimal cache usage for shards with replication factor > 1.+`random`, the default, randomly shuffles replicas for each request.+This distributes requests evenly, but can result in sub-optimal cache usage for shards with replication factor > 1.+`stable:dividend:_paramName_` parses an integer from the value associated with the given parameter name; this integer is used as the dividend (mod equivalent replica count) to determine (via list rotation) order of preference among equivalent replicas.+@@ -306,7 +313,8 @@ For extremely short autoCommit intervals, consider disabling caching and autowar== Configuring the ShardHandlerFactory-For finer-grained control, you can directly configure and tune aspects of the concurrency and thread-pooling used within distributed search in Solr. The default configuration favors throughput over latency.+For finer-grained control, you can directly configure and tune aspects of the concurrency and thread-pooling used within distributed search in Solr.+The default configuration favors throughput over latency.This is done by defining a `shardHandlerFactory` in the configuration for your search handler.diff --git a/solr/solr-ref-guide/src/solrcloud-recoveries-and-write-tolerance.adoc b/solr/solr-ref-guide/src/solrcloud-recoveries-and-write-tolerance.adocindex 94ab897427e..6b539a93e32 100644--- a/solr/solr-ref-guide/src/solrcloud-recoveries-and-write-tolerance.adoc+++ b/solr/solr-ref-guide/src/solrcloud-recoveries-and-write-tolerance.adoc@@ -20,24 +20,42 @@ SolrCloud is highly available and fault tolerant in reads and writes.== Write Side Fault Tolerance-SolrCloud is designed to replicate documents to ensure redundancy for your data, and enable you to send update requests to any node in the cluster. That node will determine if it hosts the leader for the appropriate shard, and if not it will forward the request to the the leader, which will then forward it to all existing replicas, using versioning to make sure every replica has the most up-to-date version. If the leader goes down, another replica can take its place. This architecture enables you to be certain that your data can be recovered in the event of a disaster.+SolrCloud is designed to replicate documents to ensure redundancy for your data, and enable you to send update requests to any node in the cluster.+That node will determine if it hosts the leader for the appropriate shard, and if not it will forward the request to the the leader, which will then forward it to all existing replicas, using versioning to make sure every replica has the most up-to-date version.+If the leader goes down, another replica can take its place.+This architecture enables you to be certain that your data can be recovered in the event of a disaster.=== Recovery-A Transaction Log is created for each node so that every change to content or organization is noted. The log is used to determine which content in the node should be included in a replica. When a new replica is created, it refers to the Leader and the Transaction Log to know which content to include. If it fails, it retries.+A Transaction Log is created for each node so that every change to content or organization is noted.+The log is used to determine which content in the node should be included in a replica.+When a new replica is created, it refers to the Leader and the Transaction Log to know which content to include.+If it fails, it retries.Since the Transaction Log consists of a record of updates, it allows for more robust indexing because it includes redoing the uncommitted updates if indexing is interrupted.-If a leader goes down, it may have sent requests to some replicas and not others. So when a new potential leader is identified, it runs a synch process against the other replicas. If this is successful, everything should be consistent, the leader registers as active, and normal actions proceed. If a replica is too far out of sync, the system asks for a full replication/replay-based recovery.+If a leader goes down, it may have sent requests to some replicas and not others.+So when a new potential leader is identified, it runs a synch process against the other replicas.+If this is successful, everything should be consistent, the leader registers as active, and normal actions proceed.+If a replica is too far out of sync, the system asks for a full replication/replay-based recovery.If an update fails because cores are reloading schemas and some have finished but others have not, the leader tells the nodes that the update failed and starts the recovery procedure.=== Achieved Replication Factor-When using a replication factor greater than one, an update request may succeed on the shard leader but fail on one or more of the replicas. For instance, consider a collection with one shard and a replication factor of three. In this case, you have a shard leader and two additional replicas. If an update request succeeds on the leader but fails on both replicas, for whatever reason, the update request is still considered successful from the perspective of the client. The replicas that missed the update will sync with the leader when they recover.+When using a replication factor greater than one, an update request may succeed on the shard leader but fail on one or more of the replicas.+For instance, consider a collection with one shard and a replication factor of three.+In this case, you have a shard leader and two additional replicas.+If an update request succeeds on the leader but fails on both replicas, for whatever reason, the update request is still considered successful from the perspective of the client.+The replicas that missed the update will sync with the leader when they recover.-Behind the scenes, this means that Solr has accepted updates that are only on one of the nodes (the current leader). To make the client aware of this, Solr includes in the response header the "Achieved Replication Factor" (`rf`). The achieved replication factor is the number of replicas of the shard that actually received the update request (including the leader), in the above example, 1. In the case of multi-shard update requests, the achieved replication factor is the minimum achieved replication factor across all shards.+Behind the scenes, this means that Solr has accepted updates that are only on one of the nodes (the current leader).+To make the client aware of this, Solr includes in the response header the "Achieved Replication Factor" (`rf`).+The achieved replication factor is the number of replicas of the shard that actually received the update request (including the leader), in the above example, 1.+In the case of multi-shard update requests, the achieved replication factor is the minimum achieved replication factor across all shards.-On the client side, if the achieved replication factor is less than the acceptable level, then the client application can take additional measures to handle the degraded state. For instance, a client application may want to keep a log of which update requests were sent while the state of the collection was degraded and then resend the updates once the problem has been resolved.+On the client side, if the achieved replication factor is less than the acceptable level, then the client application can take additional measures to handle the degraded state.+For instance, a client application may want to keep a log of which update requests were sent while the state of the collection was degraded and then resend the updates once the problem has been resolved.-NOTE: In previous version of Solr, the `min_rf` parameter had to be specified to ask Solr for the achieved replication factor. Now it is always included in the response.+NOTE: In previous version of Solr, the `min_rf` parameter had to be specified to ask Solr for the achieved replication factor.+Now it is always included in the response.diff --git a/solr/solr-ref-guide/src/solrcloud-shards-indexing.adoc b/solr/solr-ref-guide/src/solrcloud-shards-indexing.adocindex 2f360bf2f99..1b0d16e951b 100644--- a/solr/solr-ref-guide/src/solrcloud-shards-indexing.adoc+++ b/solr/solr-ref-guide/src/solrcloud-shards-indexing.adoc@@ -18,37 +18,60 @@When your collection is too large for one node, you can break it up and store it in sections by creating multiple *shards*.-A Shard is a logical partition of the collection, containing a subset of documents from the collection, such that every document in a collection is contained in exactly one Shard. Which shard contains each document in a collection depends on the overall "Sharding" strategy for that collection.+A Shard is a logical partition of the collection, containing a subset of documents from the collection, such that every document in a collection is contained in exactly one shard.+Which shard contains each document in a collection depends on the overall sharding strategy for that collection.-For example, you might have a collection where the "country" field of each document determines which shard it is part of, so documents from the same country are co-located. A different collection might simply use a "hash" on the uniqueKey of each document to determine its Shard.+For example, you might have a collection where the "country" field of each document determines which shard it is part of, so documents from the same country are co-located.+A different collection might simply use a "hash" on the uniqueKey of each document to determine its Shard.-Before SolrCloud, Solr supported Distributed Search, which allowed one query to be executed across multiple shards, so the query was executed against the entire Solr index and no documents would be missed from the search results. So splitting an index across shards is not exclusively a SolrCloud concept. There were, however, several problems with the distributed approach that necessitated improvement with SolrCloud:+Before SolrCloud, Solr supported Distributed Search, which allowed one query to be executed across multiple shards, so the query was executed against the entire Solr index and no documents would be missed from the search results.+So splitting an index across shards is not exclusively a SolrCloud concept.+There were, however, several problems with the distributed approach that necessitated improvement with SolrCloud:. Splitting an index into shards was somewhat manual.. There was no support for distributed indexing, which meant that you needed to explicitly send documents to a specific shard; Solr couldn't figure out on its own what shards to send documents to.. There was no load balancing or failover, so if you got a high number of queries, you needed to figure out where to send them and if one shard died it was just gone.-SolrCloud addresses those limitations. There is support for distributing both the index process and the queries automatically, and ZooKeeper provides failover and load balancing. Additionally, every shard can have multiple replicas for additional robustness.+SolrCloud addresses those limitations.+There is support for distributing both the index process and the queries automatically, and ZooKeeper provides failover and load balancing.+Additionally, every shard can have multiple replicas for additional robustness.== Leaders and Replicas-In SolrCloud there are no leaders or followers. Instead, every shard consists of at least one physical *replica*, exactly one of which is a *leader*. Leaders are automatically elected, initially on a first-come-first-served basis, and then based on the ZooKeeper process described at http://zookeeper.apache.org/doc/r{ivy-zookeeper-version}/recipes.html#sc_leaderElection.+In SolrCloud there are no leaders or followers.+Instead, every shard consists of at least one physical *replica*, exactly one of which is a *leader*.+Leaders are automatically elected, initially on a first-come-first-served basis, and then based on the ZooKeeper process described at http://zookeeper.apache.org/doc/r{ivy-zookeeper-version}/recipes.html#sc_leaderElection.If a leader goes down, one of the other replicas is automatically elected as the new leader.-When a document is sent to a Solr node for indexing, the system first determines which Shard that document belongs to, and then which node is currently hosting the leader for that shard. The document is then forwarded to the current leader for indexing, and the leader forwards the update to all of the other replicas.+When a document is sent to a Solr node for indexing, the system first determines which Shard that document belongs to, and then which node is currently hosting the leader for that shard.+The document is then forwarded to the current leader for indexing, and the leader forwards the update to all of the other replicas.=== Types of Replicas-By default, all replicas are eligible to become leaders if their leader goes down. However, this comes at a cost: if all replicas could become a leader at any time, every replica must be in sync with its leader at all times. New documents added to the leader must be routed to the replicas, and each replica must do a commit. If a replica goes down, or is temporarily unavailable, and then rejoins the cluster, recovery may be slow if it has missed a large number of updates.--These issues are not a problem for most users. However, some use cases would perform better if the replicas behaved a bit more like the former model, either by not syncing in real-time or by not being eligible to become leaders at all.--Solr accomplishes this by allowing you to set the replica type when creating a new collection or when adding a replica. The available types are:--* *NRT*: This is the default. A NRT replica (NRT = NearRealTime) maintains a transaction log and writes new documents to its indexes locally. Any replica of this type is eligible to become a leader. Traditionally, this was the only type supported by Solr.-* *TLOG*: This type of replica maintains a transaction log but does not index document changes locally. This type helps speed up indexing since no commits need to occur in the replicas. When this type of replica needs to update its index, it does so by replicating the index from the leader. This type of replica is also eligible to become a shard leader; it would do so by first processing its transaction log. If it does become a leader, it will behave the same as if it was a NRT type of replica.-* *PULL*: This type of replica does not maintain a transaction log nor index document changes locally. It only replicates the index from the shard leader. It is not eligible to become a shard leader and doesn't participate in shard leader election at all.+By default, all replicas are eligible to become leaders if their leader goes down.+However, this comes at a cost: if all replicas could become a leader at any time, every replica must be in sync with its leader at all times.+New documents added to the leader must be routed to the replicas, and each replica must do a commit.+If a replica goes down, or is temporarily unavailable, and then rejoins the cluster, recovery may be slow if it has missed a large number of updates.++These issues are not a problem for most users.+However, some use cases would perform better if the replicas behaved a bit more like the former model, either by not syncing in real-time or by not being eligible to become leaders at all.++Solr accomplishes this by allowing you to set the replica type when creating a new collection or when adding a replica.+The available types are:++* *NRT*: This is the default.+A NRT replica (NRT = NearRealTime) maintains a transaction log and writes new documents to its indexes locally.+Any replica of this type is eligible to become a leader.+Traditionally, this was the only type supported by Solr.+* *TLOG*: This type of replica maintains a transaction log but does not index document changes locally.+This type helps speed up indexing since no commits need to occur in the replicas.+When this type of replica needs to update its index, it does so by replicating the index from the leader.+This type of replica is also eligible to become a shard leader; it would do so by first processing its transaction log.+If it does become a leader, it will behave the same as if it was a NRT type of replica.+* *PULL*: This type of replica does not maintain a transaction log nor index document changes locally.+It only replicates the index from the shard leader.+It is not eligible to become a shard leader and doesn't participate in shard leader election at all.If you do not specify the type of replica when it is created, it will be NRT type.@@ -62,7 +85,8 @@ There are three combinations of replica types that are recommended:==== All NRT Replicas-Use this for small to medium clusters, or even big clusters where the update (index) throughput is not too high. NRT is the only type of replica that supports soft-commits, so also use this combination when NearRealTime is needed.+Use this for small to medium clusters, or even big clusters where the update (index) throughput is not too high.+NRT is the only type of replica that supports soft-commits, so also use this combination when NearRealTime is needed.==== All TLOG Replicas@@ -74,57 +98,81 @@ Use this combination if NearRealTime is not needed, the number of replicas per s==== Other Combinations of Replica Types-Other combinations of replica types are not recommended. If more than one replica in the shard is writing its own index instead of replicating from an NRT replica, a leader election can cause all replicas of the shard to become out of sync with the leader, and all would have to replicate the full index.+Other combinations of replica types are not recommended.+If more than one replica in the shard is writing its own index instead of replicating from an NRT replica, a leader election can cause all replicas of the shard to become out of sync with the leader, and all would have to replicate the full index.=== Recovery with PULL ReplicasIf a PULL replica goes down or leaves the cluster, there are a few scenarios to consider.-If the PULL replica cannot sync to the leader because the leader is down, replication would not occur. However, it would continue to serve queries. Once it can connect to the leader again, replication would resume.+If the PULL replica cannot sync to the leader because the leader is down, replication would not occur.+However, it would continue to serve queries.+Once it can connect to the leader again, replication would resume.If the PULL replica cannot connect to ZooKeeper, it would be removed from the cluster and queries would not be routed to it from the cluster.-If the PULL replica dies or is unreachable for any other reason, it won't be query-able. When it rejoins the cluster, it would replicate from the leader and when that is complete, it would be ready to serve queries again.+If the PULL replica dies or is unreachable for any other reason, it won't be query-able.+When it rejoins the cluster, it would replicate from the leader and when that is complete, it would be ready to serve queries again.=== Queries with Preferred Replica Types-By default all replicas serve queries. See the section <<solrcloud-distributed-requests.adoc#shards-preference-parameter,shards.preference Parameter>> for details on how to indicate preferred replica types for queries.+By default all replicas serve queries.+See the section <<solrcloud-distributed-requests.adoc#shards-preference-parameter,shards.preference Parameter>> for details on how to indicate preferred replica types for queries.== Document RoutingSolr offers the ability to specify the router implementation used by a collection by specifying the `router.name` parameter when <<collection-management.adoc#create,creating your collection>>.-If you use the `compositeId` router (the default), you can send documents with a prefix in the document ID which will be used to calculate the hash Solr uses to determine the shard a document is sent to for indexing. The prefix can be anything you'd like it to be (it doesn't have to be the shard name, for example), but it must be consistent so Solr behaves consistently.+If you use the `compositeId` router (the default), you can send documents with a prefix in the document ID which will be used to calculate the hash Solr uses to determine the shard a document is sent to for indexing.+The prefix can be anything you'd like it to be (it doesn't have to be the shard name, for example), but it must be consistent so Solr behaves consistently.-For example, if you want to co-locate documents for a customer, you could use the customer name or ID as the prefix. If your customer is "IBM", for example, with a document with the ID "12345", you would insert the prefix into the document id field: "IBM!12345". The exclamation mark ('!') is critical here, as it distinguishes the prefix used to determine which shard to direct the document to.+For example, if you want to co-locate documents for a customer, you could use the customer name or ID as the prefix.+If your customer is "IBM", for example, with a document with the ID "12345", you would insert the prefix into the document id field: "IBM!12345". The exclamation mark ('!') is critical here, as it distinguishes the prefix used to determine which shard to direct the document to.-Then at query time, you include the prefix(es) into your query with the `\_route_` parameter (i.e., `q=solr&_route_=IBM!`) to direct queries to specific shards. In some situations, this may improve query performance because it overcomes network latency when querying all the shards.+Then at query time, you include the prefix(es) into your query with the `\_route_` parameter (i.e., `q=solr&_route_=IBM!`) to direct queries to specific shards.+In some situations, this may improve query performance because it overcomes network latency when querying all the shards.-The `compositeId` router supports prefixes containing up to 2 levels of routing. For example: a prefix routing first by region, then by customer: "USA!IBM!12345"+The `compositeId` router supports prefixes containing up to 2 levels of routing.+For example: a prefix routing first by region, then by customer: "USA!IBM!12345"-Another use case could be if the customer "IBM" has a lot of documents and you want to spread it across multiple shards. The syntax for such a use case would be: `shard_key/num!document_id` where the `/num` is the number of bits from the shard key to use in the composite hash.+Another use case could be if the customer "IBM" has a lot of documents and you want to spread it across multiple shards.+The syntax for such a use case would be: `shard_key/num!document_id` where the `/num` is the number of bits from the shard key to use in the composite hash.-So `IBM/3!12345` will take 3 bits from the shard key and 29 bits from the unique doc id, spreading the tenant over 1/8th of the shards in the collection. Likewise if the num value was 2 it would spread the documents across 1/4th the number of shards. At query time, you include the prefix(es) along with the number of bits into your query with the `\_route_` parameter (i.e., `q=solr&_route_=IBM/3!`) to direct queries to specific shards.+So `IBM/3!12345` will take 3 bits from the shard key and 29 bits from the unique doc id, spreading the tenant over 1/8th of the shards in the collection.+Likewise if the num value was 2 it would spread the documents across 1/4th the number of shards.+At query time, you include the prefix(es) along with the number of bits into your query with the `\_route_` parameter (i.e., `q=solr&_route_=IBM/3!`) to direct queries to specific shards.If you do not want to influence how documents are stored, you don't need to specify a prefix in your document ID.-If you created the collection and defined the "implicit" router at the time of creation, you can additionally define a `router.field` parameter to use a field from each document to identify a shard where the document belongs. If the field specified is missing in the document, however, the document will be rejected. You could also use the `\_route_` parameter to name a specific shard.+If you created the collection and defined the "implicit" router at the time of creation, you can additionally define a `router.field` parameter to use a field from each document to identify a shard where the document belongs.+If the field specified is missing in the document, however, the document will be rejected.+You could also use the `\_route_` parameter to name a specific shard.== Shard Splitting-When you create a collection in SolrCloud, you decide on the initial number shards to be used. But it can be difficult to know in advance the number of shards that you need, particularly when organizational requirements can change at a moment's notice, and the cost of finding out later that you chose wrong can be high, involving creating new cores and reindexing all of your data.+When you create a collection in SolrCloud, you decide on the initial number shards to be used.+But it can be difficult to know in advance the number of shards that you need, particularly when organizational requirements can change at a moment's notice, and the cost of finding out later that you chose wrong can be high, involving creating new cores and reindexing all of your data.-The ability to split shards is in the Collections API. It currently allows splitting a shard into two pieces. The existing shard is left as-is, so the split action effectively makes two copies of the data as new shards. You can delete the old shard at a later time when you're ready.+The ability to split shards is in the Collections API.+It currently allows splitting a shard into two pieces.+The existing shard is left as-is, so the split action effectively makes two copies of the data as new shards.+You can delete the old shard at a later time when you're ready.More details on how to use shard splitting is in the section on the Collection API's <<shard-management.adoc#splitshard,SPLITSHARD command>>.== Ignoring Commits from Client Applications in SolrCloud-In most cases, when running in SolrCloud mode, indexing client applications should not send explicit commit requests. Rather, you should configure auto commits with `openSearcher=false` and `autoSoftCommit` to make recent updates visible in search requests. This ensures that auto commits occur on a regular schedule in the cluster.+In most cases, when running in SolrCloud mode, indexing client applications should not send explicit commit requests.+Rather, you should configure auto commits with `openSearcher=false` and `autoSoftCommit` to make recent updates visible in search requests.+This ensures that auto commits occur on a regular schedule in the cluster.-NOTE: Using `autoSoftCommit` or `commitWithin` requires the client app to embrace the realities of "eventual consistency". Solr will make documents searchable at _roughly_ the same time across replicas of a collection but there are no hard guarantees. Consequently, in rare cases, it's possible for a document to show up in one search only for it not to appear in a subsequent search occurring immediately after the first search when the second search is routed to a different replica. Also, documents added in a particular order (even in the same batch) might become searchable out of the order of submission when there is sharding. The document will become visible on all replicas of a shard after the next `autoCommit` or `commitWithin` interval expires.+NOTE: Using `autoSoftCommit` or `commitWithin` requires the client app to embrace the realities of "eventual consistency". Solr will make documents searchable at _roughly_ the same time across replicas of a collection but there are no hard guarantees.+Consequently, in rare cases, it's possible for a document to show up in one search only for it not to appear in a subsequent search occurring immediately after the first search when the second search is routed to a different replica.+Also, documents added in a particular order (even in the same batch) might become searchable out of the order of submission when there is sharding.+The document will become visible on all replicas of a shard after the next `autoCommit` or `commitWithin` interval expires.-To enforce a policy where client applications should not send explicit commits, you should update all client applications that index data into SolrCloud. However, that is not always feasible, so Solr provides the `IgnoreCommitOptimizeUpdateProcessorFactory`, which allows you to ignore explicit commits and/or optimize requests from client applications without having refactor your client application code.+To enforce a policy where client applications should not send explicit commits, you should update all client applications that index data into SolrCloud.+However, that is not always feasible, so Solr provides the `IgnoreCommitOptimizeUpdateProcessorFactory`, which allows you to ignore explicit commits and/or optimize requests from client applications without having refactor your client application code.To activate this request processor you'll need to add the following to your `solrconfig.xml`:@@ -140,7 +188,8 @@ To activate this request processor you'll need to add the following to your `sol</updateRequestProcessorChain>-----As shown in the example above, the processor will return 200 to the client but will ignore the commit or optimize request. Notice that you need to wire-in the implicit processors needed by SolrCloud as well, since this custom chain is taking the place of the default chain.+As shown in the example above, the processor will return 200 to the client but will ignore the commit or optimize request.+Notice that you need to wire-in the implicit processors needed by SolrCloud as well, since this custom chain is taking the place of the default chain.In the following example, the processor will raise an exception with a 403 code with a customized error message:diff --git a/solr/solr-ref-guide/src/solrcloud-with-legacy-configuration-files.adoc b/solr/solr-ref-guide/src/solrcloud-with-legacy-configuration-files.adocindex 1f765e6713b..cc9e8e792db 100644--- a/solr/solr-ref-guide/src/solrcloud-with-legacy-configuration-files.adoc+++ b/solr/solr-ref-guide/src/solrcloud-with-legacy-configuration-files.adoc@@ -18,7 +18,9 @@If you are migrating from a user-managed cluster to SolrCloud, this information may be helpful.-All of the required configuration is already set up in the sample configurations shipped with Solr. You only need to add the following if you are migrating old configuration files. Do not remove these files and parameters from a new Solr instance if you intend to use Solr in SolrCloud mode.+All of the required configuration is already set up in the sample configurations shipped with Solr.+You only need to add the following if you are migrating old configuration files.+Do not remove these files and parameters from a new Solr instance if you intend to use Solr in SolrCloud mode.These properties exist in 3 files: `schema.xml`, `solrconfig.xml`, and `solr.xml`.@@ -29,7 +31,8 @@ These properties exist in 3 files: `schema.xml`, `solrconfig.xml`, and `solr.xml<field name="_version_" type="long" indexed="true" stored="true" multiValued="false"/>----+-. In `solrconfig.xml`, you must have an `UpdateLog` defined. This should be defined in the `updateHandler` section.+. In `solrconfig.xml`, you must have an `UpdateLog` defined.+This should be defined in the `updateHandler` section.+[source,xml]----@@ -42,7 +45,9 @@ These properties exist in 3 files: `schema.xml`, `solrconfig.xml`, and `solr.xml</updateHandler>----+-. The DistributedUpdateProcessor is part of the default update chain and is automatically injected into any of your custom update chains, so you don't actually need to make any changes for this capability. However, should you wish to add it explicitly, you can still add it to the `solrconfig.xml` file as part of an `updateRequestProcessorChain`. For example:+. The DistributedUpdateProcessor is part of the default update chain and is automatically injected into any of your custom update chains, so you don't actually need to make any changes for this capability.+However, should you wish to add it explicitly, you can still add it to the `solrconfig.xml` file as part of an `updateRequestProcessorChain`.+For example:+[source,xml]----diff --git a/solr/solr-ref-guide/src/solrj.adoc b/solr/solr-ref-guide/src/solrj.adocindex 6098a28fb41..b4bd862b958 100644--- a/solr/solr-ref-guide/src/solrj.adoc+++ b/solr/solr-ref-guide/src/solrj.adoc@@ -60,7 +60,8 @@ compile group: 'org.apache.solr', name: 'solr-solrj', version: '{solr-docs-versiIf you are not using one of the above build system, it's still easy to add SolrJ to your build.-At build time, all that is required is the SolrJ jar itself: `solr-solrj-{solr-docs-version}.0.jar`. To compile code manually that uses SolrJ, use a `javac` command similar to:+At build time, all that is required is the SolrJ jar itself: `solr-solrj-{solr-docs-version}.0.jar`.+To compile code manually that uses SolrJ, use a `javac` command similar to:[source,bash,subs="attributes"]----@@ -93,7 +94,8 @@ Requests are sent in the form of {solr-javadocs}/solrj/org/apache/solr/client/so- {solr-javadocs}/solrj/org/apache/solr/client/solrj/impl/HttpSolrClient.html[`HttpSolrClient`] - geared towards query-centric workloads, though also a good general-purpose client.Communicates directly with a single Solr node.-- {solr-javadocs}/solrj/org/apache/solr/client/solrj/impl/Http2SolrClient.html[`Http2SolrClient`] - async, non-blocking and general-purpose client that leverage HTTP/2. This class is experimental therefore its API's might change or be removed in minor versions of SolrJ.+- {solr-javadocs}/solrj/org/apache/solr/client/solrj/impl/Http2SolrClient.html[`Http2SolrClient`] - async, non-blocking and general-purpose client that leverage HTTP/2.+This class is experimental therefore its API's might change or be removed in minor versions of SolrJ.- {solr-javadocs}/solrj/org/apache/solr/client/solrj/impl/LBHttpSolrClient.html[`LBHttpSolrClient`] - balances request load across a list of Solr nodes.Adjusts the list of "in-service" nodes based on node health.- {solr-javadocs}/solrj/org/apache/solr/client/solrj/impl/LBHttp2SolrClient.html[`LBHttp2SolrClient`] - just like `LBHttpSolrClient` but using `Http2SolrClient` instead.@@ -115,9 +117,11 @@ For comprehensive information on how to tweak your `SolrClient`, see the JavadocMost `SolrClient` implementations (except for `CloudSolrClient` and `Http2SolrClient`) require users to specify one or more Solr base URLs, which the client then uses to send HTTP requests to Solr.The path users include on the base URL they provide has an effect on the behavior of the created client from that point on.-. A URL with a path pointing to a specific core or collection (e.g., `\http://hostname:8983/solr/core1`). When a core or collection is specified in the base URL, subsequent requests made with that client are not required to re-specify the affected collection.+. A URL with a path pointing to a specific core or collection (e.g., `\http://hostname:8983/solr/core1`).+When a core or collection is specified in the base URL, subsequent requests made with that client are not required to re-specify the affected collection.However, the client is limited to sending requests to that core/collection, and can not send requests to any others.-. A URL pointing to the root Solr path (e.g., `\http://hostname:8983/solr`). When no core or collection is specified in the base URL, requests can be made to any core/collection, but the affected core/collection must be specified on all requests.+. A URL pointing to the root Solr path (e.g., `\http://hostname:8983/solr`).+When no core or collection is specified in the base URL, requests can be made to any core/collection, but the affected core/collection must be specified on all requests.Generally speaking, if your `SolrClient` will only be used on a single core/collection, including that entity in the path is the most convenient.Where more flexibility is required, the collection/core should be excluded.@@ -131,7 +135,8 @@ If not an `IllegalArgumentException` will be thrown.==== Base URLs of CloudSolrClient-It is also possible to specify base URLs for `CloudSolrClient`, but URLs are expected to point to the root Solr path (e.g., `\http://hostname:8983/solr`). They should not include any collections, cores, or other path components.+It is also possible to specify base URLs for `CloudSolrClient`, but URLs are expected to point to the root Solr path (e.g., `\http://hostname:8983/solr`).+They should not include any collections, cores, or other path components.[source,java,indent=0]----@@ -245,7 +250,8 @@ include::{example-source-dir}UsingSolrJRefGuideExamplesTest.java[tag=solrj-query== Other APIsSolrJ allows more than just querying and indexing.It supports all of Solr's APIs.-Accessing Solr's other APIs is as easy as finding the appropriate request object, providing any necessary parameters, and passing it to the `request()` method of your `SolrClient`. `request()` will return a `NamedList`: a generic object which mirrors the hierarchical structure of the JSON or XML returned by their request.+Accessing Solr's other APIs is as easy as finding the appropriate request object, providing any necessary parameters, and passing it to the `request()` method of your `SolrClient`.+`request()` will return a `NamedList`: a generic object which mirrors the hierarchical structure of the JSON or XML returned by their request.The example below shows how SolrJ users can call the CLUSTERSTATUS API of SolrCloud deployments, and manipulate the returned `NamedList`:diff --git a/solr/solr-ref-guide/src/spatial-search.adoc b/solr/solr-ref-guide/src/spatial-search.adocindex 1a87e1d3e86..7325bc746a1 100644--- a/solr/solr-ref-guide/src/spatial-search.adoc+++ b/solr/solr-ref-guide/src/spatial-search.adoc@@ -50,7 +50,9 @@ Here's how `LatLonPointSpatialField` (LLPSF) should usually be configured in the[source,xml]<fieldType name="location" class="solr.LatLonPointSpatialField" docValues="true"/>-LLPSF supports toggling `indexed`, `stored`, `docValues`, and `multiValued`. LLPSF internally uses a 2-dimensional Lucene "Points" (BDK tree) index when "indexed" is enabled (the default). When "docValues" is enabled, a latitude and longitudes pair are bit-interleaved into 64 bits and put into Lucene DocValues.+LLPSF supports toggling `indexed`, `stored`, `docValues`, and `multiValued`.+LLPSF internally uses a 2-dimensional Lucene "Points" (BDK tree) index when "indexed" is enabled (the default).+When "docValues" is enabled, a latitude and longitudes pair are bit-interleaved into 64 bits and put into Lucene DocValues.The accuracy of the docValues data is about a centimeter.== Indexing Points@@ -79,7 +81,8 @@ The format of the file to pass in.== Searching with Query Parsers-There are two spatial Solr "query parsers" for geospatial search: `geofilt` and `bbox`. They take the following parameters:+There are two spatial Solr "query parsers" for geospatial search: `geofilt` and `bbox`.+They take the following parameters:`d`::The radial distance, usually in kilometers.@@ -96,7 +99,7 @@ A spatial indexed field.(Advanced option; not supported by LatLonType (deprecated) or PointType) If the query is used in a scoring context (e.g., as the main query in `q`), this _<<local-params.adoc#,local param>>_ determines what scores will be produced.Valid values are:-* `none`: A fixed score of 1.0. (the default)+* `none`: A fixed score of `1.0` (the default).* `kilometers`: distance in kilometers between the field value and the specified center point* `miles`: distance in miles between the field value and the specified center point* `degrees`: distance in degrees between the field value and the specified center point@@ -105,7 +108,8 @@ Valid values are:+[WARNING]====-Don't use this for indexed non-point shapes (e.g., polygons). The results will be erroneous.+Don't use this for indexed non-point shapes (e.g., polygons).+The results will be erroneous.And with RPT, it's only recommended for multi-valued point data, as the implementation doesn't scale very well and for single-valued fields, you should instead use a separate non-RPT field purely for distance sorting.====+@@ -116,7 +120,8 @@ When used with `BBoxField`, additional options are supported:* `area2D`: cartesian coordinates based area of the overlapping shapes expressed in terms of the `distanceUnits` configured for this field`filter`::-(Advanced option; not supported by LatLonType (deprecated) or PointType). If you only want the query to score (with the above `score` local parameter), not filter, then set this local parameter to false.+(Advanced option; not supported by LatLonType (deprecated) or PointType).+If you only want the query to score (with the above `score` local parameter), not filter, then set this local parameter to false.=== geofilt@@ -145,7 +150,8 @@ Here's a sample query:&q=*:*&fq={!bbox sfield=store}&pt=45.15,-93.85&d=5The rectangular shape is faster to compute and so it's sometimes used as an alternative to `geofilt` when it's acceptable to return points outside of the radius.-However, if the ideal goal is a circle but you want it to run faster, then instead consider using the RPT field and try a large `distErrPct` value like `0.1` (10% radius). This will return results outside the radius but it will do so somewhat uniformly around the shape.+However, if the ideal goal is a circle but you want it to run faster, then instead consider using the RPT field and try a large `distErrPct` value like `0.1` (10% radius).+This will return results outside the radius but it will do so somewhat uniformly around the shape.image::images/spatial-search/bbox.png[Bounding box]@@ -175,7 +181,9 @@ It's most common to put a spatial query into an "fq" parameter – a filter querBy default, Solr will cache the query in the filter cache.If you know the filter query (be it spatial or not) is fairly unique and not likely to get a cache hit then specify `cache="false"` as a local-param as seen in the following example.-The only spatial types which stand to benefit from this technique are LatLonPointSpatialField and LatLonType (deprecated). Enable docValues on the field (if it isn't already). LatLonType (deprecated) additionally requires a `cost="100"` (or more) local-param.+The only spatial types which stand to benefit from this technique are LatLonPointSpatialField and LatLonType (deprecated).+Enable docValues on the field (if it isn't already).+LatLonType (deprecated) additionally requires a `cost="100"` (or more) local-param.[source,text]&q=...mykeywords...&fq=...someotherfilters...&fq={!geofilt cache=false}&sfield=store&pt=45.15,-93.85&d=5@@ -193,7 +201,8 @@ For more information about these function queries, see the section on <<function=== geodist-`geodist` is a distance function that takes three optional parameters: `(sfield,latitude,longitude)`. You can use the `geodist` function to sort results by distance or score return results.+`geodist` is a distance function that takes three optional parameters: `(sfield,latitude,longitude)`.+You can use the `geodist` function to sort results by distance or score return results.For example, to sort your results by ascending distance, use a request like:@@ -250,7 +259,8 @@ RPT offers several functional improvements over LatLonPointSpatialField:* Ability to index non-point shapes (e.g., polygons) as well as points – see RptWithGeometrySpatialField* Heatmap grid faceting-RPT _shares_ various features in common with `LatLonPointSpatialField`. Some are listed here:+RPT _shares_ various features in common with `LatLonPointSpatialField`.+Some are listed here:* Latitude/Longitude indexed point data; possibly multi-valued* Fast filtering with `geofilt`, `bbox` filters, and range query syntax (dateline crossing is supported)@@ -267,13 +277,15 @@ Use LLPSF for the distance sorting/boosting; it only needs to have docValues for=== Schema Configuration for RPT-To use RPT, the field type must be registered and configured in `schema.xml`. There are many options for this field type.+To use RPT, the field type must be registered and configured in `schema.xml`.+There are many options for this field type.`name`::The name of the field type.`class`::-This should be `solr.SpatialRecursivePrefixTreeFieldType`. But be aware that the Lucene spatial module includes some other so-called "spatial strategies" other than RPT, notably TermQueryPT*, BBox, PointVector*, and SerializedDV.+This should be `solr.SpatialRecursivePrefixTreeFieldType`.+But be aware that the Lucene spatial module includes some other so-called "spatial strategies" other than RPT, notably TermQueryPT*, BBox, PointVector*, and SerializedDV.Solr requires a field type to parallel these in order to use them.The asterisked ones have them.@@ -292,14 +304,17 @@ Spatial4j governs this feature and supports https://locationtech.github.io/spati`distanceUnits`::This is used to specify the units for distance measurements used throughout the use of this field.-This can be `degrees`, `kilometers` or `miles`. It is applied to nearly all distance measurements involving the field: `maxDistErr`, `distErr`, `d`, `geodist` and the `score` when score is `distance`, `area`, or `area2d`. However, it doesn't affect distances embedded in WKT strings, (e.g., `BUFFER(POINT(200 10),0.2)`), which are still in degrees.+This can be `degrees`, `kilometers` or `miles`.+It is applied to nearly all distance measurements involving the field: `maxDistErr`, `distErr`, `d`, `geodist` and the `score` when score is `distance`, `area`, or `area2d`.+However, it doesn't affect distances embedded in WKT strings, (e.g., `BUFFER(POINT(200 10),0.2)`), which are still in degrees.+`distanceUnits` defaults to either `kilometers` if `geo` is `true`, or `degrees` if `geo` is `false`.+`distanceUnits` replaces the `units` attribute; which is now deprecated and mutually exclusive with this attribute.`distErrPct`::-Defines the default precision of non-point shapes (both index & query), as a fraction between `0.0` (fully precise) to `0.5`. The closer this number is to zero, the more accurate the shape will be.+Defines the default precision of non-point shapes (both index & query), as a fraction between `0.0` (fully precise) to `0.5`.+The closer this number is to zero, the more accurate the shape will be.However, more precise indexed shapes use more disk space and take longer to index.+Bigger `distErrPct` values will make queries faster but less accurate.@@ -314,7 +329,8 @@ If left blank, the default is one meter – just a bit less than 0.000009 degreeThis setting is used internally to compute an appropriate maxLevels (see below).`worldBounds`::-Defines the valid numerical ranges for x and y, in the format of `ENVELOPE(minX, maxX, maxY, minY)`. If `geo="true"`, the standard lat-lon world boundaries are assumed.+Defines the valid numerical ranges for x and y, in the format of `ENVELOPE(minX, maxX, maxY, minY)`.+If `geo="true"`, the standard lat-lon world boundaries are assumed.If `geo=false`, you should define your boundaries.`distCalculator`::@@ -326,14 +342,17 @@ Other possible values are `lawOfCosines`, `vincentySphere` and `cartesian^2`.`prefixTree`:: Defines the spatial grid implementation.Since a PrefixTree (such as RecursivePrefixTree) maps the world as a grid, each grid cell is decomposed to another set of grid cells at the next level.+-If `geo=true` then the default prefix tree is `geohash`, otherwise it's `quad`. Geohash has 32 children at each level, quad has 4. Geohash can only be used for `geo=true` as it's strictly geospatial.+If `geo=true` then the default prefix tree is `geohash`, otherwise it's `quad`.+Geohash has 32 children at each level, quad has 4.+Geohash can only be used for `geo=true` as it's strictly geospatial.+A third choice is `packedQuad`, which is generally more efficient than `quad`, provided there are many levels -- perhaps 20 or more.`maxLevels`:: Sets the maximum grid depth for indexed data.Instead, it's usually more intuitive to compute an appropriate maxLevels by specifying `maxDistErr`.-*_And there are others:_* `normWrapLongitude`, `datelineRule`, `validationRule`, `autoIndex`, `allowMultiOverlap`, `precisionModel`. For further info, see notes below about `spatialContextFactory` implementations referenced above, especially the link to the JTS based one.+*_And there are others:_* `normWrapLongitude`, `datelineRule`, `validationRule`, `autoIndex`, `allowMultiOverlap`, `precisionModel`.+For further info, see notes below about `spatialContextFactory` implementations referenced above, especially the link to the JTS based one.=== Standard Shapes@@ -466,7 +485,8 @@ Defaults to being computed via `distErrPct` (or `distErr`).`facet.heatmap.distErrPct`::A fraction of the size of geom used to compute gridLevel.-Defaults to 0.15. It's computed the same as a similarly named parameter for RPT.+Defaults to 0.15.+It's computed the same as a similarly named parameter for RPT.`facet.heatmap.distErr`::A cell error distance used to pick the grid level indirectly.@@ -503,10 +523,12 @@ Note: Don't divide an on-screen projected map rectangle evenly to plot these recThis could be arranged to be the same as an on-screen map but won't necessarily be.The `counts_ints2D` key has a 2D array of integers.-The initial outer level is in row order (top-down), then the inner arrays are the columns (left-right). If any array would be all zeros, a null is returned instead for efficiency reasons.+The initial outer level is in row order (top-down), then the inner arrays are the columns (left-right).+If any array would be all zeros, a null is returned instead for efficiency reasons.The entire value is null if there is no matching spatial data.-If `format=png` then the output key is `counts_png`. It's a base-64 encoded string of a 4-byte PNG.+If `format=png` then the output key is `counts_png`.+It's a base-64 encoded string of a 4-byte PNG.The PNG logically holds exactly the same data that the ints2D format does.Note that the alpha channel byte is flipped to make it easier to view the PNG for diagnostic purposes, since otherwise counts would have to exceed 2^24 before it becomes non-opague.Thus counts greater than this value will become opaque.@@ -551,7 +573,8 @@ Now to sort the results by one of the relevancy modes, use it like this:&q={!field f=bbox score=overlapRatio}Intersects(ENVELOPE(-10, 20, 15, 10))-The `score` local parameter can be one of `overlapRatio`, `area`, and `area2D`. `area` scores by the document area using surface-of-a-sphere (assuming `geo=true`) math, while `area2D` uses simple width * height.+The `score` local parameter can be one of `overlapRatio`, `area`, and `area2D`.+`area` scores by the document area using surface-of-a-sphere (assuming `geo=true`) math, while `area2D` uses simple width * height.`overlapRatio` computes a [0-1] ranged score based on how much overlap exists relative to the document's area and the query area.The javadocs of {lucene-javadocs}/spatial-extras/org/apache/lucene/spatial/bbox/BBoxOverlapRatioValueSource.html[BBoxOverlapRatioValueSource] have more info on the formula.There is an additional parameter `queryTargetProportion` that allows you to weight the query side of the formula to the index (target) side of the formula.diff --git a/solr/solr-ref-guide/src/spell-checking.adoc b/solr/solr-ref-guide/src/spell-checking.adocindex f24eae64b23..1c46664b31e 100644--- a/solr/solr-ref-guide/src/spell-checking.adoc+++ b/solr/solr-ref-guide/src/spell-checking.adoc@@ -24,11 +24,14 @@ The basis for these suggestions can be terms in a field in Solr, externally crea=== Define Spell Check in solrconfig.xml-The first step is to specify the source of terms in `solrconfig.xml`. There are three approaches to spell checking in Solr, discussed below.+The first step is to specify the source of terms in `solrconfig.xml`.+There are three approaches to spell checking in Solr, discussed below.==== IndexBasedSpellChecker-The `IndexBasedSpellChecker` uses a Solr index as the basis for a parallel index used for spell checking. It requires defining a field as the basis for the index terms; a common practice is to copy terms from some fields (such as `title`, `body`, etc.) to another field created for spell checking. Here is a simple example of configuring `solrconfig.xml` with the `IndexBasedSpellChecker`:+The `IndexBasedSpellChecker` uses a Solr index as the basis for a parallel index used for spell checking.+It requires defining a field as the basis for the index terms; a common practice is to copy terms from some fields (such as `title`, `body`, etc.) to another field created for spell checking.+Here is a simple example of configuring `solrconfig.xml` with the `IndexBasedSpellChecker`:[source,xml]----@@ -46,15 +49,22 @@ The `IndexBasedSpellChecker` uses a Solr index as the basis for a parallel index</searchComponent>-----The first element defines the `searchComponent` to use the `solr.SpellCheckComponent`. The `classname` is the specific implementation of the SpellCheckComponent, in this case `solr.IndexBasedSpellChecker`. Defining the `classname` is optional; if not defined, it will default to `IndexBasedSpellChecker`.+The first element defines the `searchComponent` to use the `solr.SpellCheckComponent`.+The `classname` is the specific implementation of the SpellCheckComponent, in this case `solr.IndexBasedSpellChecker`.+Defining the `classname` is optional; if not defined, it will default to `IndexBasedSpellChecker`.-The `spellcheckIndexDir` defines the location of the directory that holds the spellcheck index, while the `field` defines the source field (defined in the Schema) for spell check terms. When choosing a field for the spellcheck index, it's best to avoid a heavily processed field to get more accurate results. If the field has many word variations from processing synonyms and/or stemming, the dictionary will be created with those variations in addition to more valid spelling data.+The `spellcheckIndexDir` defines the location of the directory that holds the spellcheck index, while the `field` defines the source field (defined in the Schema) for spell check terms.+When choosing a field for the spellcheck index, it's best to avoid a heavily processed field to get more accurate results.+If the field has many word variations from processing synonyms and/or stemming, the dictionary will be created with those variations in addition to more valid spelling data.-Finally, _buildOnCommit_ defines whether to build the spell check index at every commit (that is, every time new documents are added to the index). It is optional, and can be omitted if you would rather set it to `false`.+Finally, _buildOnCommit_ defines whether to build the spell check index at every commit (that is, every time new documents are added to the index).+It is optional, and can be omitted if you would rather set it to `false`.==== DirectSolrSpellChecker-The `DirectSolrSpellChecker` uses terms from the Solr index without building a parallel index like the `IndexBasedSpellChecker`. This spell checker has the benefit of not having to be built regularly, meaning that the terms are always up-to-date with terms in the index. Here is how this might be configured in `solrconfig.xml`+The `DirectSolrSpellChecker` uses terms from the Solr index without building a parallel index like the `IndexBasedSpellChecker`.+This spell checker has the benefit of not having to be built regularly, meaning that the terms are always up-to-date with terms in the index.+Here is how this might be configured in `solrconfig.xml`[source,xml]----@@ -76,19 +86,35 @@ The `DirectSolrSpellChecker` uses terms from the Solr index without building a p</searchComponent>-----When choosing a `field` to query for this spell checker, you want one which has relatively little analysis performed on it (particularly analysis such as stemming). Note that you need to specify a field to use for the suggestions, so like the `IndexBasedSpellChecker`, you may want to copy data from fields like `title`, `body`, etc., to a field dedicated to providing spelling suggestions.+When choosing a `field` to query for this spell checker, you want one which has relatively little analysis performed on it (particularly analysis such as stemming).+Note that you need to specify a field to use for the suggestions, so like the `IndexBasedSpellChecker`, you may want to copy data from fields like `title`, `body`, etc., to a field dedicated to providing spelling suggestions.-Many of the parameters relate to how this spell checker should query the index for term suggestions. The `distanceMeasure` defines the metric to use during the spell check query. The value "internal" uses the default Levenshtein metric, which is the same metric used with the other spell checker implementations.+Many of the parameters relate to how this spell checker should query the index for term suggestions.+The `distanceMeasure` defines the metric to use during the spell check query.+The value "internal" uses the default Levenshtein metric, which is the same metric used with the other spell checker implementations.-Because this spell checker is querying the main index, you may want to limit how often it queries the index to be sure to avoid any performance conflicts with user queries. The `accuracy` setting defines the threshold for a valid suggestion, while `maxEdits` defines the number of changes to the term to allow. Since most spelling mistakes are only 1 letter off, setting this to 1 will reduce the number of possible suggestions (the default, however, is 2); the value can only be 1 or 2. `minPrefix` defines the minimum number of characters the terms should share. Setting this to 1 means that the spelling suggestions will all start with the same letter, for example.+Because this spell checker is querying the main index, you may want to limit how often it queries the index to be sure to avoid any performance conflicts with user queries.+The `accuracy` setting defines the threshold for a valid suggestion, while `maxEdits` defines the number of changes to the term to allow.+Since most spelling mistakes are only 1 letter off, setting this to 1 will reduce the number of possible suggestions (the default, however, is 2); the value can only be 1 or 2.+`minPrefix` defines the minimum number of characters the terms should share.+Setting this to 1 means that the spelling suggestions will all start with the same letter, for example.-The `maxInspections` parameter defines the maximum number of possible matches to review before returning results; the default is 5. `minQueryLength` defines how many characters must be in the query before suggestions are provided; the default is 4. `maxQueryLength` enables the spell checker to skip over very long query terms, which can avoid expensive operations or exceptions. There is no limit to term length by default.+The `maxInspections` parameter defines the maximum number of possible matches to review before returning results; the default is 5.+`minQueryLength` defines how many characters must be in the query before suggestions are provided; the default is 4.+`maxQueryLength` enables the spell checker to skip over very long query terms, which can avoid expensive operations or exceptions.+There is no limit to term length by default.-At first, spellchecker analyses incoming query words by looking up them in the index. Only query words, which are absent in index or too rare ones (below `maxQueryFrequency`) are considered as misspelled and used for finding suggestions. Words which are frequent than `maxQueryFrequency` bypass spellchecker unchanged. After suggestions for every misspelled word are found they are filtered for enough frequency with `thresholdTokenFrequency` as boundary value. These parameters (`maxQueryFrequency` and `thresholdTokenFrequency`) can be a percentage (such as .01, or 1%) or an absolute value (such as 4).+At first, spellchecker analyses incoming query words by looking up them in the index.+Only query words, which are absent in index or too rare ones (below `maxQueryFrequency`) are considered as misspelled and used for finding suggestions.+Words which are frequent than `maxQueryFrequency` bypass spellchecker unchanged.+After suggestions for every misspelled word are found they are filtered for enough frequency with `thresholdTokenFrequency` as boundary value.+These parameters (`maxQueryFrequency` and `thresholdTokenFrequency`) can be a percentage (such as .01, or 1%) or an absolute value (such as 4).==== FileBasedSpellChecker-The `FileBasedSpellChecker` uses an external file as a spelling dictionary. This can be useful if using Solr as a spelling server, or if spelling suggestions don't need to be based on actual terms in the index. In `solrconfig.xml`, you would define the searchComponent as so:+The `FileBasedSpellChecker` uses an external file as a spelling dictionary.+This can be useful if using Solr as a spelling server, or if spelling suggestions don't need to be based on actual terms in the index.+In `solrconfig.xml`, you would define the searchComponent as so:[source,xml]----@@ -111,12 +137,17 @@ The differences here are the use of the `sourceLocation` to define the location[TIP]====-In the previous example, _name_ is used to name this specific definition of the spellchecker. Multiple definitions can co-exist in a single `solrconfig.xml`, and the _name_ helps to differentiate them. If only defining one spellchecker, no name is required.+In the previous example, _name_ is used to name this specific definition of the spellchecker.+Multiple definitions can co-exist in a single `solrconfig.xml`, and the _name_ helps to differentiate them.+If only defining one spellchecker, no name is required.======== WordBreakSolrSpellChecker-`WordBreakSolrSpellChecker` offers suggestions by combining adjacent query terms and/or breaking terms into multiple words. It is a `SpellCheckComponent` enhancement, leveraging Lucene's `WordBreakSpellChecker`. It can detect spelling errors resulting from misplaced whitespace without the use of shingle-based dictionaries and provides collation support for word-break errors, including cases where the user has a mix of single-word spelling errors and word-break errors in the same query. It also provides shard support.+`WordBreakSolrSpellChecker` offers suggestions by combining adjacent query terms and/or breaking terms into multiple words.+It is a `SpellCheckComponent` enhancement, leveraging Lucene's `WordBreakSpellChecker`.+It can detect spelling errors resulting from misplaced whitespace without the use of shingle-based dictionaries and provides collation support for word-break errors, including cases where the user has a mix of single-word spelling errors and word-break errors in the same query.+It also provides shard support.Here is how it might be configured in `solrconfig.xml`:@@ -134,9 +165,11 @@ Here is how it might be configured in `solrconfig.xml`:</searchComponent>-----Some of the parameters will be familiar from the discussion of the other spell checkers, such as `name`, `classname`, and `field`. New for this spell checker is `combineWords`, which defines whether words should be combined in a dictionary search (default is true); `breakWords`, which defines if words should be broken during a dictionary search (default is true); and `maxChanges`, an integer which defines how many times the spell checker should check collation possibilities against the index (default is 10).+Some of the parameters will be familiar from the discussion of the other spell checkers, such as `name`, `classname`, and `field`.+New for this spell checker is `combineWords`, which defines whether words should be combined in a dictionary search (default is true); `breakWords`, which defines if words should be broken during a dictionary search (default is true); and `maxChanges`, an integer which defines how many times the spell checker should check collation possibilities against the index (default is 10).-The spellchecker can be configured with a traditional checker (i.e., `DirectSolrSpellChecker`). The results are combined and collations can contain a mix of corrections from both spellcheckers.+The spellchecker can be configured with a traditional checker (i.e., `DirectSolrSpellChecker`).+The results are combined and collations can contain a mix of corrections from both spellcheckers.=== Add It to a Request Handler@@ -147,7 +180,9 @@ Queries will be sent to a <<query-syntax-and-parsers.adoc#,RequestHandler>>. If<str name="spellcheck">true</str>-----One of the possible parameters is the `spellcheck.dictionary` to use, and multiples can be defined. With multiple dictionaries, all specified dictionaries are consulted and results are interleaved. Collations are created with combinations from the different spellcheckers, with care taken that multiple overlapping corrections do not occur in the same collation.+One of the possible parameters is the `spellcheck.dictionary` to use, and multiples can be defined.+With multiple dictionaries, all specified dictionaries are consulted and results are interleaved.+Collations are created with combinations from the different spellcheckers, with care taken that multiple overlapping corrections do not occur in the same collation.Here is an example with multiple dictionaries:@@ -170,80 +205,122 @@ Here is an example with multiple dictionaries:The SpellCheck component accepts the parameters described below.`spellcheck`::-This parameter turns on SpellCheck suggestions for the request. If `true`, then spelling suggestions will be generated. This is required if spell checking is desired.+This parameter turns on SpellCheck suggestions for the request.+If `true`, then spelling suggestions will be generated.+This is required if spell checking is desired.`spellcheck.q` or `q`::This parameter specifies the query to spellcheck.+-If `spellcheck.q` is defined, then it is used; otherwise the original input query is used. The `spellcheck.q` parameter is intended to be the original query, minus any extra markup like field names, boosts, and so on. If the `q` parameter is specified, then the `SpellingQueryConverter` class is used to parse it into tokens; otherwise the <<tokenizers.adoc#white-space-tokenizer,`WhitespaceTokenizer`>> is used.+If `spellcheck.q` is defined, then it is used; otherwise the original input query is used.+The `spellcheck.q` parameter is intended to be the original query, minus any extra markup like field names, boosts, and so on.+If the `q` parameter is specified, then the `SpellingQueryConverter` class is used to parse it into tokens; otherwise the <<tokenizers.adoc#white-space-tokenizer,`WhitespaceTokenizer`>> is used.+-The choice of which one to use is up to the application. Essentially, if you have a spelling "ready" version in your application, then it is probably better to use `spellcheck.q`. Otherwise, if you just want Solr to do the job, use the `q` parameter.+The choice of which one to use is up to the application.+Essentially, if you have a spelling "ready" version in your application, then it is probably better to use `spellcheck.q`.+Otherwise, if you just want Solr to do the job, use the `q` parameter.-NOTE: The `SpellingQueryConverter` class does not deal properly with non-ASCII characters. In this case, you have either to use `spellcheck.q`, or implement your own QueryConverter.+NOTE: The `SpellingQueryConverter` class does not deal properly with non-ASCII characters.+In this case, you have either to use `spellcheck.q`, or implement your own QueryConverter.`spellcheck.build`::-If set to `true`, this parameter creates the dictionary to be used for spell-checking. In a typical search application, you will need to build the dictionary before using the spell check. However, it's not always necessary to build a dictionary first. For example, you can configure the spellchecker to use a dictionary that already exists.+If set to `true`, this parameter creates the dictionary to be used for spell-checking.+In a typical search application, you will need to build the dictionary before using the spell check.+However, it's not always necessary to build a dictionary first.+For example, you can configure the spellchecker to use a dictionary that already exists.+The dictionary will take some time to build, so this parameter should not be sent with every request.`spellcheck.reload`::-If set to `true`, this parameter reloads the spellchecker. The results depend on the implementation of `SolrSpellChecker.reload()`. In a typical implementation, reloading the spellchecker means reloading the dictionary.+If set to `true`, this parameter reloads the spellchecker.+The results depend on the implementation of `SolrSpellChecker.reload()`.+In a typical implementation, reloading the spellchecker means reloading the dictionary.`spellcheck.count`::-This parameter specifies the maximum number of suggestions that the spellchecker should return for a term. If this parameter isn't set, the value defaults to `1`. If the parameter is set but not assigned a number, the value defaults to `5`. If the parameter is set to a positive integer, that number becomes the maximum number of suggestions returned by the spellchecker.+This parameter specifies the maximum number of suggestions that the spellchecker should return for a term.+If this parameter isn't set, the value defaults to `1`.+If the parameter is set but not assigned a number, the value defaults to `5`.+If the parameter is set to a positive integer, that number becomes the maximum number of suggestions returned by the spellchecker.`spellcheck.queryAnalyzerFieldType`::-A field type from Solr's schema. The analyzer configured for the provided field type is used by the QueryConverter to tokenize the value for "q" parameter. The field type specified by this parameter should do minimal transformations. It's usually a best practice to avoid types that aggressively stem or NGram, for instance, since those types of analysis can throw off spell checking.+A field type from Solr's schema.+The analyzer configured for the provided field type is used by the QueryConverter to tokenize the value for "q" parameter.+The field type specified by this parameter should do minimal transformations.+It's usually a best practice to avoid types that aggressively stem or NGram, for instance, since those types of analysis can throw off spell checking.`spellcheck.onlyMorePopular`::-If `true`, Solr will return suggestions that result in more hits for the query than the existing query. Note that this will return more popular suggestions even when the given query term is present in the index and considered "correct".+If `true`, Solr will return suggestions that result in more hits for the query than the existing query.+Note that this will return more popular suggestions even when the given query term is present in the index and considered "correct".`spellcheck.maxResultsForSuggest`::-If, for example, this is set to `5` and the user's query returns 5 or fewer results, the spellchecker will report "correctlySpelled=false" and also offer suggestions (and collations if requested). Setting this greater than zero is useful for creating "did-you-mean?" suggestions for queries that return a low number of hits.+If, for example, this is set to `5` and the user's query returns 5 or fewer results, the spellchecker will report "correctlySpelled=false" and also offer suggestions (and collations if requested).+Setting this greater than zero is useful for creating "did-you-mean?" suggestions for queries that return a low number of hits.`spellcheck.alternativeTermCount`::-Defines the number of suggestions to return for each query term existing in the index and/or dictionary. Presumably, users will want fewer suggestions for words with docFrequency>0. Also, setting this value enables context-sensitive spell suggestions.+Defines the number of suggestions to return for each query term existing in the index and/or dictionary.+Presumably, users will want fewer suggestions for words with docFrequency>0.+Also, setting this value enables context-sensitive spell suggestions.`spellcheck.extendedResults`::-If `true`, this parameter causes to Solr to return additional information about spellcheck results, such as the frequency of each original term in the index (`origFreq`) as well as the frequency of each suggestion in the index (`frequency`). Note that this result format differs from the non-extended one as the returned suggestion for a word is actually an array of lists, where each list holds the suggested term and its frequency.+If `true`, this parameter causes to Solr to return additional information about spellcheck results, such as the frequency of each original term in the index (`origFreq`) as well as the frequency of each suggestion in the index (`frequency`).+Note that this result format differs from the non-extended one as the returned suggestion for a word is actually an array of lists, where each list holds the suggested term and its frequency.`spellcheck.collate`::If `true`, this parameter directs Solr to take the best suggestion for each token (if one exists) and construct a new query from the suggestions.+For example, if the input query was "jawa class lording" and the best suggestion for "jawa" was "java" and "lording" was "loading", then the resulting collation would be "java class loading".+-The `spellcheck.collate` parameter only returns collations that are guaranteed to result in hits if re-queried, even when applying original `fq` parameters. This is especially helpful when there is more than one correction per query.+The `spellcheck.collate` parameter only returns collations that are guaranteed to result in hits if re-queried, even when applying original `fq` parameters.+This is especially helpful when there is more than one correction per query.-NOTE: This only returns a query to be used. It does not actually run the suggested query.+NOTE: This only returns a query to be used.+It does not actually run the suggested query.`spellcheck.maxCollations`::-The maximum number of collations to return. The default is `1`. This parameter is ignored if `spellcheck.collate` is false.+The maximum number of collations to return.+The default is `1`.+This parameter is ignored if `spellcheck.collate` is false.`spellcheck.maxCollationTries`::-This parameter specifies the number of collation possibilities for Solr to try before giving up. Lower values ensure better performance. Higher values may be necessary to find a collation that can return results. The default value is `0`, which is equivalent to not checking collations. This parameter is ignored if `spellcheck.collate` is false.+This parameter specifies the number of collation possibilities for Solr to try before giving up.+Lower values ensure better performance.+Higher values may be necessary to find a collation that can return results.+The default value is `0`, which is equivalent to not checking collations.+This parameter is ignored if `spellcheck.collate` is false.`spellcheck.maxCollationEvaluations`::-This parameter specifies the maximum number of word correction combinations to rank and evaluate prior to deciding which collation candidates to test against the index. This is a performance safety-net in case a user enters a query with many misspelled words. The default is `10000` combinations, which should work well in most situations.+This parameter specifies the maximum number of word correction combinations to rank and evaluate prior to deciding which collation candidates to test against the index.+This is a performance safety-net in case a user enters a query with many misspelled words.+The default is `10000` combinations, which should work well in most situations.`spellcheck.collateExtendedResults`::-If `true`, this parameter returns an expanded response format detailing the collations Solr found. The default value is `false` and this is ignored if `spellcheck.collate` is false.+If `true`, this parameter returns an expanded response format detailing the collations Solr found.+The default value is `false` and this is ignored if `spellcheck.collate` is false.`spellcheck.collateMaxCollectDocs`::-This parameter specifies the maximum number of documents that should be collected when testing potential collations against the index. A value of `0` indicates that all documents should be collected, resulting in exact hit-counts. Otherwise an estimation is provided as a performance optimization in cases where exact hit-counts are unnecessary – the higher the value specified, the more precise the estimation.+This parameter specifies the maximum number of documents that should be collected when testing potential collations against the index.+A value of `0` indicates that all documents should be collected, resulting in exact hit-counts.+Otherwise an estimation is provided as a performance optimization in cases where exact hit-counts are unnecessary – the higher the value specified, the more precise the estimation.+The default value for this parameter is `0`, but when `spellcheck.collateExtendedResults` is false, the optimization is always used as if `1` had been specified.`spellcheck.collateParam.*` Prefix::-This parameter prefix can be used to specify any additional parameters that you wish to the Spellchecker to use when internally validating collation queries. For example, even if your regular search results allow for loose matching of one or more query terms via parameters like `q.op=OR` and `mm=20%` you can specify override parameters such as `spellcheck.collateParam.q.op=AND&spellcheck.collateParam.mm=100%` to require that only collations consisting of words that are all found in at least one document may be returned.+This parameter prefix can be used to specify any additional parameters that you wish to the Spellchecker to use when internally validating collation queries.+For example, even if your regular search results allow for loose matching of one or more query terms via parameters like `q.op=OR` and `mm=20%` you can specify override parameters such as `spellcheck.collateParam.q.op=AND&spellcheck.collateParam.mm=100%` to require that only collations consisting of words that are all found in at least one document may be returned.`spellcheck.dictionary`::-This parameter causes Solr to use the dictionary named in the parameter's argument. The default setting is `default`. This parameter can be used to invoke a specific spellchecker on a per request basis.+This parameter causes Solr to use the dictionary named in the parameter's argument.+The default setting is `default`.+This parameter can be used to invoke a specific spellchecker on a per request basis.`spellcheck.accuracy`::-Specifies an accuracy value to be used by the spell checking implementation to decide whether a result is worthwhile or not. The value is a float between 0 and 1. Defaults to `Float.MIN_VALUE`.+Specifies an accuracy value to be used by the spell checking implementation to decide whether a result is worthwhile or not.+The value is a float between 0 and 1.+Defaults to `Float.MIN_VALUE`.`spellcheck.<DICT_NAME>.key`::-Specifies a key/value pair for the implementation handling a given dictionary. The value that is passed through is just `key=value` (`spellcheck.<DICT_NAME>.` is stripped off).+Specifies a key/value pair for the implementation handling a given dictionary.+The value that is passed through is just `key=value` (`spellcheck.<DICT_NAME>.` is stripped off).+For example, given a dictionary called `foo`, `spellcheck.foo.myKey=myValue` would result in `myKey=myValue` being passed through to the implementation handling the dictionary `foo`.@@ -300,7 +377,8 @@ Results:== Distributed SpellCheck-The `SpellCheckComponent` also supports spellchecking on distributed indexes. If you are using the SpellCheckComponent on a request handler other than "/select", you must provide the following two parameters:+The `SpellCheckComponent` also supports spellchecking on distributed indexes.+If you are using the SpellCheckComponent on a request handler other than "/select", you must provide the following two parameters:`shards`::Specifies the shards in your distributed indexing configuration.@@ -315,4 +393,5 @@ For example:[source,text]http://localhost:8983/solr/techproducts/spell?spellcheck=true&spellcheck.build=true&spellcheck.q=toyata&shards.qt=/spell&shards=solr-shard1:8983/solr/techproducts,solr-shard2:8983/solr/techproducts-In case of a distributed request to the SpellCheckComponent, the shards are requested for at least five suggestions even if the `spellcheck.count` parameter value is less than five. Once the suggestions are collected, they are ranked by the configured distance measure (Levenstein Distance by default) and then by aggregate frequency.+In case of a distributed request to the SpellCheckComponent, the shards are requested for at least five suggestions even if the `spellcheck.count` parameter value is less than five.+Once the suggestions are collected, they are ranked by the configured distance measure (Levenstein Distance by default) and then by aggregate frequency.diff --git a/solr/solr-ref-guide/src/standard-query-parser.adoc b/solr/solr-ref-guide/src/standard-query-parser.adocindex 5e810cad1e8..2d0bcc0d126 100644--- a/solr/solr-ref-guide/src/standard-query-parser.adoc+++ b/solr/solr-ref-guide/src/standard-query-parser.adoc@@ -18,29 +18,36 @@Solr's default Query Parser is also known as the "```lucene```" parser.-The key advantage of the standard query parser is that it supports a robust and fairly intuitive syntax allowing you to create a variety of structured queries. The largest disadvantage is that it's very intolerant of syntax errors, as compared with something like the <<dismax-query-parser.adoc#,DisMax>> query parser which is designed to throw as few errors as possible.+The key advantage of the standard query parser is that it supports a robust and fairly intuitive syntax allowing you to create a variety of structured queries.+The largest disadvantage is that it's very intolerant of syntax errors, as compared with something like the <<dismax-query-parser.adoc#,DisMax>> query parser which is designed to throw as few errors as possible.== Standard Query Parser ParametersIn addition to the <<common-query-parameters.adoc#,Common Query Parameters>>, <<faceting.adoc#,Faceting Parameters>>, <<highlighting.adoc#,Highlighting Parameters>>, and <<morelikethis.adoc#,MoreLikeThis Parameters>>, the standard query parser supports the parameters described in the table below.`q`::-Defines a query using standard query syntax. This parameter is mandatory.+Defines a query using standard query syntax.+This parameter is mandatory.`q.op`::-Specifies the default operator for query expressions. Possible values are "AND" or "OR".+Specifies the default operator for query expressions.+Possible values are "AND" or "OR".`df`::Specifies a default searchable field.`sow`::-Split on whitespace. If set to `true`, text analysis is invoked separately for each individual whitespace-separated term. The default is `false`; whitespace-separated term sequences will be provided to text analysis in one shot, enabling proper function of analysis filters that operate over term sequences, e.g., multi-word synonyms and shingles.+Split on whitespace.+If set to `true`, text analysis is invoked separately for each individual whitespace-separated term.+The default is `false`; whitespace-separated term sequences will be provided to text analysis in one shot, enabling proper function of analysis filters that operate over term sequences, e.g., multi-word synonyms and shingles.Default parameter values are specified in `solrconfig.xml`, or overridden by query-time values in the request.== Standard Query Parser Response-By default, the response from the standard query parser contains one `<result>` block, which is unnamed. If the <<common-query-parameters.adoc#debug-parameter,`debug` parameter>> is used, then an additional `<lst>` block will be returned, using the name "debug". This will contain useful debugging info, including the original query string, the parsed query string, and explain info for each document in the <result> block. If the <<common-query-parameters.adoc#explainother-parameter,`explainOther` parameter>> is also used, then additional explain info will be provided for all the documents matching that query.+By default, the response from the standard query parser contains one `<result>` block, which is unnamed.+If the <<common-query-parameters.adoc#debug-parameter,`debug` parameter>> is used, then an additional `<lst>` block will be returned, using the name "debug". This will contain useful debugging info, including the original query string, the parsed query string, and explain info for each document in the <result> block.+If the <<common-query-parameters.adoc#explainother-parameter,`explainOther` parameter>> is also used, then additional explain info will be provided for all the documents matching that query.=== Sample Responses@@ -94,7 +101,8 @@ Results:== Specifying Terms for the Standard Query Parser-A query to the standard query parser is broken up into terms and operators. There are two types of terms: single terms and phrases.+A query to the standard query parser is broken up into terms and operators.+There are two types of terms: single terms and phrases.* A single term is a single word such as "test" or "hello"* A phrase is a group of words surrounded by double quotes such as "hello dolly"@@ -105,28 +113,39 @@ IMPORTANT: It is important that the analyzer used for queries parses terms and p=== Term Modifiers-Solr supports a variety of term modifiers that add flexibility or precision, as needed, to searches. These modifiers include wildcard characters, characters for making a search "fuzzy" or more general, and so on. The sections below describe these modifiers in detail.+Solr supports a variety of term modifiers that add flexibility or precision, as needed, to searches.+These modifiers include wildcard characters, characters for making a search "fuzzy" or more general, and so on.+The sections below describe these modifiers in detail.=== Wildcard Searches-Solr's standard query parser supports single and multiple character wildcard searches within single terms. Wildcard characters can be applied to single terms, but not to search phrases.+Solr's standard query parser supports single and multiple character wildcard searches within single terms.+Wildcard characters can be applied to single terms, but not to search phrases.[%autowidth.stretch,options="header"]|===|Wildcard Search Type |Special Character |Example|Single character (matches a single character) | `?` |The search string `te?t` would match both test and text.-|Multiple characters (matches zero or more sequential characters) | `*` |The wildcard search: `tes*` would match test, testing, and tester. You can also use wildcard characters in the middle of a term. For example: `te*t` would match test and text. `*est` would match pest and test.+|Multiple characters (matches zero or more sequential characters) | `*` |The wildcard search: `tes*` would match test, testing, and tester.+You can also use wildcard characters in the middle of a term.+For example: `te*t` would match test and text.+`*est` would match pest and test.|====== Fuzzy Searches-Solr's standard query parser supports fuzzy searches based on the Damerau-Levenshtein Distance or Edit Distance algorithm. Fuzzy searches discover terms that are similar to a specified term without necessarily being an exact match. To perform a fuzzy search, use the tilde ~ symbol at the end of a single-word term. For example, to search for a term similar in spelling to "roam," use the fuzzy search:+Solr's standard query parser supports fuzzy searches based on the Damerau-Levenshtein Distance or Edit Distance algorithm.+Fuzzy searches discover terms that are similar to a specified term without necessarily being an exact match.+To perform a fuzzy search, use the tilde ~ symbol at the end of a single-word term.+For example, to search for a term similar in spelling to "roam," use the fuzzy search:`roam~`-This search will match terms like roams, foam, & foams. It will also match the word "roam" itself.+This search will match terms like roams, foam, & foams.+It will also match the word "roam" itself.-An optional distance parameter specifies the maximum number of edits allowed, between 0 and 2, defaulting to 2. For example:+An optional distance parameter specifies the maximum number of edits allowed, between 0 and 2, defaulting to 2.+For example:`roam~1`@@ -138,11 +157,13 @@ IMPORTANT: In many cases, stemming (reducing terms to a common stem) can produceA proximity search looks for terms that are within a specific distance from one another.-To perform a proximity search, add the tilde character ~ and a numeric value to the end of a search phrase. For example, to search for a "apache" and "jakarta" within 10 words of each other in a document, use the search:+To perform a proximity search, add the tilde character ~ and a numeric value to the end of a search phrase.+For example, to search for a "apache" and "jakarta" within 10 words of each other in a document, use the search:`"jakarta apache"~10`-The distance referred to here is the number of term movements needed to match the specified phrase. In the example above, if "apache" and "jakarta" were 10 spaces apart in a field, but "apache" appeared before "jakarta", more than 10 term movements would be required to move the terms together and position "apache" to the right of "jakarta" with a space in between.+The distance referred to here is the number of term movements needed to match the specified phrase.+In the example above, if "apache" and "jakarta" were 10 spaces apart in a field, but "apache" appeared before "jakarta", more than 10 term movements would be required to move the terms together and position "apache" to the right of "jakarta" with a space in between.=== Existence Searches@@ -155,11 +176,16 @@ A field will be considered to "exist" if it has any value, even values which are=== Range Searches-A range search specifies a range of values for a field (a range with an upper bound and a lower bound). The query matches documents whose values for the specified field or fields fall within the range. Range queries can be inclusive or exclusive of the upper and lower bounds. Sorting is done lexicographically, except on numeric fields. For example, the range query below matches all documents whose `popularity` field has a value between 52 and 10,000, inclusive.+A range search specifies a range of values for a field (a range with an upper bound and a lower bound).+The query matches documents whose values for the specified field or fields fall within the range.+Range queries can be inclusive or exclusive of the upper and lower bounds.+Sorting is done lexicographically, except on numeric fields.+For example, the range query below matches all documents whose `popularity` field has a value between 52 and 10,000, inclusive.`popularity:[52 TO 10000]`-Range queries are not limited to date fields or even numerical fields. You could also use range queries with non-date fields:+Range queries are not limited to date fields or even numerical fields.+You could also use range queries with non-date fields:`title:{Aida TO Carmen}`@@ -169,7 +195,8 @@ The brackets around a query determine its inclusiveness.* Square brackets `[` & `]` denote an inclusive range query that matches values including the upper and lower bound.* Curly brackets `{` & `}` denote an exclusive range query that matches values between the upper and lower bounds, but excluding the upper and lower bounds themselves.-* You can mix these types so one end of the range is inclusive and the other is exclusive. Here's an example: `count:{1 TO 10]`+* You can mix these types so one end of the range is inclusive and the other is exclusive.+Here's an example: `count:{1 TO 10]`Wildcards, `*`, can also be used for either or both endpoints to specify an open-ended range query.This is a <<#differences-between-lucenes-classic-query-parser-and-solrs-standard-query-parser,divergence from Lucene's Classic Query Parser>>.@@ -192,24 +219,31 @@ However for float/double types that support `NaN` values, these two queries perf=== Boosting a Term with "^"-Lucene/Solr provides the relevance level of matching documents based on the terms found. To boost a term use the caret symbol `^` with a boost factor (a number) at the end of the term you are searching. The higher the boost factor, the more relevant the term will be.+Lucene/Solr provides the relevance level of matching documents based on the terms found.+To boost a term use the caret symbol `^` with a boost factor (a number) at the end of the term you are searching.+The higher the boost factor, the more relevant the term will be.-Boosting allows you to control the relevance of a document by boosting its term. For example, if you are searching for+Boosting allows you to control the relevance of a document by boosting its term.+For example, if you are searching for-"jakarta apache" and you want the term "jakarta" to be more relevant, you can boost it by adding the ^ symbol along with the boost factor immediately after the term. For example, you could type:+"jakarta apache" and you want the term "jakarta" to be more relevant, you can boost it by adding the ^ symbol along with the boost factor immediately after the term.+For example, you could type:`jakarta^4 apache`-This will make documents with the term jakarta appear more relevant. You can also boost Phrase Terms as in the example:+This will make documents with the term jakarta appear more relevant.+You can also boost Phrase Terms as in the example:`"jakarta apache"^4 "Apache Lucene"`-By default, the boost factor is 1. Although the boost factor must be positive, it can be less than 1 (for example, it could be 0.2).+By default, the boost factor is 1.+Although the boost factor must be positive, it can be less than 1 (for example, it could be 0.2).=== Constant Score with "^="-Constant score queries are created with `<query_clause>^=<score>`, which sets the entire clause to the specified score for any documents matching that clause. This is desirable when you only care about matches for a particular clause and don't want other relevancy factors such as term frequency (the number of times the term appears in the field) or inverse document frequency (a measure across the whole index for how rare a term is in a field).+Constant score queries are created with `<query_clause>^=<score>`, which sets the entire clause to the specified score for any documents matching that clause.+This is desirable when you only care about matches for a particular clause and don't want other relevancy factors such as term frequency (the number of times the term appears in the field) or inverse document frequency (a measure across the whole index for how rare a term is in a field).Example:@@ -228,7 +262,8 @@ Alternatively, you can specify a different field or a combination of fields in aTo specify a field, type the field name followed by a colon ":" and then the term you are searching for within the field.-For example, suppose an index contains two fields, title and text,and that text is the default field. If you want to find a document called "The Right Way" which contains the text "don't go this way," you could include either of the following terms in your search query:+For example, suppose an index contains two fields, title and text,and that text is the default field.+If you want to find a document called "The Right Way" which contains the text "don't go this way," you could include either of the following terms in your search query:`title:"The Right Way" AND text:go`@@ -236,11 +271,13 @@ For example, suppose an index contains two fields, title and text,and that textSince text is the default field, the field indicator is not required; hence the second query above omits it.-The field is only valid for the term that it directly precedes, so the query `title:Do it right` will find only "Do" in the title field. It will find "it" and "right" in the default field (in this case the text field).+The field is only valid for the term that it directly precedes, so the query `title:Do it right` will find only "Do" in the title field.+It will find "it" and "right" in the default field (in this case the text field).== Boolean Operators Supported by the Standard Query Parser-Boolean operators allow you to apply Boolean logic to queries, requiring the presence or absence of specific terms or conditions in fields in order to match documents. The table below summarizes the Boolean operators supported by the standard query parser.+Boolean operators allow you to apply Boolean logic to queries, requiring the presence or absence of specific terms or conditions in fields in order to match documents.+The table below summarizes the Boolean operators supported by the standard query parser.[%autowidth.stretch,options="header"]|===@@ -249,16 +286,23 @@ Boolean operators allow you to apply Boolean logic to queries, requiring the pre|NOT |`!` |Requires that the following term not be present.|OR |`\|\|` |Requires that either term (or both terms) be present for a match.| |`+` |Requires that the following term be present.-| |`-` |Prohibits the following term (that is, matches on fields or documents that do not include that term). The `-` operator is functionally similar to the Boolean operator `!`. Because it's used by popular search engines such as Google, it may be more familiar to some user communities.+| |`-` |Prohibits the following term (that is, matches on fields or documents that do not include that term).+The `-` operator is functionally similar to the Boolean operator `!`. Because it's used by popular search engines such as Google, it may be more familiar to some user communities.|===-Boolean operators allow terms to be combined through logic operators. Lucene supports AND, "`+`", OR, NOT and "`-`" as Boolean operators.+Boolean operators allow terms to be combined through logic operators.+Lucene supports AND, "`+`", OR, NOT and "`-`" as Boolean operators.IMPORTANT: When specifying Boolean operators with keywords such as AND or NOT, the keywords must appear in all uppercase.-NOTE: The standard query parser supports all the Boolean operators listed in the table above. The DisMax query parser supports only `+` and `-`.+NOTE: The standard query parser supports all the Boolean operators listed in the table above.+The DisMax query parser supports only `+` and `-`.-The OR operator is the default conjunction operator. This means that if there is no Boolean operator between two terms, the OR operator is used. The OR operator links two terms and finds a matching document if either of the terms exist in a document. This is equivalent to a union using sets. The symbol || can be used in place of the word OR.+The OR operator is the default conjunction operator.+This means that if there is no Boolean operator between two terms, the OR operator is used.+The OR operator links two terms and finds a matching document if either of the terms exist in a document.+This is equivalent to a union using sets.+The symbol || can be used in place of the word OR.To search for documents that contain either "jakarta apache" or just "jakarta," use the query:@@ -280,7 +324,9 @@ NOTE: This operator is supported by both the standard query parser and the DisMa=== The Boolean Operator AND ("&&")-The AND operator matches documents where both terms exist anywhere in the text of a single document. This is equivalent to an intersection using sets. The symbol `&&` can be used in place of the word AND.+The AND operator matches documents where both terms exist anywhere in the text of a single document.+This is equivalent to an intersection using sets.+The symbol `&&` can be used in place of the word AND.To search for documents that contain "jakarta apache" and "Apache Lucene," use either of the following queries:@@ -291,7 +337,9 @@ To search for documents that contain "jakarta apache" and "Apache Lucene," use e=== The Boolean Operator NOT ("!")-The NOT operator excludes documents that contain the term after NOT. This is equivalent to a difference using sets. The symbol `!` can be used in place of the word NOT.+The NOT operator excludes documents that contain the term after NOT.+This is equivalent to a difference using sets.+The symbol `!` can be used in place of the word NOT.The following queries search for documents that contain the phrase "jakarta apache" but do not contain the phrase "Apache Lucene":@@ -313,7 +361,8 @@ Solr gives the following characters special meaning when they appear in a query:`+` `-` `&&` `||` `!` `(` `)` `{` `}` `[` `]` `^` `"` `~` `*` `?` `:` `/`-To make Solr interpret any of these characters literally, rather as a special character, precede the character with a backslash character `\`. For example, to search for (1+1):2 without having Solr interpret the plus sign and parentheses as special characters for formulating a sub-query with two terms, escape the characters by preceding each one with a backslash:+To make Solr interpret any of these characters literally, rather as a special character, precede the character with a backslash character `\`.+For example, to search for (1+1):2 without having Solr interpret the plus sign and parentheses as special characters for formulating a sub-query with two terms, escape the characters by preceding each one with a backslash:[source,plain]----@@ -322,7 +371,8 @@ To make Solr interpret any of these characters literally, rather as a special ch== Grouping Terms to Form Sub-Queries-Lucene/Solr supports using parentheses to group clauses to form sub-queries. This can be very useful if you want to control the Boolean logic for a query.+Lucene/Solr supports using parentheses to group clauses to form sub-queries.+This can be very useful if you want to control the Boolean logic for a query.The query below searches for either "jakarta" or "apache" and "website":@@ -332,7 +382,8 @@ This adds precision to the query, requiring that the term "website" exist, along=== Grouping Clauses within a Field-To apply two or more Boolean operators to a single field in a search, group the Boolean clauses within parentheses. For example, the query below searches for a title field that contains both the word "return" and the phrase "pink panther":+To apply two or more Boolean operators to a single field in a search, group the Boolean clauses within parentheses.+For example, the query below searches for a title field that contains both the word "return" and the phrase "pink panther":`title:(+return +"pink panther")`@@ -348,7 +399,8 @@ Comments may be nested.== Differences between Lucene's Classic Query Parser and Solr's Standard Query Parser-Solr's standard query parser originated as a variation of Lucene's "classic" QueryParser. It diverges in the following ways:+Solr's standard query parser originated as a variation of Lucene's "classic" QueryParser.+It diverges in the following ways:* A `*` may be used for either or both endpoints to specify an open-ended range query, or by itself as an existence query.** `field:[* TO 100]` finds all field values less than or equal to 100@@ -361,21 +413,25 @@ Solr's standard query parser originated as a variation of Lucene's "classic" Que* Support for embedded Solr queries (sub-queries) using any type of query parser as a nested clause using the local-params syntax.** `inStock:true OR {!dismax qf='name manu' v='ipod'}`+-Gotcha: Be careful not to start your query with `{!` at the very beginning, which changes the parsing of the entire-query string, which may not be what you want if there are additional clauses. So flipping the example above so the-sub-query comes first would fail to work as expected without a leading space.+Gotcha: Be careful not to start your query with `{!` at the very beginning, which changes the parsing of the entire query string, which may not be what you want if there are additional clauses.+So flipping the example above so the sub-query comes first would fail to work as expected without a leading space.+-Sub-queries can also be done with the magic field `\_query_` and for function queries with the magic field `\_val_` but it-should be considered deprecated since it is less clear. Example: `\_val_:"recip(rord(myfield),1,2,3)"`-* Support for a special `filter(...)` syntax to indicate that some query clauses should be cached in the filter cache (as a constant score boolean query). This allows sub-queries to be cached and re-used in other queries. For example `inStock:true` will be cached and re-used in all three of the queries below:+Sub-queries can also be done with the magic field `\_query_` and for function queries with the magic field `\_val_` but it should be considered deprecated since it is less clear.+Example: `\_val_:"recip(rord(myfield),1,2,3)"`+* Support for a special `filter(...)` syntax to indicate that some query clauses should be cached in the filter cache (as a constant score boolean query).+This allows sub-queries to be cached and re-used in other queries.+For example `inStock:true` will be cached and re-used in all three of the queries below:** `q=features:songs OR filter(inStock:true)`** `q=+manu:Apple +filter(inStock:true)`** `q=+manu:Apple & fq=inStock:true`+-This can even be used to cache individual clauses of complex filter queries. In the first query below, 3 items will be added to the filter cache (the top level `fq` and both `filter(...)` clauses) and in the second query, there will be 2 cache hits, and one new cache insertion (for the new top level `fq`):+This can even be used to cache individual clauses of complex filter queries.+In the first query below, 3 items will be added to the filter cache (the top level `fq` and both `filter(...)` clauses) and in the second query, there will be 2 cache hits, and one new cache insertion (for the new top level `fq`):** `q=features:songs & fq=+filter(inStock:true) +filter(price:[* TO 100])`** `q=manu:Apple & fq=-filter(inStock:true) -filter(price:[* TO 100])`-* Range queries ("[a TO z]"), prefix queries ("a*"), and wildcard queries ("a*b") are constant-scoring (all matching documents get an equal score). The scoring factors TF, IDF, index boost, and "coord" are not used. There is no limitation on the number of terms that match (as there was in past versions of Lucene).+* Range queries ("[a TO z]"), prefix queries ("a*"), and wildcard queries ("a*b") are constant-scoring (all matching documents get an equal score).+The scoring factors TF, IDF, index boost, and "coord" are not used.+There is no limitation on the number of terms that match (as there was in past versions of Lucene).* Constant score queries are created with `<query_clause>^=<score>`, which sets the entire clause to the specified score for any documents matching that clause:** `q=(description:blue color:blue)^=1.0 title:blue^=5.0`diff --git a/solr/solr-ref-guide/src/statistics.adoc b/solr/solr-ref-guide/src/statistics.adocindex d391e4b5e77..4423b721052 100644--- a/solr/solr-ref-guide/src/statistics.adoc+++ b/solr/solr-ref-guide/src/statistics.adoc@@ -16,18 +16,15 @@// specific language governing permissions and limitations// under the License.-This section of the user guide covers the core statistical functions-available in math expressions.+This section of the user guide covers the core statistical functions available in math expressions.== Descriptive Statistics-The `describe` function returns descriptive statistics for a-numeric array. The `describe` function returns a single *tuple* with name/value-pairs containing the descriptive statistics.+The `describe` function returns descriptive statistics for a numeric array.+The `describe` function returns a single *tuple* with name/value pairs containing the descriptive statistics.Below is a simple example that selects a random sample of documents from the *logs* collection,-vectorizes the `response_d` field in the result set and uses the `describe` function to-return descriptive statistics about the vector.+vectorizes the `response_d` field in the result set and uses the `describe` function to return descriptive statistics about the vector.[source,text]----@@ -66,11 +63,8 @@ When this expression is sent to the `/stream` handler it responds with:}-----Notice that the random sample contains 50,000 records and the response-time is only 430 milliseconds. Samples of this size can be used to-reliably estimate the statistics for very large underlying-data sets with sub-second performance.-+Notice that the random sample contains 50,000 records and the response time is only 430 milliseconds.+Samples of this size can be used to reliably estimate the statistics for very large underlying data sets with sub-second performance.The `describe` function can also be visualized in a table with Zeppelin-Solr:@@ -79,20 +73,17 @@ image::images/math-expressions/describe.png[]== Histograms and Frequency Tables-Histograms and frequency tables are tools for visualizing the distribution-of a random variable.+Histograms and frequency tables are tools for visualizing the distribution of a random variable.-The `hist` function creates a histogram designed for usage with continuous data. The-`freqTable` function creates a frequency table for use with discrete data.+The `hist` function creates a histogram designed for usage with continuous data.+The `freqTable` function creates a frequency table for use with discrete data.=== histograms-In the example below a histogram is used to visualize a random sample of-response times from the logs collection. The example retrieves the-random sample with the `random` function and creates a vector from the `response_d` field-in the result set. Then the `hist` function is applied to the vector-to return a histogram with 22 bins. The `hist` function returns a-list of tuples with summary statistics for each bin.+In the example below a histogram is used to visualize a random sample of response times from the logs collection.+The example retrieves the random sample with the `random` function and creates a vector from the `response_d` field in the result set.+Then the `hist` function is applied to the vector to return a histogram with 22 bins.+The `hist` function returns a list of tuples with summary statistics for each bin.[source,text]----@@ -149,8 +140,7 @@ With Zeppelin-Solr the histogram can be first visualized as a table:image::images/math-expressions/histtable.png[]-Then the histogram can be visualized with an area chart by plotting the *mean* of-the bins on the *x-axis* and the *prob* (probability) on the *y-axis*:+Then the histogram can be visualized with an area chart by plotting the *mean* of the bins on the *x-axis* and the *prob* (probability) on the *y-axis*:image::images/math-expressions/hist.png[]@@ -160,28 +150,24 @@ image::images/math-expressions/cumProb.png[]=== Custom Histograms-Custom histograms can be defined and visualized by combining the output from multiple-`stats` functions into a single histogram. Instead of automatically binning a numeric-field the custom histogram allows for comparison of bins based on queries.+Custom histograms can be defined and visualized by combining the output from multiple `stats` functions into a single histogram.+Instead of automatically binning a numeric field the custom histogram allows for comparison of bins based on queries.-NOTE: The `stats` function is first discussed in the *Searching, Sampling and Aggregation* section of the-user guide.+NOTE: The `stats` function is first discussed in the *Searching, Sampling and Aggregation* section of the user guide.A simple example will illustrate how to define and visualize a custom histogram.-In this example, three `stats` functions are wrapped in a `plist` function. The-`plist` (parallel list) function executes each of its internal functions in parallel-and concatenates the results into a single stream. `plist` also maintains the order-of the outputs from each of the sub-functions. In this example each `stats` function-computes the count of documents that match a specific query. In this case they count the-number of documents that contain the terms copper, gold and silver. The list of tuples-with the counts is then stored in variable *a*.+In this example, three `stats` functions are wrapped in a `plist` function.+The `plist` (parallel list) function executes each of its internal functions in parallel and concatenates the results into a single stream.+`plist` also maintains the order of the outputs from each of the sub-functions.+In this example each `stats` function computes the count of documents that match a specific query.+In this case they count the number of documents that contain the terms copper, gold and silver.+The list of tuples with the counts is then stored in variable *a*.Then an `array` of labels is created and set to variable *l*.Finally the `zplot` function is used to plot the labels vector and the `count(*)` column.-Notice the `col` function is used inside of the `zplot` function to extract the-counts from the `stats` results.+Notice the `col` function is used inside of the `zplot` function to extract the counts from the `stats` results.image::images/math-expressions/custom-hist.png[]@@ -189,33 +175,25 @@ image::images/math-expressions/custom-hist.png[]=== Frequency TablesThe `freqTable` function returns a frequency distribution for a discrete data set.-The `freqTable` function doesn't create bins like the histogram. Instead it counts-the occurrence of each discrete data value and returns a list of tuples with the-frequency statistics for each value.+The `freqTable` function doesn't create bins like the histogram.+Instead it counts the occurrence of each discrete data value and returns a list of tuples with the frequency statistics for each value.-Below is an example of a frequency table built from a result set-of rounded *differences* in daily opening stock prices for the stock ticker *amzn*.+Below is an example of a frequency table built from a result set of rounded *differences* in daily opening stock prices for the stock ticker *amzn*.-This example is interesting because it shows a multi-step process to arrive-at the result. The first step is to *search* for records in the the *stocks*-collection with a ticker of *amzn*. Notice that the result set is sorted by-date ascending and it returns the `open_d` field which is the opening price for-the day.+This example is interesting because it shows a multi-step process to arrive at the result.+The first step is to *search* for records in the the *stocks* collection with a ticker of *amzn*.+Notice that the result set is sorted by date ascending and it returns the `open_d` field which is the opening price for the day.-The `open_d` field is then vectorized and set to variable *b*, which now contains-a vector of opening prices ordered by date ascending.+The `open_d` field is then vectorized and set to variable *b*, which now contains a vector of opening prices ordered by date ascending.-The `diff` function is then used to calculate the *first difference* for the-vector of opening prices. The first difference simply subtracts the previous value-from each value in the array. This will provide an array of price differences-for each day which will show daily change in opening price.+The `diff` function is then used to calculate the *first difference* for the vector of opening prices.+The first difference simply subtracts the previous value from each value in the array.+This will provide an array of price differences for each day which will show daily change in opening price.-Then the `round` function is used to round the price differences to the nearest-integer to create a vector of discrete values. The `round` function in this-example is effectively *binning* continuous data at integer boundaries.+Then the `round` function is used to round the price differences to the nearest integer to create a vector of discrete values.+The `round` function in this example is effectively *binning* continuous data at integer boundaries.-Finally the `freqTable` function is run on the discrete values to calculate-the frequency table.+Finally the `freqTable` function is run on the discrete values to calculate the frequency table.[source,text]----@@ -266,22 +244,18 @@ With Zeppelin-Solr the frequency table can be first visualized as a table:image::images/math-expressions/freqTable.png[]-The frequency table can then be plotted by switching to a scatter chart and selecting-the *value* column for the *x-axis* and the *count* column for the *y-axis*+The frequency table can then be plotted by switching to a scatter chart and selecting the *value* column for the *x-axis* and the *count* column for the *y-axis*image::images/math-expressions/freqTable1.png[]-Notice that the visualization nicely displays the frequency of daily change in stock prices-rounded to integers. The most frequently occurring value is 0 with 1494 occurrences followed by- -1 and 1 with around 700 occurrences.-+Notice that the visualization nicely displays the frequency of daily change in stock prices rounded to integers.+The most frequently occurring value is 0 with 1494 occurrences followed by -1 and 1 with around 700 occurrences.== Percentiles-The `percentile` function returns the estimated value for a specific percentile in-a sample set. The example below returns a random sample containing the `response_d` field-from the logs collection. The `response_d` field is vectorized and the 20th percentile-is calculated for the vector:+The `percentile` function returns the estimated value for a specific percentile in a sample set.+The example below returns a random sample containing the `response_d` field from the logs collection.+The `response_d` field is vectorized and the 20th percentile is calculated for the vector:[source,text]----@@ -310,8 +284,7 @@ When this expression is sent to the `/stream` handler it responds with:----The `percentile` function can also compute an array of percentile values.-The example below is computing the 20th, 40th, 60th and 80th percentiles for a random sample-of the `response_d` field:+The example below is computing the 20th, 40th, 60th and 80th percentiles for a random sample of the `response_d` field:[source,text]----@@ -348,43 +321,39 @@ When this expression is sent to the `/stream` handler it responds with:Quantile plots or QQ Plots are powerful tools for visually comparing two or more distributions.-A quantile plot, plots the percentiles from two or more distributions in the same visualization. This allows-for visual comparison of the distributions at each percentile. A simple example will help illustrate the power-of quantile plots.+A quantile plot, plots the percentiles from two or more distributions in the same visualization.+This allows for visual comparison of the distributions at each percentile.+A simple example will help illustrate the power of quantile plots.-In this example the distribution of daily stock price changes for two stock tickers, *goog* and-*amzn*, are visualized with a quantile plot.+In this example the distribution of daily stock price changes for two stock tickers, *goog* and *amzn*, are visualized with a quantile plot.-The example first creates an array of values representing the percentiles that will be calculated and sets this array-to variable *p*. Then random samples of the `change_d` field are drawn for the tickers *amzn* and *goog*. The `change_d` field-represents the change in stock price for one day. Then the `change_d` field is vectorized for both samples and placed-in the variables *amzn* and *goog*. The `percentile` function is then used to calculate the percentiles for both vectors. Notice that-the variable *p* is used to specify the list of percentiles that are calculated.+The example first creates an array of values representing the percentiles that will be calculated and sets this array to variable *p*.+Then random samples of the `change_d` field are drawn for the tickers *amzn* and *goog*.+The `change_d` field represents the change in stock price for one day.+Then the `change_d` field is vectorized for both samples and placed in the variables *amzn* and *goog*.+The `percentile` function is then used to calculate the percentiles for both vectors.+Notice that the variable *p* is used to specify the list of percentiles that are calculated.-Finally `zplot` is used to plot the percentiles sequence on the *x-axis* and the calculated-percentile values for both distributions on the *y-axis*. And a line plot is used-to visualize the QQ plot.+Finally `zplot` is used to plot the percentiles sequence on the *x-axis* and the calculated percentile values for both distributions on the *y-axis*.+And a line plot is used to visualize the QQ plot.image::images/math-expressions/quantile-plot.png[]-This quantile plot provides a clear picture of the distributions of daily price changes for *amzn*-and *googl*. In the plot the *x-axis* is the percentiles and the *y-axis* is the percentile value calculated.--Notice that the *goog* percentile value starts lower and ends higher than the *amzn* plot and that there is a-steeper slope. This shows the greater variability in the *goog* price change distribution. The plot gives a clear picture-of the difference-in the distributions across the full range of percentiles.+This quantile plot provides a clear picture of the distributions of daily price changes for *amzn* and *googl*.+In the plot the *x-axis* is the percentiles and the *y-axis* is the percentile value calculated.+Notice that the *goog* percentile value starts lower and ends higher than the *amzn* plot and that there is a steeper slope.+This shows the greater variability in the *goog* price change distribution.+The plot gives a clear picture of the difference in the distributions across the full range of percentiles.== Correlation and Covariance-Correlation and Covariance measure how random variables fluctuate-together.+Correlation and Covariance measure how random variables fluctuate together.=== Correlation and Correlation Matrices-Correlation is a measure of the linear correlation between two vectors. Correlation is scaled between--1 and 1.+Correlation is a measure of the linear correlation between two vectors.+Correlation is scaled between -1 and 1.Three correlation types are supported:@@ -392,56 +361,45 @@ Three correlation types are supported:* *kendalls** *spearmans*-The type of correlation is specified by adding the *type* named parameter in the-function call.+The type of correlation is specified by adding the *type* named parameter in the function call.-In the example below a random sample containing two fields, `filesize_d` and `response_d`, is drawn from-the logs collection using the `random` function. The fields are vectorized into the-variables *x* and *y* and then *Spearman's* correlation for-the two vectors is calculated using the `corr` function.+In the example below a random sample containing two fields, `filesize_d` and `response_d`, is drawn from the logs collection using the `random` function.+The fields are vectorized into the variables *x* and *y* and then *Spearman's* correlation for the two vectors is calculated using the `corr` function.image::images/math-expressions/correlation.png[]==== Correlation Matrices-Correlation matrices are powerful tools for visualizing the correlation between two or more-vectors.+Correlation matrices are powerful tools for visualizing the correlation between two or more vectors.-The `corr` function builds a correlation matrix-if a matrix is passed as the parameter. The correlation matrix is computed by correlating the *columns*-of the matrix.+The `corr` function builds a correlation matrix if a matrix is passed as the parameter.+The correlation matrix is computed by correlating the *columns* of the matrix.The example below demonstrates the power of correlation matrices combined with 2 dimensional faceting.-In this example the `facet2D` function is used to generate a two dimensional facet aggregation-over the fields `complaint_type_s` and `zip_s` from the *nyc311* complaints database.+In this example the `facet2D` function is used to generate a two dimensional facet aggregation over the fields `complaint_type_s` and `zip_s` from the *nyc311* complaints database.The *top 20* complaint types and the *top 25* zip codes for each complaint type are aggregated.-The result is a stream of tuples each containing the fields `complaint_type_s`, `zip_s` and-the count for the pair.+The result is a stream of tuples each containing the fields `complaint_type_s`, `zip_s` and the count for the pair.-The `pivot` function is then used to pivot the fields into a *matrix* with the `zip_s`-field as the *rows* and the `complaint_type_s` field as the *columns*. The `count(*)` field populates-the values in the cells of the matrix.+The `pivot` function is then used to pivot the fields into a *matrix* with the `zip_s` field as the *rows* and the `complaint_type_s` field as the *columns*.+The `count(*)` field populates the values in the cells of the matrix.-The `corr` function is then used correlate the *columns* of the matrix. This produces a correlation matrix-that shows how complaint types are correlated based on the zip codes they appear in. Another way to look at this-is it shows how the different complaint types tend to co-occur across zip codes.+The `corr` function is then used correlate the *columns* of the matrix.+This produces a correlation matrix that shows how complaint types are correlated based on the zip codes they appear in.+Another way to look at this is it shows how the different complaint types tend to co-occur across zip codes.Finally the `zplot` function is used to plot the correlation matrix as a heat map.image::images/math-expressions/corrmatrix.png[]-Notice in the example the correlation matrix is square with complaint types shown on both-the *x* and y-axises. The color of the cells in the heat map shows the-intensity of the correlation between the complaint types.+Notice in the example the correlation matrix is square with complaint types shown on both the *x* and y-axises.+The color of the cells in the heat map shows the intensity of the correlation between the complaint types.-The heat map is interactive, so mousing over one of the cells pops up the values-for the cell.+The heat map is interactive, so mousing over one of the cells pops up the values for the cell.image::images/math-expressions/corrmatrix2.png[]-Notice that HEAT/HOT WATER and UNSANITARY CONDITION complaints have a correlation of 8 (rounded to the nearest-tenth).+Notice that HEAT/HOT WATER and UNSANITARY CONDITION complaints have a correlation of 8 (rounded to the nearesttenth).=== Covariance and Covariance Matrices@@ -449,19 +407,15 @@ Covariance is an unscaled measure of correlation.The `cov` function calculates the covariance of two vectors of data.-In the example below a random sample containing two fields, `filesize_d` and `response_d`, is drawn from-the logs collection using the `random` function. The fields are vectorized into the-variables *x* and *y* and then the covariance for-the two vectors is calculated using the `cov` function.+In the example below a random sample containing two fields, `filesize_d` and `response_d`, is drawn from the logs collection using the `random` function.+The fields are vectorized into the variables *x* and *y* and then the covariance for the two vectors is calculated using the `cov` function.image::images/math-expressions/covariance.png[]-If a matrix is passed to the `cov` function it will automatically compute a covariance-matrix for the *columns* of the matrix.+If a matrix is passed to the `cov` function it will automatically compute a covariance matrix for the *columns* of the matrix.Notice in the example below that the *x* and *y* vectors are added to a matrix.-The matrix is then transposed to turn the rows into columns,-and the covariance matrix is computed for the columns of the matrix.+The matrix is then transposed to turn the rows into columns, and the covariance matrix is computed for the columns of the matrix.[source,text]----@@ -500,8 +454,7 @@ When this expression is sent to the `/stream` handler it responds with:}-----The covariance matrix contains both the variance for the two vectors and the covariance between the vectors-in the following format:+The covariance matrix contains both the variance for the two vectors and the covariance between the vectors in the following format:[source,text]@@ -511,44 +464,32 @@ in the following format:y [80243.3948172242, 1948.3216661122592]-----The covariance matrix is always square. So a covariance matrix created from 3 vectors will produce a 3 x 3 matrix.--+The covariance matrix is always square.+So a covariance matrix created from 3 vectors will produce a 3 x 3 matrix.== Statistical Inference Tests-Statistical inference tests test a hypothesis on *random samples* and return p-values which-can be used to infer the reliability of the test for the entire population.+Statistical inference tests test a hypothesis on *random samples* and return p-values which can be used to infer the reliability of the test for the entire population.The following statistical inference tests are available:-* `anova`: One-Way-Anova tests if there is a statistically significant difference in the-means of two or more random samples.+* `anova`: One-Way-Anova tests if there is a statistically significant difference in the means of two or more random samples.-* `ttest`: The T-test tests if there is a statistically significant difference in the means of two-random samples.+* `ttest`: The T-test tests if there is a statistically significant difference in the means of two random samples.-* `pairedTtest`: The paired t-test tests if there is a statistically significant difference-in the means of two random samples with paired data.+* `pairedTtest`: The paired t-test tests if there is a statistically significant difference in the means of two random samples with paired data.-* `gTestDataSet`: The G-test tests if two samples of binned discrete data were drawn-from the same population.+* `gTestDataSet`: The G-test tests if two samples of binned discrete data were drawn from the same population.-* `chiSquareDataset`: The Chi-Squared test tests if two samples of binned discrete data were-drawn from the same population.+* `chiSquareDataset`: The Chi-Squared test tests if two samples of binned discrete data were drawn from the same population.-* `mannWhitney`: The Mann-Whitney test is a non-parametric test that tests if two-samples of continuous data were pulled-from the same population. The Mann-Whitney test is often used instead of the T-test when the-underlying assumptions of the T-test are not-met.+* `mannWhitney`: The Mann-Whitney test is a non-parametric test that tests if two samples of continuous data were pulled from the same population.+The Mann-Whitney test is often used instead of the T-test when the underlying assumptions of the T-test are not met.-* `ks`: The Kolmogorov-Smirnov test tests if two samples of continuous data were drawn from-the same distribution.+* `ks`: The Kolmogorov-Smirnov test tests if two samples of continuous data were drawn from the same distribution.Below is a simple example of a T-test performed on two random samples.-The returned p-value of .93 means we can accept the null hypothesis-that the two samples do not have statistically significantly differences in the means.+The returned p-value of .93 means we can accept the null hypothesis that the two samples do not have statistically significantly differences in the means.[source,text]----@@ -583,12 +524,10 @@ When this expression is sent to the `/stream` handler it responds with:== Transformations-In statistical analysis its often useful to transform data sets before performing-statistical calculations. The statistical function library includes the following-commonly used transformations:+In statistical analysis its often useful to transform data sets before performing statistical calculations.+The statistical function library includes the following commonly used transformations:-* `rank`: Returns a numeric array with the rank-transformed value of each element of the original-array.+* `rank`: Returns a numeric array with the rank-transformed value of each element of the original array.* `log`: Returns a numeric array with the natural log of each element of the original array.@@ -635,11 +574,9 @@ When this expression is sent to the `/stream` handler it responds with:== Back Transformations-Vectors that have been transformed with the `log`, `log10`, `sqrt` and `cbrt` functions-can be back transformed using the `pow` function.+Vectors that have been transformed with the `log`, `log10`, `sqrt` and `cbrt` functions can be back transformed using the `pow` function.-The example below shows how to back transform data that has been transformed by the-`sqrt` function.+The example below shows how to back transform data that has been transformed by the `sqrt` function.[source,text]@@ -678,8 +615,7 @@ When this expression is sent to the `/stream` handler it responds with:}-----The example below shows how to back transform data that has been transformed by the-`log10` function.+The example below shows how to back transform data that has been transformed by the `log10` function.[source,text]@@ -718,8 +654,7 @@ When this expression is sent to the `/stream` handler it responds with:}-----Vectors that have been transformed with the `recip` function can be back-transformed by taking the reciprocal-of the reciprocal.+Vectors that have been transformed with the `recip` function can be back-transformed by taking the reciprocal of the reciprocal.The example below shows an example of the back-transformation of the `recip` function.@@ -761,12 +696,11 @@ When this expression is sent to the `/stream` handler it responds with:== Z-scores-The `zscores` function converts a numeric array to an array of z-scores. The z-score-is the number of standard deviations a number is from the mean.+The `zscores` function converts a numeric array to an array of z-scores.+The z-score is the number of standard deviations a number is from the mean.The example below computes the z-scores for the values in an array.-[source,text]----let(a=array(1,2,3),diff --git a/solr/solr-ref-guide/src/stats-component.adoc b/solr/solr-ref-guide/src/stats-component.adocindex 4b4ddd251d1..8dea0fd3345 100644--- a/solr/solr-ref-guide/src/stats-component.adoc+++ b/solr/solr-ref-guide/src/stats-component.adoc@@ -116,7 +116,8 @@ The number of documents in the set which do not have a value for this field/funcThis statistic is computed for all field types and is computed by default.`sumOfSquares`::-Sum of all values squared (a by product of computing stddev). This statistic is computed for numeric and date field types and is computed by default.+Sum of all values squared (a by product of computing stddev).+This statistic is computed for numeric and date field types and is computed by default.`mean`::The average `(v1 + v2 .... + vN)/N`.@@ -127,7 +128,8 @@ Standard deviation, measuring how widely spread the values in the data set are.This statistic is computed for numeric and date field types and is computed by default.`percentiles`::-A list of percentile values based on cut-off points specified by the parameter value, such as `1,99,99.9`. These values are an approximation, using the https://github.com/tdunning/t-digest/blob/master/docs/t-digest-paper/histo.pdf[t-digest algorithm]. This statistic is computed for numeric field types and is not computed by default.+A list of percentile values based on cut-off points specified by the parameter value, such as `1,99,99.9`.+These values are an approximation, using the https://github.com/tdunning/t-digest/blob/master/docs/t-digest-paper/histo.pdf[t-digest algorithm]. This statistic is computed for numeric field types and is not computed by default.`distinctValues`::The set of all distinct values for the field/function in all of the documents in the set.diff --git a/solr/solr-ref-guide/src/stream-decorator-reference.adoc b/solr/solr-ref-guide/src/stream-decorator-reference.adocindex 55e67a7d6e0..85ad0c1274f 100644--- a/solr/solr-ref-guide/src/stream-decorator-reference.adoc+++ b/solr/solr-ref-guide/src/stream-decorator-reference.adoc@@ -399,7 +399,8 @@ Each tuple that is classified is assigned two scores:* probability_d*: A float between 0 and 1 which describes the probability that the tuple belongs to the class.This is useful in the classification use case.-* score_d*: The score of the document that has not be squashed between 0 and 1. The score may be positive or negative.+* score_d*: The score of the document that has not be squashed between 0 and 1.+The score may be positive or negative.The higher the score the better the document fits the class.This un-squashed score will be useful in query re-ranking and recommendation use cases.This score is particularly useful when multiple high ranking documents have a probability_d score of 1, which won't provide a meaningful ranking between documents.@@ -1047,11 +1048,12 @@ Because the null stream adds minimal overhead of its own, it can be used to isolIf the /export handlers performance is not the bottleneck, then the bottleneck is likely occurring in the workers where the stream decorators are running.The null expression can be wrapped by the parallel function and sent to worker nodes.-In this scenario each worker will return one tuple with the count of tuples processed on the worker and the timing information for that worker.\This gives valuable information such as:+In this scenario each worker will return one tuple with the count of tuples processed on the worker and the timing information for that worker.+This gives valuable information such as:-1. As more workers are added does the performance of the /export handler improve or not.-2. Are tuples being evenly distributed across the workers, or is the hash partitioning sending more documents to a single worker.-3. Are all workers processing data at the same speed, or is one of the workers the source of the bottleneck.+. As more workers are added does the performance of the /export handler improve or not.+. Are tuples being evenly distributed across the workers, or is the hash partitioning sending more documents to a single worker.+. Are all workers processing data at the same speed, or is one of the workers the source of the bottleneck.=== null Parameters@@ -1347,7 +1349,8 @@ One can provide a list of operations and evaluators to perform on any fields, su* `fieldName as aliasFieldName`: aliased field name to include in the output tuple (can include multiple of these), such as `outputTuple[aliasFieldName] = incomingTuple[fieldName]`* `replace(fieldName, value, withValue=replacementValue)`: if `incomingTuple[fieldName] == value` then `outgoingTuple[fieldName]` will be set to `replacementValue`.`value` can be the string "null" to replace a null value with some other value.-* `replace(fieldName, value, withField=otherFieldName)`: if `incomingTuple[fieldName] == value` then `outgoingTuple[fieldName]` will be set to the value of `incomingTuple[otherFieldName]`. `value` can be the string "null" to replace a null value with some other value.+* `replace(fieldName, value, withField=otherFieldName)`: if `incomingTuple[fieldName] == value` then `outgoingTuple[fieldName]` will be set to the value of `incomingTuple[otherFieldName]`.+`value` can be the string "null" to replace a null value with some other value.=== select Syntaxdiff --git a/solr/solr-ref-guide/src/stream-evaluator-reference.adoc b/solr/solr-ref-guide/src/stream-evaluator-reference.adocindex 7cb6087dbb5..b2a2c68c751 100644--- a/solr/solr-ref-guide/src/stream-evaluator-reference.adoc+++ b/solr/solr-ref-guide/src/stream-evaluator-reference.adoc@@ -621,7 +621,8 @@ number | matrix: Either the distance or distance matrix.== divThe `div` function will take two numeric values and divide them.-The function will fail to execute if any of the values are non-numeric or null, or the 2nd value is 0. Returns a numeric value.+The function will fail to execute if any of the values are non-numeric or null, or the 2nd value is 0.+Returns a numeric value.=== div Parameters@@ -1253,7 +1254,8 @@ If omitted a sequence will be created for the x values.=== loess Named Parameters-* `bandwidth`: (Optional) The percent of the data points to use when drawing the local regression line, defaults to .25. Decreasing the bandwidth increases the number of curves that loess can fit.+* `bandwidth`: (Optional) The percent of the data points to use when drawing the local regression line, defaults to .25.+Decreasing the bandwidth increases the number of curves that loess can fit.* `robustIterations`: (Optional) The number of iterations used to smooth outliers, defaults to 2.=== loess Syntax@@ -1439,7 +1441,8 @@ meanDifference(numericArray, numericArray)== minMaxScaleThe `minMaxScale` function scales numeric arrays within a minimum and maximum value.-By default `minMaxScale` scales between 0 and 1. The `minMaxScale` function can operate on both numeric arrays and matrices.+By default `minMaxScale` scales between 0 and 1.+The `minMaxScale` function can operate on both numeric arrays and matrices.When operating on a matrix the `minMaxScale` function operates on each row of the matrix.diff --git a/solr/solr-ref-guide/src/stream-screen.adoc b/solr/solr-ref-guide/src/stream-screen.adocindex 19d26ebddca..04785106839 100644--- a/solr/solr-ref-guide/src/stream-screen.adoc+++ b/solr/solr-ref-guide/src/stream-screen.adoc@@ -16,11 +16,16 @@// specific language governing permissions and limitations// under the License.-The Stream screen allows you to enter a <<streaming-expressions.adoc#,streaming expression>> and see the results. It is very similar to the <<query-screen.adoc#,Query Screen>>, except the input box is at the top and all options must be declared in the expression.+The Stream screen allows you to enter a <<streaming-expressions.adoc#,streaming expression>> and see the results.+It is very similar to the <<query-screen.adoc#,Query Screen>>, except the input box is at the top and all options must be declared in the expression.-The screen will insert everything up to the streaming expression itself, so you do not need to enter the full URI with the hostname, port, collection, etc. Simply input the expression after the `expr=` part, and the URL will be constructed dynamically as appropriate.+The screen will insert everything up to the streaming expression itself, so you do not need to enter the full URI with the hostname, port, collection, etc.+Simply input the expression after the `expr=` part, and the URL will be constructed dynamically as appropriate.-Under the input box, the Execute button will run the expression. An option "with explanation" will show the parts of the streaming expression that were executed. Under this, the streamed results are shown. A URL to be able to view the output in a browser is also available.+Under the input box, the Execute button will run the expression.+An option "with explanation" will show the parts of the streaming expression that were executed.+Under this, the streamed results are shown.+A URL to be able to view the output in a browser is also available..Stream Screen with query and resultsimage::images/stream-screen/StreamScreen.png[image,height=400]diff --git a/solr/solr-ref-guide/src/stream-source-reference.adoc b/solr/solr-ref-guide/src/stream-source-reference.adocindex 0d690314603..5be176b6225 100644--- a/solr/solr-ref-guide/src/stream-source-reference.adoc+++ b/solr/solr-ref-guide/src/stream-source-reference.adoc@@ -19,9 +19,15 @@== search-The `search` function searches a SolrCloud collection and emits a stream of tuples that match the query. This is very similar to a standard Solr query, and uses many of the same parameters.+The `search` function searches a SolrCloud collection and emits a stream of tuples that match the query.+This is very similar to a standard Solr query, and uses many of the same parameters.-This expression allows you to specify a request hander using the `qt` parameter. By default, the `/select` handler is used. The `/select` handler can be used for simple rapid prototyping of expressions. For production, however, you will most likely want to use the `/export` handler which is designed to `sort` and `export` entire result sets. The `/export` handler is not used by default because it has stricter requirements then the `/select` handler so it's not as easy to get started working with. To read more about the `/export` handler requirements review the section <<exporting-result-sets.adoc#,Exporting Result Sets>>.+This expression allows you to specify a request hander using the `qt` parameter.+By default, the `/select` handler is used.+The `/select` handler can be used for simple rapid prototyping of expressions.+For production, however, you will most likely want to use the `/export` handler which is designed to `sort` and `export` entire result sets.+The `/export` handler is not used by default because it has stricter requirements then the `/select` handler so it's not as easy to get started working with.+To read more about the `/export` handler requirements review the section <<exporting-result-sets.adoc#,Exporting Result Sets>>.=== search Parameters@@ -30,9 +36,14 @@ This expression allows you to specify a request hander using the `qt` parameter.* `fl`: (Mandatory) The list of fields to return.* `sort`: (Mandatory) The sort criteria.* `zkHost`: Only needs to be defined if the collection being searched is found in a different zkHost than the local stream handler.-* `qt`: Specifies the query type, or request handler, to use. Set this to `/export` to work with large result sets. The default is `/select`.-* `rows`: (Mandatory with the `/select` handler) The rows parameter specifies how many rows to return. This parameter is only needed with the `/select` handler (which is the default) since the `/export` handler always returns all rows.-* `partitionKeys`: Comma delimited list of keys to partition the search results by. To be used with the parallel function for parallelizing operations across worker nodes. See the <<stream-decorator-reference.adoc#parallel,parallel>> function for details.+* `qt`: Specifies the query type, or request handler, to use.+Set this to `/export` to work with large result sets.+The default is `/select`.+* `rows`: (Mandatory with the `/select` handler) The rows parameter specifies how many rows to return.+This parameter is only needed with the `/select` handler (which is the default) since the `/export` handler always returns all rows.+* `partitionKeys`: Comma delimited list of keys to partition the search results by.+To be used with the parallel function for parallelizing operations across worker nodes.+See the <<stream-decorator-reference.adoc#parallel,parallel>> function for details.=== search Syntax@@ -48,25 +59,37 @@ expr=search(collection1,== jdbc-The `jdbc` function searches a JDBC datasource and emits a stream of tuples representing the JDBC result set. Each row in the result set is translated into a tuple and each tuple contains all the cell values for that row.+The `jdbc` function searches a JDBC datasource and emits a stream of tuples representing the JDBC result set.+Each row in the result set is translated into a tuple and each tuple contains all the cell values for that row.=== jdbc Parameters* `connection`: (Mandatory) JDBC formatted connection string to whatever driver you are using.* `sql`: (Mandatory) query to pass off to the JDBC endpoint* `sort`: (Mandatory) The sort criteria indicating how the data coming out of the JDBC stream is sorted-* `driver`: The name of the JDBC driver used for the connection. If provided then the driver class will attempt to be loaded into the JVM. If not provided then it is assumed that the driver is already loaded into the JVM. Some drivers require explicit loading so this option is provided.-* `[driverProperty]`: One or more properties to pass to the JDBC driver during connection. The format is `propertyName="propertyValue"`. You can provide as many of these properties as you'd like and they will all be passed to the connection.+* `driver`: The name of the JDBC driver used for the connection.+If provided then the driver class will attempt to be loaded into the JVM.+If not provided then it is assumed that the driver is already loaded into the JVM.+Some drivers require explicit loading so this option is provided.+* `[driverProperty]`: One or more properties to pass to the JDBC driver during connection.+The format is `propertyName="propertyValue"`.+You can provide as many of these properties as you'd like and they will all be passed to the connection.=== Connections and Drivers-Because some JDBC drivers require explicit loading the `driver` parameter can be used to provide the driver class name. If provided, then during stream construction the driver will be loaded. If the driver cannot be loaded because the class is not found on the classpath, then stream construction will fail.+Because some JDBC drivers require explicit loading the `driver` parameter can be used to provide the driver class name.+If provided, then during stream construction the driver will be loaded.+If the driver cannot be loaded because the class is not found on the classpath, then stream construction will fail.-When the JDBC stream is opened it will validate that a driver can be found for the provided connection string. If a driver cannot be found (because it hasn't been loaded) then the open will fail.+When the JDBC stream is opened it will validate that a driver can be found for the provided connection string.+If a driver cannot be found (because it hasn't been loaded) then the open will fail.=== Datatypes-Due to the inherent differences in datatypes across JDBC sources the following datatypes are supported. The table indicates what Java type will be used for a given JDBC type. Types marked as requiring conversion will go through a conversion for each value of that type. For performance reasons the cell data types are only considered when the stream is opened as this is when the converters are created.+Due to the inherent differences in datatypes across JDBC sources the following datatypes are supported.+The table indicates what Java type will be used for a given JDBC type.+Types marked as requiring conversion will go through a conversion for each value of that type.+For performance reasons the cell data types are only considered when the stream is opened as this is when the converters are created.[width="100%",options="header",]|===@@ -124,7 +147,8 @@ merge the aggregates.* `fl`: (Mandatory) The list of fields to return.* `sort`: (Mandatory) The sort criteria.* `expr`: The streaming expression that is sent to the export handler that operates over the sorted-result set. The `input()` function provides the stream of sorted tuples from the export handler (see examples below).+result set.+The `input()` function provides the stream of sorted tuples from the export handler (see examples below).=== drill Syntax@@ -166,22 +190,36 @@ echo("Hello world")== facet-The `facet` function provides aggregations that are rolled up over buckets. Under the covers the facet function pushes down the aggregation into the search engine using Solr's JSON Facet API. This provides sub-second performance for many use cases. The facet function is appropriate for use with a low to moderate number of distinct values in the bucket fields. To support high cardinality aggregations see the rollup function.+The `facet` function provides aggregations that are rolled up over buckets.+Under the covers the facet function pushes down the aggregation into the search engine using Solr's JSON Facet API.+This provides sub-second performance for many use cases.+The facet function is appropriate for use with a low to moderate number of distinct values in the bucket fields.+To support high cardinality aggregations see the rollup function.=== facet Parameters* `collection`: (Mandatory) Collection the facets will be aggregated from.* `q`: (Mandatory) The query to build the aggregations from.-* `buckets`: (Mandatory) Comma separated list of fields to rollup over. The comma separated list represents the dimensions in a multi-dimensional rollup.-* `bucketSorts`: (Mandatory) Comma separated list of sorts to apply to each dimension in the buckets parameters. Sorts can be on the computed metrics or on the bucket values.-* `rows`: (Default 10) The number of rows to return. '-1' will return all rows.+* `buckets`: (Mandatory) Comma separated list of fields to rollup over.+The comma separated list represents the dimensions in a multi-dimensional rollup.+* `bucketSorts`: (Mandatory) Comma separated list of sorts to apply to each dimension in the buckets parameters.+Sorts can be on the computed metrics or on the bucket values.+* `rows`: (Default 10) The number of rows to return.+'-1' will return all rows.* `offset`:(Default 0) The offset in the result set to start from.* `overfetch`: (Default 150) Over-fetching is used to provide accurate aggregations over high cardinality fields.* `method`: The JSON facet API aggregation method.-* `bucketSizeLimit`: Sets the absolute number of rows to fetch. This is incompatible with rows, offset and overfetch. This value is applied to each dimension. '-1' will fetch all the buckets.-* `metrics`: List of metrics to compute for the buckets. Currently supported metrics are `sum(col)`, `avg(col)`, `min(col)`, `max(col)`, `count(*)`, `per(col, 50)`. The `per` metric calculates a percentile+* `bucketSizeLimit`: Sets the absolute number of rows to fetch.+This is incompatible with rows, offset and overfetch.+This value is applied to each dimension.+'-1' will fetch all the buckets.+* `metrics`: List of metrics to compute for the buckets.+Currently supported metrics are `sum(col)`, `avg(col)`, `min(col)`, `max(col)`, `count(*)`, `per(col, 50)`.+The `per` metric calculates a percentilefor a numeric column and can be specified multiple times in the same facet function.-* `tiered`: (Default true) Flag governing whether the `facet` stream should parallelize JSON Facet requests to multiple Solr collections using a `plist` expression; this option only applies if the `collection` is an alias backed by multiple collections. If `tiered` is enabled, then a `rollup` expression is used internally to aggregate the metrics from multiple `facet` expressions into a single result; only `count`, `min`, `max`, `sum`, and `avg` metrics are supported. Client applications can disable this globally by setting the `solr.facet.stream.tiered=false` system property.+* `tiered`: (Default true) Flag governing whether the `facet` stream should parallelize JSON Facet requests to multiple Solr collections using a `plist` expression; this option only applies if the `collection` is an alias backed by multiple collections.+If `tiered` is enabled, then a `rollup` expression is used internally to aggregate the metrics from multiple `facet` expressions into a single result; only `count`, `min`, `max`, `sum`, and `avg` metrics are supported.+Client applications can disable this globally by setting the `solr.facet.stream.tiered=false` system property.=== facet Syntax@@ -237,17 +275,24 @@ The `rows` parameter returns 10 rows and the `offset` parameter starts returning== features-The `features` function extracts the key terms from a text field in a classification training set stored in a SolrCloud collection. It uses an algorithm known as *Information Gain*, to select the important terms from the training set. The `features` function was designed to work specifically with the <<train,train>> function, which uses the extracted features to train a text classifier.+The `features` function extracts the key terms from a text field in a classification training set stored in a SolrCloud collection.+It uses an algorithm known as *Information Gain*, to select the important terms from the training set.+The `features` function was designed to work specifically with the <<train,train>> function, which uses the extracted features to train a text classifier.-The `features` function is designed to work with a training set that provides both positive and negative examples of a class. It emits a tuple for each feature term that is extracted along with the inverse document frequency (IDF) for the term in the training set.+The `features` function is designed to work with a training set that provides both positive and negative examples of a class.+It emits a tuple for each feature term that is extracted along with the inverse document frequency (IDF) for the term in the training set.-The `features` function uses a query to select the training set from a collection. The IDF for each selected feature is calculated relative to the training set matching the query. This allows multiple training sets to be stored in the same SolrCloud collection without polluting the IDF across training sets.+The `features` function uses a query to select the training set from a collection.+The IDF for each selected feature is calculated relative to the training set matching the query.+This allows multiple training sets to be stored in the same SolrCloud collection without polluting the IDF across training sets.=== features Parameters* `collection`: (Mandatory) The collection that holds the training set-* `q`: (Mandatory) The query that defines the training set. The IDF for the features will be generated specific to the result set matching the query.-* `featureSet`: (Mandatory) The name of the feature set. This can be used to retrieve the features if they are stored in a SolrCloud collection.+* `q`: (Mandatory) The query that defines the training set.+The IDF for the features will be generated specific to the result set matching the query.+* `featureSet`: (Mandatory) The name of the feature set.+This can be used to retrieve the features if they are stored in a SolrCloud collection.* `field`: (Mandatory) The text field to extract the features from.* `outcome`: (Mandatory) The field that defines the class, positive or negative* `numTerms`: (Mandatory) How many feature terms to extract.@@ -269,14 +314,20 @@ features(collection1,The `cat` function reads the specified files or directories and emits each line in the file(s) as a tuple.-Each emitted tuple contains two fields: `file` and `line`. `file` contains the path to the file being read from relative to the `userfiles` chroot (directly under `$SOLR_HOME`), and `line` contains a line in that file.+Each emitted tuple contains two fields: `file` and `line`.+`file` contains the path to the file being read from relative to the `userfiles` chroot (directly under `$SOLR_HOME`), and `line` contains a line in that file.`cat` is ideally used with the `update` stream to index data from the specified documents, or with the `analyze` stream to further split the lines into individual tokens for statistical processing or visualization.=== cat Parameters-* `filePaths`: (Mandatory) a comma separated list of filepaths to read lines from. If the specified path is a directory, it will be crawled recursively and all contained files will be read. To prevent malicious users from reading arbitrary files from Solr nodes, `filePaths` must be a relative path measured from a chroot of `$SOLR_HOME/userfiles` on the node running the streaming expression.-* `maxLines`: (defaults to -1) The maximum number of lines to read (and tuples to emit). If a negative value is specified, all lines in the specified files will be emitted as tuples. Files are read in the order that they appear in the comma-separated `filePaths` argument. If the line-limit is hit, it will be these later files that are partially emitted or not read at all.+* `filePaths`: (Mandatory) a comma separated list of filepaths to read lines from.+If the specified path is a directory, it will be crawled recursively and all contained files will be read.+To prevent malicious users from reading arbitrary files from Solr nodes, `filePaths` must be a relative path measured from a chroot of `$SOLR_HOME/userfiles` on the node running the streaming expression.+* `maxLines`: (defaults to -1) The maximum number of lines to read (and tuples to emit).+If a negative value is specified, all lines in the specified files will be emitted as tuples.+Files are read in the order that they appear in the comma-separated `filePaths` argument.+If the line-limit is hit, it will be these later files that are partially emitted or not read at all.=== cat Examples@@ -286,7 +337,8 @@ The following example emits all lines from a single text file located at `$SOLR_cat("authors.txt")-----This example will read lines from `$SOLR_HOME/userfiles/authors.txt`, as well as all files (recursively) found under `$SOLR_HOME/userfiles/fiction/scifi`. Only 500 lines will be emitted, meaning that some files may be partially emitted or not read at all:+This example will read lines from `$SOLR_HOME/userfiles/authors.txt`, as well as all files (recursively) found under `$SOLR_HOME/userfiles/fiction/scifi`.+ Only 500 lines will be emitted, meaning that some files may be partially emitted or not read at all:[source,text]----cat("authors.txt,fiction/scifi/", maxLines=500)@@ -294,11 +346,13 @@ cat("authors.txt,fiction/scifi/", maxLines=500)== nodes-The `nodes` function provides breadth-first graph traversal. For details, see the section <<graph-traversal.adoc#,Graph Traversal>>.+The `nodes` function provides breadth-first graph traversal.+For details, see the section <<graph-traversal.adoc#,Graph Traversal>>.== knnSearch-The `knnSearch` function returns the k-nearest neighbors for a document based on text similarity. Under the covers the `knnSearch` function+The `knnSearch` function returns the k-nearest neighbors for a document based on text similarity.+Under the covers the `knnSearch` functionuses the More Like This query parser plugin.=== knnSearch Parameters@@ -328,28 +382,40 @@ knnSearch(collection1,== model-The `model` function retrieves and caches logistic regression text classification models that are stored in a SolrCloud collection. The `model` function is designed to work with models that are created by the <<train,train function>>, but can also be used to retrieve text classification models trained outside of Solr, as long as they conform to the specified format. After the model is retrieved it can be used by the <<stream-decorator-reference.adoc#classify,classify function>> to classify documents.+The `model` function retrieves and caches logistic regression text classification models that are stored in a SolrCloud collection.+The `model` function is designed to work with models that are created by the <<train,train function>>, but can also be used to retrieve text classification models trained outside of Solr, as long as they conform to the specified format.+After the model is retrieved it can be used by the <<stream-decorator-reference.adoc#classify,classify function>> to classify documents.-A single model tuple is fetched and returned based on the *id* parameter. The model is retrieved by matching the *id* parameter with a model name in the index. If more then one iteration of the named model is stored in the index, the highest iteration is selected.+A single model tuple is fetched and returned based on the *id* parameter.+The model is retrieved by matching the *id* parameter with a model name in the index.+If more then one iteration of the named model is stored in the index, the highest iteration is selected.=== Caching with model-The `model` function has an internal LRU (least-recently-used) cache so models do not have to be retrieved with each invocation of the `model` function. The time to cache for each model ID can be passed as a parameter to the function call. Retrieving a cached model does not reset the time for expiring the model ID in the cache.+The `model` function has an internal LRU (least-recently-used) cache so models do not have to be retrieved with each invocation of the `model` function.+The time to cache for each model ID can be passed as a parameter to the function call.+Retrieving a cached model does not reset the time for expiring the model ID in the cache.=== Model Storage-The storage format of the models in Solr is below. The `train` function outputs the format below so you only need to know schema details if you plan to use the `model` function with logistic regression models trained outside of Solr.+The storage format of the models in Solr is below.+The `train` function outputs the format below so you only need to know schema details if you plan to use the `model` function with logistic regression models trained outside of Solr.* `name_s` (Single value, String, Stored): The name of the model.-* `iteration_i` (Single value, Integer, Stored): The iteration number of the model. Solr can store all iterations of the models generated by the train function.+* `iteration_i` (Single value, Integer, Stored): The iteration number of the model.+Solr can store all iterations of the models generated by the train function.* `terms_ss` (Multi value, String, Stored: The array of terms/features of the model.-* `weights_ds` (Multi value, double, Stored): The array of term weights. Each weight corresponds by array index to a term.-* `idfs_ds` (Multi value, double, Stored): The array of term IDFs (Inverse document frequency). Each IDF corresponds by array index to a term.+* `weights_ds` (Multi value, double, Stored): The array of term weights.+Each weight corresponds by array index to a term.+* `idfs_ds` (Multi value, double, Stored): The array of term IDFs (Inverse document frequency).+Each IDF corresponds by array index to a term.=== model Parameters* `collection`: (Mandatory) The collection where the model is stored.-* `id`: (Mandatory) The id/name of the model. The model function always returns one model. If there are multiple iterations of the name, the highest iteration is returned.+* `id`: (Mandatory) The id/name of the model.+The model function always returns one model.+If there are multiple iterations of the name, the highest iteration is returned.* `cacheMillis`: (Optional) The amount of time to cache the model in the LRU cache.=== model Syntax@@ -363,7 +429,8 @@ model(modelCollection,== random-The `random` function searches a SolrCloud collection and emits a pseudo-random set of results that match the query. Each invocation of random will return a different pseudo-random result set.+The `random` function searches a SolrCloud collection and emits a pseudo-random set of results that match the query.+Each invocation of random will return a different pseudo-random result set.=== random Parameters@@ -383,11 +450,17 @@ random(baskets,fl="basketID")-----In the example above the `random` function is searching the baskets collections for all rows where "productID:productX". It will return 100 pseudo-random results. The field list returned is the basketID.+In the example above the `random` function is searching the baskets collections for all rows where "productID:productX". It will return 100 pseudo-random results.+The field list returned is the basketID.== significantTerms-The `significantTerms` function queries a SolrCloud collection, but instead of returning documents, it returns significant terms found in documents in the result set. The `significantTerms` function scores terms based on how frequently they appear in the result set and how rarely they appear in the entire corpus. The `significantTerms` function emits a tuple for each term which contains the term, the score, the foreground count and the background count. The foreground count is how many documents the term appears in in the result set. The background count is how many documents the term appears in in the entire corpus. The foreground and background counts are global for the collection.+The `significantTerms` function queries a SolrCloud collection, but instead of returning documents, it returns significant terms found in documents in the result set.+The `significantTerms` function scores terms based on how frequently they appear in the result set and how rarely they appear in the entire corpus.+The `significantTerms` function emits a tuple for each term which contains the term, the score, the foreground count and the background count.+The foreground count is how many documents the term appears in in the result set.+The background count is how many documents the term appears in in the entire corpus.+The foreground and background counts are global for the collection.=== significantTerms Parameters@@ -395,8 +468,14 @@ The `significantTerms` function queries a SolrCloud collection, but instead of r* `q`: (Mandatory) The query that describes the foreground document set.* `field`: (Mandatory) The field to extract the terms from.* `limit`: (Optional, Default 20) The max number of terms to return.-* `minDocFreq`: (Optional, Defaults to 5 documents) The minimum number of documents the term must appear in on a shard. This is a float value. If greater then 1.0 then it's considered the absolute number of documents. If less then 1.0 it's treated as a percentage of documents.-* `maxDocFreq`: (Optional, Defaults to 30% of documents) The maximum number of documents the term can appear in on a shard. This is a float value. If greater then 1.0 then it's considered the absolute number of documents. If less then 1.0 it's treated as a percentage of documents.+* `minDocFreq`: (Optional, Defaults to 5 documents) The minimum number of documents the term must appear in on a shard.+This is a float value.+If greater then 1.0 then it's considered the absolute number of documents.+If less then 1.0 it's treated as a percentage of documents.+* `maxDocFreq`: (Optional, Defaults to 30% of documents) The maximum number of documents the term can appear in on a shard.+This is a float value.+If greater then 1.0 then it's considered the absolute number of documents.+If less then 1.0 it's treated as a percentage of documents.* `minTermLength`: (Optional, Default 4) The minimum length of the term to be considered significant.=== significantTerms Syntax@@ -416,14 +495,20 @@ In the example above the `significantTerms` function is querying `collection1` a== shortestPath-The `shortestPath` function is an implementation of a shortest path graph traversal. The `shortestPath` function performs an iterative breadth-first search through an unweighted graph to find the shortest paths between two nodes in a graph. The `shortestPath` function emits a tuple for each path found. Each tuple emitted will contain a `path` key which points to a `List` of nodeIDs comprising the path.+The `shortestPath` function is an implementation of a shortest path graph traversal.+The `shortestPath` function performs an iterative breadth-first search through an unweighted graph to find the shortest paths between two nodes in a graph.+The `shortestPath` function emits a tuple for each path found.+Each tuple emitted will contain a `path` key which points to a `List` of nodeIDs comprising the path.=== shortestPath Parameters* `collection`: (Mandatory) The collection that the topic query will be run on.* `from`: (Mandatory) The nodeID to start the search from* `to`: (Mandatory) The nodeID to end the search at-* `edge`: (Mandatory) Syntax: `from_field=to_field`. The `from_field` defines which field to search from. The `to_field` defines which field to search to. See example below for a detailed explanation.+* `edge`: (Mandatory) Syntax: `from_field=to_field`.+The `from_field` defines which field to search from.+The `to_field` defines which field to search to.+See example below for a detailed explanation.* `threads`: (Optional: Default 6) The number of threads used to perform the partitioned join in the traversal.* `partitionSize`: (Optional: Default 250) The number of nodes in each partition of the join.* `fq`: (Optional) Filter query@@ -445,14 +530,23 @@ shortestPath(collection,The expression above performs a breadth-first search to find the shortest paths in an unweighted, directed graph.-The search starts from the nodeID "\[email protected]" in the `from_address` field and searches for the nodeID "\[email protected]" in the `to_address` field. This search is performed iteratively until the `maxDepth` has been reached. Each level in the traversal is implemented as a parallel partitioned nested loop join across the entire collection. The `threads` parameter controls the number of threads performing the join at each level, while the `partitionSize` parameter controls the of number of nodes in each join partition. The `maxDepth` parameter controls the number of levels to traverse. `fq` is a limiting query applied to each level in the traversal.+The search starts from the nodeID "\[email protected]" in the `from_address` field and searches for the nodeID "\[email protected]" in the `to_address` field.+This search is performed iteratively until the `maxDepth` has been reached.+Each level in the traversal is implemented as a parallel partitioned nested loop join across the entire collection.+The `threads` parameter controls the number of threads performing the join at each level, while the `partitionSize` parameter controls the of number of nodes in each join partition.+The `maxDepth` parameter controls the number of levels to traverse.+`fq` is a limiting query applied to each level in the traversal.== shuffle-The `shuffle` expression sorts and exports entire result sets. The `shuffle` expression is similar to the `search` expression except that-under the covers `shuffle` always uses the /export handler. The `shuffle` expression is designed to be combined with the relational algebra-decorators that require complete, sorted result sets. Shuffled result sets can be partitioned across worker nodes with the parallel-stream decorator to perform parallel relational algebra. When used in parallel mode the partitionKeys parameter must be provided.+The `shuffle` expression sorts and exports entire result sets.+The `shuffle` expression is similar to the `search` expression except that+under the covers `shuffle` always uses the /export handler.+The `shuffle` expression is designed to be combined with the relational algebra+decorators that require complete, sorted result sets.+Shuffled result sets can be partitioned across worker nodes with the parallel+stream decorator to perform parallel relational algebra.+When used in parallel mode the partitionKeys parameter must be provided.=== shuffle Parameters@@ -461,7 +555,9 @@ stream decorator to perform parallel relational algebra. When used in parallel m* `fl`: (Mandatory) The list of fields to return.* `sort`: (Mandatory) The sort criteria.* `zkHost`: Only needs to be defined if the collection being searched is found in a different zkHost than the local stream handler.-* `partitionKeys`: Comma delimited list of keys to partition the search results by. To be used with the parallel function for parallelizing operations across worker nodes. See the <<stream-decorator-reference.adoc#parallel,parallel>> function for details.+* `partitionKeys`: Comma delimited list of keys to partition the search results by.+To be used with the parallel function for parallelizing operations across worker nodes.+See the <<stream-decorator-reference.adoc#parallel,parallel>> function for details.=== shuffle Syntax@@ -475,13 +571,18 @@ shuffle(collection1,== stats-The `stats` function gathers simple aggregations for a search result set. The stats function does not support rollups over buckets, so the stats stream always returns a single tuple with the rolled up stats. Under the covers the stats function pushes down the generation of the stats into the search engine using the StatsComponent. The stats function currently supports the following metrics: `count(*)`, `sum()`, `avg()`, `min()`, and `max()`.+The `stats` function gathers simple aggregations for a search result set.+The stats function does not support rollups over buckets, so the stats stream always returns a single tuple with the rolled up stats.+Under the covers the stats function pushes down the generation of the stats into the search engine using the StatsComponent.+The stats function currently supports the following metrics: `count(*)`, `sum()`, `avg()`, `min()`, and `max()`.=== stats Parameters* `collection`: (Mandatory) Collection the stats will be aggregated from.* `q`: (Mandatory) The query to build the aggregations from.-* `metrics`: (Mandatory) The metrics to include in the result tuple. Current supported metrics are `sum(col)`, `avg(col)`, `min(col)`, `max(col)`, `count(*)`, `per(col, 50)`. The `per` metric calculates a percentile+* `metrics`: (Mandatory) The metrics to include in the result tuple.+Current supported metrics are `sum(col)`, `avg(col)`, `min(col)`, `max(col)`, `count(*)`, `per(col, 50)`.+The `per` metric calculates a percentilefor a numeric column and can be specified multiple times in the same stats function.@@ -506,7 +607,8 @@ stats(collection1,== timeseries-The `timeseries` function builds a time series aggregation. Under the covers the `timeseries` function uses the+The `timeseries` function builds a time series aggregation.+Under the covers the `timeseries` function uses theJSON Facet API as its high performance aggregation engine.=== timeseries Parameters@@ -517,8 +619,11 @@ JSON Facet API as its high performance aggregation engine.* `start`: (Mandatory) The start of the time series expressed in Solr date or date math syntax.* `end`: (Mandatory) The end of the time series expressed in Solr date or date math syntax.* `gap`: (Mandatory) The time gap between time series aggregation points expressed in Solr date math syntax.-* `format`: (Optional) Date template to format the date field in the output tuples. Formatting is performed by Java's SimpleDateFormat class.-* `metrics`: (Mandatory) The metrics to include in the result tuple. Current supported metrics are `sum(col)`, `avg(col)`, `min(col)`, `max(col)`, `count(*)`, `per(col, 50)`. The `per` metric calculates a percentile+* `format`: (Optional) Date template to format the date field in the output tuples.+Formatting is performed by Java's SimpleDateFormat class.+* `metrics`: (Mandatory) The metrics to include in the result tuple.+Current supported metrics are `sum(col)`, `avg(col)`, `min(col)`, `max(col)`, `count(*)`, `per(col, 50)`.+The `per` metric calculates a percentilefor a numeric column and can be specified multiple times in the same timeseries function.@@ -545,17 +650,24 @@ timeseries(collection1,== train-The `train` function trains a Logistic Regression text classifier on a training set stored in a SolrCloud collection. It uses a parallel iterative, batch Gradient Descent approach to train the model. The training algorithm is embedded inside Solr so with each iteration only the model is streamed across the network.+The `train` function trains a Logistic Regression text classifier on a training set stored in a SolrCloud collection.+It uses a parallel iterative, batch Gradient Descent approach to train the model.+The training algorithm is embedded inside Solr so with each iteration only the model is streamed across the network.-The `train` function wraps a <<features,features>> function which provides the terms and inverse document frequency (IDF) used to train the model. The `train` function operates over the same training set as the `features` function, which includes both positive and negative examples of the class.+The `train` function wraps a <<features,features>> function which provides the terms and inverse document frequency (IDF) used to train the model.+The `train` function operates over the same training set as the `features` function, which includes both positive and negative examples of the class.-With each iteration the `train` function emits a tuple with the model. The model contains the feature terms, weights, and the confusion matrix for the model. The optimized model can then be used to classify documents based on their feature terms.+With each iteration the `train` function emits a tuple with the model.+The model contains the feature terms, weights, and the confusion matrix for the model.+The optimized model can then be used to classify documents based on their feature terms.=== train Parameters* `collection`: (Mandatory) Collection that holds the training set-* `q`: (Mandatory) The query that defines the training set. The IDF for the features will be generated on the-* `name`: (Mandatory) The name of model. This can be used to retrieve the model if they stored in a SolrCloud collection.+* `q`: (Mandatory) The query that defines the training set.+The IDF for the features will be generated on the+* `name`: (Mandatory) The name of model.+This can be used to retrieve the model if they stored in a SolrCloud collection.* `field`: (Mandatory) The text field to extract the features from.* `outcome`: (Mandatory) The field that defines the class, positive or negative* `maxIterations`: (Mandatory) How many training iterations to perform.@@ -576,7 +688,13 @@ train(collection1,== topic-The `topic` function provides publish/subscribe messaging capabilities built on top of SolrCloud. The topic function allows users to subscribe to a query. The function then provides one-time delivery of new or updated documents that match the topic query. The initial call to the topic function establishes the checkpoints for the specific topic ID. Subsequent calls to the same topic ID will return documents added or updated after the initial checkpoint. Each run of the topic query updates the checkpoints for the topic ID. Setting the initialCheckpoint parameter to 0 will cause the topic to process all documents in the index that match the topic query.+The `topic` function provides publish/subscribe messaging capabilities built on top of SolrCloud.+The topic function allows users to subscribe to a query.+The function then provides one-time delivery of new or updated documents that match the topic query.+The initial call to the topic function establishes the checkpoints for the specific topic ID.+Subsequent calls to the same topic ID will return documents added or updated after the initial checkpoint.+Each run of the topic query updates the checkpoints for the topic ID.+Setting the initialCheckpoint parameter to 0 will cause the topic to process all documents in the index that match the topic query.[WARNING]====@@ -587,10 +705,13 @@ The topic function should be considered in beta until https://issues.apache.org/* `checkpointCollection`: (Mandatory) The collection where the topic checkpoints are stored.* `collection`: (Mandatory) The collection that the topic query will be run on.-* `id`: (Mandatory) The unique ID for the topic. The checkpoints will be saved under this id.+* `id`: (Mandatory) The unique ID for the topic.+The checkpoints will be saved under this id.* `q`: (Mandatory) The topic query.* `fl`: (Mandatory) The field list returned by the topic function.-* `initialCheckpoint`: (Optional) Sets the initial Solr `\_version_` number to start reading from the queue. If not set, it defaults to the highest version in the index. Setting to 0 will process all records that match query in the index.+* `initialCheckpoint`: (Optional) Sets the initial Solr `\_version_` number to start reading from the queue.+If not set, it defaults to the highest version in the index.+Setting to 0 will process all records that match query in the index.* `zkHost`: (Optional) Only needs to be defined if the collection being searched is found in a different zkHost than the local stream handler.=== topic Syntaxdiff --git a/solr/solr-ref-guide/src/streaming-expressions.adoc b/solr/solr-ref-guide/src/streaming-expressions.adocindex 05d6b1a2335..0d5dc494ff6 100644--- a/solr/solr-ref-guide/src/streaming-expressions.adoc+++ b/solr/solr-ref-guide/src/streaming-expressions.adoc@@ -91,7 +91,8 @@ For the above example the `/stream` handler responded with the following JSON re}-----Note the last tuple in the above example stream is `{"EOF":true,"RESPONSE_TIME":33}`. The `EOF` indicates the end of the stream.+Note the last tuple in the above example stream is `{"EOF":true,"RESPONSE_TIME":33}`.+The `EOF` indicates the end of the stream.To process the JSON response, you'll need to use a streaming JSON implementation because streaming expressions are designed to return the entire result set which may have millions of records.In your JSON client you'll need to iterate each doc (tuple) and check for the EOF tuple to determine the end of stream.diff --git a/solr/solr-ref-guide/src/suggester.adoc b/solr/solr-ref-guide/src/suggester.adocindex 689655b127a..ddcdafbe97e 100644--- a/solr/solr-ref-guide/src/suggester.adoc+++ b/solr/solr-ref-guide/src/suggester.adoc@@ -30,13 +30,16 @@ The main features of this Suggester are:* Term dictionary pluggability, giving you the flexibility to choose the dictionary implementation* Distributed support-The `solrconfig.xml` found in Solr's "```techproducts```" example has a Suggester implementation configured already. For more on search components, see the section <<requesthandlers-searchcomponents.adoc#,Request Handlers and Search Components>>.+The `solrconfig.xml` found in Solr's "```techproducts```" example has a Suggester implementation configured already.+For more on search components, see the section <<requesthandlers-searchcomponents.adoc#,Request Handlers and Search Components>>.-The "```techproducts```" example `solrconfig.xml` has a `suggest` search component and a `/suggest` request handler already configured. You can use that as the basis for your configuration, or create it from scratch, as detailed below.+The "```techproducts```" example `solrconfig.xml` has a `suggest` search component and a `/suggest` request handler already configured.+You can use that as the basis for your configuration, or create it from scratch, as detailed below.== Adding the Suggest Search Component-The first step is to add a search component to `solrconfig.xml` and tell it to use the SuggestComponent. Here is some sample code that could be used.+The first step is to add a search component to `solrconfig.xml` and tell it to use the SuggestComponent.+Here is some sample code that could be used.[source,xml]----@@ -59,26 +62,36 @@ The Suggester search component takes several configuration parameters.The choice of the lookup implementation (`lookupImpl`, how terms are found in the suggestion dictionary) and the dictionary implementation (`dictionaryImpl`, how terms are stored in the suggestion dictionary) will dictate some of the parameters required.-Below are the main parameters that can be used no matter what lookup or dictionary implementation is used. In the following sections additional parameters are provided for each implementation.+Below are the main parameters that can be used no matter what lookup or dictionary implementation is used.+In the following sections additional parameters are provided for each implementation.`searchComponent name`::Arbitrary name for the search component.`name`::-A symbolic name for this suggester. You can refer to this name in the URL parameters and in the SearchHandler configuration. It is possible to have multiples of these in one `solrconfig.xml` file.+A symbolic name for this suggester.+You can refer to this name in the URL parameters and in the SearchHandler configuration.+It is possible to have multiples of these in one `solrconfig.xml` file.`lookupImpl`::-Lookup implementation. There are several possible implementations, described below in the section <<Lookup Implementations>>. If not set, the default lookup is `JaspellLookupFactory`.+Lookup implementation.+There are several possible implementations, described below in the section <<Lookup Implementations>>. If not set, the default lookup is `JaspellLookupFactory`.`dictionaryImpl`::-The dictionary implementation to use. There are several possible implementations, described below in the section <<Dictionary Implementations>>.+The dictionary implementation to use.+There are several possible implementations, described below in the section <<Dictionary Implementations>>.+-If not set, the default dictionary implementation is `HighFrequencyDictionaryFactory`. However, if a `sourceLocation` is used, the dictionary implementation will be `FileDictionaryFactory`.+If not set, the default dictionary implementation is `HighFrequencyDictionaryFactory`.+However, if a `sourceLocation` is used, the dictionary implementation will be `FileDictionaryFactory`.`field`::-A field from the index to use as the basis of suggestion terms. If `sourceLocation` is empty (meaning any dictionary implementation other than `FileDictionaryFactory`), then terms from this field in the index will be used.+A field from the index to use as the basis of suggestion terms.+If `sourceLocation` is empty (meaning any dictionary implementation other than `FileDictionaryFactory`), then terms from this field in the index will be used.+-To be used as the basis for a suggestion, the field must be stored. You may want to <<copy-fields.adoc#,use copyField rules>> to create a special 'suggest' field comprised of terms from other fields in documents. In any event, you very likely want a minimal amount of analysis on the field, so an additional option is to create a field type in your schema that only uses basic tokenizers or filters. One option for such a field type is shown here:+To be used as the basis for a suggestion, the field must be stored.+You may want to <<copy-fields.adoc#,use copyField rules>> to create a special 'suggest' field comprised of terms from other fields in documents.+In any event, you very likely want a minimal amount of analysis on the field, so an additional option is to create a field type in your schema that only uses basic tokenizers or filters.+One option for such a field type is shown here:+[source,xml]----@@ -90,27 +103,35 @@ To be used as the basis for a suggestion, the field must be stored. You may want</fieldType>----+-However, this minimal analysis is not required if you want more analysis to occur on terms. If using the `AnalyzingLookupFactory` as your `lookupImpl`, however, you have the option of defining the field type rules to use for index and query time analysis.+However, this minimal analysis is not required if you want more analysis to occur on terms.+If using the `AnalyzingLookupFactory` as your `lookupImpl`, however, you have the option of defining the field type rules to use for index and query time analysis.`sourceLocation`::-The path to the dictionary file if using the `FileDictionaryFactory`. If this value is empty then the main index will be used as a source of terms and weights.+The path to the dictionary file if using the `FileDictionaryFactory`.+If this value is empty then the main index will be used as a source of terms and weights.`storeDir`::The location to store the dictionary file.`buildOnCommit` and `buildOnOptimize`::-If `true`, the lookup data structure will be rebuilt after soft-commit. If `false`, the default, then the lookup data will be built only when requested by URL parameter `suggest.build=true`. Use `buildOnCommit` to rebuild the dictionary with every soft-commit, or `buildOnOptimize` to build the dictionary only when the index is optimized.+If `true`, the lookup data structure will be rebuilt after soft-commit.+If `false`, the default, then the lookup data will be built only when requested by URL parameter `suggest.build=true`.+Use `buildOnCommit` to rebuild the dictionary with every soft-commit, or `buildOnOptimize` to build the dictionary only when the index is optimized.+-Some lookup implementations may take a long time to build, especially with large indexes. In such cases, using `buildOnCommit` or `buildOnOptimize`, particularly with a high frequency of softCommits is not recommended; it's recommended instead to build the suggester at a lower frequency by manually issuing requests with `suggest.build=true`.+Some lookup implementations may take a long time to build, especially with large indexes.+In such cases, using `buildOnCommit` or `buildOnOptimize`, particularly with a high frequency of softCommits is not recommended; it's recommended instead to build the suggester at a lower frequency by manually issuing requests with `suggest.build=true`.`buildOnStartup`::-If `true,` then the lookup data structure will be built when Solr starts or when the core is reloaded. If this parameter is not specified, the suggester will check if the lookup data structure is present on disk and build it if not found.+If `true,` then the lookup data structure will be built when Solr starts or when the core is reloaded.+If this parameter is not specified, the suggester will check if the lookup data structure is present on disk and build it if not found.+-Enabling this to `true` could lead to the core talking longer to load (or reload) as the suggester data structure needs to be built, which can sometimes take a long time. It’s usually preferred to have this setting set to `false`, the default, and build suggesters manually issuing requests with `suggest.build=true`.+Enabling this to `true` could lead to the core talking longer to load (or reload) as the suggester data structure needs to be built, which can sometimes take a long time.+It’s usually preferred to have this setting set to `false`, the default, and build suggesters manually issuing requests with `suggest.build=true`.=== Lookup Implementations-The `lookupImpl` parameter defines the algorithms used to look up terms in the suggest index. There are several possible implementations to choose from, and some require additional parameters to be configured.+The `lookupImpl` parameter defines the algorithms used to look up terms in the suggest index.+There are several possible implementations to choose from, and some require additional parameters to be configured.==== AnalyzingLookupFactory@@ -125,14 +146,18 @@ The field type to use for the query-time and build-time term suggestion analysisIf `true`, the default, exact suggestions are returned first, even if they are prefixes or other strings in the FST have larger weights.`preserveSep`::-If `true`, the default, then a separator between tokens is preserved. This means that suggestions are sensitive to tokenization (e.g., baseball is different from base ball).+If `true`, the default, then a separator between tokens is preserved.+This means that suggestions are sensitive to tokenization (e.g., baseball is different from base ball).`preservePositionIncrements`::-If `true`, the suggester will preserve position increments. This means that token filters which leave gaps (for example, when StopFilter matches a stopword) the position would be respected when building the suggester. The default is `false`.+If `true`, the suggester will preserve position increments.+This means that token filters which leave gaps (for example, when StopFilter matches a stopword) the position would be respected when building the suggester.+The default is `false`.==== FuzzyLookupFactory-This is a suggester which is an extension of the AnalyzingSuggester but is fuzzy in nature. The similarity is measured by the Levenshtein algorithm.+This is a suggester which is an extension of the AnalyzingSuggester but is fuzzy in nature.+The similarity is measured by the Levenshtein algorithm.This implementation uses the following additional properties:@@ -140,95 +165,127 @@ This implementation uses the following additional properties:If `true`, the default, exact suggestions are returned first, even if they are prefixes or other strings in the FST have larger weights.`preserveSep`::-If `true`, the default, then a separator between tokens is preserved. This means that suggestions are sensitive to tokenization (e.g., baseball is different from base ball).+If `true`, the default, then a separator between tokens is preserved.+This means that suggestions are sensitive to tokenization (e.g., baseball is different from base ball).`maxSurfaceFormsPerAnalyzedForm`::-The maximum number of surface forms to keep for a single analyzed form. When there are too many surface forms we discard the lowest weighted ones.+The maximum number of surface forms to keep for a single analyzed form.+When there are too many surface forms we discard the lowest weighted ones.`maxGraphExpansions`::-When building the FST ("index-time"), we add each path through the tokenstream graph as an individual entry. This places an upper-bound on how many expansions will be added for a single suggestion. The default is `-1` which means there is no limit.+When building the FST ("index-time"), we add each path through the tokenstream graph as an individual entry.+This places an upper-bound on how many expansions will be added for a single suggestion.+The default is `-1` which means there is no limit.`preservePositionIncrements`::-If `true`, the suggester will preserve position increments. This means that token filters which leave gaps (for example, when StopFilter matches a stopword) the position would be respected when building the suggester. The default is `false`.+If `true`, the suggester will preserve position increments.+This means that token filters which leave gaps (for example, when StopFilter matches a stopword) the position would be respected when building the suggester.+The default is `false`.`maxEdits`::-The maximum number of string edits allowed. The system's hard limit is `2`. The default is `1`.+The maximum number of string edits allowed.+The system's hard limit is `2`.+The default is `1`.`transpositions`::If `true`, the default, transpositions should be treated as a primitive edit operation.`nonFuzzyPrefix`::-The length of the common non fuzzy prefix match which must match a suggestion. The default is `1`.+The length of the common non fuzzy prefix match which must match a suggestion.+The default is `1`.`minFuzzyLength`::-The minimum length of query before which any string edits will be allowed. The default is `3`.+The minimum length of query before which any string edits will be allowed.+The default is `3`.`unicodeAware`::-If `true`, the `maxEdits`, `minFuzzyLength`, `transpositions` and `nonFuzzyPrefix` parameters will be measured in unicode code points (actual letters) instead of bytes. The default is `false`.+If `true`, the `maxEdits`, `minFuzzyLength`, `transpositions` and `nonFuzzyPrefix` parameters will be measured in unicode code points (actual letters) instead of bytes.+The default is `false`.==== AnalyzingInfixLookupFactory-Analyzes the input text and then suggests matches based on prefix matches to any tokens in the indexed text. This uses a Lucene index for its dictionary.+Analyzes the input text and then suggests matches based on prefix matches to any tokens in the indexed text.+This uses a Lucene index for its dictionary.This implementation uses the following additional properties.`indexPath`::-When using `AnalyzingInfixSuggester` you can provide your own path where the index will get built. The default is `analyzingInfixSuggesterIndexDir` and will be created in your collection's `data/` directory.+When using `AnalyzingInfixSuggester` you can provide your own path where the index will get built.+The default is `analyzingInfixSuggesterIndexDir` and will be created in your collection's `data/` directory.`minPrefixChars`::-Minimum number of leading characters before PrefixQuery is used (default is `4`). Prefixes shorter than this are indexed as character ngrams (increasing index size but making lookups faster).+Minimum number of leading characters before PrefixQuery is used (default is `4`).+Prefixes shorter than this are indexed as character ngrams (increasing index size but making lookups faster).`allTermsRequired`::-Boolean option for multiple terms. The default is `true`, all terms will be required.+Boolean option for multiple terms.+The default is `true`, all terms will be required.`highlight`::-Highlight suggest terms. Default is `true`.+Highlight suggest terms.+Default is `true`.This implementation supports <<Context Filtering>>.==== BlendedInfixLookupFactory-An extension of the `AnalyzingInfixSuggester` which provides additional functionality to weight prefix matches across the matched documents. It scores higher if a hit is closer to the start of the suggestion.+An extension of the `AnalyzingInfixSuggester` which provides additional functionality to weight prefix matches across the matched documents.+It scores higher if a hit is closer to the start of the suggestion.This implementation uses the following additional properties:`blenderType`::-Used to calculate weight coefficient using the position of the first matching word. Available options are:+Used to calculate weight coefficient using the position of the first matching word.+Available options are:`position_linear`:::-`weightFieldValue * (1 - 0.10*position)`: Matches to the start will be given a higher score. This is the default.+`weightFieldValue * (1 - 0.10*position)`: Matches to the start will be given a higher score.+This is the default.`position_reciprocal`:::-`weightFieldValue / (1 + position)`: Matches to the start will be given a higher score. The score of matches positioned far from the start of the suggestion decays faster than linear.+`weightFieldValue / (1 + position)`: Matches to the start will be given a higher score.+The score of matches positioned far from the start of the suggestion decays faster than linear.`position_exponential_reciprocal`:::-`weightFieldValue / pow(1 + position,exponent)`: Matches to the start will be given a higher score. The score of matches positioned far from the start of the suggestion decays faster than reciprocal.+`weightFieldValue / pow(1 + position,exponent)`: Matches to the start will be given a higher score.+The score of matches positioned far from the start of the suggestion decays faster than reciprocal.`exponent`::::-An optional configuration variable for `position_exponential_reciprocal` to control how fast the score will decrease. Default `2.0`.+An optional configuration variable for `position_exponential_reciprocal` to control how fast the score will decrease.+Default `2.0`.`numFactor`::-The factor to multiply the number of searched elements from which results will be pruned. Default is `10`.+The factor to multiply the number of searched elements from which results will be pruned.+Default is `10`.`indexPath`::-When using `BlendedInfixSuggester` you can provide your own path where the index will get built. The default directory name is `blendedInfixSuggesterIndexDir` and will be created in your collection's data directory.+When using `BlendedInfixSuggester` you can provide your own path where the index will get built.+The default directory name is `blendedInfixSuggesterIndexDir` and will be created in your collection's data directory.`minPrefixChars`::-Minimum number of leading characters before PrefixQuery is used (the default is `4`). Prefixes shorter than this are indexed as character ngrams, which increases index size but makes lookups faster.+Minimum number of leading characters before PrefixQuery is used (the default is `4`).+Prefixes shorter than this are indexed as character ngrams, which increases index size but makes lookups faster.This implementation supports <<Context Filtering>>.==== FreeTextLookupFactory-It looks at the last tokens plus the prefix of whatever final token the user is typing, if present, to predict the most likely next token. The number of previous tokens that need to be considered can also be specified. This suggester would only be used as a fallback, when the primary suggester fails to find any suggestions.+It looks at the last tokens plus the prefix of whatever final token the user is typing, if present, to predict the most likely next token.+The number of previous tokens that need to be considered can also be specified.+This suggester would only be used as a fallback, when the primary suggester fails to find any suggestions.This implementation uses the following additional properties:`suggestFreeTextAnalyzerFieldType`::-The analyzer used at "query-time" and "build-time" to analyze suggestions. This parameter is required.+The analyzer used at "query-time" and "build-time" to analyze suggestions.+This parameter is required.`ngrams`::-The max number of tokens out of which singles will be made the dictionary. The default value is `2`. Increasing this would mean you want more than the previous 2 tokens to be taken into consideration when making the suggestions.+The max number of tokens out of which singles will be made the dictionary.+The default value is `2`.+Increasing this would mean you want more than the previous 2 tokens to be taken into consideration when making the suggestions.==== FSTLookupFactory-An automaton-based lookup. This implementation is slower to build, but provides the lowest memory cost. We recommend using this implementation unless you need more sophisticated matching results, in which case you should use the Jaspell implementation.+An automaton-based lookup.+This implementation is slower to build, but provides the lowest memory cost.+We recommend using this implementation unless you need more sophisticated matching results, in which case you should use the Jaspell implementation.This implementation uses the following additional properties:@@ -244,17 +301,22 @@ A simple compact ternary trie based lookup.==== WFSTLookupFactory-A weighted automaton representation which is an alternative to `FSTLookup` for more fine-grained ranking. `WFSTLookup` does not use buckets, but instead a shortest path algorithm.+A weighted automaton representation which is an alternative to `FSTLookup` for more fine-grained ranking.+`WFSTLookup` does not use buckets, but instead a shortest path algorithm.-Note that it expects weights to be whole numbers. If weight is missing it's assumed to be `1.0`. Weights affect the sorting of matching suggestions when `spellcheck.onlyMorePopular=true` is selected: weights are treated as "popularity" score, with higher weights preferred over suggestions with lower weights.+Note that it expects weights to be whole numbers.+If weight is missing it's assumed to be `1.0`.+Weights affect the sorting of matching suggestions when `spellcheck.onlyMorePopular=true` is selected: weights are treated as "popularity" score, with higher weights preferred over suggestions with lower weights.==== JaspellLookupFactory-A more complex lookup based on a ternary trie from the http://jaspell.sourceforge.net/[JaSpell] project. Use this implementation if you need more sophisticated matching results.+A more complex lookup based on a ternary trie from the http://jaspell.sourceforge.net/[JaSpell] project.+Use this implementation if you need more sophisticated matching results.=== Dictionary Implementations-The dictionary implementations define how terms are stored. There are several options, and multiple dictionaries can be used in a single request if necessary.+The dictionary implementations define how terms are stored.+There are several options, and multiple dictionaries can be used in a single request if necessary.==== DocumentDictionaryFactory@@ -263,13 +325,16 @@ A dictionary with terms, weights, and an optional payload taken from the index.This dictionary implementation takes the following parameters in addition to parameters described for the Suggester generally and for the lookup implementation:`weightField`::-A field that is stored or a numeric DocValue field. This parameter is optional.+A field that is stored or a numeric DocValue field.+This parameter is optional.`payloadField`::-The `payloadField` should be a field that is stored. This parameter is optional.+The `payloadField` should be a field that is stored.+This parameter is optional.`contextField`::-Field to be used for context filtering. Note that only some lookup implementations support filtering.+Field to be used for context filtering.+Note that only some lookup implementations support filtering.==== DocumentExpressionDictionaryFactory@@ -278,13 +343,17 @@ This dictionary implementation is the same as the `DocumentDictionaryFactory` buThis dictionary implementation takes the following parameters in addition to parameters described for the Suggester generally and for the lookup implementation:`payloadField`::-The `payloadField` should be a field that is stored. This parameter is optional.+The `payloadField` should be a field that is stored.+This parameter is optional.`weightExpression`::-An arbitrary expression used for scoring the suggestions. The fields used must be numeric fields. This parameter is required.+An arbitrary expression used for scoring the suggestions.+The fields used must be numeric fields.+This parameter is required.`contextField`::-Field to be used for context filtering. Note that only some lookup implementations support filtering.+Field to be used for context filtering.+Note that only some lookup implementations support filtering.==== HighFrequencyDictionaryFactory@@ -297,14 +366,19 @@ A value between zero and one representing the minimum fraction of the total docu==== FileDictionaryFactory-This dictionary implementation allows using an external file that contains suggest entries. Weights and payloads can also be used.+This dictionary implementation allows using an external file that contains suggest entries.+Weights and payloads can also be used.-If using a dictionary file, it should be a plain text file in UTF-8 encoding. You can use both single terms and phrases in the dictionary file. If adding weights or payloads, those should be separated from terms using the delimiter defined with the `fieldDelimiter` property (the default is '\t', the tab representation). If using payloads, the first line in the file *must* specify a payload.+If using a dictionary file, it should be a plain text file in UTF-8 encoding.+You can use both single terms and phrases in the dictionary file.+If adding weights or payloads, those should be separated from terms using the delimiter defined with the `fieldDelimiter` property (the default is '\t', the tab representation).+If using payloads, the first line in the file *must* specify a payload.This dictionary implementation takes one parameter in addition to parameters described for the Suggester generally and for the lookup implementation:`fieldDelimiter`::-Specifies the delimiter to be used separating the entries, weights and payloads. The default is tab (`\t`).+Specifies the delimiter to be used separating the entries, weights and payloads.+The default is tab (`\t`).+.Example File[source,text]@@ -345,11 +419,15 @@ To do this, simply define separate suggesters, as in this example:</searchComponent>-----When using these Suggesters in a query, you would define multiple `suggest.dictionary` parameters in the request, referring to the names given for each Suggester in the search component definition. The response will include the terms in sections for each Suggester. See the <<Example Usages>> section below for an example request and response.+When using these Suggesters in a query, you would define multiple `suggest.dictionary` parameters in the request, referring to the names given for each Suggester in the search component definition.+The response will include the terms in sections for each Suggester.+See the <<Example Usages>> section below for an example request and response.== Adding the Suggest Request Handler-After adding the search component, a request handler must be added to `solrconfig.xml`. This request handler works the <<requesthandlers-searchcomponents.adoc#,same as any other request handler>>, and allows you to configure default parameters for serving suggestion requests. The request handler definition must incorporate the "suggest" search component defined previously.+After adding the search component, a request handler must be added to `solrconfig.xml`.+This request handler works the <<requesthandlers-searchcomponents.adoc#,same as any other request handler>>, and allows you to configure default parameters for serving suggestion requests.+The request handler definition must incorporate the "suggest" search component defined previously.[source,xml]----@@ -372,7 +450,9 @@ The following parameters allow you to set defaults for the Suggest request handlThis parameter should always be `true`, because we always want to run the Suggester for queries submitted to this handler.`suggest.dictionary`::-The name of the dictionary component configured in the search component. This is a mandatory parameter. It can be set in the request handler, or sent as a parameter at query time.+The name of the dictionary component configured in the search component.+This is a mandatory parameter.+It can be set in the request handler, or sent as a parameter at query time.`suggest.q`::The query to use for suggestion lookups.@@ -384,7 +464,9 @@ Specifies the number of suggestions for Solr to return.A Context Filter Query used to filter suggestions based on the context field, if supported by the suggester.`suggest.build`::-If `true`, it will build the suggester index. This is likely useful only for initial requests; you would probably not want to build the dictionary on every request, particularly in a production system. If you would like to keep your dictionary up to date, you should use the `buildOnCommit` or `buildOnOptimize` parameter for the search component.+If `true`, it will build the suggester index.+This is likely useful only for initial requests; you would probably not want to build the dictionary on every request, particularly in a production system.+If you would like to keep your dictionary up to date, you should use the `buildOnCommit` or `buildOnOptimize` parameter for the search component.`suggest.reload`::If `true`, it will reload the suggester index.@@ -400,7 +482,8 @@ These properties can also be overridden at query time, or not set in the request.Context Filtering[IMPORTANT]====-Context filtering (`suggest.cfq`) is currently only supported by `AnalyzingInfixLookupFactory` and `BlendedInfixLookupFactory`, and only when backed by a `Document*Dictionary`. All other implementations will return unfiltered matches as if filtering was not requested.+Context filtering (`suggest.cfq`) is currently only supported by `AnalyzingInfixLookupFactory` and `BlendedInfixLookupFactory`, and only when backed by a `Document*Dictionary`.+All other implementations will return unfiltered matches as if filtering was not requested.====== Example Usages@@ -508,9 +591,11 @@ Example response:=== Context Filtering-Context filtering lets you filter suggestions by a separate context field, such as category, department or any other token. The `AnalyzingInfixLookupFactory` and `BlendedInfixLookupFactory` currently support this feature, when backed by `DocumentDictionaryFactory`.+Context filtering lets you filter suggestions by a separate context field, such as category, department or any other token.+The `AnalyzingInfixLookupFactory` and `BlendedInfixLookupFactory` currently support this feature, when backed by `DocumentDictionaryFactory`.-Add `contextField` to your suggester configuration. This example will suggest names and allow to filter by category:+Add `contextField` to your suggester configuration.+This example will suggest names and allow to filter by category:.solrconfig.xml[source,xml]diff --git a/solr/solr-ref-guide/src/system-requirements.adoc b/solr/solr-ref-guide/src/system-requirements.adocindex 00b524ec41a..f0c494e571b 100644--- a/solr/solr-ref-guide/src/system-requirements.adoc+++ b/solr/solr-ref-guide/src/system-requirements.adoc@@ -39,7 +39,8 @@ Java HotSpot(TM) 64-Bit Server VM (build 25.60-b23, mixed mode)----The exact output will vary, but you need to make sure you meet the minimum version requirement.-We also recommend choosing a version that is not end-of-life from its vendor. Oracle/OpenJDK are the most tested JREs and are preferred.+We also recommend choosing a version that is not end-of-life from its vendor.+Oracle/OpenJDK are the most tested JREs and are preferred.It's also preferred to use the latest available official release.Some versions of Java VM have bugs that may impact your implementation.@@ -49,7 +50,8 @@ To be sure, check the page https://cwiki.apache.org/confluence/display/LUCENE/JaJava is available from a number of providers.Lucene and Solr regularly test with https://jdk.java.net/[OpenJDK] and Oracle versions of Java.-Some are free, others have a cost, some provide security patches and support, others do not. We recommend you read the article https://medium.com/@javachampions/java-is-still-free-2-0-0-6b9aa8d6d244[Java is still free by Java Champions] to help you decide.+Some are free, others have a cost, some provide security patches and support, others do not.+We recommend you read the article https://medium.com/@javachampions/java-is-still-free-2-0-0-6b9aa8d6d244[Java is still free by Java Champions] to help you decide.The Lucene project does not endorse any particular provider of Java.@@ -78,7 +80,7 @@ In addition, some organizations also maintain their own test infrastructure andOur continuous testing is against the two code lines under active development, Solr 8x and the future Solr 9.0:-* Lucene/Solr 8.x is the current stable release line and will have "point releases", i.e., 8.1, 8.2, etc. until Lucene/Solr 9.0 is released.+* Lucene/Solr 8.x is the current stable release line and will have "point releases", i.e., 8.1, 8.2, etc., until Lucene/Solr 9.0 is released.** Solr 8.x is currently tested against Java 8, 9, 10, 11, 12 and (pre-release) 13.* There is also development and testing with the future Lucene/Solr 9.x release line, which will require Java 11 as a minimum version.This line is currently tested against Java 11, 12 and (pre-release) 13.@@ -89,7 +91,8 @@ The success rate in our automated tests is similar with all the Java versions te==== Lucene/Solr Prior to 7.0-* Lucene/Solr 7.0 was the first version that successfully passed our tests using Java 9 and higher. You should avoid Java 9 or later for Lucene/Solr 6.x or earlier.+* Lucene/Solr 7.0 was the first version that successfully passed our tests using Java 9 and higher.+You should avoid Java 9 or later for Lucene/Solr 6.x or earlier.==== Lucene/Solr 7.xdiff --git a/solr/solr-ref-guide/src/tagger-handler.adoc b/solr/solr-ref-guide/src/tagger-handler.adocindex 3606c3baf94..b739e79dd71 100644--- a/solr/solr-ref-guide/src/tagger-handler.adoc+++ b/solr/solr-ref-guide/src/tagger-handler.adoc@@ -63,7 +63,8 @@ For configuration examples, jump to the <<tutorial-with-geonames,tutorial>> belo== Tagger Parameters-The tagger's execution is completely configurable with request parameters. Only `field` is required.+The tagger's execution is completely configurable with request parameters.+ Only `field` is required.`field`::The tag field that serves as the dictionary.@@ -126,13 +127,15 @@ Solr's parameters for controlling the response format are also supported, such a== Tutorial with GeonamesThis is a tutorial that demonstrates how to configure and use the text-tagger with the popular http://www.geonames.org/[Geonames] data set. It's more than a tutorial;+tagger with the popular http://www.geonames.org/[Geonames] data set.+It's more than a tutorial;it's a how-to with information that wasn't described above.=== Create and Configure a Solr CollectionCreate a Solr collection named "geonames". For the tutorial, we'll-assume the default "data-driven" configuration. It's good for+assume the default "data-driven" configuration.+It's good forexperimentation and getting going fast but not for production or beingoptimal.@@ -141,15 +144,19 @@ bin/solr create -c geonames==== Configuring the Tagger-We need to configure the schema first. The "data driven" mode we're+We need to configure the schema first.+The "data driven" mode we'reusing allows us to keep this step fairly minimal -- we just need todeclare a field type, 2 fields, and a copy-field.The critical part-up-front is to define the "tag" field type. There are many many ways to+up-front is to define the "tag" field type.+There are many many ways toconfigure text analysis; and we're not going to get into those choices-here. But an important bit is the `ConcatenateGraphFilterFactory` at the-end of the index analyzer chain. Another important bit for performance+here.+But an important bit is the `ConcatenateGraphFilterFactory` at the+end of the index analyzer chain.+Another important bit for performanceis `postingsFormat=FST50` resulting in a compact FST based in-memory datastructure that is especially beneficial for the text tagger.@@ -207,8 +214,10 @@ curl -X POST -H 'Content-type:application/json' http://localhost:8983/solr/geona[[tagger-load-some-sample-data]]=== Load Some Sample Data-We'll go with some Geonames.org data in CSV format. Solr is quite-flexible in loading data in a variety of formats. This+We'll go with some Geonames.org data in CSV format.+Solr is quite+flexible in loading data in a variety of formats.+Thishttp://download.geonames.org/export/dump/cities1000.zip[cities1000.zip]should be almost 7MB file expanding to a cities1000.txt file around22.2MB containing 145k lines, each a city in the world of at least 1000@@ -230,17 +239,20 @@ curl -X POST --data-binary @/path/to/cities1000.txt -H 'Content-type:application'http://localhost:8983/solr/geonames/update?commit=true&optimize=true&maxSegments=1&separator=%09&encapsulator=%00&fieldnames=id,name,,alternative_names,latitude,longitude,,,countrycode,,,,,,population,elevation,,timezone,lastupdate'-----That might take around 35 seconds; it depends. It can be a lot faster if+That might take around 35 seconds; it depends.+It can be a lot faster ifthe schema were tuned to only have what we truly need (no text search ifnot needed).In that command we said `optimize=true&maxSegments=1` to put the index in a state that-will make tagging faster. The `encapsulator=%00` is a bit of a hack to+will make tagging faster.+The `encapsulator=%00` is a bit of a hack todisable the default double-quote.=== Tag Time!-This is a trivial example tagging a small piece of text. For more+This is a trivial example tagging a small piece of text.+For moreoptions, see the earlier documentation.[source,bash]diff --git a/solr/solr-ref-guide/src/taking-solr-to-production.adoc b/solr/solr-ref-guide/src/taking-solr-to-production.adocindex db5ad47c004..e60904139b1 100644--- a/solr/solr-ref-guide/src/taking-solr-to-production.adoc+++ b/solr/solr-ref-guide/src/taking-solr-to-production.adoc@@ -72,7 +72,8 @@ tar xzf solr-{solr-docs-version}.0.tgz solr-{solr-docs-version}.0/bin/install_so----The previous command extracts the `install_solr_service.sh` script from the archive into the current directory.-If installing on Red Hat, please make sure *lsof* is installed before running the Solr installation script (`sudo yum install lsof`). The installation script must be run as root:+If installing on Red Hat, please make sure *lsof* is installed before running the Solr installation script (`sudo yum install lsof`).+The installation script must be run as root:[source,bash,subs="attributes"]----@@ -95,7 +96,8 @@ To see available options, simply do:sudo bash ./install_solr_service.sh -help-----Once the script completes, Solr will be installed as a service and running in the background on your server (on port 8983). To verify, you can do:+Once the script completes, Solr will be installed as a service and running in the background on your server (on port 8983).+To verify, you can do:[source,bash]----@@ -239,9 +241,11 @@ See <<jvm-settings.adoc#choosing-memory-heap-settings,Choosing Memory Heap SettiAlso, the <<solr-control-script-reference.adoc#,Solr Control Script>> comes with a set of pre-configured Garbage First Garbage Collection settings that have shown to work well with Solr for a number of different workloads.However, these settings may not work well for your specific use of Solr.Consequently, you may need to change the GC settings, which should also be done with the `GC_TUNE` variable in the `/etc/default/solr.in.sh` include file.+For more information about garbage collection settings refer to following articles:-1. https://cwiki.apache.org/confluence/display/solr/ShawnHeisey-2. https://www.oracle.com/technetwork/articles/java/g1gc-1984535.html+. https://cwiki.apache.org/confluence/display/solr/ShawnHeisey+. https://www.oracle.com/technetwork/articles/java/g1gc-1984535.html+You can also refer to <<jvm-settings.adoc#,JVM Settings>> for tuning your memory and garbage collection settings.==== Out-of-Memory Shutdown Hookdiff --git a/solr/solr-ref-guide/src/task-management.adoc b/solr/solr-ref-guide/src/task-management.adocindex 208c70416e7..fe60480287e 100644--- a/solr/solr-ref-guide/src/task-management.adoc+++ b/solr/solr-ref-guide/src/task-management.adoc@@ -19,14 +19,15 @@Solr allows users to control their running tasks by monitoring them, specifying tasks as cancellation enabled and allowingcancellation of the same.-This is achieved using the task management interface. Currently, this is supported for queries.+This is achieved using the task management interface.+Currently, this is supported for queries.== Types of OperationsTask management interface supports the following types of operations:-1. List all currently running cancellable tasks.-2. Cancel a specific task.-3. Query the status of a specific task.+. List all currently running cancellable tasks.+. Cancel a specific task.+. Query the status of a specific task.== Listing All Active Cancellable TasksTo list all the active cancellable tasks currently running, please use the following syntax:diff --git a/solr/solr-ref-guide/src/term-vector-component.adoc b/solr/solr-ref-guide/src/term-vector-component.adocindex 145b093e147..c1afcd2d8e3 100644--- a/solr/solr-ref-guide/src/term-vector-component.adoc+++ b/solr/solr-ref-guide/src/term-vector-component.adoc@@ -22,7 +22,8 @@ For each document in the response, the TermVectorCcomponent can return the term== Term Vector Component Configuration-The TermVectorComponent is not enabled implicitly in Solr - it must be explicitly configured in your `solrconfig.xml` file. The examples on this page show how it is configured in Solr's "```techproducts```" example:+The TermVectorComponent is not enabled implicitly in Solr - it must be explicitly configured in your `solrconfig.xml` file.+The examples on this page show how it is configured in Solr's "```techproducts```" example:[source,bash]----@@ -36,7 +37,8 @@ To enable the this component, you need to configure it using a `searchComponent`<searchComponent name="tvComponent" class="org.apache.solr.handler.component.TermVectorComponent"/>-----A request handler must then be configured to use this component name. In the `techproducts` example, the component is associated with a special request handler named `/tvrh`, that enables term vectors by default using the `tv=true` parameter; but you can associate it with any request handler:+A request handler must then be configured to use this component name.+In the `techproducts` example, the component is associated with a special request handler named `/tvrh`, that enables term vectors by default using the `tv=true` parameter; but you can associate it with any request handler:[source,xml]----@@ -50,7 +52,8 @@ A request handler must then be configured to use this component name. In the `te</requestHandler>-----Once your handler is defined, you may use in conjunction with any schema (that has a `uniqueKeyField)` to fetch term vectors for fields configured with the `termVector` attribute, such as in the `techproducts` sample schema. For example:+Once your handler is defined, you may use in conjunction with any schema (that has a `uniqueKeyField)` to fetch term vectors for fields configured with the `termVector` attribute, such as in the `techproducts` sample schema.+For example:[source,xml]----@@ -135,13 +138,15 @@ If `true`, the Term Vector Component will run.For a given comma-separated list of Lucene document IDs (*not* the Solr Unique Key), term vectors will be returned.`tv.fl`::-For a given comma-separated list of fields, term vectors will be returned. If not specified, the `fl` parameter is used.+For a given comma-separated list of fields, term vectors will be returned.+If not specified, the `fl` parameter is used.`tv.all`::If `true`, all the boolean parameters listed below (`tv.df`, `tv.offsets`, `tv.positions`, `tv.payloads`, `tv.tf` and `tv.tf_idf`) will be enabled.`tv.df`::-If `true`, returns the Document Frequency (DF) of the term in the collection. This can be computationally expensive.+If `true`, returns the Document Frequency (DF) of the term in the collection.+This can be computationally expensive.`tv.offsets`::If `true`, returns offset information for each term in the document.@@ -156,9 +161,11 @@ If `true`, returns payload information.If `true`, returns document term frequency info for each term in the document.`tv.tf_idf`::-If `true`, calculates TF / DF (i.e.,: TF * IDF) for each term. Please note that this is a _literal_ calculation of "Term Frequency multiplied by Inverse Document Frequency" and *not* a classical TF-IDF similarity measure.+If `true`, calculates TF / DF (i.e.,: TF * IDF) for each term.+Please note that this is a _literal_ calculation of "Term Frequency multiplied by Inverse Document Frequency" and *not* a classical TF-IDF similarity measure.+-This parameter requires both `tv.tf` and `tv.df` to be "true". This can be computationally expensive. (The results are not shown in example output)+This parameter requires both `tv.tf` and `tv.df` to be "true". This can be computationally expensive.+(The results are not shown in example output)To see an example of TermVector component output, see the Wiki page: https://cwiki.apache.org/confluence/display/solr/TermVectorComponentExampleOptions@@ -166,4 +173,5 @@ For schema requirements, see also the section <<field-properties-by-use-case.ad== SolrJ and the Term Vector Component-Neither the `SolrQuery` class nor the `QueryResponse` class offer specific method calls to set Term Vector Component parameters or get the "termVectors" output. However, there is a patch for it: https://issues.apache.org/jira/browse/SOLR-949[SOLR-949].+Neither the `SolrQuery` class nor the `QueryResponse` class offer specific method calls to set Term Vector Component parameters or get the "termVectors" output.+However, there is a patch for it: https://issues.apache.org/jira/browse/SOLR-949[SOLR-949].diff --git a/solr/solr-ref-guide/src/term-vectors.adoc b/solr/solr-ref-guide/src/term-vectors.adocindex 84047267ece..9136c8cf1be 100644--- a/solr/solr-ref-guide/src/term-vectors.adoc+++ b/solr/solr-ref-guide/src/term-vectors.adoc@@ -16,18 +16,15 @@// specific language governing permissions and limitations// under the License.-This section of the user guide presents an overview of the text analysis, text analytics-and TF-IDF term vector functions in math expressions.+This section of the user guide presents an overview of the text analysis, text analytics and TF-IDF term vector functions in math expressions.== Text Analysis-The `analyze` function applies a Solr analyzer to a text field and returns the tokens-emitted by the analyzer in an array. Any analyzer chain that is attached to a field in Solr's-schema can be used with the `analyze` function.+The `analyze` function applies a Solr analyzer to a text field and returns the tokens emitted by the analyzer in an array.+Any analyzer chain that is attached to a field in Solr's schema can be used with the `analyze` function.-In the example below, the text "hello world" is analyzed using the analyzer chain attached to the `subject` field in-the schema. The `subject` field is defined as the field type `text_general` and the text is analyzed using the-analysis chain configured for the `text_general` field type.+In the example below, the text "hello world" is analyzed using the analyzer chain attached to the `subject` field in the schema.+The `subject` field is defined as the field type `text_general` and the text is analyzed using the analysis chain configured for the `text_general` field type.[source,text]----@@ -107,9 +104,7 @@ Notice in the output that an array of bigram terms have been added to the tuples=== Text Analytics-The `cartesianProduct` function can be used in conjunction-with the `analyze` function to perform a wide range-of text analytics.+The `cartesianProduct` function can be used in conjunction with the `analyze` function to perform a wide range of text analytics.The `cartesianProduct` function explodes a multivalued field into a stream of tuples.When the `analyze` function is used to create the multivalued field, the `cartesianProduct` function will explode the analyzed tokens into a stream of tuples.@@ -121,8 +116,7 @@ An example performing phrase aggregation is used to illustrate the power of combIn this example the `search` expression is performed over a collection of movie reviews.The phrase query "Man on Fire" is searched for and the top 5000 results, by score are returned.-A single field from the results is return which is the `review_t` field that-contains text of the movie review.+A single field from the results is return which is the `review_t` field that contains text of the movie review.Then `cartesianProduct` function is run over the search results.The `cartesianProduct` function applies the `analyze` function, which takes the `review_t` field and analyzes it with the Lucene/Solr analyzer attached to the `text_bigrams` schema field.@@ -130,8 +124,7 @@ This analyzer emits the bigrams found in the text field.The `cartesianProduct` function explodes each bigram into its own tuple with the bigram stored in the field `term`.The stream of tuples, each containing a bigram, is then filtered by the `having` function-using regular expressions to select bigrams with a length of 12 or greater and to filter-out bigrams that contain specific characters.+using regular expressions to select bigrams with a length of 12 or greater and to filter out bigrams that contain specific characters.The `hashRollup` function then aggregates the bigrams and the `top` function emits the top 10 bigrams by count.@@ -139,9 +132,7 @@ Then Zeppelin-Solr is used to visualize the top 10 ten bigrams.image::images/math-expressions/text-analytics.png[]-Lucene/Solr analyzers can be configured in many different ways to support-aggregations over NLP entities (people, places, companies, etc.) as well as-tokens extracted with regular expressions or dictionaries.+Lucene/Solr analyzers can be configured in many different ways to support aggregations over NLP entities (people, places, companies, etc.) as well as tokens extracted with regular expressions or dictionaries.== TF-IDF Term Vectors@@ -169,12 +160,11 @@ The example below builds on the document annotation example.<1> The `echo` parameter will echo variables `c` and `d`, so the output includesthe row and column labels, which will be defined later in the expression.-<2> The list of tuples are stored in variable `a`. The `termVectors` function-operates over variable `a` and builds a matrix with 2 rows and 4 columns.+<2> The list of tuples are stored in variable `a`.+The `termVectors` function operates over variable `a` and builds a matrix with 2 rows and 4 columns.<3> The `termVectors` function sets the row and column labels of the term vectors matrix as variable `b`.The row labels are the document ids and the column labels are the terms.-<4> The `getRowLabels` and `getColumnLabels` functions return-the row and column labels which are then stored in variables *`c`* and *`d`*.+<4> The `getRowLabels` and `getColumnLabels` functions return the row and column labels which are then stored in variables *`c`* and *`d`*.When this expression is sent to the `/stream` handler it responds with:@@ -267,5 +257,5 @@ The minimum percentage, expressed as a number between 0 and 1, of documents theThe maximum percentage, expressed as a number between 0 and 1, of documents the term can appear in to be included in the index.`exclude`::-A comma-delimited list of strings used to exclude terms. If a term contains any of the excluded strings that-term will be excluded from the term vector.+A comma-delimited list of strings used to exclude terms.+If a term contains any of the excluded strings that term will be excluded from the term vector.diff --git a/solr/solr-ref-guide/src/terms-component.adoc b/solr/solr-ref-guide/src/terms-component.adocindex b7c84d81fb9..412134df48c 100644--- a/solr/solr-ref-guide/src/terms-component.adoc+++ b/solr/solr-ref-guide/src/terms-component.adoc@@ -16,9 +16,12 @@// specific language governing permissions and limitations// under the License.-The Terms Component provides access to the indexed terms in a field and the number of documents that match each term. This can be useful for building an auto-suggest feature or any other feature that operates at the term level instead of the search or document level. Retrieving terms in index order is very fast since the implementation directly uses Lucene's TermEnum to iterate over the term dictionary.+The Terms Component provides access to the indexed terms in a field and the number of documents that match each term.+This can be useful for building an auto-suggest feature or any other feature that operates at the term level instead of the search or document level.+Retrieving terms in index order is very fast since the implementation directly uses Lucene's TermEnum to iterate over the term dictionary.-In a sense, this search component provides fast field-faceting over the whole index, not restricted by the base query or any filters. The document frequencies returned are the number of documents that match the term, including any documents that have been marked for deletion but not yet removed from the index.+In a sense, this search component provides fast field-faceting over the whole index, not restricted by the base query or any filters.+The document frequencies returned are the number of documents that match the term, including any documents that have been marked for deletion but not yet removed from the index.== Configuring the Terms Component@@ -40,32 +43,45 @@ If you want to enable Terms component when using another Request Handler, `terms=== Terms Component Parameters-The parameters below allow you to control what terms are returned. You can also configure any of these with the request handler if you'd like to set them permanently. Or, you can add them to the query request. These parameters are:+The parameters below allow you to control what terms are returned.+You can also configure any of these with the request handler if you'd like to set them permanently.+Or, you can add them to the query request.+These parameters are:`terms`::-If set to `true`, enables the Terms Component. By default, the Terms Component is off (`false`).+If set to `true`, enables the Terms Component.+By default, the Terms Component is off (`false`).+Example: `terms=true``terms.fl`::-Specifies the field from which to retrieve terms. This parameter is required if `terms=true`.+Specifies the field from which to retrieve terms.+This parameter is required if `terms=true`.+Example: `terms.fl=title``terms.list`::-Fetches the document frequency for a comma-delimited list of terms. Terms are always returned in index order. If `terms.ttf` is set to true, also returns their total term frequency. If multiple `terms.fl` are defined, these statistics will be returned for each term in each requested field.+Fetches the document frequency for a comma-delimited list of terms.+Terms are always returned in index order.+If `terms.ttf` is set to true, also returns their total term frequency.+If multiple `terms.fl` are defined, these statistics will be returned for each term in each requested field.+Example: `terms.list=termA,termB,termC`+-NOTE: When `terms.list` is specified, then terms are always sorted by `index`. Except `terms.ttf`, none of other terms parameters are supported when `terms.list` is specified.+NOTE: When `terms.list` is specified, then terms are always sorted by `index`.+Except `terms.ttf`, none of other terms parameters are supported when `terms.list` is specified.`terms.limit`::-Specifies the maximum number of terms to return. The default is `10`. If the limit is set to a number less than 0, then no maximum limit is enforced. Although this is not required, either this parameter or `terms.upper` must be defined.+Specifies the maximum number of terms to return.+The default is `10`.+If the limit is set to a number less than 0, then no maximum limit is enforced.+Although this is not required, either this parameter or `terms.upper` must be defined.+Example: `terms.limit=20``terms.lower`::-Specifies the term at which to start. If not specified, the empty string is used, causing Solr to start at the beginning of the field.+Specifies the term at which to start.+If not specified, the empty string is used, causing Solr to start at the beginning of the field.+Example: `terms.lower=orange`@@ -75,12 +91,15 @@ If set to true, includes the lower-bound term (specified with `terms.lower` in tExample: `terms.lower.incl=false``terms.mincount`::-Specifies the minimum document frequency to return in order for a term to be included in a query response. Results are inclusive of the mincount (that is, >= mincount).+Specifies the minimum document frequency to return in order for a term to be included in a query response.+Results are inclusive of the mincount (that is, >= mincount).+Example: `terms.mincount=5``terms.maxcount`::-Specifies the maximum document frequency a term must have in order to be included in a query response. The default setting is -1, which sets no upper bound. Results are inclusive of the maxcount (that is, \<= maxcount).+Specifies the maximum document frequency a term must have in order to be included in a query response.+The default setting is -1, which sets no upper bound.+Results are inclusive of the maxcount (that is, \<= maxcount).+Example: `terms.maxcount=25`@@ -90,7 +109,8 @@ Restricts matches to terms that begin with the specified string.Example: `terms.prefix=inter``terms.raw`::-If set to true, returns the raw characters of the indexed term, regardless of whether it is human-readable. For instance, the indexed form of numeric numbers is not human-readable.+If set to true, returns the raw characters of the indexed term, regardless of whether it is human-readable.+For instance, the indexed form of numeric numbers is not human-readable.+Example: `terms.raw=true`@@ -100,7 +120,9 @@ Restricts matches to terms that match the regular expression.Example: `terms.regex=.*pedist``terms.regex.flag`::-Defines a Java regex flag to use when evaluating the regular expression defined with `terms.regex`. See http://docs.oracle.com/javase/tutorial/essential/regex/pattern.html for details of each flag. Valid options are:+Defines a Java regex flag to use when evaluating the regular expression defined with `terms.regex`.+See http://docs.oracle.com/javase/tutorial/essential/regex/pattern.html for details of each flag.+Valid options are:* `case_insensitive`* `comments`@@ -114,15 +136,19 @@ Defines a Java regex flag to use when evaluating the regular expression definedExample: `terms.regex.flag=case_insensitive``terms.stats`::-Include index statistics in the results. Currently returns only the *numDocs* for a collection. When combined with `terms.list` it provides enough information to compute inverse document frequency (IDF) for a list of terms.+Include index statistics in the results.+Currently returns only the *numDocs* for a collection.+When combined with `terms.list` it provides enough information to compute inverse document frequency (IDF) for a list of terms.`terms.sort`::-Defines how to sort the terms returned. Valid options are `count`, which sorts by the term frequency, with the highest term frequency first, or `index`, which sorts in index order.+Defines how to sort the terms returned.+Valid options are `count`, which sorts by the term frequency, with the highest term frequency first, or `index`, which sorts in index order.+Example: `terms.sort=index``terms.ttf`::-If set to true, returns both `df` (docFreq) and `ttf` (totalTermFreq) statistics for each requested term in `terms.list`. In this case, the response format is:+If set to true, returns both `df` (docFreq) and `ttf` (totalTermFreq) statistics for each requested term in `terms.list`.+In this case, the response format is:XML:@@ -156,12 +182,14 @@ JSON:----`terms.upper`::-Specifies the term to stop at. Although this parameter is not required, either this parameter or `terms.limit` must be defined.+Specifies the term to stop at.+Although this parameter is not required, either this parameter or `terms.limit` must be defined.+Example: `terms.upper=plum``terms.upper.incl`::-If set to true, the upper bound term is included in the result set. The default is false.+If set to true, the upper bound term is included in the result set.+The default is false.+Example: `terms.upper.incl=true`@@ -298,7 +326,9 @@ Results (notice that the term counts are not affected by the query):== Using the Terms Component for an Auto-Suggest Feature-If the <<suggester.adoc#,Suggester>> doesn't suit your needs, you can use the Terms component in Solr to build a similar feature for your own search application. Simply submit a query specifying whatever characters the user has typed so far as a prefix. For example, if the user has typed "at", the search engine's interface would submit the following query:+If the <<suggester.adoc#,Suggester>> doesn't suit your needs, you can use the Terms component in Solr to build a similar feature for your own search application.+Simply submit a query specifying whatever characters the user has typed so far as a prefix.+For example, if the user has typed "at", the search engine's interface would submit the following query:[source,text]http://localhost:8983/solr/techproducts/terms?terms.fl=name&terms.prefix=at&wt=xmldiff --git a/solr/solr-ref-guide/src/thread-dump.adoc b/solr/solr-ref-guide/src/thread-dump.adocindex e6e7eb35523..ee1a907d5b1 100644--- a/solr/solr-ref-guide/src/thread-dump.adoc+++ b/solr/solr-ref-guide/src/thread-dump.adoc@@ -25,7 +25,8 @@ On the right of the thread name, a down-arrow means you can expand to see the st.List of Threadsimage::images/thread-dump/thread_dump_1.png[image,width=484,height=250]-When you move your cursor over a thread name, a box floats over the name with the state for that thread. Thread states can be:+When you move your cursor over a thread name, a box floats over the name with the state for that thread.+Thread states can be:[%autowidth.stretch,options="header"]|===diff --git a/solr/solr-ref-guide/src/time-series.adoc b/solr/solr-ref-guide/src/time-series.adocindex abd920c9dbc..c24cb752809 100644--- a/solr/solr-ref-guide/src/time-series.adoc+++ b/solr/solr-ref-guide/src/time-series.adoc@@ -16,17 +16,14 @@// specific language governing permissions and limitations// under the License.-This section of the user guide provides an overview of some of the time series capabilities available-in Streaming Expressions and Math Expressions.+This section of the user guide provides an overview of some of the time series capabilities available in Streaming Expressions and Math Expressions.== Time Series Aggregation-The `timeseries` function performs fast, distributed time-series aggregation leveraging Solr's built-in faceting and date math capabilities.+The `timeseries` function performs fast, distributed time series aggregation leveraging Solr's built-in faceting and date math capabilities.The example below performs a monthly time series aggregation over a collection of daily stock price data.-In this example the average monthly closing price is calculated for the stock-ticker *AMZN* between a specific date range.+In this example the average monthly closing price is calculated for the stock ticker *AMZN* between a specific date range.[source,text]----@@ -86,8 +83,7 @@ image::images/math-expressions/timeseries1.png[]== Vectorizing the Time SeriesBefore a time series can be smoothed or modeled the data will need to be vectorized.-The `col` function can be used-to copy a column of data from a list of tuples into an array.+The `col` function can be used to copy a column of data from a list of tuples into an array.The expression below demonstrates the vectorization of the `date_dt` and `avg(close_d)` fields.The `zplot` function is then used to plot the months on the x-axis and the average closing prices on the y-axis.@@ -98,42 +94,35 @@ image::images/math-expressions/timeseries2.png[]== SmoothingTime series smoothing is often used to remove the noise from a time series and help spot the underlying trend.-The math expressions library has three *sliding window* approaches-for time series smoothing.+The math expressions library has three *sliding window* approaches for time series smoothing.These approaches use a summary value from a sliding window of the data to calculate a new set of smoothed data points.-The three *sliding window* functions are lagging indicators, which means-they don't start to move in the direction of the trend until the trend effects-the summary value of the sliding window.+The three *sliding window* functions are lagging indicators, which means they don't start to move in the direction of the trend until the trend effects the summary value of the sliding window.Because of this lagging quality these smoothing functions are often used to confirm the direction of the trend.=== Moving AverageThe `movingAvg` function computes a simple moving average over a sliding window of data.-The example below generates a time series, vectorizes the `avg(close_d)` field and computes the-moving average with a window size of 5.+The example below generates a time series, vectorizes the `avg(close_d)` field and computes the moving average with a window size of 5.-The moving average function returns an array that is of shorter length-then the original vector. This is because results are generated only when a full window of data-is available for computing the average. With a window size of five the moving average will-begin generating results at the 5th value. The prior values are not included in the result.+The moving average function returns an array that is of shorter length then the original vector.+This is because results are generated only when a full window of data is available for computing the average.+With a window size of five the moving average will begin generating results at the 5th value.+The prior values are not included in the result.-The `zplot` function is then used to plot the months on the x-axis, and the average close and moving-average on the y-axis. Notice that the `ltrim` function is used to trim the first 4 values from-the x-axis and the average closing prices. This is done to line up the three arrays so they start-from the 5th value.+The `zplot` function is then used to plot the months on the x-axis, and the average close and moving average on the y-axis.+Notice that the `ltrim` function is used to trim the first 4 values from the x-axis and the average closing prices.+This is done to line up the three arrays so they start from the 5th value.image::images/math-expressions/movingavg.png[]=== Exponential Moving Average-The `expMovingAvg` function uses a different formula for computing the moving average that-responds faster to changes in the underlying data. This means that it is-less of a lagging indicator than the simple moving average.+The `expMovingAvg` function uses a different formula for computing the moving average that responds faster to changes in the underlying data.+This means that it is less of a lagging indicator than the simple moving average.-Below is an example that computes a moving average and exponential moving average and plots them-along with the original y values. Notice how the exponential moving average is more sensitive-to changes in the y values.+Below is an example that computes a moving average and exponential moving average and plots them along with the original y values.+Notice how the exponential moving average is more sensitive to changes in the y values.image::images/math-expressions/expmoving.png[]@@ -150,18 +139,15 @@ image::images/math-expressions/movingMedian.png[]== Differencing-Differencing can be used to make-a time series stationary by removing the trend or seasonality from the series.+Differencing can be used to make a time series stationary by removing the trend or seasonality from the series.=== First Difference-The technique used in differencing is to use the difference between values rather than the-original values. The *first difference* takes the difference between a value and the value-that came directly before it. The first difference is often used to remove the trend-from a time series.+The technique used in differencing is to use the difference between values rather than the original values.+The *first difference* takes the difference between a value and the value that came directly before it.+The first difference is often used to remove the trend from a time series.-The examples below uses the first difference to make two time series stationary so they can be compared-without the trend.+The examples below uses the first difference to make two time series stationary so they can be compared without the trend.In this example we'll be comparing the average monthly closing price for two stocks: Amazon and Google.The image below plots both time series before differencing is applied.@@ -169,17 +155,15 @@ The image below plots both time series before differencing is applied.image::images/math-expressions/timecompare.png[]In the next example the `diff` function is applied to both time series inside the `zplot` function.-The `diff` can be applied inside the `zplot` function or like any other function inside of the `let`-function.+The `diff` can be applied inside the `zplot` function or like any other function inside of the `let` function.-Notice that both time series now have the trend removed and the monthly movements of the stock price-can be studied without being influenced by the trend.+Notice that both time series now have the trend removed and the monthly movements of the stock price can be studied without being influenced by the trend.image::images/math-expressions/diff1.png[]-In the next example the `zoom` function of the time series visualization is used to zoom into a specific-range of months. This allows for closer inspection of the data. With closer inspection of the data there appears-to be some correlation between the monthly movements of the two stocks.+In the next example the `zoom` function of the time series visualization is used to zoom into a specific range of months.+This allows for closer inspection of the data.+With closer inspection of the data there appears to be some correlation between the monthly movements of the two stocks.image::images/math-expressions/diffzoom.png[]@@ -187,60 +171,48 @@ In the final example the differenced time series are correlated with the `corr`image::images/math-expressions/diffcorr.png[]--=== Lagged DifferencesThe `diff` function has an optional second parameter to specify a lag in the difference.-If a lag is specified the difference is taken between a value and the value at a specified-lag in the past. Lagged differences are often used to remove seasonality from a time series.+If a lag is specified the difference is taken between a value and the value at a specified lag in the past.+Lagged differences are often used to remove seasonality from a time series.The simple example below demonstrates how lagged differencing works.-Notice that the array in the example follows a simple repeated pattern. This type of pattern-is often displayed with seasonality.+Notice that the array in the example follows a simple repeated pattern.+This type of pattern is often displayed with seasonality.image::images/math-expressions/season.png[]-In this example we remove this pattern using-the `diff` function with a lag of 4. This will subtract the value lagging four indexes-behind the current index. Notice that the result set size is the original array size minus the lag.-This is because the `diff` function only returns results for values where the lag of 4-is possible to compute.+In this example we remove this pattern using the `diff` function with a lag of 4.+This will subtract the value lagging four indexes behind the current index.+Notice that the result set size is the original array size minus the lag.+This is because the `diff` function only returns results for values where the lag of 4 is possible to compute.image::images/math-expressions/seasondiff.png[]-== Anomaly Detection-The `movingMAD` (moving mean absolute deviation) function can be used to surface anomalies-in a time series by measuring dispersion (deviation from the mean) within a sliding window.+The `movingMAD` (moving mean absolute deviation) function can be used to surface anomalies in a time series by measuring dispersion (deviation from the mean) within a sliding window.-The `movingMAD` function operates in a similar manner as a moving average, except it-measures the mean absolute deviation within the window rather than the average. By-looking for unusually high or low dispersion we can find anomalies in the time-series.+The `movingMAD` function operates in a similar manner as a moving average, except it measures the mean absolute deviation within the window rather than the average.+By looking for unusually high or low dispersion we can find anomalies in the time series.-For this example we'll be working with daily stock prices for Amazon over a two year-period. The daily stock data will provide a larger data set to study.+For this example we'll be working with daily stock prices for Amazon over a two year period.+The daily stock data will provide a larger data set to study.-In the example below the `search` expression is used to return the daily closing price-for the ticker *AMZN* over a two year period.+In the example below the `search` expression is used to return the daily closing price for the ticker *AMZN* over a two year period.image::images/math-expressions/anomaly.png[]-The next step is to apply the `movingMAD` function to the data to calculate-the moving mean absolute deviation over a 10 day window. The example below shows the function being-applied and visualized.+The next step is to apply the `movingMAD` function to the data to calculate the moving mean absolute deviation over a 10 day window.+The example below shows the function being applied and visualized.image::images/math-expressions/mad.png[]-Once the moving MAD has been calculated we can visualize the distribution of dispersion-with the `empiricalDistribution` function. The example below plots the empirical-distribution with 10 bins, creating a 10 bin histogram of the dispersion of the-time series.+Once the moving MAD has been calculated we can visualize the distribution of dispersion with the `empiricalDistribution` function.+The example below plots the empirical distribution with 10 bins, creating a 10 bin histogram of the dispersion of the time series.-This visualization shows that most of the mean absolute deviations fall between 0 and-9.2 with the mean of the final bin at 11.94.+This visualization shows that most of the mean absolute deviations fall between 0 and 9.2 with the mean of the final bin at 11.94.image::images/math-expressions/maddist.png[]@@ -254,24 +226,22 @@ The `outliers` function takes four parameters:* High probability threshold* List of results that the numeric vector was selected from-The `outliers` function iterates the numeric vector and uses the probability-distribution to calculate the cumulative probability of each value. If the cumulative-probability is below the low probability threshold or above the high threshold it considers-the value an outlier. When the `outliers` function encounters an outlier it returns-the corresponding result from the list of results provided by the fifth parameter.+The `outliers` function iterates the numeric vector and uses the probability distribution to calculate the cumulative probability of each value.+If the cumulative probability is below the low probability threshold or above the high threshold it considers the value an outlier.+When the `outliers` function encounters an outlier it returns the corresponding result from the list of results provided by the fifth parameter.It also includes the cumulative probability and the value of the outlier.The example below shows the `outliers` function applied to the Amazon stock-price data set. The empirical distribution of the moving mean absolute deviation is-the first parameter. The vector containing the moving mean absolute-deviations is the second parameter. `-1` is the low and `.99` is the high probability-thresholds. `-1` means that low outliers will not be considered. The final parameter-is the original result set containing the `close_d` and `date_dt` fields.+price data set.+The empirical distribution of the moving mean absolute deviation is the first parameter.+The vector containing the moving mean absolute deviations is the second parameter.+`-1` is the low and `.99` is the high probability thresholds.+`-1` means that low outliers will not be considered.+The final parameter is the original result set containing the `close_d` and `date_dt` fields.The output of the `outliers` function contains the results where an outlier was detected.In this case 5 results above the .99 probability threshold were detected.-image::images/math-expressions/outliers.png[]@@ -280,26 +250,19 @@ image::images/math-expressions/outliers.png[]Math expressions support in Solr includes a number of functions that can be used to model a time series.These functions include linear regression, polynomial and harmonic curve fitting, loess regression, and KNN regression.-Each of these functions can model a time series and be used for-interpolation (predicting values within the dataset) and several-can be used for extrapolation (predicting values beyond the data set).+Each of these functions can model a time series and be used for interpolation (predicting values within the dataset) and several can be used for extrapolation (predicting values beyond the data set).-The various regression functions are covered in detail in the Linear Regression, Curve-Fitting and Machine Learning sections of the user guide.+The various regression functions are covered in detail in the Linear Regression, Curve Fitting and Machine Learning sections of the user guide.-The example below uses the `polyfit` function (polynomial regression) to-fit a non-linear model to a time series. The data set being used is the-monthly average closing price for Amazon over an eight year period.+The example below uses the `polyfit` function (polynomial regression) to fit a non-linear model to a time series.+The data set being used is the monthly average closing price for Amazon over an eight year period.-In this example the `polyfit` function returns a fitted model for the *y*-axis, which is the average monthly closing prices, using a 4 degree polynomial.-The degree of the polynomial determines the number of curves in the-model. The fitted model is set to the variable `y1`. The fitted model-is then directly plotted with `zplot` along with the original `y`-values.+In this example the `polyfit` function returns a fitted model for the y-axis, which is the average monthly closing prices, using a 4 degree polynomial.+The degree of the polynomial determines the number of curves in the model.+The fitted model is set to the variable `y1`.+The fitted model is then directly plotted with `zplot` along with the original `y` values.-The visualization shows the smooth line fit through the average closing-price data.+The visualization shows the smooth line fit through the average closing price data.image::images/math-expressions/timemodel.png[]@@ -307,22 +270,17 @@ image::images/math-expressions/timemodel.png[]== ForecastingThe `polyfit` function can also be used to extrapolate a time series to forecast-future stock prices. The example below demonstrates a 10 month forecast.--In the example the `polyfit` function fits a model to the y-axis and the model-is set to the variable *`m`*.-Then to create a forecast 10 zeros are appended-to the y-axis to create new vector called `y10`.-Then a new x-axis is created using-the `natural` function which returns a sequence of whole numbers 0 to the length of `y10`.+future stock prices.+The example below demonstrates a 10 month forecast.++In the example the `polyfit` function fits a model to the y-axis and the model is set to the variable *`m`*.+Then to create a forecast 10 zeros are appended to the y-axis to create new vector called `y10`.+Then a new x-axis is created using the `natural` function which returns a sequence of whole numbers 0 to the length of `y10`.The new x-axis is stored in the variable `x10`.-The `predict` function uses the fitted model to predict values for the new x-axis stored in-variable `x10`.+The `predict` function uses the fitted model to predict values for the new x-axis stored in variable `x10`.-The `zplot` function is then used to plot the `x10` vector on the x-axis and the `y10` vector and extrapolated-model on the y-axis. Notice that the `y10` vector drops to zero where the observed data-ends, but the forecast continues along the fitted curve-of the model.+The `zplot` function is then used to plot the `x10` vector on the x-axis and the `y10` vector and extrapolated model on the y-axis.+Notice that the `y10` vector drops to zero where the observed data ends, but the forecast continues along the fitted curve of the model.image::images/math-expressions/forecast.png[]diff --git a/solr/solr-ref-guide/src/tokenizers.adoc b/solr/solr-ref-guide/src/tokenizers.adocindex 872904e6e0c..2fb30d44337 100644--- a/solr/solr-ref-guide/src/tokenizers.adoc+++ b/solr/solr-ref-guide/src/tokenizers.adoc@@ -111,7 +111,8 @@ The following sections describe the tokenizer factory classes included in this r== Standard Tokenizer-This tokenizer splits the text field into tokens, treating whitespace and punctuation as delimiters. Delimiter characters are discarded, with the following exceptions:+This tokenizer splits the text field into tokens, treating whitespace and punctuation as delimiters.+Delimiter characters are discarded, with the following exceptions:* Periods (dots) that are not followed by whitespace are kept as part of the token, including Internet domain names.* The "@" character is among the set of token-splitting punctuation, so email addresses are *not* preserved as single tokens.@@ -158,7 +159,10 @@ The Standard Tokenizer supports http://unicode.org/reports/tr29/#Word_Boundaries== Classic Tokenizer-The Classic Tokenizer preserves the same behavior as the Standard Tokenizer of Solr versions 3.1 and previous. It does not use the http://unicode.org/reports/tr29/#Word_Boundaries[Unicode standard annex UAX#29] word boundary rules that the Standard Tokenizer uses. This tokenizer splits the text field into tokens, treating whitespace and punctuation as delimiters. Delimiter characters are discarded, with the following exceptions:+The Classic Tokenizer preserves the same behavior as the Standard Tokenizer of Solr versions 3.1 and previous.+It does not use the http://unicode.org/reports/tr29/#Word_Boundaries[Unicode standard annex UAX#29] word boundary rules that the Standard Tokenizer uses.+This tokenizer splits the text field into tokens, treating whitespace and punctuation as delimiters.+Delimiter characters are discarded, with the following exceptions:* Periods (dots) that are not followed by whitespace are kept as part of the token.@@ -280,7 +284,8 @@ This tokenizer creates tokens from strings of contiguous letters, discarding all== Lower Case Tokenizer-Tokenizes the input stream by delimiting at non-letters and then converting all letters to lowercase. Whitespace and non-letters are discarded.+Tokenizes the input stream by delimiting at non-letters and then converting all letters to lowercase.+Whitespace and non-letters are discarded.*Factory class:* `solr.LowerCaseTokenizerFactory`@@ -330,7 +335,10 @@ Reads the field text and generates n-gram tokens of sizes in the given range.*Example:*-Default behavior. Note that this tokenizer operates over the whole field. It does not break the field at whitespace. As a result, the space character is included in the encoding.+Default behavior.+Note that this tokenizer operates over the whole field.+It does not break the field at whitespace.+As a result, the space character is included in the encoding.[.dynamic-tabs]--@@ -472,7 +480,8 @@ Edge n-gram range of 2 to 5This tokenizer processes multilingual text and tokenizes it appropriately based on its script attribute.-You can customize this tokenizer's behavior by specifying http://userguide.icu-project.org/boundaryanalysis#TOC-RBBI-Rules[per-script rule files]. To add per-script rules, add a `rulefiles` argument, which should contain a comma-separated list of `code:rulefile` pairs in the following format: four-letter ISO 15924 script code, followed by a colon, then a resource path. For example, to specify rules for Latin (script code "Latn") and Cyrillic (script code "Cyrl"), you would enter `Latn:my.Latin.rules.rbbi,Cyrl:my.Cyrillic.rules.rbbi`.+You can customize this tokenizer's behavior by specifying http://userguide.icu-project.org/boundaryanalysis#TOC-RBBI-Rules[per-script rule files]. To add per-script rules, add a `rulefiles` argument, which should contain a comma-separated list of `code:rulefile` pairs in the following format: four-letter ISO 15924 script code, followed by a colon, then a resource path.+For example, to specify rules for Latin (script code "Latn") and Cyrillic (script code "Cyrl"), you would enter `Latn:my.Latin.rules.rbbi,Cyrl:my.Cyrillic.rules.rbbi`.The default configuration for `solr.ICUTokenizerFactory` provides UAX#29 word break rules tokenization (like `solr.StandardTokenizer`), but also includes custom tailorings for Hebrew (specializing handling of double and single quotation marks), for syllable tokenization for Khmer, Lao, and Myanmar, and dictionary-based word segmentation for CJK characters.@@ -539,7 +548,8 @@ The default configuration for `solr.ICUTokenizerFactory` provides UAX#29 word br[IMPORTANT]====-To use this tokenizer, you must add additional .jars to Solr's classpath (as described in the section <<solr-plugins.adoc#installing-plugins,Solr Plugins>>). See the `solr/contrib/analysis-extras/README.md` for information on which jars you need to add.+To use this tokenizer, you must add additional .jars to Solr's classpath (as described in the section <<solr-plugins.adoc#installing-plugins,Solr Plugins>>).+See the `solr/contrib/analysis-extras/README.md` for information on which jars you need to add.====@@ -551,7 +561,8 @@ This tokenizer creates synonyms from file path hierarchies.*Arguments:*-`delimiter`: (character, no default) You can specify the file path delimiter and replace it with a delimiter you provide. This can be useful for working with backslash delimiters.+`delimiter`: (character, no default) You can specify the file path delimiter and replace it with a delimiter you provide.+This can be useful for working with backslash delimiters.`replace`: (character, no default) Specifies the delimiter character Solr uses in the tokenized output.@@ -591,7 +602,8 @@ This tokenizer creates synonyms from file path hierarchies.== Regular Expression Pattern Tokenizer-This tokenizer uses a Java regular expression to break the input text stream into tokens. The expression provided by the pattern argument can be interpreted either as a delimiter that separates tokens, or to match patterns that should be extracted from the text as tokens.+This tokenizer uses a Java regular expression to break the input text stream into tokens.+The expression provided by the pattern argument can be interpreted either as a delimiter that separates tokens, or to match patterns that should be extracted from the text as tokens.See {java-javadocs}java/util/regex/Pattern.html[the Javadocs for `java.util.regex.Pattern`] for more information on Java regular expression syntax.@@ -601,11 +613,15 @@ See {java-javadocs}java/util/regex/Pattern.html[the Javadocs for `java.util.rege`pattern`: (Required) The regular expression, as defined by in `java.util.regex.Pattern`.-`group`: (Optional, default -1) Specifies which regex group to extract as the token(s). The value -1 means the regex should be treated as a delimiter that separates tokens. Non-negative group numbers (>= 0) indicate that character sequences matching that regex group should be converted to tokens. Group zero refers to the entire regex, groups greater than zero refer to parenthesized sub-expressions of the regex, counted from left to right.+`group`: (Optional, default -1) Specifies which regex group to extract as the token(s).+The value -1 means the regex should be treated as a delimiter that separates tokens.+Non-negative group numbers (>= 0) indicate that character sequences matching that regex group should be converted to tokens.+Group zero refers to the entire regex, groups greater than zero refer to parenthesized sub-expressions of the regex, counted from left to right.*Example:*-A comma separated list. Tokens are separated by a sequence of zero or more spaces, a comma, and zero or more spaces.+A comma separated list.+Tokens are separated by a sequence of zero or more spaces, a comma, and zero or more spaces.[.dynamic-tabs]--@@ -637,7 +653,8 @@ A comma separated list. Tokens are separated by a sequence of zero or more space*Example:*-Extract simple, capitalized words. A sequence of at least one capital letter followed by zero or more letters of either case is extracted as a token.+Extract simple, capitalized words.+A sequence of at least one capital letter followed by zero or more letters of either case is extracted as a token.[.dynamic-tabs]--@@ -669,7 +686,10 @@ Extract simple, capitalized words. A sequence of at least one capital letter fol*Example:*-Extract part numbers which are preceded by "SKU", "Part" or "Part Number", case sensitive, with an optional semi-colon separator. Part numbers must be all numeric digits, with an optional hyphen. Regex capture groups are numbered by counting left parenthesis from left to right. Group 3 is the subexpression "[0-9-]+", which matches one or more digits or hyphens.+Extract part numbers which are preceded by "SKU", "Part" or "Part Number", case sensitive, with an optional semi-colon separator.+Part numbers must be all numeric digits, with an optional hyphen.+Regex capture groups are numbered by counting left parenthesis from left to right.+Group 3 is the subexpression "[0-9-]+", which matches one or more digits or hyphens.[.dynamic-tabs]--@@ -701,13 +721,16 @@ Extract part numbers which are preceded by "SKU", "Part" or "Part Number", case== Simplified Regular Expression Pattern Tokenizer-This tokenizer is similar to the `PatternTokenizerFactory` described above, but uses Lucene {lucene-javadocs}/core/org/apache/lucene/util/automaton/RegExp.html[`RegExp`] pattern matching to construct distinct tokens for the input stream. The syntax is more limited than `PatternTokenizerFactory`, but the tokenization is quite a bit faster.+This tokenizer is similar to the `PatternTokenizerFactory` described above, but uses Lucene {lucene-javadocs}/core/org/apache/lucene/util/automaton/RegExp.html[`RegExp`] pattern matching to construct distinct tokens for the input stream.+The syntax is more limited than `PatternTokenizerFactory`, but the tokenization is quite a bit faster.*Factory class:* `solr.SimplePatternTokenizerFactory`*Arguments:*-`pattern`: (Required) The regular expression, as defined by in the {lucene-javadocs}/core/org/apache/lucene/util/automaton/RegExp.html[`RegExp`] javadocs, identifying the characters to include in tokens. The matching is greedy such that the longest token matching at a given point is created. Empty tokens are never created.+`pattern`: (Required) The regular expression, as defined by in the {lucene-javadocs}/core/org/apache/lucene/util/automaton/RegExp.html[`RegExp`] javadocs, identifying the characters to include in tokens.+The matching is greedy such that the longest token matching at a given point is created.+Empty tokens are never created.`maxDeterminizedStates`: (Optional, default 10000) the limit on total state count for the determined automaton computed from the regexp.@@ -741,13 +764,16 @@ To match tokens delimited by simple whitespace characters:== Simplified Regular Expression Pattern Splitting Tokenizer-This tokenizer is similar to the `SimplePatternTokenizerFactory` described above, but uses Lucene {lucene-javadocs}/core/org/apache/lucene/util/automaton/RegExp.html[`RegExp`] pattern matching to identify sequences of characters that should be used to split tokens. The syntax is more limited than `PatternTokenizerFactory`, but the tokenization is quite a bit faster.+This tokenizer is similar to the `SimplePatternTokenizerFactory` described above, but uses Lucene {lucene-javadocs}/core/org/apache/lucene/util/automaton/RegExp.html[`RegExp`] pattern matching to identify sequences of characters that should be used to split tokens.+The syntax is more limited than `PatternTokenizerFactory`, but the tokenization is quite a bit faster.*Factory class:* `solr.SimplePatternSplitTokenizerFactory`*Arguments:*-`pattern`: (Required) The regular expression, as defined by in the {lucene-javadocs}/core/org/apache/lucene/util/automaton/RegExp.html[`RegExp`] javadocs, identifying the characters that should split tokens. The matching is greedy such that the longest token separator matching at a given point is matched. Empty tokens are never created.+`pattern`: (Required) The regular expression, as defined by in the {lucene-javadocs}/core/org/apache/lucene/util/automaton/RegExp.html[`RegExp`] javadocs, identifying the characters that should split tokens.+The matching is greedy such that the longest token separator matching at a given point is matched.+Empty tokens are never created.`maxDeterminizedStates`: (Optional, default 10000) the limit on total state count for the determined automaton computed from the regexp.@@ -781,7 +807,8 @@ To match tokens delimited by simple whitespace characters:== UAX29 URL Email Tokenizer-This tokenizer splits the text field into tokens, treating whitespace and punctuation as delimiters. Delimiter characters are discarded, with the following exceptions:+This tokenizer splits the text field into tokens, treating whitespace and punctuation as delimiters.+Delimiter characters are discarded, with the following exceptions:* Periods (dots) that are not followed by whitespace are kept as part of the token.@@ -833,14 +860,16 @@ The UAX29 URL Email Tokenizer supports http://unicode.org/reports/tr29/#Word_Bou== White Space Tokenizer-Simple tokenizer that splits the text stream on whitespace and returns sequences of non-whitespace characters as tokens. Note that any punctuation _will_ be included in the tokens.+Simple tokenizer that splits the text stream on whitespace and returns sequences of non-whitespace characters as tokens.+Note that any punctuation _will_ be included in the tokens.*Factory class:* `solr.WhitespaceTokenizerFactory`*Arguments:*`rule`::-Specifies how to define whitespace for the purpose of tokenization. Valid values:+Specifies how to define whitespace for the purpose of tokenization.+Valid values:* `java`: (Default) Uses {java-javadocs}java/lang/Character.html#isWhitespace-int-[Character.isWhitespace(int)]* `unicode`: Uses Unicode's WHITESPACE propertydiff --git a/solr/solr-ref-guide/src/transform.adoc b/solr/solr-ref-guide/src/transform.adocindex 61e5e063e92..f16105d27ba 100644--- a/solr/solr-ref-guide/src/transform.adoc+++ b/solr/solr-ref-guide/src/transform.adoc@@ -22,67 +22,56 @@ This section of the user guide provides an overview of useful transformations ap== Selecting and Adding Fields-The `select` function wraps another streaming expression can perform the following operations on each tuple-in the stream:+The `select` function wraps another streaming expression can perform the following operations on each tuple in the stream:* *Select* a subset of fields* *Map* fields to new names* *Compute* new fields-Below is an example showing the `select` function wrapping a `search` function-and mapping fields to new field names. The `recNum` function is a math expression-which simply returns the current record number of the tuple. The `select` expression can call-any math expression to compute new values.+Below is an example showing the `select` function wrapping a `search` function and mapping fields to new field names.+The `recNum` function is a math expression which simply returns the current record number of the tuple.+The `select` expression can call any math expression to compute new values.image::images/math-expressions/select1.png[]-Below is an example using the `div` function to compute a new field-from two existing fields:+Below is an example using the `div` function to compute a new field from two existing fields:image::images/math-expressions/select-math.png[]== Filtering Tuples-The `having` function can be used to filter tuples in the stream based on-boolean logic.+The `having` function can be used to filter tuples in the stream based on boolean logic.-In the example below the `having` function is filtering the output of the-`facet` function to only emit tuples that have `count(*)` greater than 20404.+In the example below the `having` function is filtering the output of the `facet` function to only emit tuples that have `count(*)` greater than 20404.image::images/math-expressions/having.png[]== Paging-The *record number*, added with the `recNum` function,-can be filtered on to support paging.+The *record number*, added with the `recNum` function, can be filtered on to support paging.-In the example below the `and` function with nested `lt` and `gt` functions are-used to select records within a specific record number range:+In the example below the `and` function with nested `lt` and `gt` functions are used to select records within a specific record number range:image::images/math-expressions/search-page.png[]== Handling Nulls-The `notNull` and `isNull` functions can be used to either replace null values with different values,-or to filter out tuples with null values.+The `notNull` and `isNull` functions can be used to either replace null values with different values, or to filter out tuples with null values.-The example below is using the `isNull` function inside of `select` function-to replace null values with -1. The `if` function takes 3 parameters. The first-is a boolean expression, in this case `isNull`. The `if` function returns-the second parameter if the boolean function returns true, and the third-parameter if it returns false. In this case `isNull` is always true because its-checking for a field in the tuples that is not included in the result set.+The example below is using the `isNull` function inside of `select` function to replace null values with -1.+The `if` function takes 3 parameters.+The first is a boolean expression, in this case `isNull`.+The `if` function returns the second parameter if the boolean function returns true, and the third parameter if it returns false.+In this case `isNull` is always true because its checking for a field in the tuples that is not included in the result set.image::images/math-expressions/select2.png[]-`notNull` and `isNull` can also be used with the `having` function to filter out-tuples with null values.+`notNull` and `isNull` can also be used with the `having` function to filter out tuples with null values.-The example below emits all the documents because it is evaluating `isNull` for-a field that is not in the result set, which always returns true.+The example below emits all the documents because it is evaluating `isNull` for a field that is not in the result set, which always returns true.image::images/math-expressions/having2.png[]@@ -93,37 +82,32 @@ image::images/math-expressions/having3.png[]== Regex Matching and Filtering-The `matches` function can be used inside of a `having` function-to test if a field in the record matches a specific-regular expression. This allows for sophisticated regex matching over search results.+The `matches` function can be used inside of a `having` function to test if a field in the record matches a specific regular expression.+This allows for sophisticated regex matching over search results.-The example below uses the `matches` function to return all records where-the `complaint_type_s` field ends with *Commercial*.+The example below uses the `matches` function to return all records where the `complaint_type_s` field ends with *Commercial*.image::images/math-expressions/search-matches.png[]== Sorting-The `sort` and `top` function can be used to resort a result set in memory. The `sort` function-sorts and returns the entire result set based on the sort criteria. The `top` function-can be used to return the top N values in a result set based on the sort criteria.+The `sort` and `top` function can be used to resort a result set in memory.+The `sort` function sorts and returns the entire result set based on the sort criteria.+The `top` function can be used to return the top N values in a result set based on the sort criteria.image::images/math-expressions/search-resort.png[]== Rollups-The `rollup` and `hashRollup` functions can be used to perform aggregations over result sets. This-is different then the `facet`, `facet2D` and `timeseries` aggregation functions which push the aggregations-into the search engine using the JSON facet API.+The `rollup` and `hashRollup` functions can be used to perform aggregations over result sets.+This is different then the `facet`, `facet2D` and `timeseries` aggregation functions which push the aggregations into the search engine using the JSON facet API.-The `rollup` function performs map-reduce style rollups, which requires the result stream be sorted by the-the grouping fields. This allows for aggregations over very high cardinality fields. The `hashRollup` function-performs rollups keeping all buckets in an in-memory hashmap. This requires enough memory to store all the-distinct group by fields in memory, but does not require that the underlying stream be sorted.+The `rollup` function performs map-reduce style rollups, which requires the result stream be sorted by the the grouping fields.+This allows for aggregations over very high cardinality fields.+The `hashRollup` function performs rollups keeping all buckets in an in-memory hashmap.+This requires enough memory to store all the distinct group by fields in memory, but does not require that the underlying stream be sorted.-The example below shows a visualization of the top 5 complaint types-from a random sample of the `nyc311` complaint database. The `top`-function is used to select the top 5 complaint types based on-the `count(*)` field output by the `hashRollup`.+The example below shows a visualization of the top 5 complaint types from a random sample of the `nyc311` complaint database.+The `top` function is used to select the top 5 complaint types based on the `count(*)` field output by the `hashRollup`.image::images/math-expressions/hashRollup.png[]diff --git a/solr/solr-ref-guide/src/transforming-and-indexing-custom-json.adoc b/solr/solr-ref-guide/src/transforming-and-indexing-custom-json.adocindex 1c3b5fe4010..193aa9d9476 100644--- a/solr/solr-ref-guide/src/transforming-and-indexing-custom-json.adoc+++ b/solr/solr-ref-guide/src/transforming-and-indexing-custom-json.adoc@@ -18,33 +18,48 @@If you have JSON documents that you would like to index without transforming them into Solr's structure, you can add them to Solr by including some parameters with the update request.-These parameters provide information on how to split a single JSON file into multiple Solr documents and how to map fields to Solr's schema. One or more valid JSON documents can be sent to the `/update/json/docs` path with the configuration params.+These parameters provide information on how to split a single JSON file into multiple Solr documents and how to map fields to Solr's schema.+One or more valid JSON documents can be sent to the `/update/json/docs` path with the configuration params.== Mapping ParametersThese parameters allow you to define how a JSON file should be read for multiple Solr documents.`split`::-Defines the path at which to split the input JSON into multiple Solr documents and is required if you have multiple documents in a single JSON file. If the entire JSON makes a single Solr document, the path must be “`/`”.+Defines the path at which to split the input JSON into multiple Solr documents and is required if you have multiple documents in a single JSON file.+If the entire JSON makes a single Solr document, the path must be “`/`”.+-It is possible to pass multiple `split` paths by separating them with a pipe `(|)`, for example: `split=/|/foo|/foo/bar`. If one path is a child of another, they automatically become a child document.+It is possible to pass multiple `split` paths by separating them with a pipe `(|)`, for example: `split=/|/foo|/foo/bar`.+If one path is a child of another, they automatically become a child document.`f`::-Provides multivalued mapping to map document field names to Solr field names. The format of the parameter is `target-field-name:json-path`, as in `f=first:/first`. The `json-path` is required. The `target-field-name` is the Solr document field name, and is optional. If not specified, it is automatically derived from the input JSON. The default target field name is the fully qualified name of the field.+Provides multivalued mapping to map document field names to Solr field names.+The format of the parameter is `target-field-name:json-path`, as in `f=first:/first`.+The `json-path` is required.+The `target-field-name` is the Solr document field name, and is optional.+If not specified, it is automatically derived from the input JSON.+The default target field name is the fully qualified name of the field.+Wildcards can be used here, see <<Using Wildcards for Field Names>> below for more information.`mapUniqueKeyOnly`::-(boolean) This parameter is particularly convenient when the fields in the input JSON are not available in the schema and <<schemaless-mode.adoc#,schemaless mode>> is not enabled. This will index all the fields into the default search field (using the `df` parameter, below) and only the `uniqueKey` field is mapped to the corresponding field in the schema. If the input JSON does not have a value for the `uniqueKey` field then a UUID is generated for the same.+(boolean) This parameter is particularly convenient when the fields in the input JSON are not available in the schema and <<schemaless-mode.adoc#,schemaless mode>> is not enabled.+This will index all the fields into the default search field (using the `df` parameter, below) and only the `uniqueKey` field is mapped to the corresponding field in the schema.+If the input JSON does not have a value for the `uniqueKey` field then a UUID is generated for the same.`df`::-If the `mapUniqueKeyOnly` flag is used, the update handler needs a field where the data should be indexed to. This is the same field that other handlers use as a default search field.+If the `mapUniqueKeyOnly` flag is used, the update handler needs a field where the data should be indexed to.+This is the same field that other handlers use as a default search field.`srcField`::-This is the name of the field to which the JSON source will be stored into. This can only be used if `split=/` (i.e., you want your JSON input file to be indexed as a single Solr document). Note that atomic updates will cause the field to be out-of-sync with the document.+This is the name of the field to which the JSON source will be stored into.+This can only be used if `split=/` (i.e., you want your JSON input file to be indexed as a single Solr document).+Note that atomic updates will cause the field to be out-of-sync with the document.`echo`::-This is for debugging purpose only. Set it to `true` if you want the docs to be returned as a response. Nothing will be indexed.+This is for debugging purpose only.+Set it to `true` if you want the docs to be returned as a response.+Nothing will be indexed.For example, if we have a JSON file that includes two documents, we could define an update request like this:@@ -147,7 +162,8 @@ curl 'http://localhost:8983/api/collections/techproducts/update/json/docs'\====---With this request, we have defined that "exams" contains multiple documents. In addition, we have mapped several fields from the input document to Solr fields.+With this request, we have defined that "exams" contains multiple documents.+In addition, we have mapped several fields from the input document to Solr fields.When the update request is complete, the following two documents will be added to the index:@@ -171,7 +187,8 @@ When the update request is complete, the following two documents will be added t}-----In the prior example, all of the fields we wanted to use in Solr had the same names as they did in the input JSON. When that is the case, we can simplify the request by only specifying the `json-path` portion of the `f` parameter, as in this example:+In the prior example, all of the fields we wanted to use in Solr had the same names as they did in the input JSON.+When that is the case, we can simplify the request by only specifying the `json-path` portion of the `f` parameter, as in this example:[.dynamic-tabs]--@@ -272,18 +289,22 @@ curl 'http://localhost:8983/api/collections/techproducts/update/json/docs'\====---In this example, we simply named the field paths (such as `/exams/test`). Solr will automatically attempt to add the content of the field from the JSON input to the index in a field with the same name.+In this example, we simply named the field paths (such as `/exams/test`).+Solr will automatically attempt to add the content of the field from the JSON input to the index in a field with the same name.[TIP]====-Documents will be rejected during indexing if the fields do not exist in the schema before indexing. So, if you are NOT using schemaless mode, you must pre-create all fields. If you are working in <<schemaless-mode.adoc#,Schemaless Mode>>, however, fields that don't exist will be created on the fly with Solr's best guess for the field type.+Documents will be rejected during indexing if the fields do not exist in the schema before indexing.+So, if you are NOT using schemaless mode, you must pre-create all fields.+If you are working in <<schemaless-mode.adoc#,Schemaless Mode>>, however, fields that don't exist will be created on the fly with Solr's best guess for the field type.======= Reusing Parameters in Multiple RequestsYou can store and re-use parameters with Solr's <<request-parameters-api.adoc#,Request Parameters API>>.-Say we wanted to define parameters to split documents at the `exams` field, and map several other fields. We could make an API request such as:+Say we wanted to define parameters to split documents at the `exams` field, and map several other fields.+We could make an API request such as:[.dynamic-tabs]--@@ -413,9 +434,12 @@ Instead of specifying all the field names explicitly, it is possible to specifyThere are two restrictions: wildcards can only be used at the end of the `json-path`, and the split path cannot use wildcards.-A single asterisk `\*` maps only to direct children, and a double asterisk `**` maps recursively to all descendants. The following are example wildcard path mappings:+A single asterisk `\*` maps only to direct children, and a double asterisk `**` maps recursively to all descendants.+The following are example wildcard path mappings:-* `f=$FQN:/**`: maps all fields to the fully qualified name (`$FQN`) of the JSON field. The fully qualified name is obtained by concatenating all the keys in the hierarchy with a period (`.`) as a delimiter. This is the default behavior if no `f` path mappings are specified.+* `f=$FQN:/**`: maps all fields to the fully qualified name (`$FQN`) of the JSON field.+The fully qualified name is obtained by concatenating all the keys in the hierarchy with a period (`.`) as a delimiter.+This is the default behavior if no `f` path mappings are specified.* `f=/docs/*`: maps all the fields under docs and in the name as given in JSON* `f=/docs/**`: maps all the fields under docs and its children in the name as given in JSON* `f=searchField:/docs/*`: maps all fields under /docs to a single field called ‘searchField’@@ -509,7 +533,8 @@ curl 'http://localhost:8983/api/collections/techproducts/update/json'\Because we want the fields to be indexed with the field names as they are found in the JSON input, the double wildcard in `f=/**` will map all fields and their descendants to the same fields in Solr.-It is also possible to send all the values to a single field and do a full text search on that. This is a good option to blindly index and query JSON documents without worrying about fields and schema.+It is also possible to send all the values to a single field and do a full text search on that.+This is a good option to blindly index and query JSON documents without worrying about fields and schema.[.dynamic-tabs]--@@ -597,7 +622,8 @@ curl 'http://localhost:8983/api/collections/techproducts/update/json'\In the above example, we've said all of the fields should be added to a field in Solr named 'txt'. This will add multiple fields to a single field, so whatever field you choose should be multi-valued.-The default behavior is to use the fully qualified name (FQN) of the node. So, if we don't define any field mappings, like this:+The default behavior is to use the fully qualified name (FQN) of the node.+So, if we don't define any field mappings, like this:[.dynamic-tabs]--@@ -779,9 +805,14 @@ curl 'http://localhost:8983/api/collections/techproducts/update/json' -H 'Conten== Tips for Custom JSON Indexing-. Schemaless mode: This handles field creation automatically. The field guessing may not be exactly as you expect, but it works. The best thing to do is to setup a local server in schemaless mode, index a few sample docs and create those fields in your real setup with proper field types before indexing-. Pre-created Schema: Post your docs to the `/update/json/docs` endpoint with `echo=true`. This gives you the list of field names you need to create. Create the fields before you actually index-. No schema, only full-text search: All you need to do is to do full-text search on your JSON. Set the configuration as given in the Setting JSON Defaults section.+. Schemaless mode: This handles field creation automatically.+The field guessing may not be exactly as you expect, but it works.+The best thing to do is to setup a local server in schemaless mode, index a few sample docs and create those fields in your real setup with proper field types before indexing+. Pre-created Schema: Post your docs to the `/update/json/docs` endpoint with `echo=true`.+This gives you the list of field names you need to create.+Create the fields before you actually index+. No schema, only full-text search: All you need to do is to do full-text search on your JSON.+Set the configuration as given in the Setting JSON Defaults section.== Setting JSON Defaults@@ -803,7 +834,8 @@ It is possible to send any JSON to the `/update/json/docs` endpoint and the defa</initParams>-----So, if no parameters are passed, the entire JSON file would get indexed to the `\_src_` field and all the values in the input JSON would go to a field named `text`. If there is a value for the uniqueKey it is stored and if no value could be obtained from the input JSON, a UUID is created and used as the uniqueKey field value.+So, if no parameters are passed, the entire JSON file would get indexed to the `\_src_` field and all the values in the input JSON would go to a field named `text`.+If there is a value for the uniqueKey it is stored and if no value could be obtained from the input JSON, a UUID is created and used as the uniqueKey field value.Alternately, use the Request Parameters feature to set these parameters, as shown earlier in the section <<Reusing Parameters in Multiple Requests>>.diff --git a/solr/solr-ref-guide/src/tutorial-aws.adoc b/solr/solr-ref-guide/src/tutorial-aws.adocindex 24633e87f0e..56584e6905b 100644--- a/solr/solr-ref-guide/src/tutorial-aws.adoc+++ b/solr/solr-ref-guide/src/tutorial-aws.adoc@@ -41,21 +41,23 @@ In this guide we are going to:To use this guide, you must have the following:* An https://aws.amazon.com[AWS] account.-* Familiarity with setting up a single-node SolrCloud on local machine. Refer to the <<solr-tutorial.adoc#,Solr Tutorial>> if you have never used Solr before.+* Familiarity with setting up a single-node SolrCloud on local machine.+Refer to the <<solr-tutorial.adoc#,Solr Tutorial>> if you have never used Solr before.== Launch EC2 instances=== Create new Security Group. Navigate to the https://console.aws.amazon.com/ec2/v2/home[AWS EC2 console] and to the region of your choice.- . Configure an http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/using-network-security.html[AWS security group] which will limit access to the installation and allow our launched EC2 instances to talk to each other without restrictions.- .. From the EC2 Dashboard, click btn:[Security Groups] from the left-hand menu, under "Network & Security".- .. Click btn:[Create Security Group] under the _Security Groups_ section.+. Configure an http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/using-network-security.html[AWS security group] which will limit access to the installation and allow our launched EC2 instances to talk to each other without restrictions.+.. From the EC2 Dashboard, click btn:[Security Groups] from the left-hand menu, under "Network & Security".+.. Click btn:[Create Security Group] under the _Security Groups_ section.Give your security group a descriptive name.- .. You can select one of the existing https://aws.amazon.com/vpc[VPCs] or create a new one.- .. We need two ports open for our cloud here:- ... Solr port. In this example we will use Solr's default port 8983.- ... ZooKeeper Port: We'll use Solr's embedded ZooKeeper, so we'll use the default port 9983 (see the <<Deploying with External ZooKeeper>> to configure external ZooKeeper).+.. You can select one of the existing https://aws.amazon.com/vpc[VPCs] or create a new one.+.. We need two ports open for our cloud here:+... Solr port.+In this example we will use Solr's default port 8983.+... ZooKeeper Port: We'll use Solr's embedded ZooKeeper, so we'll use the default port 9983 (see the <<Deploying with External ZooKeeper>> to configure external ZooKeeper)... Click btn:[Inbound] to set inbound network rules, then select btn:[Add Rule].Select "Custom TCP" as the type.Enter `8983` for the "Port Range" and choose "My IP for the Source, then enter your public IP.@@ -111,7 +113,8 @@ You do not have to add any tags for this quick start, but you can add them if yo. Select an existing “private key file” or create a new one and download to your local machine so you will be able to login into the instances via SSH.+image::images/tutorial-aws/aws-key.png[image,width=600,height=400]-. On the instances list, you can watch the states change. You cannot use the instances until they become *“running”*.+. On the instances list, you can watch the states change.+You cannot use the instances until they become *“running”*.== Install, Configure and Startdiff --git a/solr/solr-ref-guide/src/tutorial-diy.adoc b/solr/solr-ref-guide/src/tutorial-diy.adocindex 86d9473eb84..307225b3997 100644--- a/solr/solr-ref-guide/src/tutorial-diy.adoc+++ b/solr/solr-ref-guide/src/tutorial-diy.adoc@@ -47,7 +47,8 @@ Solr has lots of ways to index data.Choose one of the approaches below and try it out with your system:Local Files with `bin/post`::-If you have a local directory of files, the Post Tool (`bin/post`) can index a directory of files. We saw this in action in our first exercise.+If you have a local directory of files, the Post Tool (`bin/post`) can index a directory of files.+We saw this in action in our first exercise.+We used only JSON, XML and CSV in our exercises, but the Post Tool can also handle HTML, PDF, Microsoft Office formats (such as MS Word), plain text, and more.+@@ -61,7 +62,8 @@ These might be caused by the field guessing, or the file type may not be supportIndexing content such as this demonstrates the need to plan Solr for your data, which requires understanding it and perhaps also some trial and error.SolrJ::-SolrJ is a Java-based client for interacting with Solr. Use <<solrj.adoc#,SolrJ>> for JVM-based languages or other <<client-apis.adoc#,Solr clients>> to programmatically create documents to send to Solr.+SolrJ is a Java-based client for interacting with Solr.+Use <<solrj.adoc#,SolrJ>> for JVM-based languages or other <<client-apis.adoc#,Solr clients>> to programmatically create documents to send to Solr.Documents Screen::Use the Admin UI <<documents-screen.adoc#,Documents tab>> (at http://localhost:8983/solr/#/localDocs/documents) to paste in a document to be indexed, or select `Document Builder` from the `Document Type` dropdown to build a document one field at a time.@@ -79,7 +81,8 @@ You can see that that has happened by looking at the values for `numDocs` and `mThe `maxDoc` value may be larger as the `maxDoc` count includes logically deleted documents that have not yet been physically removed from the index.You can re-post the sample files over and over again as much as you want and `numDocs` will never increase, because the new documents will constantly be replacing the old.-Go ahead and edit any of the existing example data files, change some of the data, and re-run the PostTool (`bin/post`). You'll see your changes reflected in subsequent searches.+Go ahead and edit any of the existing example data files, change some of the data, and re-run the PostTool (`bin/post`).+You'll see your changes reflected in subsequent searches.=== Deleting Datadiff --git a/solr/solr-ref-guide/src/tutorial-films.adoc b/solr/solr-ref-guide/src/tutorial-films.adocindex 4a7e1e2d49e..8d9899f47ae 100644--- a/solr/solr-ref-guide/src/tutorial-films.adoc+++ b/solr/solr-ref-guide/src/tutorial-films.adoc@@ -145,7 +145,8 @@ The films data we are going to index has a small number of fields for each movieIf you look at one of the files in `example/films`, you'll see the first film is named _.45_, released in 2006.As the first document in the dataset, Solr is going to guess the field type based on the data in the record.-If we go ahead and index this data, that first film name is going to indicate to Solr that that field type is a "float" numeric field, and will create a "name" field with a type `FloatPointField`. All data after this record will be expected to be a float.+If we go ahead and index this data, that first film name is going to indicate to Solr that that field type is a "float" numeric field, and will create a "name" field with a type `FloatPointField`.+All data after this record will be expected to be a float.Well, that's not going to work.We have titles like _A Mighty Wind_ and _Chicken Run_, which are strings - decidedly not numeric and not floats.@@ -175,7 +176,8 @@ In the first exercise when we queried the documents we had indexed, we didn't haThe configuration we're using now doesn't have that rule.We would need to define a field to search for every query.-We can, however, set up a "catchall field" by defining a copy field that will take all data from all fields and index it into a field named `\_text_`. Let's do that now.+We can, however, set up a "catchall field" by defining a copy field that will take all data from all fields and index it into a field named `\_text_`.+Let's do that now.You can use either the Admin UI or the Schema API for this.@@ -278,7 +280,8 @@ On the Admin UI Query tab, if you check the `facet` checkbox, you'll see a few fimage::images/solr-tutorial/tutorial-admin-ui-facet-options.png[Solr Quick Start: Query tab facet options]To see facet counts from all documents (`q=\*:*`): turn on faceting (`facet=true`), and specify the field to facet on via the `facet.field` parameter.-If you only want facets, and no document contents, specify `rows=0`. The `curl` command below will return facet counts for the `genre_str` field:+If you only want facets, and no document contents, specify `rows=0`.+The `curl` command below will return facet counts for the `genre_str` field:`curl "http://localhost:8983/solr/films/select?q=\*:*&rows=0&facet=true&facet.field=genre_str"`@@ -449,7 +452,8 @@ We've truncated this output as well - you will see a lot of genres and directorsIn this exercise, we learned a little bit more about how Solr organizes data in the indexes, and how to work with the Schema API to manipulate the schema file.We also learned a bit about facets in Solr, including range facets and pivot facets.-In both of these things, we've only scratched the surface of the available options. If you can dream it, it might be possible!+In both of these things, we've only scratched the surface of the available options.+If you can dream it, it might be possible!Like our previous exercise, this data may not be relevant to your needs.We can clean up our work by deleting the collection.diff --git a/solr/solr-ref-guide/src/tutorial-solrcloud.adoc b/solr/solr-ref-guide/src/tutorial-solrcloud.adocindex ff983c3cb88..546a3938fbf 100644--- a/solr/solr-ref-guide/src/tutorial-solrcloud.adoc+++ b/solr/solr-ref-guide/src/tutorial-solrcloud.adoc@@ -22,25 +22,32 @@ It's a system in which data is organized into multiple pieces, or shards, that cThis section explains SolrCloud and its inner workings in detail, but before you dive in, it's best to have an idea of what it is you're trying to accomplish.-This page provides a simple tutorial to start Solr in SolrCloud mode, so you can begin to get a sense for how shards interact with each other during indexing and when serving queries. To that end, we'll use simple examples of configuring SolrCloud on a single machine, which is obviously not a real production environment, which would include several servers or virtual machines. In a real production environment, you'll also use the real machine names instead of "localhost" which we've used here.+This page provides a simple tutorial to start Solr in SolrCloud mode, so you can begin to get a sense for how shards interact with each other during indexing and when serving queries.+To that end, we'll use simple examples of configuring SolrCloud on a single machine, which is obviously not a real production environment, which would include several servers or virtual machines.+In a real production environment, you'll also use the real machine names instead of "localhost" which we've used here.In this section you will learn how to start a SolrCloud cluster using startup scripts and a specific configset.[TIP]====-This tutorial assumes that you're already familiar with the basics of using Solr. If you need a refresher, please see the <<getting-started.adoc#,Getting Started section>> to get a grounding in Solr concepts. If you load documents as part of that exercise, you should start over with a fresh Solr installation for these SolrCloud tutorials.+This tutorial assumes that you're already familiar with the basics of using Solr.+If you need a refresher, please see the <<getting-started.adoc#,Getting Started section>> to get a grounding in Solr concepts.+If you load documents as part of that exercise, you should start over with a fresh Solr installation for these SolrCloud tutorials.====[WARNING]====-For security reasons, Solr nodes only accept connections from localhost by default. Administrators setting up SolrCloud deployments with multiple nodes must override this setting. For more details see <<securing-solr.adoc#network-configuration,here>>.+For security reasons, Solr nodes only accept connections from localhost by default.+Administrators setting up SolrCloud deployments with multiple nodes must override this setting.+For more details see <<securing-solr.adoc#network-configuration,here>>.====== SolrCloud Example=== Interactive Startup-The `bin/solr` script makes it easy to get started with SolrCloud as it walks you through the process of launching Solr nodes in SolrCloud mode and adding a collection. To get started, simply do:+The `bin/solr` script makes it easy to get started with SolrCloud as it walks you through the process of launching Solr nodes in SolrCloud mode and adding a collection.+To get started, simply do:[source,bash]----@@ -59,7 +66,8 @@ This interactive session will help you launch a SolrCloud cluster on your localTo begin, how many Solr nodes would you like to run in your local cluster? (specify 1-4 nodes) [2]-----The script supports starting up to 4 nodes, but we recommend using the default of 2 when starting out. These nodes will each exist on a single machine, but will use different ports to mimic operation on different servers.+The script supports starting up to 4 nodes, but we recommend using the default of 2 when starting out.+These nodes will each exist on a single machine, but will use different ports to mimic operation on different servers.Next, the script will prompt you for the port to bind each of the Solr nodes to, such as:@@ -68,14 +76,16 @@ Next, the script will prompt you for the port to bind each of the Solr nodes to,Please enter the port for node1 [8983]-----Choose any available port for each node; the default for the first node is 8983 and 7574 for the second node. The script will start each node in order and show you the command it uses to start the server, such as:+Choose any available port for each node; the default for the first node is 8983 and 7574 for the second node.+The script will start each node in order and show you the command it uses to start the server, such as:[source,bash]----solr start -cloud -s example/cloud/node1/solr -p 8983-----The first node will also start an embedded ZooKeeper server bound to port 9983. The Solr home for the first node is in `example/cloud/node1/solr` as indicated by the `-s` option.+The first node will also start an embedded ZooKeeper server bound to port 9983.+The Solr home for the first node is in `example/cloud/node1/solr` as indicated by the `-s` option.After starting up all nodes in the cluster, the script prompts you for the name of the collection to create:@@ -92,8 +102,12 @@ Next, the script prompts you for the number of shards to distribute the collectiNext, the script will prompt you for the number of replicas to create for each shard.<<solrcloud-shards-indexing.adoc#,Replication>> is covered in more detail later in the guide, so if you're unsure, then use the default of 2 so that you can see how replication is handled in SolrCloud.-Lastly, the script will prompt you for the name of a configuration directory for your collection. You can choose *_default*, or *sample_techproducts_configs*.-The configuration directories are pulled from `server/solr/configsets/` so you can review them beforehand if you wish. The *_default* configuration is useful when you're still designing a schema for your documents and need some flexibility as you experiment with Solr, since it has schemaless functionality. However, after creating your collection, the schemaless functionality can be disabled in order to lock down the schema (so that documents indexed after doing so will not alter the schema) or to configure the schema by yourself. This can be done as follows (assuming your collection name is `mycollection`):+Lastly, the script will prompt you for the name of a configuration directory for your collection.+You can choose *_default*, or *sample_techproducts_configs*.+The configuration directories are pulled from `server/solr/configsets/` so you can review them beforehand if you wish.+The *_default* configuration is useful when you're still designing a schema for your documents and need some flexibility as you experiment with Solr, since it has schemaless functionality.+However, after creating your collection, the schemaless functionality can be disabled in order to lock down the schema (so that documents indexed after doing so will not alter the schema) or to configure the schema by yourself.+This can be done as follows (assuming your collection name is `mycollection`):[.dynamic-tabs]--@@ -116,7 +130,8 @@ curl http://host:8983/api/collections/mycollection/config -d '{"set-user-propert====---At this point, you should have a new collection created in your local SolrCloud cluster. To verify this, you can run the status command:+At this point, you should have a new collection created in your local SolrCloud cluster.+To verify this, you can run the status command:[source,bash]----@@ -125,7 +140,8 @@ bin/solr statusIf you encounter any errors during this process, check the Solr log files in `example/cloud/node1/logs` and `example/cloud/node2/logs`.-You can see how your collection is deployed across the cluster by visiting the cloud panel in the Solr Admin UI: http://localhost:8983/solr/#/~cloud. Solr also provides a way to perform basic diagnostics for a collection using the healthcheck command:+You can see how your collection is deployed across the cluster by visiting the cloud panel in the Solr Admin UI: http://localhost:8983/solr/#/~cloud.+Solr also provides a way to perform basic diagnostics for a collection using the healthcheck command:[source,bash]----@@ -154,7 +170,8 @@ bin/solr -e cloud -noprompt=== Restarting Nodes-You can restart your SolrCloud nodes using the `bin/solr` script. For instance, to restart node1 running on port 8983 (with an embedded ZooKeeper server), you would do:+You can restart your SolrCloud nodes using the `bin/solr` script.+For instance, to restart node1 running on port 8983 (with an embedded ZooKeeper server), you would do:[source,bash]----@@ -172,7 +189,8 @@ Notice that you need to specify the ZooKeeper address (`-z localhost:9983`) when=== Adding a Node to a Cluster-Adding a node to an existing cluster is a bit advanced and involves a little more understanding of Solr. Once you startup a SolrCloud cluster using the startup scripts, you can add a new node to it by:+Adding a node to an existing cluster is a bit advanced and involves a little more understanding of Solr.+Once you startup a SolrCloud cluster using the startup scripts, you can add a new node to it by:[source,bash]----@@ -181,7 +199,8 @@ cp <existing solr.xml path> <new solr.home>bin/solr start -cloud -s solr.home/solr -p <port num> -z <zk hosts string>-----Notice that the above requires you to create a Solr home directory. You either need to copy `solr.xml` to the `solr_home` directory, or keep in centrally in ZooKeeper `/solr.xml`.+Notice that the above requires you to create a Solr home directory.+You either need to copy `solr.xml` to the `solr_home` directory, or keep in centrally in ZooKeeper `/solr.xml`.Example (with directory structure) that adds a node to an example started with "bin/solr -e cloud":@@ -192,6 +211,7 @@ cp server/solr/solr.xml example/cloud/node3/solrbin/solr start -cloud -s example/cloud/node3/solr -p 8987 -z localhost:9983-----The previous command will start another Solr node on port 8987 with Solr home set to `example/cloud/node3/solr`. The new node will write its log files to `example/cloud/node3/logs`.+The previous command will start another Solr node on port 8987 with Solr home set to `example/cloud/node3/solr`.+The new node will write its log files to `example/cloud/node3/logs`.Once you're comfortable with how the SolrCloud example works, we recommend using the process described in <<taking-solr-to-production.adoc#,Taking Solr to Production>> for setting up SolrCloud nodes in production.diff --git a/solr/solr-ref-guide/src/tutorial-techproducts.adoc b/solr/solr-ref-guide/src/tutorial-techproducts.adocindex 13ac40dfcbb..a95f41d3373 100644--- a/solr/solr-ref-guide/src/tutorial-techproducts.adoc+++ b/solr/solr-ref-guide/src/tutorial-techproducts.adoc@@ -318,7 +318,8 @@ This is, again, default behavior.If you want to restrict the fields in the response, you can use the `fl` parameter, which takes a comma-separated list of field names.This is one of the available fields on the query form in the Admin UI.-Put "id" (without quotes) in the "fl" box and hit btn:[Execute Query] again. Or, to specify it with curl:+Put "id" (without quotes) in the "fl" box and hit btn:[Execute Query] agai+Or, to specify it with curl:`curl "http://localhost:8983/solr/techproducts/select?q=foundation&fl=id"`@@ -371,7 +372,8 @@ This search finds all documents that contain the term "electronics" anywhere inHowever, we can see from the above there is a `cat` field (for "category").If we limit our search for only documents with the category "electronics", the results will be more precise for our users.-Update your query in the `q` field of the Admin UI so it's `cat:electronics`. Now you get 12 results:+Update your query in the `q` field of the Admin UI so it's `cat:electronics`.+Now you get 12 results:[source,json]{diff --git a/solr/solr-ref-guide/src/update-request-processors.adoc b/solr/solr-ref-guide/src/update-request-processors.adocindex a613435c59d..35411846f88 100644--- a/solr/solr-ref-guide/src/update-request-processors.adoc+++ b/solr/solr-ref-guide/src/update-request-processors.adoc@@ -59,10 +59,10 @@ However, before we understand how to configure update processor chains, we mustIn case no update processor chains are configured in `solrconfig.xml`, Solr will automatically create a default update processor chain which will be used for all update requests.This default update processor chain consists of the following processors (in order):-1. `LogUpdateProcessorFactory` - Tracks the commands processed during this request and logs them-2. `DistributedUpdateProcessorFactory` - Responsible for distributing update requests to the right node e.g., routing requests to the leader of the right shard and distributing updates from the leader to each replica.+. `LogUpdateProcessorFactory` - Tracks the commands processed during this request and logs them+. `DistributedUpdateProcessorFactory` - Responsible for distributing update requests to the right node e.g., routing requests to the leader of the right shard and distributing updates from the leader to each replica.This processor is activated only in SolrCloud mode.-3. `RunUpdateProcessorFactory` - Executes the update using internal Solr APIs.+. `RunUpdateProcessorFactory` - Executes the update using internal Solr APIs.Each of these perform an essential function and as such any custom chain usually contain all of these processors.The `RunUpdateProcessorFactory` is usually the last update processor in any custom chain.@@ -432,7 +432,8 @@ This keeps track of all commands that have passed through the chain and prints tAlmost all processor chains should end with an instance of `RunUpdateProcessorFactory` unless the user is explicitly executing the update commands in an alternative custom `UpdateRequestProcessorFactory`.=== Update Processors That Can Be Used at Runtime-These Update processors do not need any configuration in `solrconfig.xml`. They are automatically initialized when their name is added to the `processor` parameter sent with an update request.+These Update processors do not need any configuration in `solrconfig.xml`.+They are automatically initialized when their name is added to the `processor` parameter sent with an update request.Multiple processors can be used by appending multiple processor names separated by commas.==== AtomicUpdateProcessorFactory@@ -473,7 +474,8 @@ For example:processor=template&template.field=fullName:Mr. {firstName} {lastName}-----The above example would add a new field to the document called `fullName`. The fields `firstName and` `lastName` are supplied from the document fields.+The above example would add a new field to the document called `fullName`.+The fields `firstName and` `lastName` are supplied from the document fields.If either of them is missing, that part is replaced with an empty string.If those fields are multi-valued, only the first value is used.diff --git a/solr/solr-ref-guide/src/upgrading-a-solr-cluster.adoc b/solr/solr-ref-guide/src/upgrading-a-solr-cluster.adocindex ef88cd2255c..f665ef2ee1d 100644--- a/solr/solr-ref-guide/src/upgrading-a-solr-cluster.adoc+++ b/solr/solr-ref-guide/src/upgrading-a-solr-cluster.adoc@@ -89,7 +89,8 @@ For instance, if you plan to run Solr as the "solr" user and `SOLR_HOME` is `/va=== Step 4: Start Solr-You are now ready to start the upgraded Solr node by doing: `sudo service solr start`. The upgraded instance will join the existing cluster because you're using the same `SOLR_HOME`, `SOLR_PORT`, and `SOLR_HOST` settings used by the old Solr node; thus, the new server will look like the old node to the running cluster.+You are now ready to start the upgraded Solr node by doing: `sudo service solr start`.+The upgraded instance will join the existing cluster because you're using the same `SOLR_HOME`, `SOLR_PORT`, and `SOLR_HOST` settings used by the old Solr node; thus, the new server will look like the old node to the running cluster.Be sure to look in `/var/solr/logs/solr.log` for errors during startup.=== Step 5: Run Healthcheckdiff --git a/solr/solr-ref-guide/src/user-managed-distributed-search.adoc b/solr/solr-ref-guide/src/user-managed-distributed-search.adocindex 6397cf4059d..a88a122bc0c 100644--- a/solr/solr-ref-guide/src/user-managed-distributed-search.adoc+++ b/solr/solr-ref-guide/src/user-managed-distributed-search.adoc@@ -18,15 +18,21 @@When using traditional index sharding, you will need to consider how to query your documents.-It is highly recommended that you use <<cluster-types.adoc#solrcloud-mode,SolrCloud>> when needing to scale up or scale out. The setup described below is legacy and was used prior to the existence of SolrCloud. SolrCloud provides for a truly distributed set of features with support for things like automatic routing, leader election, optimistic concurrency and other sanity checks that are expected out of a distributed system.+It is highly recommended that you use <<cluster-types.adoc#solrcloud-mode,SolrCloud>> when needing to scale up or scale out.+The setup described below is legacy and was used prior to the existence of SolrCloud.+SolrCloud provides for a truly distributed set of features with support for things like automatic routing, leader election, optimistic concurrency and other sanity checks that are expected out of a distributed system.-Everything on this page is specific to legacy setup of distributed search. Users trying out SolrCloud should not follow any of the steps or information below.+Everything on this page is specific to legacy setup of distributed search.+Users trying out SolrCloud should not follow any of the steps or information below.-Update reorders (i.e., replica A may see update X then Y, and replica B may see update Y then X). *deleteByQuery* also handles reorders the same way, to ensure replicas are consistent. All replicas of a shard are consistent, even if the updates arrive in a different order on different replicas.+Update reorders (i.e., replica A may see update X then Y, and replica B may see update Y then X).+*deleteByQuery* also handles reorders the same way, to ensure replicas are consistent.+All replicas of a shard are consistent, even if the updates arrive in a different order on different replicas.== Distributing Documents across Shards-When not using SolrCloud, it is up to you to get all your documents indexed on each shard of your server farm. Solr supports distributed indexing (routing) in its true form only in the SolrCloud mode.+When not using SolrCloud, it is up to you to get all your documents indexed on each shard of your server farm.+Solr supports distributed indexing (routing) in its true form only in the SolrCloud mode.In the legacy distributed mode, Solr does not calculate universal term/doc frequencies.For most large-scale implementations, it is not likely to matter that Solr calculates TF/IDF at the shard level.@@ -48,7 +54,8 @@ Rather than require users to include the shards parameter explicitly, it is usua[IMPORTANT]====-Do not add the `shards` parameter to the standard request handler; doing so may cause search queries may enter an infinite loop. Instead, define a new request handler that uses the `shards` parameter, and pass distributed search requests to that handler.+Do not add the `shards` parameter to the standard request handler; doing so may cause search queries may enter an infinite loop.+Instead, define a new request handler that uses the `shards` parameter, and pass distributed search requests to that handler.====With Legacy mode, only query requests are distributed.@@ -98,7 +105,7 @@ It can be avoided by following the instructions in the section <<solrcloud-dist== Load Balancing Requests-When managing a user-managed Solr cluster. query requests should be load balanced across each of the shard followers.+When managing a user-managed Solr cluster, query requests should be load balanced across each of the shard followers.This gives you both increased query handling capacity and failover if a server goes down.Because there is no centralized cluster management in this scenario, none of the leader shards in this configuration know about each other.diff --git a/solr/solr-ref-guide/src/user-managed-index-replication.adoc b/solr/solr-ref-guide/src/user-managed-index-replication.adocindex f6127881e4a..eb2cfd87ac1 100644--- a/solr/solr-ref-guide/src/user-managed-index-replication.adoc+++ b/solr/solr-ref-guide/src/user-managed-index-replication.adoc@@ -300,9 +300,9 @@ This screen will only appear if you are not running Solr in SolrCloud mode.If you are using Leader-Follower index replication, you can use this screen to:-. View the replicatable index state. (on a leader node)+. View the replicatable index state (on a leader node). View the current replication status (on a follower node)-. Disable replication. (on a leader node)+. Disable replication (on a leader node)image::images/user-managed-index-replication/replication.png[image,width=412,height=250]@@ -428,7 +428,8 @@ Create a backup on leader if there are committed index data in the server; other[source,bash]http://_leader_host:port_/solr/_core_name_/replication?command=backup+-This command is useful for making periodic backups. There are several supported request parameters:+This command is useful for making periodic backups.+There are several supported request parameters:`numberToKeep:`::: This can be used with the backup command unless the `maxNumberOfBackups` initialization parameter has been specified on the handler – in which case `maxNumberOfBackups` is always used and attempts to use the `numberToKeep` request parameter will cause an error.@@ -450,7 +451,8 @@ Restore a backup from a backup repository.[source,bash]http://_leader_host:port_/solr/_core_name_/replication?command=restore+-This command is used to restore a backup. There are several supported request parameters:+This command is used to restore a backup.+There are several supported request parameters:`name`::: (optional) Backup name.The name of the backed up index snapshot to be restored.@@ -460,7 +462,8 @@ It would choose the latest timestamp backup in that case.`repository`::: The name of the backup repository where the backup resides.When not specified, it defaults to local file system.-`location`::: Backup location. The value depends on the repository in use.+`location`::: Backup location.+The value depends on the repository in use.For file system repository, location defaults to core's dataDir (data directory), and if specified, it needs to be within `SOLR_HOME`, `SOLR_DATA_HOME` or the paths specified by `solr.xml` `allowPaths` parameter.`restorestatus`::@@ -469,7 +472,8 @@ Check the status of a running restore operation.[source,bash]http://_leader_host:port_/solr/_core_name_/replication?command=restorestatus+-This command is used to check the status of a restore operation. This command takes no parameters.+This command is used to check the status of a restore operation.+This command takes no parameters.+The status value can be "In Progress", "success", or "failed". If it failed then an "exception" will also be sent in the response.diff --git a/solr/solr-ref-guide/src/v2-api.adoc b/solr/solr-ref-guide/src/v2-api.adocindex 8f3484648e7..f89d81b4771 100644--- a/solr/solr-ref-guide/src/v2-api.adoc+++ b/solr/solr-ref-guide/src/v2-api.adoc@@ -134,7 +134,8 @@ Example of introspect for a POST API: `\http://localhost:8983/api/c/gettingstart}-----The `"commands"` section in the above example has one entry for each command supported at this endpoint. The key is the command name and the value is a JSON object describing the command structure using JSON schema (see http://json-schema.org/ for a description).+The `"commands"` section in the above example has one entry for each command supported at this endpoint.+The key is the command name and the value is a JSON object describing the command structure using JSON schema (see http://json-schema.org/ for a description).== Invocation Examplesdiff --git a/solr/solr-ref-guide/src/variables.adoc b/solr/solr-ref-guide/src/variables.adocindex 148f4d4fce1..548ed1f42e3 100644--- a/solr/solr-ref-guide/src/variables.adoc+++ b/solr/solr-ref-guide/src/variables.adoc@@ -17,20 +17,17 @@// under the License.-This section of the user guide describes how to assign and visualize-variables with math expressions.+This section of the user guide describes how to assign and visualize variables with math expressions.== The Let Expression-The `let` expression sets variables and returns-the value of the last variable by default.+The `let` expression sets variables and returns the value of the last variable by default.The output of any streaming expression or math expression can be set to a variable.Below is a simple example setting three variables `a`, `b`,-and `c`. Variables `a` and `b` are set to arrays.-The variable `c` is set-to the output of the `ebeAdd` function which performs element-by-element-addition of the two arrays.+and `c`.+Variables `a` and `b` are set to arrays.+The variable `c` is set to the output of the `ebeAdd` function which performs element-by-element addition of the two arrays.[source,text]----@@ -74,8 +71,7 @@ let(echo=true,c=ebeAdd(a, b))-----When this expression is sent to the `/stream` handler it-responds with:+When this expression is sent to the `/stream` handler it responds with:[source,json]----@@ -149,12 +145,10 @@ When this expression is sent to the `/stream` handler it responds with:== Visualizing Variables-The `let` expression can also include a `zplot` expression that can be used to visualize the-variables.+The `let` expression can also include a `zplot` expression that can be used to visualize the variables.In the example below the variables `a` and `b` are set to arrays.-The `zplot` function-outputs the variables as `x` and `y` fields in the output.+The `zplot` function outputs the variables as `x` and `y` fields in the output.[source,text]----@@ -192,13 +186,11 @@ When this expression is sent to the `/stream` handler it responds with:----Using this approach variables can by visualized using Zeppelin-Solr.-In the example below-the arrays are shown in table format.+In the example below the arrays are shown in table format.image::images/math-expressions/variables.png[]-Once in table format we can plot the variables using one of the plotting or charting-visualizations.+Once in table format we can plot the variables using one of the plotting or charting visualizations.The example below shows variables plotted on a line chart:image::images/math-expressions/variables1.png[]@@ -206,17 +198,13 @@ image::images/math-expressions/variables1.png[]== Caching Variables-Variables can be cached in-memory on the Solr node where the math expression-was run.+Variables can be cached in-memory on the Solr node where the math expression was run.A cached variable can then be used in future expressions.-Any object-that can be set to a variable, including data structures and mathematical models, can-be cached in-memory for future use.+Any object that can be set to a variable, including data structures and mathematical models, can be cached in-memory for future use.The `putCache` function adds a variable to the cache.-In the example below an array is cached in the workspace `workspace1`-and bound to the key `key1`.+In the example below an array is cached in the workspace `workspace1` and bound to the key `key1`.The workspace allows different users to cache objects in their own workspace.The `putCache` function returns the variable that was added to the cache.@@ -342,10 +330,8 @@ When this expression is sent to the `/stream` handler it responds with:}-----The `removeCache` function can be used to remove a key from a specific-workspace.-The `removeCache` function removes the key from the cache-and returns the object that was removed.+The `removeCache` function can be used to remove a key from a specific workspace.+The `removeCache` function removes the key from the cache and returns the object that was removed.In the example below the array that was cached above is removed from the cache.diff --git a/solr/solr-ref-guide/src/zookeeper-access-control.adoc b/solr/solr-ref-guide/src/zookeeper-access-control.adocindex ac539d75538..c4313d79657 100644--- a/solr/solr-ref-guide/src/zookeeper-access-control.adoc+++ b/solr/solr-ref-guide/src/zookeeper-access-control.adoc@@ -100,7 +100,8 @@ You can always make you own implementation, but Solr comes with:* `org.apache.solr.common.cloud.DefaultZkACLProvider`: It returns a list of length one for all `zNodePath`-s.The single ACL entry in the list is "open-unsafe". This is the default.* `org.apache.solr.common.cloud.VMParamsAllAndReadonlyDigestZkACLProvider`: This lets you define your ACLs using system properties.-The `getACLsToAdd()` implementation will apply only admin ACLs to pre-defined sensitive paths as defined by `SecurityAwareZkACLProvider` (`/security.json` and `/security/*`) and both admin and user ACLs to the rest of the contents. The two sets of roles will be defined as:+The `getACLsToAdd()` implementation will apply only admin ACLs to pre-defined sensitive paths as defined by `SecurityAwareZkACLProvider` (`/security.json` and `/security/*`) and both admin and user ACLs to the rest of the contents.+The two sets of roles will be defined as:** A user that is allowed to do everything.*** The permission is `ALL` (corresponding to all of `CREATE`, `READ`, `WRITE`, `DELETE`, and `ADMIN`), and the schema is "digest".*** The username and password are defined by system properties `zkDigestUsername` and `zkDigestPassword`, respectively.diff --git a/solr/solr-ref-guide/src/zookeeper-ensemble.adoc b/solr/solr-ref-guide/src/zookeeper-ensemble.adocindex 316ea984cb5..0c41f5fb929 100644--- a/solr/solr-ref-guide/src/zookeeper-ensemble.adoc+++ b/solr/solr-ref-guide/src/zookeeper-ensemble.adoc@@ -83,7 +83,9 @@ After installation, we'll first take a look at the basic configuration for ZooKe=== Initial Configuration-To configure your ZooKeeper instance, create a file named `<ZOOKEEPER_HOME>/conf/zoo.cfg`. A sample configuration file is included in your ZooKeeper installation, as `conf/zoo_sample.cfg`. You can edit and rename that file instead of creating it new if you prefer.+To configure your ZooKeeper instance, create a file named `<ZOOKEEPER_HOME>/conf/zoo.cfg`.+A sample configuration file is included in your ZooKeeper installation, as `conf/zoo_sample.cfg`.+You can edit and rename that file instead of creating it new if you prefer.The file should have the following information to start:@@ -149,7 +151,8 @@ The IDs differentiate each node of the ensemble, and allow each node to know wheThe ports can be any ports you choose; ZooKeeper's default ports are `2888:3888`.+Since we've assigned server IDs to specific hosts/ports, we must also define which server in the list this node is.-We do this with a `myid` file stored in the data directory (defined by the `dataDir` parameter). The contents of the `myid` file is only the server ID.+We do this with a `myid` file stored in the data directory (defined by the `dataDir` parameter).+The contents of the `myid` file is only the server ID.+In the case of the configuration example above, you would create the file `/var/lib/zookeeper/1/myid` with the content "1" (without quotes), as in this example:+@@ -227,7 +230,8 @@ Repeat this for servers 4 and 5 if you are creating a 5-node ensemble (a rare caTo ease troubleshooting in case of problems with the ensemble later, it's recommended to run ZooKeeper with logging enabled and with proper JVM garbage collection (GC) settings.-. Create a file named `zookeeper-env.sh` and put it in the `<ZOOKEEPER_HOME>/conf` directory (the same place you put `zoo.cfg`). This file will need to exist on each server of the ensemble.+. Create a file named `zookeeper-env.sh` and put it in the `<ZOOKEEPER_HOME>/conf` directory (the same place you put `zoo.cfg`).+This file will need to exist on each server of the ensemble.. Add the following settings to the file:+diff --git a/solr/solr-ref-guide/src/zookeeper-file-management.adoc b/solr/solr-ref-guide/src/zookeeper-file-management.adocindex 47a486f4ee4..e0852fc5c84 100644--- a/solr/solr-ref-guide/src/zookeeper-file-management.adoc+++ b/solr/solr-ref-guide/src/zookeeper-file-management.adoc@@ -44,7 +44,8 @@ You can also explicitly upload a configuration directory when creating a collectbin/solr create -c mycollection -d _default-----The create command will upload a copy of the `_default` configuration directory to ZooKeeper under `/configs/mycollection`. Refer to the <<solr-control-script-reference.adoc#,Solr Control Script Reference>> page for more details about the create command for creating collections.+The create command will upload a copy of the `_default` configuration directory to ZooKeeper under `/configs/mycollection`.+Refer to the <<solr-control-script-reference.adoc#,Solr Control Script Reference>> page for more details about the create command for creating collections.Once a configuration directory has been uploaded to ZooKeeper, you can update them using the <<solr-control-script-reference.adoc#,Solr Control Script>>@@ -82,9 +83,12 @@ To update or change your SolrCloud configuration files:== Preparing ZooKeeper before First Cluster Start-If you will share the same ZooKeeper instance with other applications you should use a _chroot_ in ZooKeeper. Please see <<taking-solr-to-production.adoc#zookeeper-chroot,ZooKeeper chroot>> for instructions.+If you will share the same ZooKeeper instance with other applications you should use a _chroot_ in ZooKeeper.+Please see <<taking-solr-to-production.adoc#zookeeper-chroot,ZooKeeper chroot>> for instructions.-There are certain configuration files containing cluster wide configuration. Since some of these are crucial for the cluster to function properly, you may need to upload such files to ZooKeeper before starting your Solr cluster for the first time. Examples of such configuration files (not exhaustive) are `solr.xml`, `security.json` and `clusterprops.json`.+There are certain configuration files containing cluster wide configuration.+Since some of these are crucial for the cluster to function properly, you may need to upload such files to ZooKeeper before starting your Solr cluster for the first time.+Examples of such configuration files (not exhaustive) are `solr.xml`, `security.json` and `clusterprops.json`.If you for example would like to keep your `solr.xml` in ZooKeeper to avoid having to copy it to every node's `solr_home` directory, you can push it to ZooKeeper with the bin/solr utility (Unix example):diff --git a/solr/solr-ref-guide/src/zookeeper-utilities.adoc b/solr/solr-ref-guide/src/zookeeper-utilities.adocindex 3bf28301d8a..db3ec44d9a2 100644--- a/solr/solr-ref-guide/src/zookeeper-utilities.adoc+++ b/solr/solr-ref-guide/src/zookeeper-utilities.adoc@@ -22,7 +22,8 @@ While Solr's Admin UI includes pages dedicated to the state of your SolrCloud clTIP: See the section <<cloud-screens.adoc#,Cloud Screens>> for more information about using the Admin UI screens.-The ZooKeeper CLI script found in `server/scripts/cloud-scripts` let you upload configuration information to ZooKeeper. It also provides a few other commands that let you link collection sets to collections, make ZooKeeper paths or clear them, and download configurations from ZooKeeper to the local filesystem.+The ZooKeeper CLI script found in `server/scripts/cloud-scripts` let you upload configuration information to ZooKeeper.+It also provides a few other commands that let you link collection sets to collections, make ZooKeeper paths or clear them, and download configurations from ZooKeeper to the local filesystem.Many of the functions provided by the zkCli.sh script are also provided by the <<solr-control-script-reference.adoc#,Solr Control Script>>, which may be more familiar as the start script ZooKeeper maintenance commands are very similar to Unix commands.@@ -31,7 +32,8 @@ Many of the functions provided by the zkCli.sh script are also provided by the <====The `zkcli.sh` provided by Solr is not the same as the https://zookeeper.apache.org/doc/current/zookeeperStarted.html#sc_ConnectingToZooKeeper[`zkCli.sh` included in ZooKeeper distributions].-ZooKeeper's `zkCli.sh` provides a completely general, application-agnostic shell for manipulating data in ZooKeeper. Solr's `zkcli.sh` – discussed in this section – is specific to Solr, and has command line arguments specific to dealing with Solr data in ZooKeeper.+ZooKeeper's `zkCli.sh` provides a completely general, application-agnostic shell for manipulating data in ZooKeeper.+Solr's `zkcli.sh` – discussed in this section – is specific to Solr, and has command line arguments specific to dealing with Solr data in ZooKeeper.====== Using Solr's ZooKeeper CLI@@ -41,7 +43,9 @@ Use the `help` option to get a list of available commands from the script itselfBoth `zkcli.sh` (for Unix environments) and `zkcli.bat` (for Windows environments) support the following command line options:`-cmd <arg>`::-The CLI Command to be executed. This parameter is *mandatory*. The following commands are supported:+The CLI Command to be executed.+This parameter is *mandatory*.+The following commands are supported:* `bootstrap`* `upconfig`@@ -56,13 +60,15 @@ The CLI Command to be executed. This parameter is *mandatory*. The following com* `clusterprop``-z` or `-zkhost <locations>`::-ZooKeeper host address. This parameter is *mandatory* for all CLI commands.+ZooKeeper host address.+This parameter is *mandatory* for all CLI commands.`-c` or `-collection <name>`::For `linkconfig`: name of the collection.`-d` or `-confdir <path>`::-For `upconfig`: a directory of configuration files. For downconfig: the destination of files pulled from ZooKeeper+For `upconfig`: a directory of configuration files.+For downconfig: the destination of files pulled from ZooKeeper`-h` or `-help`::Display help text.@@ -79,7 +85,8 @@ Run ZooKeeper internally by passing the Solr run port; only for clusters on oneFor `clusterprop`: the *mandatory* cluster property name.`-val <value>`::-For `clusterprop`: the cluster property value. If not specified, *null* will be used as value.+For `clusterprop`: the cluster property value.+If not specified, *null* will be used as value.[TIP]====@@ -146,7 +153,8 @@ This can be useful to create a chroot path in ZooKeeper before first cluster sta=== Set a Cluster Property-This command will add or modify a single cluster property in `clusterprops.json`. Use this command instead of the usual getfile -> edit -> putfile cycle.+This command will add or modify a single cluster property in `clusterprops.json`.+Use this command instead of the usual getfile -> edit -> putfile cycle.Unlike the CLUSTERPROP command on the <<cluster-node-management.adoc#clusterprop,Collections API>>, this command does *not* require a running Solr cluster.