Apache Solr open-source search software
SOLR-14444: add required/default tables to all params, part III (final) (#217)
Cassandra Targett5118648da76e
diff --git a/solr/solr-ref-guide/src/stats-component.adoc b/solr/solr-ref-guide/src/stats-component.adocindex 8dea0fd3345..371e0dc68e7 100644--- a/solr/solr-ref-guide/src/stats-component.adoc+++ b/solr/solr-ref-guide/src/stats-component.adoc@@ -30,13 +30,25 @@ bin/solr -e techproductsThe Stats Component accepts the following parameters:`stats`::+++[%autowidth,frame=none]+|===+|Optional |Default: `false`+|===++If `true`, then invokes the Stats component.`stats.field`::+++[%autowidth,frame=none]+|===+s|Required |Default: none+|===++Specifies a field for which statistics should be generated.This parameter may be invoked multiple times in a query in order to request statistics on multiple fields.+-<<local-params.adoc#,Local Params>> may be used to indicate which subset of the supported statistics should be computed, and/or that statistics should be computed over the results of an arbitrary numeric function (or query) instead of a simple field name.+<<local-params.adoc#,Local Params>> may be used to indicate a subset of the supported statistics should be computed, and/or that statistics should be computed over the results of an arbitrary numeric function (or query) instead of a simple field name.See the examples below.diff --git a/solr/solr-ref-guide/src/suggester.adoc b/solr/solr-ref-guide/src/suggester.adocindex ddcdafbe97e..f3ca4c59d3b 100644--- a/solr/solr-ref-guide/src/suggester.adoc+++ b/solr/solr-ref-guide/src/suggester.adoc@@ -66,32 +66,74 @@ Below are the main parameters that can be used no matter what lookup or dictionaIn the following sections additional parameters are provided for each implementation.`searchComponent name`::-Arbitrary name for the search component.+++[%autowidth,frame=none]+|===+s|Required |Default: none+|===+++Arbitrary name for the overall search component definition.++`searchComponent class`::+++[%autowidth,frame=none]+|===+s|Required |Default: none+|===+++The Suggester class.+This should be defined as `solr.SuggestComponent`.`name`::+++[%autowidth,frame=none]+|===+s|Required |Default: none+|===++A symbolic name for this suggester.You can refer to this name in the URL parameters and in the SearchHandler configuration.It is possible to have multiples of these in one `solrconfig.xml` file.+In the example configuration above, this is the name referenced in the line: `<str name="name">mySuggester</str>`.`lookupImpl`::+++[%autowidth,frame=none]+|===+|Optional |Default: `JaspellLookupFactory`+|===++Lookup implementation.-There are several possible implementations, described below in the section <<Lookup Implementations>>. If not set, the default lookup is `JaspellLookupFactory`.+There are several possible implementations, described in the section <<Lookup Implementations>>.`dictionaryImpl`::+++[%autowidth,frame=none]+|===+|Optional |Default: _see description_+|===++The dictionary implementation to use.-There are several possible implementations, described below in the section <<Dictionary Implementations>>.+There are several possible implementations, described in the section <<Dictionary Implementations>>.+If not set, the default dictionary implementation is `HighFrequencyDictionaryFactory`.However, if a `sourceLocation` is used, the dictionary implementation will be `FileDictionaryFactory`.`field`::+++[%autowidth,frame=none]+|===+|Optional |Default: none+|===++A field from the index to use as the basis of suggestion terms.If `sourceLocation` is empty (meaning any dictionary implementation other than `FileDictionaryFactory`), then terms from this field in the index will be used.+To be used as the basis for a suggestion, the field must be stored.You may want to <<copy-fields.adoc#,use copyField rules>> to create a special 'suggest' field comprised of terms from other fields in documents.-In any event, you very likely want a minimal amount of analysis on the field, so an additional option is to create a field type in your schema that only uses basic tokenizers or filters.-One option for such a field type is shown here:+++You usually want a minimal amount of analysis on the field (no stemming, no synonyms, etc.), so an option is to create a field type in your schema that only uses basic tokenizers or filters.+An example of such a field type is shown here:+[source,xml]----@@ -104,29 +146,55 @@ One option for such a field type is shown here:----+However, this minimal analysis is not required if you want more analysis to occur on terms.++If using the `AnalyzingLookupFactory` as your `lookupImpl`, however, you have the option of defining the field type rules to use for index and query time analysis.`sourceLocation`::+++[%autowidth,frame=none]+|===+|Optional |Default: _see description_+|===++The path to the dictionary file if using the `FileDictionaryFactory`.-If this value is empty then the main index will be used as a source of terms and weights.+If this value is empty, the main index will be used as a source of terms and weights.`storeDir`::+++[%autowidth,frame=none]+|===+|Optional |Default: none+|===++The location to store the dictionary file.`buildOnCommit` and `buildOnOptimize`::+++[%autowidth,frame=none]+|===+|Optional |Default: `false`+|===++If `true`, the lookup data structure will be rebuilt after soft-commit.-If `false`, the default, then the lookup data will be built only when requested by URL parameter `suggest.build=true`.-Use `buildOnCommit` to rebuild the dictionary with every soft-commit, or `buildOnOptimize` to build the dictionary only when the index is optimized.+If `false`,then the lookup data will be built only when requested by URL parameter `suggest.build=true`.+Use `buildOnCommit` to rebuild the dictionary with every soft commit, or `buildOnOptimize` to build the dictionary only when the index is optimized.+Some lookup implementations may take a long time to build, especially with large indexes.-In such cases, using `buildOnCommit` or `buildOnOptimize`, particularly with a high frequency of softCommits is not recommended; it's recommended instead to build the suggester at a lower frequency by manually issuing requests with `suggest.build=true`.+In such cases, using `buildOnCommit` or `buildOnOptimize`, particularly with a high frequency of soft commits is not recommended.+Instead build the suggester at a lower frequency by manually issuing requests with `suggest.build=true`.`buildOnStartup`::+++[%autowidth,frame=none]+|===+|Optional |Default: `false`+|===++If `true,` then the lookup data structure will be built when Solr starts or when the core is reloaded.If this parameter is not specified, the suggester will check if the lookup data structure is present on disk and build it if not found.+-Enabling this to `true` could lead to the core talking longer to load (or reload) as the suggester data structure needs to be built, which can sometimes take a long time.-It’s usually preferred to have this setting set to `false`, the default, and build suggesters manually issuing requests with `suggest.build=true`.+Enabling this to `true` could lead to Solr taking longer to load (or reload) cores as the suggester data structure is built, which can sometimes take a long time.+It’s usually preferred to leave this set to `false` and build suggesters manually with `suggest.build=true`.=== Lookup Implementations@@ -140,19 +208,42 @@ A lookup that first analyzes the incoming text and adds the analyzed form to a wThis implementation uses the following additional properties:`suggestAnalyzerFieldType`::+++[%autowidth,frame=none]+|===+s|Required |Default: none+|===++The field type to use for the query-time and build-time term suggestion analysis.`exactMatchFirst`::-If `true`, the default, exact suggestions are returned first, even if they are prefixes or other strings in the FST have larger weights.+++[%autowidth,frame=none]+|===+|Optional |Default: `true`+|===+++If `true`, exact suggestions are returned first, even if they are prefixes or other strings in the FST have larger weights.`preserveSep`::-If `true`, the default, then a separator between tokens is preserved.+++[%autowidth,frame=none]+|===+|Optional |Default: `true`+|===+++If `true`, then a separator between tokens is preserved.This means that suggestions are sensitive to tokenization (e.g., baseball is different from base ball).`preservePositionIncrements`::+++[%autowidth,frame=none]+|===+|Optional |Default: `false`+|===++If `true`, the suggester will preserve position increments.This means that token filters which leave gaps (for example, when StopFilter matches a stopword) the position would be respected when building the suggester.-The default is `false`.==== FuzzyLookupFactory@@ -162,45 +253,99 @@ The similarity is measured by the Levenshtein algorithm.This implementation uses the following additional properties:`exactMatchFirst`::-If `true`, the default, exact suggestions are returned first, even if they are prefixes or other strings in the FST have larger weights.+++[%autowidth,frame=none]+|===+|Optional |Default: `true`+|===+++If `true`, exact suggestions are returned first, even if they are prefixes or other strings in the FST have larger weights.`preserveSep`::-If `true`, the default, then a separator between tokens is preserved.-This means that suggestions are sensitive to tokenization (e.g., baseball is different from base ball).+++[%autowidth,frame=none]+|===+|Optional |Default: `true`+|===+++If `true`, then a separator between tokens is preserved.+This means that suggestions are sensitive to tokenization (e.g., "baseball" is different from "base ball").`maxSurfaceFormsPerAnalyzedForm`::+++[%autowidth,frame=none]+|===+|Optional |Default: `256`+|===++The maximum number of surface forms to keep for a single analyzed form.When there are too many surface forms we discard the lowest weighted ones.`maxGraphExpansions`::+++[%autowidth,frame=none]+|===+|Optional |Default: `-1`+|===++When building the FST ("index-time"), we add each path through the tokenstream graph as an individual entry.This places an upper-bound on how many expansions will be added for a single suggestion.-The default is `-1` which means there is no limit.`preservePositionIncrements`::+++[%autowidth,frame=none]+|===+|Optional |Default: `false`+|===++If `true`, the suggester will preserve position increments.This means that token filters which leave gaps (for example, when StopFilter matches a stopword) the position would be respected when building the suggester.-The default is `false`.`maxEdits`::+++[%autowidth,frame=none]+|===+|Optional |Default: `1`+|===++The maximum number of string edits allowed.-The system's hard limit is `2`.-The default is `1`.+Solr's hard limit is `2`.`transpositions`::-If `true`, the default, transpositions should be treated as a primitive edit operation.+++[%autowidth,frame=none]+|===+|Optional |Default: `true`+|===+++If `true`, transpositions should be treated as a primitive edit operation.`nonFuzzyPrefix`::+++[%autowidth,frame=none]+|===+|Optional |Default: `1`+|===++The length of the common non fuzzy prefix match which must match a suggestion.-The default is `1`.`minFuzzyLength`::+++[%autowidth,frame=none]+|===+|Optional |Default: `3`+|===++The minimum length of query before which any string edits will be allowed.-The default is `3`.`unicodeAware`::+++[%autowidth,frame=none]+|===+|Optional |Default: `false`+|===++If `true`, the `maxEdits`, `minFuzzyLength`, `transpositions` and `nonFuzzyPrefix` parameters will be measured in unicode code points (actual letters) instead of bytes.-The default is `false`.==== AnalyzingInfixLookupFactory@@ -210,20 +355,42 @@ This uses a Lucene index for its dictionary.This implementation uses the following additional properties.`indexPath`::+++[%autowidth,frame=none]+|===+|Optional |Default: _see description_+|===++When using `AnalyzingInfixSuggester` you can provide your own path where the index will get built.The default is `analyzingInfixSuggesterIndexDir` and will be created in your collection's `data/` directory.`minPrefixChars`::-Minimum number of leading characters before PrefixQuery is used (default is `4`).+++[%autowidth,frame=none]+|===+|Optional |Default: `4`+|===+++Minimum number of leading characters before PrefixQuery is used.Prefixes shorter than this are indexed as character ngrams (increasing index size but making lookups faster).`allTermsRequired`::-Boolean option for multiple terms.-The default is `true`, all terms will be required.+++[%autowidth,frame=none]+|===+|Optional |Default: `true`+|===+++If `true`, all terms will be required.`highlight`::+++[%autowidth,frame=none]+|===+|Optional |Default: `true`+|===++Highlight suggest terms.-Default is `true`.This implementation supports <<Context Filtering>>.@@ -235,31 +402,61 @@ It scores higher if a hit is closer to the start of the suggestion.This implementation uses the following additional properties:`blenderType`::+++[%autowidth,frame=none]+|===+|Optional |Default: `position_linear`+|===++Used to calculate weight coefficient using the position of the first matching word.Available options are:-`position_linear`:::-`weightFieldValue * (1 - 0.10*position)`: Matches to the start will be given a higher score.-This is the default.-`position_reciprocal`:::-`weightFieldValue / (1 + position)`: Matches to the start will be given a higher score.++* `position_linear`: Matches to the start will be given a higher score.+++`weightFieldValue * (1 - 0.10*position)`++* `position_reciprocal`: Matches to the start will be given a higher score.The score of matches positioned far from the start of the suggestion decays faster than linear.-`position_exponential_reciprocal`:::-`weightFieldValue / pow(1 + position,exponent)`: Matches to the start will be given a higher score.+++`weightFieldValue / (1 + position)`++* `position_exponential_reciprocal`: Matches to the start will be given a higher score.The score of matches positioned far from the start of the suggestion decays faster than reciprocal.-`exponent`::::-An optional configuration variable for `position_exponential_reciprocal` to control how fast the score will decrease.-Default `2.0`.+++`weightFieldValue / pow(1 + position,exponent)`+++When using this blender type, an additional parameter is available:++** `exponent`: Controls how fast the score will decrease.+The default `2.0`.`numFactor`::+++[%autowidth,frame=none]+|===+|Optional |Default: `10`+|===++The factor to multiply the number of searched elements from which results will be pruned.-Default is `10`.`indexPath`::+++[%autowidth,frame=none]+|===+|Optional |Default: _see description_+|===++When using `BlendedInfixSuggester` you can provide your own path where the index will get built.The default directory name is `blendedInfixSuggesterIndexDir` and will be created in your collection's data directory.`minPrefixChars`::-Minimum number of leading characters before PrefixQuery is used (the default is `4`).+++[%autowidth,frame=none]+|===+|Optional |Default: `4`+|===+++Minimum number of leading characters before PrefixQuery is used.Prefixes shorter than this are indexed as character ngrams, which increases index size but makes lookups faster.This implementation supports <<Context Filtering>>.@@ -273,12 +470,22 @@ This suggester would only be used as a fallback, when the primary suggester failThis implementation uses the following additional properties:`suggestFreeTextAnalyzerFieldType`::-The analyzer used at "query-time" and "build-time" to analyze suggestions.-This parameter is required.+++[%autowidth,frame=none]+|===+s|Required |Default: none+|===+++The field type used at "query-time" and "build-time" to analyze suggestions.`ngrams`::+++[%autowidth,frame=none]+|===+|Optional |Default: `2`+|===++The max number of tokens out of which singles will be made the dictionary.-The default value is `2`.Increasing this would mean you want more than the previous 2 tokens to be taken into consideration when making the suggestions.==== FSTLookupFactory@@ -290,9 +497,21 @@ We recommend using this implementation unless you need more sophisticated matchiThis implementation uses the following additional properties:`exactMatchFirst`::+++[%autowidth,frame=none]+|===+|Optional |Default: `true`+|===++If `true`, the default, exact suggestions are returned first, even if they are prefixes or other strings in the FST have larger weights.`weightBuckets`::+++[%autowidth,frame=none]+|===+|Optional |Default: none+|===++The number of separate buckets for weights which the suggester will use while building its dictionary.==== TSTLookupFactory@@ -325,15 +544,31 @@ A dictionary with terms, weights, and an optional payload taken from the index.This dictionary implementation takes the following parameters in addition to parameters described for the Suggester generally and for the lookup implementation:`weightField`::+++[%autowidth,frame=none]+|===+|Optional |Default: none+|===++A field that is stored or a numeric DocValue field.-This parameter is optional.`payloadField`::+++[%autowidth,frame=none]+|===+|Optional |Default: none+|===++The `payloadField` should be a field that is stored.-This parameter is optional.`contextField`::-Field to be used for context filtering.+++[%autowidth,frame=none]+|===+|Optional |Default: none+|===+++Field to be used for <<Context Filtering>>.Note that only some lookup implementations support filtering.==== DocumentExpressionDictionaryFactory@@ -343,16 +578,32 @@ This dictionary implementation is the same as the `DocumentDictionaryFactory` buThis dictionary implementation takes the following parameters in addition to parameters described for the Suggester generally and for the lookup implementation:`payloadField`::+++[%autowidth,frame=none]+|===+|Optional |Default: none+|===++The `payloadField` should be a field that is stored.-This parameter is optional.`weightExpression`::+++[%autowidth,frame=none]+|===+s|Required |Default: none+|===++An arbitrary expression used for scoring the suggestions.The fields used must be numeric fields.-This parameter is required.`contextField`::-Field to be used for context filtering.+++[%autowidth,frame=none]+|===+|Optional |Default: none+|===+++Field to be used for <<Context Filtering>>.Note that only some lookup implementations support filtering.==== HighFrequencyDictionaryFactory@@ -362,7 +613,13 @@ This dictionary implementation allows adding a threshold to prune out less frequThis dictionary implementation takes one parameter in addition to parameters described for the Suggester generally and for the lookup implementation:`threshold`::-A value between zero and one representing the minimum fraction of the total documents where a term should appear in order to be added to the lookup dictionary.+++[%autowidth,frame=none]+|===+|Optional |Default: `0`+|===+++A value between `0` and `1` representing the minimum fraction of the total documents where a term should appear in order to be added to the lookup dictionary.==== FileDictionaryFactory@@ -371,12 +628,18 @@ Weights and payloads can also be used.If using a dictionary file, it should be a plain text file in UTF-8 encoding.You can use both single terms and phrases in the dictionary file.-If adding weights or payloads, those should be separated from terms using the delimiter defined with the `fieldDelimiter` property (the default is '\t', the tab representation).+If adding weights or payloads, those should be separated from terms using the delimiter defined with the `fieldDelimiter` property (the default is `\t`, the tab representation).If using payloads, the first line in the file *must* specify a payload.This dictionary implementation takes one parameter in addition to parameters described for the Suggester generally and for the lookup implementation:`fieldDelimiter`::+++[%autowidth,frame=none]+|===+|Optional |Default: `\t`+|===++Specifies the delimiter to be used separating the entries, weights and payloads.The default is tab (`\t`).+@@ -446,46 +709,96 @@ The request handler definition must incorporate the "suggest" search component dThe following parameters allow you to set defaults for the Suggest request handler:-`suggest=true`::+`suggest`::+++[%autowidth,frame=none]+|===+|Optional |Default: `false`+|===++This parameter should always be `true`, because we always want to run the Suggester for queries submitted to this handler.`suggest.dictionary`::+++[%autowidth,frame=none]+|===+s|Required |Default: none+|===++The name of the dictionary component configured in the search component.-This is a mandatory parameter.It can be set in the request handler, or sent as a parameter at query time.`suggest.q`::+++[%autowidth,frame=none]+|===+|Optional |Default: none+|===++The query to use for suggestion lookups.+If not provided, the `q` parameter is used.`suggest.count`::+++[%autowidth,frame=none]+|===+|Optional |Default: `1`+|===++Specifies the number of suggestions for Solr to return.`suggest.cfq`::-A Context Filter Query used to filter suggestions based on the context field, if supported by the suggester.+++[%autowidth,frame=none]+|===+|Optional |Default: none+|===+++A <<Context Filtering, context filter>> query used to filter suggestions based on the context field, if supported by the suggester.+++Context filtering is currently only supported by `AnalyzingInfixLookupFactory` and `BlendedInfixLookupFactory`, and only when backed by a `Document*Dictionary`.+All other implementations will return unfiltered matches as if filtering was not requested.`suggest.build`::+++[%autowidth,frame=none]+|===+|Optional |Default: `false`+|===++If `true`, it will build the suggester index.This is likely useful only for initial requests; you would probably not want to build the dictionary on every request, particularly in a production system.If you would like to keep your dictionary up to date, you should use the `buildOnCommit` or `buildOnOptimize` parameter for the search component.`suggest.reload`::+++[%autowidth,frame=none]+|===+|Optional |Default: `false`+|===++If `true`, it will reload the suggester index.`suggest.buildAll`::+++[%autowidth,frame=none]+|===+|Optional |Default: `false`+|===++If `true`, it will build all suggester indexes.`suggest.reloadAll`::+++[%autowidth,frame=none]+|===+|Optional |Default: `false`+|===++If `true`, it will reload all suggester indexes.These properties can also be overridden at query time, or not set in the request handler at all and always sent at query time.-.Context Filtering-[IMPORTANT]-====-Context filtering (`suggest.cfq`) is currently only supported by `AnalyzingInfixLookupFactory` and `BlendedInfixLookupFactory`, and only when backed by a `Document*Dictionary`.-All other implementations will return unfiltered matches as if filtering was not requested.-====-== Example Usages=== Get Suggestions with Weightsdiff --git a/solr/solr-ref-guide/src/tagger-handler.adoc b/solr/solr-ref-guide/src/tagger-handler.adocindex b739e79dd71..94060308f3b 100644--- a/solr/solr-ref-guide/src/tagger-handler.adoc+++ b/solr/solr-ref-guide/src/tagger-handler.adoc@@ -67,23 +67,53 @@ The tagger's execution is completely configurable with request parameters.Only `field` is required.`field`::+++[%autowidth,frame=none]+|===+s|Required |Default: none+|===++The tag field that serves as the dictionary.This is required; you'll probably specify it in the request handler.`fq`::+++[%autowidth,frame=none]+|===+|Optional |Default: none+|===++You can specify some number of _filter queries_ to limit the dictionary used for tagging.This parameter is the same one used by the `solr.SearchHandler`.`rows`::-The maximum number of documents to return, but defaulting to 10000 for a tag request.+++[%autowidth,frame=none]+|===+|Optional |Default: `10000`+|===+++The maximum number of documents to return.This parameter is the same as is used by the `solr.SearchHandler`.`fl`::+++[%autowidth,frame=none]+|===+|Optional |Default: none+|===++Solr's standard parameter for listing the fields to return.This parameter is the same one used by the `solr.SearchHandler`.`overlaps`::-Choose the algorithm to determine which tags in an overlapping set should be retained, versus being pruned away.+++[%autowidth,frame=none]+|===+|Optional |Default: `NO_SUB`+|===+++Choose the algorithm to determine which tags in an overlapping set should be retained versus being pruned away.Options are:* `ALL`: Emit all tags.@@ -93,31 +123,55 @@ If there is a tie, pick the right-most.Remove any tags overlapping with this tag then repeat the algorithm to potentially find other tags that can be emitted in the cluster.`matchText`::-A boolean indicating whether to return the matched text in the tag response.+++[%autowidth,frame=none]+|===+|Optional |Default: `false`+|===+++If `true`, return the matched text in the tag response.This will trigger the tagger to fully buffer the input before tagging.`tagsLimit`::+++[%autowidth,frame=none]+|===+|Optional |Default: `1000`+|===++The maximum number of tags to return in the response.Tagging effectively stops after this point.-By default this is `1000`.`skipAltTokens`::-A boolean flag used to suppress errors that can occur if, for example,- you enable synonym expansion at query time in the analyzer, which you normally shouldn't do.-Let this default to false unless you know that such tokens can't be avoided.+++[%autowidth,frame=none]+|===+|Optional |Default: `false`+|===+++If `true`, suppress errors that can occur if, for example, you enable synonym expansion at query time in the analyzer, which you normally shouldn't do.+Let this default to `false` unless you know that such tokens can't be avoided.`ignoreStopwords`::-A boolean flag that causes stopwords (or any condition causing positions to skip like >255 char words)-to be ignored as if they aren't there.-Otherwise, the behavior is to treat them as breaks in tagging on the presumption your indexed text-analysis- configuration doesn't have a `StopWordFilter` defined.-By default the indexed analysis chain is checked for the presence of a `StopWordFilter` and if found- then `ignoreStopWords` is true if unspecified.+++[%autowidth,frame=none]+|===+|Optional |Default: `false`+|===+++A boolean flag that causes stopwords (or any condition causing positions to skip like >255 char words) to be ignored as if they aren't there.+Otherwise, the behavior is to treat them as breaks in tagging on the presumption your indexed text-analysis configuration doesn't have a `StopWordFilter` defined.+By default the indexed analysis chain is checked for the presence of a `StopWordFilter` and if found then `ignoreStopWords` is true if unspecified.You probably shouldn't have a `StopWordFilter` configured and probably won't need to set this parameter either.`xmlOffsetAdjust`::-A boolean indicating that the input is XML and furthermore that the offsets of returned tags should be adjusted as- necessary to allow for the client to insert an opening and closing element at the tag offset pair.+++[%autowidth,frame=none]+|===+|Optional |Default: `false`+|===+++If `true`, indicates the input is XML and the offsets of returned tags should be adjusted as necessary to allow for the client to insert an opening and closing element at the tag offset pair.If it isn't possible to do so then the tag will be omitted.You are expected to configure `HTMLStripCharFilterFactory` in the schema when using this option.This will trigger the tagger to fully buffer the input before tagging.diff --git a/solr/solr-ref-guide/src/term-vector-component.adoc b/solr/solr-ref-guide/src/term-vector-component.adocindex c1afcd2d8e3..dfed603085d 100644--- a/solr/solr-ref-guide/src/term-vector-component.adoc+++ b/solr/solr-ref-guide/src/term-vector-component.adoc@@ -132,35 +132,95 @@ The example below shows some of the available request parameters for this componhttp://localhost:8983/solr/techproducts/tvrh?q=includes:[* TO *]&rows=10&indent=true&tv=true&tv.tf=true&tv.df=true&tv.positions=true&tv.offsets=true&tv.payloads=true&tv.fl=includes`tv`::+++[%autowidth,frame=none]+|===+|Optional |Default: `false`+|===++If `true`, the Term Vector Component will run.`tv.docIds`::+++[%autowidth,frame=none]+|===+|Optional |Default: none+|===++For a given comma-separated list of Lucene document IDs (*not* the Solr Unique Key), term vectors will be returned.`tv.fl`::+++[%autowidth,frame=none]+|===+|Optional |Default: none+|===++For a given comma-separated list of fields, term vectors will be returned.If not specified, the `fl` parameter is used.`tv.all`::+++[%autowidth,frame=none]+|===+|Optional |Default: `false`+|===++If `true`, all the boolean parameters listed below (`tv.df`, `tv.offsets`, `tv.positions`, `tv.payloads`, `tv.tf` and `tv.tf_idf`) will be enabled.`tv.df`::-If `true`, returns the Document Frequency (DF) of the term in the collection.+++[%autowidth,frame=none]+|===+|Optional |Default: `false`+|===+++If `true`, returns the document frequency (DF) of the term in the collection.This can be computationally expensive.`tv.offsets`::+++[%autowidth,frame=none]+|===+|Optional |Default: `false`+|===++If `true`, returns offset information for each term in the document.`tv.positions`::+++[%autowidth,frame=none]+|===+|Optional |Default: `false`+|===++If `true`, returns position information.`tv.payloads`::+++[%autowidth,frame=none]+|===+|Optional |Default: `false`+|===++If `true`, returns payload information.`tv.tf`::+++[%autowidth,frame=none]+|===+|Optional |Default: `false`+|===++If `true`, returns document term frequency info for each term in the document.`tv.tf_idf`::+++[%autowidth,frame=none]+|===+|Optional |Default: `false`+|===++If `true`, calculates TF / DF (i.e.,: TF * IDF) for each term.Please note that this is a _literal_ calculation of "Term Frequency multiplied by Inverse Document Frequency" and *not* a classical TF-IDF similarity measure.+diff --git a/solr/solr-ref-guide/src/term-vectors.adoc b/solr/solr-ref-guide/src/term-vectors.adocindex 9136c8cf1be..4a7c348556c 100644--- a/solr/solr-ref-guide/src/term-vectors.adoc+++ b/solr/solr-ref-guide/src/term-vectors.adoc@@ -248,14 +248,38 @@ This is also important because eliminating the noisy terms helps keep the term vThere are four parameters designed to filter noisy terms from the term vector matrix:`minTermLength`::+++[%autowidth,frame=none]+|===+|Optional |Default: `3`+|===++The minimum term length required to include the term in the matrix.`minDocFreq`::-The minimum percentage, expressed as a number between 0 and 1, of documents the term must appear in to be included in the index.+++[%autowidth,frame=none]+|===+|Optional |Default: `.05`+|===+++The minimum percentage, expressed as a number between `0` and `1`, of documents the term must appear in to be included in the index.`maxDocFreq`::-The maximum percentage, expressed as a number between 0 and 1, of documents the term can appear in to be included in the index.+++[%autowidth,frame=none]+|===+|Optional |Default: `.5`+|===+++The maximum percentage, expressed as a number between `0` and `1`, of documents the term can appear in to be included in the index.`exclude`::+++[%autowidth,frame=none]+|===+|Optional |Default: none+|===++A comma-delimited list of strings used to exclude terms.If a term contains any of the excluded strings that term will be excluded from the term vector.diff --git a/solr/solr-ref-guide/src/terms-component.adoc b/solr/solr-ref-guide/src/terms-component.adocindex 412134df48c..897891c9114 100644--- a/solr/solr-ref-guide/src/terms-component.adoc+++ b/solr/solr-ref-guide/src/terms-component.adoc@@ -49,21 +49,38 @@ Or, you can add them to the query request.These parameters are:`terms`::+++[%autowidth,frame=none]+|===+|Optional |Default: `false`+|===++If set to `true`, enables the Terms Component.-By default, the Terms Component is off (`false`).+Example: `terms=true``terms.fl`::+++[%autowidth,frame=none]+|===+s|Required |Default: none+|===++Specifies the field from which to retrieve terms.This parameter is required if `terms=true`.+Example: `terms.fl=title``terms.list`::+++[%autowidth,frame=none]+|===+|Optional |Default: none+|===++Fetches the document frequency for a comma-delimited list of terms.Terms are always returned in index order.-If `terms.ttf` is set to true, also returns their total term frequency.+If `terms.ttf` is set to `true`, also returns their total term frequency.If multiple `terms.fl` are defined, these statistics will be returned for each term in each requested field.+Example: `terms.list=termA,termB,termC`@@ -72,54 +89,107 @@ NOTE: When `terms.list` is specified, then terms are always sorted by `index`.Except `terms.ttf`, none of other terms parameters are supported when `terms.list` is specified.`terms.limit`::+++[%autowidth,frame=none]+|===+|Optional |Default: `10`+|===++Specifies the maximum number of terms to return.-The default is `10`.-If the limit is set to a number less than 0, then no maximum limit is enforced.+If the limit is set to a number less than `0`, then no maximum limit is enforced.Although this is not required, either this parameter or `terms.upper` must be defined.+Example: `terms.limit=20``terms.lower`::+++[%autowidth,frame=none]+|===+|Optional |Default: _see description_+|===++Specifies the term at which to start.If not specified, the empty string is used, causing Solr to start at the beginning of the field.+Example: `terms.lower=orange``terms.lower.incl`::-If set to true, includes the lower-bound term (specified with `terms.lower` in the result set.+++[%autowidth,frame=none]+|===+|Optional |Default: `true`+|===+++If set to `true`, includes the lower-bound term (specified with `terms.lower` in the result set.+Example: `terms.lower.incl=false``terms.mincount`::+++[%autowidth,frame=none]+|===+|Optional |Default: `1`+|===++Specifies the minimum document frequency to return in order for a term to be included in a query response.Results are inclusive of the mincount (that is, >= mincount).+Example: `terms.mincount=5``terms.maxcount`::+++[%autowidth,frame=none]+|===+|Optional |Default: `-1`+|===++Specifies the maximum document frequency a term must have in order to be included in a query response.-The default setting is -1, which sets no upper bound.+The default setting is `-1`, which sets no upper bound.Results are inclusive of the maxcount (that is, \<= maxcount).+Example: `terms.maxcount=25``terms.prefix`::+++[%autowidth,frame=none]+|===+|Optional |Default: none+|===++Restricts matches to terms that begin with the specified string.+Example: `terms.prefix=inter``terms.raw`::-If set to true, returns the raw characters of the indexed term, regardless of whether it is human-readable.+++[%autowidth,frame=none]+|===+|Optional |Default: `false`+|===+++If set to `true`, returns the raw characters of the indexed term, regardless of whether it is human-readable.For instance, the indexed form of numeric numbers is not human-readable.+Example: `terms.raw=true``terms.regex`::+++[%autowidth,frame=none]+|===+|Optional |Default: none+|===++Restricts matches to terms that match the regular expression.+Example: `terms.regex=.*pedist``terms.regex.flag`::+++[%autowidth,frame=none]+|===+|Optional |Default: none+|===++Defines a Java regex flag to use when evaluating the regular expression defined with `terms.regex`.See http://docs.oracle.com/javase/tutorial/essential/regex/pattern.html for details of each flag.Valid options are:@@ -136,22 +206,40 @@ Valid options are:Example: `terms.regex.flag=case_insensitive``terms.stats`::-Include index statistics in the results.-Currently returns only the *numDocs* for a collection.+++[%autowidth,frame=none]+|===+|Optional |Default: `false`+|===+++If `true`, include index statistics in the results.+Currently returns only the number of documents for a collection.When combined with `terms.list` it provides enough information to compute inverse document frequency (IDF) for a list of terms.`terms.sort`::+++[%autowidth,frame=none]+|===+|Optional |Default: `count`+|===++Defines how to sort the terms returned.Valid options are `count`, which sorts by the term frequency, with the highest term frequency first, or `index`, which sorts in index order.+Example: `terms.sort=index``terms.ttf`::-If set to true, returns both `df` (docFreq) and `ttf` (totalTermFreq) statistics for each requested term in `terms.list`.+++[%autowidth,frame=none]+|===+|Optional |Default: `false`+|===+++If set to `true`, returns both `df` (docFreq) and `ttf` (totalTermFreq) statistics for each requested term in `terms.list`.In this case, the response format is:-++XML:-++[source,xml]----<lst name="terms">@@ -163,9 +251,9 @@ XML:</lst></lst>-----++JSON:-++[source,json]----{@@ -182,14 +270,25 @@ JSON:----`terms.upper`::+++[%autowidth,frame=none]+|===+|Optional |Default: none+|===++Specifies the term to stop at.Although this parameter is not required, either this parameter or `terms.limit` must be defined.+Example: `terms.upper=plum``terms.upper.incl`::+++[%autowidth,frame=none]+|===+|Optional |Default: `false`+|===++If set to true, the upper bound term is included in the result set.-The default is false.+Example: `terms.upper.incl=true`@@ -267,7 +366,7 @@ Results:</response>-----=== Using Terms Component as part of another handler+=== Using Terms Component in a Request HandlerThis query augments a regular search with terms information.diff --git a/solr/solr-ref-guide/src/tokenizers.adoc b/solr/solr-ref-guide/src/tokenizers.adocindex 2fb30d44337..f8bf47e5d09 100644--- a/solr/solr-ref-guide/src/tokenizers.adoc+++ b/solr/solr-ref-guide/src/tokenizers.adoc@@ -868,10 +868,16 @@ Note that any punctuation _will_ be included in the tokens.*Arguments:*`rule`::+++[%autowidth,frame=none]+|===+|Optional |Default: `java`+|===++Specifies how to define whitespace for the purpose of tokenization.Valid values:-* `java`: (Default) Uses {java-javadocs}java/lang/Character.html#isWhitespace-int-[Character.isWhitespace(int)]+* `java`: Uses {java-javadocs}java/lang/Character.html#isWhitespace-int-[Character.isWhitespace(int)]* `unicode`: Uses Unicode's WHITESPACE property*Example:*diff --git a/solr/solr-ref-guide/src/transforming-and-indexing-custom-json.adoc b/solr/solr-ref-guide/src/transforming-and-indexing-custom-json.adocindex 193aa9d9476..eef7373f1b6 100644--- a/solr/solr-ref-guide/src/transforming-and-indexing-custom-json.adoc+++ b/solr/solr-ref-guide/src/transforming-and-indexing-custom-json.adoc@@ -26,6 +26,12 @@ One or more valid JSON documents can be sent to the `/update/json/docs` path witThese parameters allow you to define how a JSON file should be read for multiple Solr documents.`split`::+++[%autowidth,frame=none]+|===+|Optional |Default: none+|===++Defines the path at which to split the input JSON into multiple Solr documents and is required if you have multiple documents in a single JSON file.If the entire JSON makes a single Solr document, the path must be “`/`”.+@@ -33,6 +39,12 @@ It is possible to pass multiple `split` paths by separating them with a pipe `(|If one path is a child of another, they automatically become a child document.`f`::+++[%autowidth,frame=none]+|===+s|Required |Default: none+|===++Provides multivalued mapping to map document field names to Solr field names.The format of the parameter is `target-field-name:json-path`, as in `f=first:/first`.The `json-path` is required.@@ -43,20 +55,44 @@ The default target field name is the fully qualified name of the field.Wildcards can be used here, see <<Using Wildcards for Field Names>> below for more information.`mapUniqueKeyOnly`::-(boolean) This parameter is particularly convenient when the fields in the input JSON are not available in the schema and <<schemaless-mode.adoc#,schemaless mode>> is not enabled.-This will index all the fields into the default search field (using the `df` parameter, below) and only the `uniqueKey` field is mapped to the corresponding field in the schema.+++[%autowidth,frame=none]+|===+|Optional |Default: `false`+|===+++This parameter is particularly convenient when the fields in the input JSON are not available in the schema and <<schemaless-mode.adoc#,schemaless mode>> is not enabled.+This will index all the fields into the default search field (using the `df` parameter) and only the `uniqueKey` field is mapped to the corresponding field in the schema.If the input JSON does not have a value for the `uniqueKey` field then a UUID is generated for the same.`df`::-If the `mapUniqueKeyOnly` flag is used, the update handler needs a field where the data should be indexed to.+++[%autowidth,frame=none]+|===+|Optional |Default: none+|===+++If the `mapUniqueKeyOnly` flag is enabled, the update handler needs a field where the data should be indexed to.This is the same field that other handlers use as a default search field.`srcField`::+++[%autowidth,frame=none]+|===+|Optional |Default: none+|===++This is the name of the field to which the JSON source will be stored into.This can only be used if `split=/` (i.e., you want your JSON input file to be indexed as a single Solr document).Note that atomic updates will cause the field to be out-of-sync with the document.`echo`::+++[%autowidth,frame=none]+|===+|Optional |Default: `false`+|===++This is for debugging purpose only.Set it to `true` if you want the docs to be returned as a response.Nothing will be indexed.diff --git a/solr/solr-ref-guide/src/zookeeper-ensemble.adoc b/solr/solr-ref-guide/src/zookeeper-ensemble.adocindex 0c41f5fb929..63b668d3c21 100644--- a/solr/solr-ref-guide/src/zookeeper-ensemble.adoc+++ b/solr/solr-ref-guide/src/zookeeper-ensemble.adoc@@ -99,15 +99,44 @@ clientPort=2181The parameters are as follows:-`tickTime`:: Part of what ZooKeeper does is determine which servers are up and running at any given time, and the minimum session time out is defined as two "ticks". The `tickTime` parameter specifies in milliseconds how long each tick should be.+`tickTime`::+++[%autowidth,frame=none]+|===+s|Required |Default: none+|===+++Part of what ZooKeeper does is determine which servers are up and running at any given time, and the minimum session timeout is defined as two "ticks".+The `tickTime` parameter specifies in milliseconds how long each tick should be.-`dataDir`:: This is the directory in which ZooKeeper will store data about the cluster.+`dataDir`::+++[%autowidth,frame=none]+|===+s|Required |Default: none+|===+++This is the directory in which ZooKeeper will store data about the cluster.This directory must be empty before starting ZooKeeper for the first time.-`clientPort`:: This is the port on which Solr will access ZooKeeper.+`clientPort`::+++[%autowidth,frame=none]+|===+s|Required |Default: none+|===+++This is the port on which Solr will access ZooKeeper.-`4lw.commands.whitelist`:: This allows the Solr admin UI to query ZooKeeper.-Optionally use "*" to enable all "4 letter words", the three listed will enable the admin UI.+`4lw.commands.whitelist`::+++[%autowidth,frame=none]+|===+|Optional |Default: none+|===+++This allows the Solr admin UI to query ZooKeeper.+Optionally use `*` to enable all "4 letter words", the three listed (`mntr`, `conf`, `ruok`) will enable the admin UI.These are the basic parameters that need to be in use on each ZooKeeper node, so this file must be copied to or created on each node.@@ -140,13 +169,34 @@ autopurge.purgeInterval=1We've added these parameters to the three we had already:-`initLimit`:: Amount of time, in ticks, to allow followers to connect and sync to a leader.-In this case, you have 5 ticks, each of which is 2000 milliseconds long, so the server will wait as long as 10 seconds to connect and sync with the leader.+`initLimit`::+++[%autowidth,frame=none]+|===+s|Required |Default: none+|===+++Amount of time, in ticks, to allow followers to connect and sync to a leader.+In this case, you have `5` ticks, each of which is 2000 milliseconds long, so the server will wait as long as `10` seconds to connect and sync with the leader.-`syncLimit`:: Amount of time, in ticks, to allow followers to sync with ZooKeeper.+`syncLimit`::+++[%autowidth,frame=none]+|===+s|Required |Default: none+|===+++Amount of time, in ticks, to allow followers to sync with ZooKeeper.If followers fall too far behind a leader, they will be dropped.-`server._X_`:: These are the server IDs (the `_X_` part), hostnames (or IP addresses) and ports for all servers in the ensemble.+`server._X_`::+++[%autowidth,frame=none]+|===+s|Required |Default: none+|===+++These are the server IDs (the `_X_` part), hostnames (or IP addresses) and ports for all servers in the ensemble.The IDs differentiate each node of the ensemble, and allow each node to know where each of the other node is located.The ports can be any ports you choose; ZooKeeper's default ports are `2888:3888`.+@@ -159,17 +209,31 @@ In the case of the configuration example above, you would create the file `/var/[source,bash]1-`autopurge.snapRetainCount`:: The number of snapshots and corresponding transaction logs to retain when purging old snapshots and transaction logs.+`autopurge.snapRetainCount`::+++[%autowidth,frame=none]+|===+|Optional |Default: `3`+|===+++The number of snapshots and corresponding transaction logs to retain when purging old snapshots and transaction logs.+ZooKeeper automatically keeps a transaction log and writes to it as changes are made.A snapshot of the current state is taken periodically, and this snapshot supersedes transaction logs older than the snapshot.However, ZooKeeper never cleans up either the old snapshots or the old transaction logs; over time they will silently fill available disk space on each server.+To avoid this, set the `autopurge.snapRetainCount` and `autopurge.purgeInterval` parameters to enable an automatic clean up (purge) to occur at regular intervals.-The `autopurge.snapRetainCount` parameter will keep the set number of snapshots and transaction logs when a clean up occurs.+The `autopurge.snapRetainCount` parameter will keep the defined number of snapshots and transaction logs when a clean up occurs.This parameter can be configured higher than `3`, but cannot be set lower than 3.-`autopurge.purgeInterval`:: The time in hours between purge tasks.+`autopurge.purgeInterval`::+++[%autowidth,frame=none]+|===+|Optional |Default: `0`+|===+++The time in hours between purge tasks.The default for this parameter is `0`, so must be set to `1` or higher to enable automatic clean up of snapshots and transaction logs.Setting it as high as `24`, for once a day, is acceptable if preferred.