Apache Solr open-source search software
SOLR-16974: Documentation and changes entry for global CB (#2069)
Jan Høydahl2bbb06aac63e
diff --git a/solr/CHANGES.txt b/solr/CHANGES.txtindex 3bcc8d79a9b..e4bae4d67f8 100644--- a/solr/CHANGES.txt+++ b/solr/CHANGES.txt@@ -82,6 +82,8 @@ New Features* SOLR-17006: Collection creation & adding replicas: User-defined properties are persisted to state.json andapplied to new replicas, available for use as property substitution in configuration files. (Vincent Primault)+* SOLR-16974: Circuit Breakers can now be configured globally (janhoy, Christine Poerschke)+* SOLR-17079: Allow to declare replica placement plugins in solr.xml (Vincent Primault)Improvementsdiff --git a/solr/bin/solr b/solr/bin/solrindex 4c8c1393c0f..8837f858d29 100755--- a/solr/bin/solr+++ b/solr/bin/solr@@ -1426,8 +1426,6 @@ if [ -n "${SOLR_CIRCUITBREAKER_QUERY_LOADAVG:-}" ]; thenSOLR_OPTS+=("-Dsolr.circuitbreaker.query.loadavg=$SOLR_CIRCUITBREAKER_QUERY_LOADAVG")fi-echo "SOLR_OPTS is now: ${SOLR_OPTS[*]}"-: ${SOLR_SERVER_DIR:=$DEFAULT_SERVER_DIR}if [ ! -e "$SOLR_SERVER_DIR" ]; thendiff --git a/solr/solr-ref-guide/modules/deployment-guide/pages/circuit-breakers.adoc b/solr/solr-ref-guide/modules/deployment-guide/pages/circuit-breakers.adocindex c074cc64bf9..d8813dd4140 100644--- a/solr/solr-ref-guide/modules/deployment-guide/pages/circuit-breakers.adoc+++ b/solr/solr-ref-guide/modules/deployment-guide/pages/circuit-breakers.adoc@@ -32,8 +32,26 @@ Setting the `shards.tolerant=true` parameter on requests can help with gracefulcircuit breaker thresholds are reached on some nodes. See the xref:deployment-guide:solrcloud-distributed-requests.adoc#shards-tolerant-parameter[`shards.tolerant` Parameter] for details.== Circuit Breaker Configurations-All circuit breaker configurations are listed as independent `<circuitBreaker>` entries in `solrconfig.xml` as shown below.-A circuit breaker can register itself to trip for query requests and/or update requests. By default, only search requests are affected. A user may register multiple circuit breakers of the same type with different thresholds for each request type.+Circuit breakers can be configured globally for the entire node, or for each collection individually, or a combination. Per-collection circuit breakers are checked before global circuit breakers, and if there is a conflict, the per-collection circuit breaker takes precedence.+Typically, any per-collection circuit breaker thresholds are set lower than global thresholds.++A circuit breaker can register itself to be checked for query requests and/or update requests. A user may register circuit breakers of the same type with different thresholds for each request type.++=== Global Circuit Breakers+Circuit breakers can be configured globally using environment variables, e.g. in `solr.in.sh`, or system properties. The variables available are:++[options="header"]+|===+|Name |Environment Variable Name |System Property Name+|JVM Heap Usage |`SOLR_CIRCUITBREAKER_QUERY_MEM`, `SOLR_CIRCUITBREAKER_UPDATE_MEM` |`solr.circuitbreaker.query.mem`, `solr.circuitbreaker.update.mem`+|System CPU Usage |`SOLR_CIRCUITBREAKER_QUERY_CPU`, `SOLR_CIRCUITBREAKER_UPDATE_CPU` |`solr.circuitbreaker.query.cpu`, `solr.circuitbreaker.update.cpu`+|System Load Average |`SOLR_CIRCUITBREAKER_QUERY_LOADAVG`, `SOLR_CIRCUITBREAKER_UPDATE_LOADAVG` |`solr.circuitbreaker.query.loadavg`, `solr.circuitbreaker.update.loadavg`+|===++For example, you can enable a global CPU circuit breaker that rejects update requests when above 95% CPU load, by setting the following environment variable: `SOLR_CIRCUITBREAKER_UPDATE_CPU=95`.++=== Per Collection Circuit Breakers+Circuit breakers are configured as independent `<circuitBreaker>` entries in `solrconfig.xml` as shown in the below examples. By default, only search requests are affected.[TIP]====@@ -49,6 +67,7 @@ The main configuration for this circuit breaker is controlling the threshold perTo enable and configure the JVM heap usage based circuit breaker, add the following:+.Per collection in `solrconfig.xml`[source,xml]----<circuitBreaker class="org.apache.solr.util.circuitbreaker.MemoryCircuitBreaker">@@ -56,6 +75,12 @@ To enable and configure the JVM heap usage based circuit breaker, add the follow</circuitBreaker>----+.Global in `solr.in.sh`+[source,bash]+----+SOLR_CIRCUITBREAKER_QUERY_MEM=75+----+The `threshold` is defined as a percentage of the max heap allocated to the JVM.For the circuit breaker configuration, a value of "0" maps to 0% usage and a value of "100" maps to 100% usage.@@ -78,6 +103,7 @@ disabled and log an error message. An alternative can then be to use the <<systeTo enable and configure the CPU utilization based circuit breaker:+.Per collection in `solrconfig.xml`[source,xml]----<circuitBreaker class="org.apache.solr.util.circuitbreaker.CPUCircuitBreaker">@@ -85,6 +111,12 @@ To enable and configure the CPU utilization based circuit breaker:</circuitBreaker>----+.Global in `solr.in.sh`+[source,bash]+----+SOLR_CIRCUITBREAKER_QUERY_CPU=75+----+The triggering threshold is defined in percent CPU usage. A value of "0" maps to 0% usageand a value of "100" maps to 100% usage. The example above will trip when the CPU usage isequal to or greater than 75%.@@ -100,6 +132,7 @@ https://en.wikipedia.org/wiki/Load_(computing)[Wikipedia page for Load],To enable and configure the Load average circuit breaker:+.Per collection in `solrconfig.xml`[source,xml]----<circuitBreaker class="org.apache.solr.util.circuitbreaker.LoadAverageCircuitBreaker">@@ -107,6 +140,12 @@ To enable and configure the Load average circuit breaker:</circuitBreaker>----+.Global in `solr.in.sh`+[source,bash]+----+SOLR_CIRCUITBREAKER_QUERY_LOADAVG=8.0+----+The triggering threshold is a floating point number matching load average.The example circuit breaker above will trip when the load average is equal to or greater than 8.0.@@ -120,6 +159,7 @@ The System Load Average Circuit breaker behavior is dependent on the operating sIn this example we will prevent update requests above 80% CPU load, and prevent query requests above 95% CPU load. Supported request types are `query` and `update`.This would prevent expensive bulk updates from impacting search. Note also the support for short-form class name.+.Per collection in `solrconfig.xml`[source,xml]----<config>@@ -139,6 +179,13 @@ This would prevent expensive bulk updates from impacting search. Note also the s</config>----+.Global in `solr.in.sh`+[source,bash]+----+SOLR_CIRCUITBREAKER_UPDATE_CPU=80+SOLR_CIRCUITBREAKER_QUERY_CPU=95+----+== Performance ConsiderationsWhile JVM or CPU circuit breakers do not add any noticeable overhead per request, having too many circuit breakers checked for a single request can cause a performance overhead.diff --git a/solr/solr-ref-guide/modules/upgrade-notes/pages/major-changes-in-solr-9.adoc b/solr/solr-ref-guide/modules/upgrade-notes/pages/major-changes-in-solr-9.adocindex fc83e288e41..19a5feaf4b4 100644--- a/solr/solr-ref-guide/modules/upgrade-notes/pages/major-changes-in-solr-9.adoc+++ b/solr/solr-ref-guide/modules/upgrade-notes/pages/major-changes-in-solr-9.adoc@@ -71,6 +71,9 @@ Due to changes in Lucene 9, that isn't possible any more.=== Curator upgraded to 5.5.0 and requires Zookeeper 3.5.x or higher<<#solr-8-2,Solr 8.2 recommended using Zookeeper 3.5.5>> and now with Curator 5.5.0 requires https://curator.apache.org/docs/breaking-changes/[Zookeeper 3.5.x or higher]. This primarily affects users of `hadoop-auth`, but usage of Curator could affect other parts of Solr.+=== Global Circuit Breakers+* Circuit breakers can now be configured globally, not only per collection. See xref:deployment-guide:circuit-breakers.adoc[Configuring Circuit Breakers] for more information.+== Solr 9.4=== The Built-In Config Sets* The build in ConfigSets (`_default` and `sample_techproducts_configs`), now use a default `autoSoftCommit` time of 3 seconds,