Apache Solr open-source search software
SOLR-18417: Have a proper page on healthchecks (#4886)
Eric Pughef22eb5cecda
diff --git a/solr/solr-ref-guide/modules/configuration-guide/pages/implicit-requesthandlers.adoc b/solr/solr-ref-guide/modules/configuration-guide/pages/implicit-requesthandlers.adocindex c99fb342feb..6522beb7f10 100644--- a/solr/solr-ref-guide/modules/configuration-guide/pages/implicit-requesthandlers.adoc+++ b/solr/solr-ref-guide/modules/configuration-guide/pages/implicit-requesthandlers.adoc@@ -40,24 +40,8 @@ This handler must have a collection name in the path to the endpoint.|`solr/<collection>/admin/file` |{solr-javadocs}/core/org/apache/solr/handler/admin/ShowFileRequestHandler.html[ShowFileRequestHandler] |`_ADMIN_FILE`|===-Health:: Report the health of the node.-+-[cols="3*.",frame=none,grid=cols,options="header"]-|===-|API Endpoints |Class & Javadocs |Paramset-|v1: `solr/admin/info/health`--v2: `api/node/health` |v1: {solr-javadocs}/core/org/apache/solr/handler/admin/HealthCheckHandler.html[HealthCheckHandler]--v2: {solr-javadocs}/core/org/apache/solr/handler/admin/api/NodeHealth.html[NodeHealth] |-|===-+-In SolrCloud mode the handler checks that the node is connected to ZooKeeper and is listed in live nodes.-The optional `requireHealthyCores=true` parameter additionally requires that all local replicas be in an active state, which is useful for rolling-restart probes.-+-In user-managed (leader-follower) mode the handler checks replication lag.-The optional `maxGenerationLag=<n>` parameter specifies the maximum number of Lucene commit generations by which a follower is allowed to trail its leader; the endpoint returns HTTP 503 if any core exceeds this threshold.-See xref:deployment-guide:user-managed-index-replication.adoc#monitoring-follower-replication-lag[Monitoring Follower Replication Lag] for details and examples.+Health:: Report the health of the node, via the `api/node/health` endpoint ({solr-javadocs}/core/org/apache/solr/handler/admin/api/NodeHealth.html[NodeHealth]).+See xref:deployment-guide:healthcheck.adoc[] for details, parameters, and examples.Logging:: Retrieve and modify registered loggers.+diff --git a/solr/solr-ref-guide/modules/deployment-guide/deployment-nav.adoc b/solr/solr-ref-guide/modules/deployment-guide/deployment-nav.adocindex 83bb2e286ed..9b81acaf3e7 100644--- a/solr/solr-ref-guide/modules/deployment-guide/deployment-nav.adoc+++ b/solr/solr-ref-guide/modules/deployment-guide/deployment-nav.adoc@@ -58,6 +58,7 @@* Monitoring Solr** xref:configuring-logging.adoc[]** xref:ping.adoc[]+** xref:healthcheck.adoc[]** xref:metrics-reporting.adoc[]** xref:performance-statistics-reference.adoc[]** xref:plugins-stats-screen.adoc[]diff --git a/solr/solr-ref-guide/modules/deployment-guide/pages/healthcheck.adoc b/solr/solr-ref-guide/modules/deployment-guide/pages/healthcheck.adocnew file mode 100644index 00000000000..7e5bc472214--- /dev/null+++ b/solr/solr-ref-guide/modules/deployment-guide/pages/healthcheck.adoc@@ -0,0 +1,169 @@+= Healthchecks+// Licensed to the Apache Software Foundation (ASF) under one+// or more contributor license agreements. See the NOTICE file+// distributed with this work for additional information+// regarding copyright ownership. The ASF licenses this file+// to you under the Apache License, Version 2.0 (the+// "License"); you may not use this file except in compliance+// with the License. You may obtain a copy of the License at+//+// http://www.apache.org/licenses/LICENSE-2.0+//+// Unless required by applicable law or agreed to in writing,+// software distributed under the License is distributed on an+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY+// KIND, either express or implied. See the License for the+// specific language governing permissions and limitations+// under the License.++Solr provides two distinct tools for checking whether things are working, and they answer different questions.+Confusing them is easy, since both are casually called a "healthcheck", so this page describes each and helps you pick the right one.++[cols="1,2,2"]+|===+| |Node Health Endpoint |`bin/solr healthcheck` command++|Answers+|"Is this node alive and part of the cluster?"+|"Does this collection actually have data, and is every replica in sync?"++|Scope+|A single node+|A single collection, across all its shards and replicas++|Checks the index/data?+|No -- metadata only+|Yes -- runs real queries++|Interface+|HTTP endpoint+|Command-line tool++|Typical use+|Load balancer / orchestrator liveness or readiness probe+|Manual or scripted diagnosis after deployment, restart, or an incident+|===++== Node Healthcheck++The `api/node/health` endpoint ({solr-javadocs}/core/org/apache/solr/handler/admin/api/NodeHealth.html[NodeHealth]) reports whether a single Solr node is alive and able to participate in the cluster.+It is designed to be cheap enough to poll frequently from a load balancer or an orchestrator like Kubernetes.++[IMPORTANT]+====+None of these checks touch the index.+A node with a collection that has zero documents, or a collection whose query results are wrong, will still report a healthy `200 OK` here.+If you need to confirm a collection actually has data, see <<Collection Data Healthcheck>> below.+====++What the endpoint checks -- and how it fails -- depends on whether the node is running in SolrCloud mode or user-managed (leader-follower) mode.++=== SolrCloud Mode++In SolrCloud mode, the endpoint returns HTTP `200 OK` only if all of the following are true, and HTTP `503 Unavailable` otherwise:++* The node's core container has finished starting up.+* The node is connected to ZooKeeper.+* The node is listed in ZooKeeper's `live_nodes`.++[source,bash]+----+curl "http://localhost:8983/api/node/health"+----++Healthy node:++[source,json]+----+{+ "responseHeader":{"status":0,"QTime":1},+ "status":"OK"+}+----++Unhealthy node (HTTP 503):++[source,json]+----+{+ "responseHeader":{"status":503,"QTime":1},+ "status":"FAILURE",+ "error":{"msg":"Not connected to ZK"}+}+----++==== Rolling Restarts: `requireHealthyCores`++Add `requireHealthyCores=true` to additionally require that every local replica belonging to an active shard has finished initializing -- i.e., none are in the `RECOVERING` or `DOWN` state.+This is useful as a readiness probe during a rolling restart, so an orchestrator doesn't move on to the next node while the one it just restarted still has replicas recovering.++[source,bash]+----+curl "http://localhost:8983/api/node/health?requireHealthyCores=true"+----++=== User-Managed (Leader-Follower) Mode++In user-managed replication, the endpoint instead checks how far behind each local follower core is from its leader, in Lucene commit generations.+Set `maxGenerationLag=<n>` to fail the healthcheck once a follower falls more than `<n>` generations behind; without it, the check simply reports `OK` once a follower has replicated at least once, even if it later falls arbitrarily far behind.++[source,bash]+----+curl "http://localhost:8983/api/node/health?maxGenerationLag=100"+----++Healthy node (all followers within the allowed lag):++[source,json]+----+{+ "responseHeader":{"status":0,"QTime":1},+ "status":"OK"+}+----++Unhealthy node (a follower has fallen too far behind, HTTP 503):++[source,json]+----+{+ "responseHeader":{"status":503,"QTime":1},+ "status":"FAILURE",+ "error":{"msg":"Cores violating maxGenerationLag:100.\nCore collection1 is lagging by 137 generations"}+}+----++See xref:user-managed-index-replication.adoc#monitoring-follower-replication-lag[Monitoring Follower Replication Lag] for details.++== Collection Healthcheck++While the Node Health endpoint checks process liveness, the `bin/solr healthcheck` tool verifies whether a specific collection is functioning properly. It queries every replica directly (non-distributed) to compare document counts, confirm each shard has a leader, and verify all replicas are in the ACTIVE state.++Because it executes real queries against individual replicas, it is a heavier operation than the Node Health endpoint and is meant as an on-demand CLI tool rather than a continuously polled HTTP healthcheck for load balancers. Use it after deployments, restarts, or incidents to confirm collection integrity.+[source,bash]+----+bin/solr healthcheck -c gettingstarted+----++[source,json]+----+{+ "collection":"gettingstarted",+ "status":"healthy",+ "numDocs":42,+ "numShards":2,+ "shards":[+ {+ "shard":"shard1",+ "status":"healthy",+ "replicas":[+ {"name":"core_node1", "url":"...", "numDocs":21, "status":"active", "leader":true}+ ]+ }+ ]+}+----++TIP: Healthy, well-functioning replicas often disagree on exact document counts due to slightly staggered autoCommit periods or in-flight updates. Small, momentary count discrepancies between replicas are normal and should not automatically be treated as an issue unless the mismatch persists.++See xref:solr-control-script-reference.adoc#healthcheck[the `healthcheck` command reference] for the full list of options.diff --git a/solr/solr-ref-guide/modules/deployment-guide/pages/taking-solr-to-production.adoc b/solr/solr-ref-guide/modules/deployment-guide/pages/taking-solr-to-production.adocindex e91be0fd62c..f3574a4b443 100644--- a/solr/solr-ref-guide/modules/deployment-guide/pages/taking-solr-to-production.adoc+++ b/solr/solr-ref-guide/modules/deployment-guide/pages/taking-solr-to-production.adoc@@ -205,6 +205,8 @@ If the `status` command is not successful, look for error messages in `/var/solrAdditionally, if you can't connect to running Solr from an external location, check your `SOLR_HOST_BIND` setting to make sure Solr is accessible to the network.+Once Solr is running, see xref:healthcheck.adoc[] for how to wire up a load balancer or orchestrator health probe, and how to confirm a collection is actually serving data rather than just that the node is up.+== Fine-Tune Your Production Setup=== Dynamic Defaults for ConcurrentMergeScheduler