Apache Solr open-source search software
SOLR-15428: Integrate the OpenJDK JMH micro benchmark framework for micro benchmarks and performance comparisons & investigation. (#214)
Mark Robert Miller890ef78421f8
diff --git a/.gitignore b/.gitignoreindex 67d83ced896..4ff1ee02594 100644--- a/.gitignore+++ b/.gitignore@@ -20,6 +20,8 @@ out/# Eclipse Gradle oddity**/bin/default/+.editorconfig+# macOS.DS_Storediff --git a/settings.gradle b/settings.gradleindex 918696f7426..bd79fa1a248 100644--- a/settings.gradle+++ b/settings.gradle@@ -34,6 +34,7 @@ if (!(System.getProperty("skip.solr") as Boolean)) {include "solr:contrib:ltr"include "solr:contrib:gcs-repository"include "solr:webapp"+ include "solr:benchmark"include "solr:test-framework"include "solr:solr-ref-guide"include "solr:example"diff --git a/solr/CHANGES.txt b/solr/CHANGES.txtindex 9570c17f959..a1b3526ece4 100644--- a/solr/CHANGES.txt+++ b/solr/CHANGES.txt@@ -331,6 +331,9 @@ Other Changes* SOLR-15309: Add missing IntelliJ IDEA entries to the .gitignore file (Pushkar Raste via Anshum Gupta)+* SOLR-15428: Integrate the OpenJDK JMH micro benchmark framework for micro benchmarks and performance comparisons and investigation.+ (Mark Miller)+Bug Fixes---------------------* SOLR-14546: Fix for a relatively hard to hit issue in OverseerTaskProcessor that could lead to out of order executiondiff --git a/solr/benchmark/.gitignore b/solr/benchmark/.gitignorenew file mode 100644index 00000000000..101cafeb9f4--- /dev/null+++ b/solr/benchmark/.gitignore@@ -0,0 +1,3 @@+/work/+/lib/+jmh-result.json\ No newline at end of filediff --git a/solr/benchmark/README.md b/solr/benchmark/README.mdnew file mode 100644index 00000000000..81e063354be--- /dev/null+++ b/solr/benchmark/README.md@@ -0,0 +1,159 @@+### JMH-Benchmarks module++This module contains benchmarks written using [JMH](https://openjdk.java.net/projects/code-tools/jmh/) from OpenJDK.+Writing correct micro-benchmarks in Java (or another JVM language) is difficult and there are many non-obvious pitfalls (many+due to compiler optimizations). JMH is a framework for running and analyzing benchmarks (micro or macro) written in Java (or+another JVM language).++### Running benchmarks++If you want to set specific JMH flags or only run certain benchmarks, passing arguments via+gradle tasks is cumbersome. The process has been simplified by the provided `jmh.sh` script.++The default behavior is to run all benchmarks:++ ./jmh.sh++Pass a pattern or name after the command to select the benchmarks:++ ./jmh.sh CloudIndexing++Check which benchmarks match the provided pattern:++ ./jmh.sh -l CloudIndexing++Run a specific test and override the number of forks, iterations and warm-up iteration to `2`:++ ./jmh.sh -f 2 -i 2 -wi 2 CloudIndexing++Run a specific test with async and GC profilers on Linux and flame graph output:++ ./jmh.sh -prof gc -prof async:libPath=/path/to/libasyncProfiler.so\;output=flamegraph\;dir=profile-results CloudIndexing++### Using JMH with async profiler++It's good practice to check profiler output for micro-benchmarks in order to verify that they represent the expected+application behavior and measure what you expect to measure. Some example pitfalls include the use of expensive mocks+or accidental inclusion of test setup code in the benchmarked code. JMH includes+[async-profiler](https://github.com/jvm-profiling-tools/async-profiler) integration that makes this easy:++ ./jmh.sh -prof async:libPath=/path/to/libasyncProfiler.so\;dir=profile-results++With flame graph output:++ ./jmh.sh -prof async:libPath=/path/to/libasyncProfiler.so\;output=flamegraph\;dir=profile-results++Simultaneous cpu, allocation and lock profiling with async profiler 2.0 and jfr output:++ ./jmh.sh -prof async:libPath=/path/to/libasyncProfiler.so\;output=jfr\;alloc\;lock\;dir=profile-results CloudIndexing++A number of arguments can be passed to configure async profiler, run the following for a description:++ ./jmh.sh -prof async:help++You can also skip specifying libPath if you place the async profiler lib in a predefined location such as /usr/java/packages/lib++### Using JMH GC profiler++You can run a benchmark with `-prof gc` to measure its allocation rate:++ ./jmh.sh -prof gc:dir=profile-results++Of particular importance is the `norm` alloc rates, which measure the allocations per operation rather than allocations per second.++### Using JMH Java Flight Recorder profiler++JMH comes with a variety of built-in profilers. Here is an example of using JFR:++ ./jmh.sh -prof jfr:dir=profile-results\;configName=jfr-profile.jfc++In this example we point to the included configuration file with configName, but you could also do something like settings=default or settings=profile.++### Benchmark Repeatability++Indexes created for the benchmarks often involve randomness when generating terms, term length and number of terms in a field. In order to make benchmarks repeatable, a static seed is used for randoms. This allows for generating varying data while ensuring that data is consistent across runs.++You can vary that seed by setting a system property to explore a wider range of variation in the benchmark:++ -jvmArgsAppend -Dsolr.bench.seed=6624420638116043983++The seed used for a given benchmark run will be printed out near the top of the output.++ --> benchmark random seed: 6624420638116043983++You can also specify where to place the mini-cluster with a system property:++ -jvmArgsAppend -DminiClusterBaseDir=/benchmark-data/mini-cluster++In this case, new data will not be generated for the benchmark, even if you change parameters. The use case for this if you are running a query based benchmark and want to create a large index for testing and reuse (say hundreds of GB's). Be aware that with this system property set, that same mini-cluster will be reused for any benchmarks run, regardless of if that makes sense or not.++### Writing benchmarks++For help in writing correct JMH tests, the best place to start is the [sample code](https://hg.openjdk.java.net/code-tools/jmh/file/tip/jmh-samples/src/main/java/org/openjdk/jmh/samples/) provided+by the JMH project.++JMH is highly configurable and users are encouraged to look through the samples for suggestions+on what options are available. A good tutorial for using JMH can be found [here](http://tutorials.jenkov.com/java-performance/jmh.html#return-value-from-benchmark-method)++Many Solr JMH benchmarks are actually closer to a full integration benchmark in that they run a single action against a full Solr mini cluster.++See org.apache.solr.bench.index.CloudIndexing for an example of this.++#### MiniCluster Metrics++After every iteration, the metrics collected by Solr will be dumped to the build/work/metrics-results folder. You can disable metrics collection using the metricsEnabled method of the MiniClusterState, in which case the same output files will be dumped, but the values will all be 0/null.++### JMH Options+Some common JMH options are:++```text++ -e <regexp+> Benchmarks to exclude from the run.++ -f <int> How many times to fork a single benchmark. Use 0 to+ disable forking altogether. Warning: disabling+ forking may have detrimental impact on benchmark+ and infrastructure reliability, you might want+ to use different warmup mode instead.++ -i <int> Number of measurement iterations to do. Measurement+ iterations are counted towards the benchmark score.+ (default: 1 for SingleShotTime, and 5 for all other+ modes)++ -l List the benchmarks that match a filter, and exit.++ -lprof List profilers, and exit.++ -o <filename> Redirect human-readable output to a given file.++ -prof <profiler> Use profilers to collect additional benchmark data.+ Some profilers are not available on all JVMs and/or+ all OSes. Please see the list of available profilers+ with -lprof.++ -v <mode> Verbosity mode. Available modes are: [SILENT, NORMAL,+ EXTRA]++ -wi <int> Number of warmup iterations to do. Warmup iterations+ are not counted towards the benchmark score. (default:+ 0 for SingleShotTime, and 5 for all other modes)++ -t <int> Number of worker threads to run with. 'max' means+ the maximum number of hardware threads available+ on the machine, figured out by JMH itself.++ -rf <type> Format type for machine-readable results. These+ results are written to a separate file (see -rff).+ See the list of available result formats with -lrf.++ -rff <filename> Write machine-readable results to a given file.+ The file format is controlled by -rf option. Please+ see the list of result formats for available formats.++ -bm <mode> Benchmark mode. Available modes are: [Throughput/thrpt,+ AverageTime/avgt, SampleTime/sample, SingleShotTime/ss,+ All/all]+```++To view all options run jmh with the -h flag.\ No newline at end of filediff --git a/solr/benchmark/build.gradle b/solr/benchmark/build.gradlenew file mode 100644index 00000000000..50bfd1bdba3--- /dev/null+++ b/solr/benchmark/build.gradle@@ -0,0 +1,75 @@+/*+ * Licensed to the Apache Software Foundation (ASF) under one or more+ * contributor license agreements. See the NOTICE file distributed with+ * this work for additional information regarding copyright ownership.+ * The ASF licenses this file to You under the Apache License, Version 2.0+ * (the "License"); you may not use this file except in compliance with+ * the License. You may obtain a copy of the License at+ *+ * http://www.apache.org/licenses/LICENSE-2.0+ *+ * Unless required by applicable law or agreed to in writing, software+ * distributed under the License is distributed on an "AS IS" BASIS,+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.+ * See the License for the specific language governing permissions and+ * limitations under the License.+ */++plugins {+ id "io.morethan.jmhreport" version "0.9.0"+ // id 'com.github.johnrengelman.shadow' version '6.1.0'+}++// generally, you use or have the option to use jmh with a 'fat' jar - currently there+// are some issues to track down to allow that, but this sets up for it+//apply plugin: 'com.github.johnrengelman.shadow'++apply plugin: 'java-library'++description = 'Solr Benchmark Framework'++//shadowJar {+// archiveBaseName = 'solr-jmh-benchmarks'+//}++jmhReport {+ jmhResultPath = project.file('build/reports/jmh/result.json')+ jmhReportOutput = project.file('build/reports/jmh')+}++compileJava {+ doFirst {+ options.compilerArgs.remove("-Werror")+ options.compilerArgs.remove("-proc:none")+ options.compilerArgs.add("-Xlint:none")++ }+}++// forbiddenApis scans code that JMH generates+forbiddenApis {+ ignoreFailures = true+}++task echoCp {+ doLast {+ print toPath((sourceSets.main.runtimeClasspath + sourceSets.main.resources).files)+ }+}++private static String toPath(Set<File> classpathUnderTest) {+ StringBuilder sb = new StringBuilder();+ for (File entry : classpathUnderTest) {+ sb.append(entry.getAbsolutePath());+ sb.append(File.pathSeparatorChar);+ }+ return sb.toString();+}++dependencies {+ implementation project(':solr:test-framework')++ implementation 'org.openjdk.jmh:jmh-core'+ annotationProcessor 'org.openjdk.jmh:jmh-generator-annprocess'++}diff --git a/solr/benchmark/jfr-profile.jfc b/solr/benchmark/jfr-profile.jfcnew file mode 100644index 00000000000..56d1fef7908--- /dev/null+++ b/solr/benchmark/jfr-profile.jfc@@ -0,0 +1,793 @@+<?xml version="1.0" encoding="UTF-8"?>+<configuration version="2.0" label="jfr" description="none" provider="Oracle">++ <category label="Java Development Kit">++ <category label="Security">++ <event name="jdk.SecurityPropertyModification" label="Security Property Modification" description="Modification of Security property">+ <setting name="enabled" label="Enabled" description="Record event" contentType="jdk.jfr.Flag">false</setting>+ <setting name="stackTrace" label="Stack Trace" description="Record stack traces" contentType="jdk.jfr.Flag">true</setting>+ <setting name="threshold" label="Threshold" description="Record event with duration above or equal to threshold" contentType="jdk.jfr.Timespan">0 ns</setting>+ </event>++ <event name="jdk.X509Validation" label="X509 Validation" description="Serial numbers from X.509 Certificates forming chain of trust">+ <setting name="enabled" label="Enabled" description="Record event" contentType="jdk.jfr.Flag">false</setting>+ <setting name="stackTrace" label="Stack Trace" description="Record stack traces" contentType="jdk.jfr.Flag">true</setting>+ <setting name="threshold" label="Threshold" description="Record event with duration above or equal to threshold" contentType="jdk.jfr.Timespan">0 ns</setting>+ </event>++ <event name="jdk.X509Certificate" label="X509 Certificate" description="Details of X.509 Certificate parsed by JDK">+ <setting name="enabled" label="Enabled" description="Record event" contentType="jdk.jfr.Flag">false</setting>+ <setting name="stackTrace" label="Stack Trace" description="Record stack traces" contentType="jdk.jfr.Flag">true</setting>+ <setting name="threshold" label="Threshold" description="Record event with duration above or equal to threshold" contentType="jdk.jfr.Timespan">0 ns</setting>+ </event>++ <event name="jdk.TLSHandshake" label="TLS Handshake" description="Parameters used in TLS Handshake">+ <setting name="enabled" label="Enabled" description="Record event" contentType="jdk.jfr.Flag">false</setting>+ <setting name="stackTrace" label="Stack Trace" description="Record stack traces" contentType="jdk.jfr.Flag">true</setting>+ <setting name="threshold" label="Threshold" description="Record event with duration above or equal to threshold" contentType="jdk.jfr.Timespan">0 ns</setting>+ </event>++ </category>++ </category>++ <category label="Java Virtual Machine">++ <category label="GC">++ <category label="Detailed">++ <event name="jdk.ObjectCountAfterGC" label="Object Count after GC">+ <setting name="enabled" label="Enabled" description="Record event" contentType="jdk.jfr.Flag">false</setting>+ </event>++ <event name="jdk.G1HeapRegionTypeChange" label="G1 Heap Region Type Change" description="Information about a G1 heap region type change">+ <setting name="enabled" label="Enabled" description="Record event" contentType="jdk.jfr.Flag">true</setting>+ </event>++ <event name="jdk.AllocationRequiringGC" label="Allocation Requiring GC">+ <setting name="enabled" label="Enabled" description="Record event" contentType="jdk.jfr.Flag">true</setting>+ <setting name="stackTrace" label="Stack Trace" description="Record stack traces" contentType="jdk.jfr.Flag">true</setting>+ </event>++ <event name="jdk.EvacuationInformation" label="Evacuation Information">+ <setting name="enabled" label="Enabled" description="Record event" contentType="jdk.jfr.Flag">true</setting>+ </event>++ <event name="jdk.G1BasicIHOP" label="G1 Basic IHOP Statistics" description="Basic statistics related to current IHOP calculation">+ <setting name="enabled" label="Enabled" description="Record event" contentType="jdk.jfr.Flag">true</setting>+ </event>++ <event name="jdk.ZStatisticsSampler" label="Z Statistics Sampler">+ <setting name="enabled" label="Enabled" description="Record event" contentType="jdk.jfr.Flag">true</setting>+ <setting name="threshold" label="Threshold" description="Record event with duration above or equal to threshold" contentType="jdk.jfr.Timespan">10 ms</setting>+ </event>++ <event name="jdk.ZStatisticsCounter" label="Z Statistics Counter">+ <setting name="threshold" label="Threshold" description="Record event with duration above or equal to threshold" contentType="jdk.jfr.Timespan">10 ms</setting>+ <setting name="enabled" label="Enabled" description="Record event" contentType="jdk.jfr.Flag">true</setting>+ </event>++ <event name="jdk.ZPageAllocation" label="ZPage Allocation" description="Allocation of a ZPage">+ <setting name="enabled" label="Enabled" description="Record event" contentType="jdk.jfr.Flag">true</setting>+ <setting name="threshold" label="Threshold" description="Record event with duration above or equal to threshold" contentType="jdk.jfr.Timespan">10 ms</setting>+ </event>++ <event name="jdk.ObjectCount" label="Object Count">+ <setting name="enabled" label="Enabled" description="Record event" contentType="jdk.jfr.Flag">true</setting>+ <setting name="period" label="Period" description="Record event at interval" contentType="jdk.jfr.Period">everyChunk</setting>+ </event>++ <event name="jdk.G1EvacuationOldStatistics" label="G1 Evacuation Memory Statistics for Old" description="Memory related evacuation statistics during GC for the old generation">+ <setting name="enabled" label="Enabled" description="Record event" contentType="jdk.jfr.Flag">true</setting>+ </event>++ <event name="jdk.ConcurrentModeFailure" label="Concurrent Mode Failure" description="Concurrent Mode failed">+ <setting name="enabled" label="Enabled" description="Record event" contentType="jdk.jfr.Flag">true</setting>+ </event>++ <event name="jdk.ZThreadPhase" label="ZGC Thread Phase">+ <setting name="enabled" label="Enabled" description="Record event" contentType="jdk.jfr.Flag">true</setting>+ <setting name="threshold" label="Threshold" description="Record event with duration above or equal to threshold" contentType="jdk.jfr.Timespan">0 ms</setting>+ </event>++ <event name="jdk.PromoteObjectInNewPLAB" label="Promotion in new PLAB" description="Object survived scavenge and was copied to a new Promotion Local Allocation Buffer (PLAB). Supported GCs are Parallel Scavange, G1 and CMS with Parallel New. Due to promotion being done in parallel an object might be reported multiple times as the GC threads race to copy all objects.">+ <setting name="enabled" label="Enabled" description="Record event" contentType="jdk.jfr.Flag">true</setting>+ </event>++ <event name="jdk.G1HeapRegionInformation" label="G1 Heap Region Information" description="Information about a specific heap region in the G1 GC">+ <setting name="enabled" label="Enabled" description="Record event" contentType="jdk.jfr.Flag">true</setting>+ <setting name="period" label="Period" description="Record event at interval" contentType="jdk.jfr.Period">everyChunk</setting>+ </event>++ <event name="jdk.ShenandoahHeapRegionInformation" label="Shenandoah Heap Region Information" description="Information about a specific heap region in the Shenandoah GC">+ <setting name="enabled" label="Enabled" description="Record event" contentType="jdk.jfr.Flag">true</setting>+ <setting name="period" label="Period" description="Record event at interval" contentType="jdk.jfr.Period">everyChunk</setting>+ </event>++ <event name="jdk.PromoteObjectOutsidePLAB" label="Promotion outside PLAB" description="Object survived scavenge and was copied directly to the heap. Supported GCs are Parallel Scavange, G1 and CMS with Parallel New. Due to promotion being done in parallel an object might be reported multiple times as the GC threads race to copy all objects.">+ <setting name="enabled" label="Enabled" description="Record event" contentType="jdk.jfr.Flag">true</setting>+ </event>++ <event name="jdk.PromotionFailed" label="Promotion Failed" description="Promotion of an object failed">+ <setting name="enabled" label="Enabled" description="Record event" contentType="jdk.jfr.Flag">true</setting>+ </event>++ <event name="jdk.ShenandoahHeapRegionStateChange" label="Shenandoah Heap Region State Change" description="Information about a Shenandoah heap region state change">+ <setting name="enabled" label="Enabled" description="Record event" contentType="jdk.jfr.Flag">true</setting>+ </event>++ <event name="jdk.G1AdaptiveIHOP" label="G1 Adaptive IHOP Statistics" description="Statistics related to current adaptive IHOP calculation">+ <setting name="enabled" label="Enabled" description="Record event" contentType="jdk.jfr.Flag">true</setting>+ </event>++ <event name="jdk.G1EvacuationYoungStatistics" label="G1 Evacuation Statistics for Young" description="Memory related evacuation statistics during GC for the young generation">+ <setting name="enabled" label="Enabled" description="Record event" contentType="jdk.jfr.Flag">true</setting>+ </event>++ <event name="jdk.TenuringDistribution" label="Tenuring Distribution">+ <setting name="enabled" label="Enabled" description="Record event" contentType="jdk.jfr.Flag">true</setting>+ </event>++ <event name="jdk.G1MMU" label="G1 MMU Information">+ <setting name="enabled" label="Enabled" description="Record event" contentType="jdk.jfr.Flag">true</setting>+ </event>++ <event name="jdk.EvacuationFailed" label="Evacuation Failed" description="Evacuation of an object failed">+ <setting name="enabled" label="Enabled" description="Record event" contentType="jdk.jfr.Flag">true</setting>+ </event>++ </category>++ <category label="Collector">++ <event name="jdk.GarbageCollection" label="Garbage Collection" description="Garbage collection performed by the JVM">+ <setting name="enabled" label="Enabled" description="Record event" contentType="jdk.jfr.Flag">true</setting>+ <setting name="threshold" label="Threshold" description="Record event with duration above or equal to threshold" contentType="jdk.jfr.Timespan">0 ms</setting>+ </event>++ <event name="jdk.G1GarbageCollection" label="G1 Garbage Collection" description="Extra information specific to G1 Garbage Collections">+ <setting name="enabled" label="Enabled" description="Record event" contentType="jdk.jfr.Flag">true</setting>+ <setting name="threshold" label="Threshold" description="Record event with duration above or equal to threshold" contentType="jdk.jfr.Timespan">0 ms</setting>+ </event>++ <event name="jdk.OldGarbageCollection" label="Old Garbage Collection" description="Extra information specific to Old Garbage Collections">+ <setting name="enabled" label="Enabled" description="Record event" contentType="jdk.jfr.Flag">true</setting>+ <setting name="threshold" label="Threshold" description="Record event with duration above or equal to threshold" contentType="jdk.jfr.Timespan">0 ms</setting>+ </event>++ <event name="jdk.YoungGarbageCollection" label="Young Garbage Collection" description="Extra information specific to Young Garbage Collections">+ <setting name="enabled" label="Enabled" description="Record event" contentType="jdk.jfr.Flag">true</setting>+ <setting name="threshold" label="Threshold" description="Record event with duration above or equal to threshold" contentType="jdk.jfr.Timespan">0 ms</setting>+ </event>++ <event name="jdk.ParallelOldGarbageCollection" label="Parallel Old Garbage Collection" description="Extra information specific to Parallel Old Garbage Collections">+ <setting name="enabled" label="Enabled" description="Record event" contentType="jdk.jfr.Flag">true</setting>+ <setting name="threshold" label="Threshold" description="Record event with duration above or equal to threshold" contentType="jdk.jfr.Timespan">0 ms</setting>+ </event>++ </category>++ <category label="Heap">++ <event name="jdk.GCHeapSummary" label="Heap Summary">+ <setting name="enabled" label="Enabled" description="Record event" contentType="jdk.jfr.Flag">true</setting>+ </event>++ <event name="jdk.G1HeapSummary" label="G1 Heap Summary">+ <setting name="enabled" label="Enabled" description="Record event" contentType="jdk.jfr.Flag">true</setting>+ </event>++ <event name="jdk.MetaspaceSummary" label="Metaspace Summary">+ <setting name="enabled" label="Enabled" description="Record event" contentType="jdk.jfr.Flag">true</setting>+ </event>++ <event name="jdk.PSHeapSummary" label="Parallel Scavenge Heap Summary">+ <setting name="enabled" label="Enabled" description="Record event" contentType="jdk.jfr.Flag">true</setting>+ </event>++ </category>++ <category label="Configuration">++ <event name="jdk.YoungGenerationConfiguration" label="Young Generation Configuration" description="The configuration of the young generation of the garbage collected heap">+ <setting name="enabled" label="Enabled" description="Record event" contentType="jdk.jfr.Flag">true</setting>+ <setting name="period" label="Period" description="Record event at interval" contentType="jdk.jfr.Period">beginChunk</setting>+ </event>++ <event name="jdk.GCSurvivorConfiguration" label="GC Survivor Configuration" description="The configuration of the survivors of garbage collection">+ <setting name="enabled" label="Enabled" description="Record event" contentType="jdk.jfr.Flag">true</setting>+ <setting name="period" label="Period" description="Record event at interval" contentType="jdk.jfr.Period">beginChunk</setting>+ </event>++ <event name="jdk.GCHeapConfiguration" label="GC Heap Configuration" description="The configuration of the garbage collected heap">+ <setting name="enabled" label="Enabled" description="Record event" contentType="jdk.jfr.Flag">true</setting>+ <setting name="period" label="Period" description="Record event at interval" contentType="jdk.jfr.Period">beginChunk</setting>+ </event>++ <event name="jdk.GCTLABConfiguration" label="TLAB Configuration" description="The configuration of the Thread Local Allocation Buffers (TLABs)">+ <setting name="enabled" label="Enabled" description="Record event" contentType="jdk.jfr.Flag">true</setting>+ <setting name="period" label="Period" description="Record event at interval" contentType="jdk.jfr.Period">beginChunk</setting>+ </event>++ <event name="jdk.GCConfiguration" label="GC Configuration" description="The configuration of the garbage collector">+ <setting name="enabled" label="Enabled" description="Record event" contentType="jdk.jfr.Flag">true</setting>+ <setting name="period" label="Period" description="Record event at interval" contentType="jdk.jfr.Period">everyChunk</setting>+ </event>++ </category>++ <category label="Metaspace">++ <event name="jdk.MetaspaceOOM" label="Metaspace Out of Memory">+ <setting name="enabled" label="Enabled" description="Record event" contentType="jdk.jfr.Flag">true</setting>+ <setting name="stackTrace" label="Stack Trace" description="Record stack traces" contentType="jdk.jfr.Flag">true</setting>+ </event>++ <event name="jdk.MetaspaceAllocationFailure" label="Metaspace Allocation Failure">+ <setting name="enabled" label="Enabled" description="Record event" contentType="jdk.jfr.Flag">true</setting>+ <setting name="stackTrace" label="Stack Trace" description="Record stack traces" contentType="jdk.jfr.Flag">true</setting>+ </event>++ <event name="jdk.MetaspaceChunkFreeListSummary" label="Metaspace Chunk Free List Summary">+ <setting name="enabled" label="Enabled" description="Record event" contentType="jdk.jfr.Flag">true</setting>+ </event>++ <event name="jdk.MetaspaceGCThreshold" label="Metaspace GC Threshold">+ <setting name="enabled" label="Enabled" description="Record event" contentType="jdk.jfr.Flag">true</setting>+ </event>++ </category>++ <category label="Phases">++ <event name="jdk.GCPhasePause" label="GC Phase Pause">+ <setting name="enabled" label="Enabled" description="Record event" contentType="jdk.jfr.Flag">true</setting>+ <setting name="threshold" label="Threshold" description="Record event with duration above or equal to threshold" contentType="jdk.jfr.Timespan">0 ms</setting>+ </event>++ <event name="jdk.GCPhaseConcurrent" label="GC Phase Concurrent">+ <setting name="enabled" label="Enabled" description="Record event" contentType="jdk.jfr.Flag">true</setting>+ <setting name="threshold" label="Threshold" description="Record event with duration above or equal to threshold" contentType="jdk.jfr.Timespan">0 ms</setting>+ </event>++ <event name="jdk.GCPhasePauseLevel4" label="GC Phase Pause Level 4">+ <setting name="enabled" label="Enabled" description="Record event" contentType="jdk.jfr.Flag">true</setting>+ <setting name="threshold" label="Threshold" description="Record event with duration above or equal to threshold" contentType="jdk.jfr.Timespan">0 ms</setting>+ </event>++ <event name="jdk.GCPhasePauseLevel1" label="GC Phase Pause Level 1">+ <setting name="enabled" label="Enabled" description="Record event" contentType="jdk.jfr.Flag">true</setting>+ <setting name="threshold" label="Threshold" description="Record event with duration above or equal to threshold" contentType="jdk.jfr.Timespan">0 ms</setting>+ </event>++ <event name="jdk.GCPhasePauseLevel3" label="GC Phase Pause Level 3">+ <setting name="enabled" label="Enabled" description="Record event" contentType="jdk.jfr.Flag">true</setting>+ <setting name="threshold" label="Threshold" description="Record event with duration above or equal to threshold" contentType="jdk.jfr.Timespan">0 ms</setting>+ </event>++ <event name="jdk.GCPhasePauseLevel2" label="GC Phase Pause Level 2">+ <setting name="enabled" label="Enabled" description="Record event" contentType="jdk.jfr.Flag">true</setting>+ <setting name="threshold" label="Threshold" description="Record event with duration above or equal to threshold" contentType="jdk.jfr.Timespan">0 ms</setting>+ </event>++ </category>++ <category label="Reference">++ <event name="jdk.GCReferenceStatistics" label="GC Reference Statistics" description="Total count of processed references during GC">+ <setting name="enabled" label="Enabled" description="Record event" contentType="jdk.jfr.Flag">true</setting>+ </event>++ </category>++ </category>++ <category label="Flag">++ <event name="jdk.DoubleFlag" label="Double Flag">+ <setting name="enabled" label="Enabled" description="Record event" contentType="jdk.jfr.Flag">true</setting>+ <setting name="period" label="Period" description="Record event at interval" contentType="jdk.jfr.Period">beginChunk</setting>+ </event>++ <event name="jdk.IntFlag" label="Int Flag">+ <setting name="enabled" label="Enabled" description="Record event" contentType="jdk.jfr.Flag">true</setting>+ <setting name="period" label="Period" description="Record event at interval" contentType="jdk.jfr.Period">beginChunk</setting>+ </event>++ <event name="jdk.LongFlag" label="Long Flag">+ <setting name="enabled" label="Enabled" description="Record event" contentType="jdk.jfr.Flag">true</setting>+ <setting name="period" label="Period" description="Record event at interval" contentType="jdk.jfr.Period">beginChunk</setting>+ </event>++ <event name="jdk.BooleanFlagChanged" label="Boolean Flag Changed">+ <setting name="enabled" label="Enabled" description="Record event" contentType="jdk.jfr.Flag">true</setting>+ </event>++ <event name="jdk.DoubleFlagChanged" label="Double Flag Changed">+ <setting name="enabled" label="Enabled" description="Record event" contentType="jdk.jfr.Flag">true</setting>+ </event>++ <event name="jdk.StringFlagChanged" label="String Flag Changed">+ <setting name="enabled" label="Enabled" description="Record event" contentType="jdk.jfr.Flag">true</setting>+ </event>++ <event name="jdk.UnsignedIntFlagChanged" label="Unsigned Int Flag Changed">+ <setting name="enabled" label="Enabled" description="Record event" contentType="jdk.jfr.Flag">true</setting>+ </event>++ <event name="jdk.BooleanFlag" label="Boolean Flag">+ <setting name="enabled" label="Enabled" description="Record event" contentType="jdk.jfr.Flag">true</setting>+ <setting name="period" label="Period" description="Record event at interval" contentType="jdk.jfr.Period">beginChunk</setting>+ </event>++ <event name="jdk.UnsignedLongFlagChanged" label="Unsigned Long Flag Changed">+ <setting name="enabled" label="Enabled" description="Record event" contentType="jdk.jfr.Flag">true</setting>+ </event>++ <event name="jdk.UnsignedIntFlag" label="Unsigned Int Flag">+ <setting name="enabled" label="Enabled" description="Record event" contentType="jdk.jfr.Flag">true</setting>+ <setting name="period" label="Period" description="Record event at interval" contentType="jdk.jfr.Period">beginChunk</setting>+ </event>++ <event name="jdk.StringFlag" label="String Flag">+ <setting name="enabled" label="Enabled" description="Record event" contentType="jdk.jfr.Flag">true</setting>+ <setting name="period" label="Period" description="Record event at interval" contentType="jdk.jfr.Period">beginChunk</setting>+ </event>++ <event name="jdk.LongFlagChanged" label="Long Flag Changed">+ <setting name="enabled" label="Enabled" description="Record event" contentType="jdk.jfr.Flag">true</setting>+ </event>++ <event name="jdk.UnsignedLongFlag" label="Unsigned Long Flag">+ <setting name="enabled" label="Enabled" description="Record event" contentType="jdk.jfr.Flag">true</setting>+ <setting name="period" label="Period" description="Record event at interval" contentType="jdk.jfr.Period">beginChunk</setting>+ </event>++ <event name="jdk.IntFlagChanged" label="Int Flag Changed">+ <setting name="enabled" label="Enabled" description="Record event" contentType="jdk.jfr.Flag">true</setting>+ </event>++ </category>++ <category label="Code Cache">++ <event name="jdk.CodeCacheConfiguration" label="Code Cache Configuration">+ <setting name="enabled" label="Enabled" description="Record event" contentType="jdk.jfr.Flag">true</setting>+ <setting name="period" label="Period" description="Record event at interval" contentType="jdk.jfr.Period">beginChunk</setting>+ </event>++ <event name="jdk.CodeCacheFull" label="Code Cache Full">+ <setting name="enabled" label="Enabled" description="Record event" contentType="jdk.jfr.Flag">true</setting>+ </event>++ <event name="jdk.CodeCacheStatistics" label="Code Cache Statistics">+ <setting name="enabled" label="Enabled" description="Record event" contentType="jdk.jfr.Flag">true</setting>+ <setting name="period" label="Period" description="Record event at interval" contentType="jdk.jfr.Period">everyChunk</setting>+ </event>++ </category>++ <category label="Runtime">++ <category label="Modules">++ <event name="jdk.ModuleRequire" label="Module Require" description="A directed edge representing a dependency">+ <setting name="enabled" label="Enabled" description="Record event" contentType="jdk.jfr.Flag">true</setting>+ <setting name="period" label="Period" description="Record event at interval" contentType="jdk.jfr.Period">endChunk</setting>+ </event>++ <event name="jdk.ModuleExport" label="Module Export">+ <setting name="enabled" label="Enabled" description="Record event" contentType="jdk.jfr.Flag">true</setting>+ <setting name="period" label="Period" description="Record event at interval" contentType="jdk.jfr.Period">endChunk</setting>+ </event>++ </category>++ <category label="Safepoint">++ <event name="jdk.SafepointBegin" label="Safepoint Begin" description="Safepointing begin">+ <setting name="enabled" label="Enabled" description="Record event" contentType="jdk.jfr.Flag">true</setting>+ <setting name="threshold" label="Threshold" description="Record event with duration above or equal to threshold" contentType="jdk.jfr.Timespan">0 ms</setting>+ </event>++ <event name="jdk.SafepointStateSynchronization" label="Safepoint State Synchronization" description="Synchronize run state of threads">+ <setting name="enabled" label="Enabled" description="Record event" contentType="jdk.jfr.Flag">false</setting>+ <setting name="threshold" label="Threshold" description="Record event with duration above or equal to threshold" contentType="jdk.jfr.Timespan">0 ms</setting>+ </event>++ <event name="jdk.SafepointEnd" label="Safepoint End" description="Safepointing end">+ <setting name="enabled" label="Enabled" description="Record event" contentType="jdk.jfr.Flag">false</setting>+ <setting name="threshold" label="Threshold" description="Record event with duration above or equal to threshold" contentType="jdk.jfr.Timespan">0 ms</setting>+ </event>++ <event name="jdk.SafepointCleanup" label="Safepoint Cleanup" description="Safepointing begin running cleanup tasks">+ <setting name="enabled" label="Enabled" description="Record event" contentType="jdk.jfr.Flag">false</setting>+ <setting name="threshold" label="Threshold" description="Record event with duration above or equal to threshold" contentType="jdk.jfr.Timespan">0 ms</setting>+ </event>++ <event name="jdk.SafepointWaitBlocked" label="Safepoint Wait Blocked" description="Safepointing begin waiting on running threads to block">+ <setting name="enabled" label="Enabled" description="Record event" contentType="jdk.jfr.Flag">false</setting>+ <setting name="threshold" label="Threshold" description="Record event with duration above or equal to threshold" contentType="jdk.jfr.Timespan">0 ms</setting>+ </event>++ <event name="jdk.SafepointCleanupTask" label="Safepoint Cleanup Task" description="Safepointing begin running cleanup tasks">+ <setting name="enabled" label="Enabled" description="Record event" contentType="jdk.jfr.Flag">false</setting>+ <setting name="threshold" label="Threshold" description="Record event with duration above or equal to threshold" contentType="jdk.jfr.Timespan">0 ms</setting>+ </event>++ </category>++ <event name="jdk.ThreadDump" label="Thread Dump">+ <setting name="enabled" label="Enabled" description="Record event" contentType="jdk.jfr.Flag">true</setting>+ <setting name="period" label="Period" description="Record event at interval" contentType="jdk.jfr.Period">60 s</setting>+ </event>++ <event name="jdk.BiasedLockClassRevocation" label="Biased Lock Class Revocation" description="Revoked biases for all instances of a class">+ <setting name="enabled" label="Enabled" description="Record event" contentType="jdk.jfr.Flag">true</setting>+ <setting name="stackTrace" label="Stack Trace" description="Record stack traces" contentType="jdk.jfr.Flag">true</setting>+ <setting name="threshold" label="Threshold" description="Record event with duration above or equal to threshold" contentType="jdk.jfr.Timespan">0 ms</setting>+ </event>++ <event name="jdk.ReservedStackActivation" label="Reserved Stack Activation" description="Activation of Reserved Stack Area caused by stack overflow with ReservedStackAccess annotated method in call stack">+ <setting name="enabled" label="Enabled" description="Record event" contentType="jdk.jfr.Flag">true</setting>+ <setting name="stackTrace" label="Stack Trace" description="Record stack traces" contentType="jdk.jfr.Flag">true</setting>+ </event>++ <event name="jdk.NativeLibrary" label="Native Library">+ <setting name="enabled" label="Enabled" description="Record event" contentType="jdk.jfr.Flag">true</setting>+ <setting name="period" label="Period" description="Record event at interval" contentType="jdk.jfr.Period">everyChunk</setting>+ </event>++ <event name="jdk.ExecuteVMOperation" label="VM Operation" description="Execution of a VM Operation">+ <setting name="enabled" label="Enabled" description="Record event" contentType="jdk.jfr.Flag">true</setting>+ <setting name="threshold" label="Threshold" description="Record event with duration above or equal to threshold" contentType="jdk.jfr.Timespan">0 ms</setting>+ </event>++ <event name="jdk.BiasedLockRevocation" label="Biased Lock Revocation" description="Revoked bias of object">+ <setting name="enabled" label="Enabled" description="Record event" contentType="jdk.jfr.Flag">true</setting>+ <setting name="stackTrace" label="Stack Trace" description="Record stack traces" contentType="jdk.jfr.Flag">true</setting>+ <setting name="threshold" label="Threshold" description="Record event with duration above or equal to threshold" contentType="jdk.jfr.Timespan">0 ms</setting>+ </event>++ <event name="jdk.BiasedLockSelfRevocation" label="Biased Lock Self Revocation" description="Revoked bias of object biased towards own thread">+ <setting name="enabled" label="Enabled" description="Record event" contentType="jdk.jfr.Flag">true</setting>+ <setting name="stackTrace" label="Stack Trace" description="Record stack traces" contentType="jdk.jfr.Flag">true</setting>+ <setting name="threshold" label="Threshold" description="Record event with duration above or equal to threshold" contentType="jdk.jfr.Timespan">0 ms</setting>+ </event>++ <event name="jdk.Shutdown" label="JVM Shutdown" description="JVM shutting down">+ <setting name="enabled" label="Enabled" description="Record event" contentType="jdk.jfr.Flag">true</setting>+ <setting name="stackTrace" label="Stack Trace" description="Record stack traces" contentType="jdk.jfr.Flag">true</setting>+ </event>++ </category>++ <category label="Code Sweeper">++ <event name="jdk.CodeSweeperConfiguration" label="Code Sweeper Configuration">+ <setting name="enabled" label="Enabled" description="Record event" contentType="jdk.jfr.Flag">true</setting>+ <setting name="period" label="Period" description="Record event at interval" contentType="jdk.jfr.Period">beginChunk</setting>+ </event>++ <event name="jdk.CodeSweeperStatistics" label="Code Sweeper Statistics">+ <setting name="enabled" label="Enabled" description="Record event" contentType="jdk.jfr.Flag">true</setting>+ <setting name="period" label="Period" description="Record event at interval" contentType="jdk.jfr.Period">everyChunk</setting>+ </event>++ <event name="jdk.SweepCodeCache" label="Sweep Code Cache">+ <setting name="enabled" label="Enabled" description="Record event" contentType="jdk.jfr.Flag">true</setting>+ <setting name="threshold" label="Threshold" description="Record event with duration above or equal to threshold" contentType="jdk.jfr.Timespan">100 ms</setting>+ </event>++ </category>++ <category label="Class Loading">++ <event name="jdk.ClassLoad" label="Class Load">+ <setting name="enabled" label="Enabled" description="Record event" contentType="jdk.jfr.Flag">true</setting>+ <setting name="stackTrace" label="Stack Trace" description="Record stack traces" contentType="jdk.jfr.Flag">true</setting>+ <setting name="threshold" label="Threshold" description="Record event with duration above or equal to threshold" contentType="jdk.jfr.Timespan">0 ms</setting>+ </event>++ <event name="jdk.ClassDefine" label="Class Define">+ <setting name="enabled" label="Enabled" description="Record event" contentType="jdk.jfr.Flag">true</setting>+ <setting name="stackTrace" label="Stack Trace" description="Record stack traces" contentType="jdk.jfr.Flag">true</setting>+ </event>++ <event name="jdk.ClassUnload" label="Class Unload">+ <setting name="enabled" label="Enabled" description="Record event" contentType="jdk.jfr.Flag">true</setting>+ </event>++ </category>++ <category label="Compiler">++ <event name="jdk.Compilation" label="Compilation">+ <setting name="enabled" label="Enabled" description="Record event" contentType="jdk.jfr.Flag">true</setting>+ <setting name="threshold" label="Threshold" description="Record event with duration above or equal to threshold" contentType="jdk.jfr.Timespan">100 ms</setting>+ </event>++ <event name="jdk.CompilerStatistics" label="Compiler Statistics">+ <setting name="enabled" label="Enabled" description="Record event" contentType="jdk.jfr.Flag">true</setting>+ <setting name="period" label="Period" description="Record event at interval" contentType="jdk.jfr.Period">1000 ms</setting>+ </event>++ <event name="jdk.CompilationFailure" label="Compilation Failure">+ <setting name="enabled" label="Enabled" description="Record event" contentType="jdk.jfr.Flag">true</setting>+ </event>++ <category label="Optimization">++ <event name="jdk.CompilerInlining" label="Method Inlining">+ <setting name="enabled" label="Enabled" description="Record event" contentType="jdk.jfr.Flag">true</setting>+ </event>++ </category>++ <event name="jdk.CompilerPhase" label="Compiler Phase">+ <setting name="enabled" label="Enabled" description="Record event" contentType="jdk.jfr.Flag">true</setting>+ <setting name="threshold" label="Threshold" description="Record event with duration above or equal to threshold" contentType="jdk.jfr.Timespan">10 s</setting>+ </event>++ <event name="jdk.CompilerConfiguration" label="Compiler Configuration">+ <setting name="enabled" label="Enabled" description="Record event" contentType="jdk.jfr.Flag">true</setting>+ <setting name="period" label="Period" description="Record event at interval" contentType="jdk.jfr.Period">beginChunk</setting>+ </event>++ </category>++ <category label="Profiling">++ <event name="jdk.OldObjectSample" label="Old Object Sample" description="A potential memory leak">+ <setting name="enabled" label="Enabled" description="Record event" contentType="jdk.jfr.Flag">true</setting>+ <setting name="stackTrace" label="Stack Trace" description="Record stack traces" contentType="jdk.jfr.Flag">true</setting>+ <setting name="cutoff" label="Cutoff" description="Limit running time of event">1 h</setting>+ </event>++ <event name="jdk.ExecutionSample" label="Method Profiling Sample" description="Snapshot of a threads state">+ <setting name="enabled" label="Enabled" description="Record event" contentType="jdk.jfr.Flag">true</setting>+ <setting name="period" label="Period" description="Record event at interval" contentType="jdk.jfr.Period">10 ms</setting>+ </event>++ <event name="jdk.NativeMethodSample" label="Method Profiling Sample Native" description="Snapshot of a threads state when in native">+ <setting name="enabled" label="Enabled" description="Record event" contentType="jdk.jfr.Flag">true</setting>+ <setting name="period" label="Period" description="Record event at interval" contentType="jdk.jfr.Period">20 ms</setting>+ </event>++ </category>++ <event name="jdk.JVMInformation" label="JVM Information" description="Description of JVM and the Java application">+ <setting name="enabled" label="Enabled" description="Record event" contentType="jdk.jfr.Flag">true</setting>+ <setting name="period" label="Period" description="Record event at interval" contentType="jdk.jfr.Period">beginChunk</setting>+ </event>++ <event name="jdk.InitialSystemProperty" label="Initial System Property" description="System Property at JVM start">+ <setting name="enabled" label="Enabled" description="Record event" contentType="jdk.jfr.Flag">true</setting>+ <setting name="period" label="Period" description="Record event at interval" contentType="jdk.jfr.Period">beginChunk</setting>+ </event>++ </category>++ <category label="Java Application">++ <event name="jdk.JavaMonitorEnter" label="Java Monitor Blocked">+ <setting name="enabled" label="Enabled" description="Record event" contentType="jdk.jfr.Flag">true</setting>+ <setting name="stackTrace" label="Stack Trace" description="Record stack traces" contentType="jdk.jfr.Flag">true</setting>+ <setting name="threshold" label="Threshold" description="Record event with duration above or equal to threshold" contentType="jdk.jfr.Timespan">10 ms</setting>+ </event>++ <event name="jdk.JavaExceptionThrow" label="Java Exception" description="An object derived from java.lang.Exception has been created">+ <setting name="enabled" label="Enabled" description="Record event" contentType="jdk.jfr.Flag">true</setting>+ <setting name="stackTrace" label="Stack Trace" description="Record stack traces" contentType="jdk.jfr.Flag">true</setting>+ <setting name="threshold" label="Threshold" description="Record event with duration above or equal to threshold" contentType="jdk.jfr.Timespan">0 ns</setting>+ </event>++ <category label="Statistics">++ <event name="jdk.ClassLoadingStatistics" label="Class Loading Statistics">+ <setting name="enabled" label="Enabled" description="Record event" contentType="jdk.jfr.Flag">true</setting>+ <setting name="period" label="Period" description="Record event at interval" contentType="jdk.jfr.Period">1000 ms</setting>+ </event>++ <event name="jdk.JavaThreadStatistics" label="Java Thread Statistics">+ <setting name="enabled" label="Enabled" description="Record event" contentType="jdk.jfr.Flag">true</setting>+ <setting name="period" label="Period" description="Record event at interval" contentType="jdk.jfr.Period">1000 ms</setting>+ </event>++ <event name="jdk.ClassLoaderStatistics" label="Class Loader Statistics">+ <setting name="enabled" label="Enabled" description="Record event" contentType="jdk.jfr.Flag">true</setting>+ <setting name="period" label="Period" description="Record event at interval" contentType="jdk.jfr.Period">everyChunk</setting>+ </event>++ <event name="jdk.ThreadAllocationStatistics" label="Thread Allocation Statistics">+ <setting name="enabled" label="Enabled" description="Record event" contentType="jdk.jfr.Flag">true</setting>+ <setting name="period" label="Period" description="Record event at interval" contentType="jdk.jfr.Period">everyChunk</setting>+ </event>++ <event name="jdk.ExceptionStatistics" label="Exception Statistics" description="Number of objects derived from java.lang.Throwable that have been created">+ <setting name="enabled" label="Enabled" description="Record event" contentType="jdk.jfr.Flag">true</setting>+ <setting name="period" label="Period" description="Record event at interval" contentType="jdk.jfr.Period">1000 ms</setting>+ </event>++ </category>++ <event name="jdk.ThreadPark" label="Java Thread Park">+ <setting name="enabled" label="Enabled" description="Record event" contentType="jdk.jfr.Flag">true</setting>+ <setting name="stackTrace" label="Stack Trace" description="Record stack traces" contentType="jdk.jfr.Flag">true</setting>+ <setting name="threshold" label="Threshold" description="Record event with duration above or equal to threshold" contentType="jdk.jfr.Timespan">10 ms</setting>+ </event>++ <event name="jdk.ThreadStart" label="Java Thread Start">+ <setting name="enabled" label="Enabled" description="Record event" contentType="jdk.jfr.Flag">true</setting>+ <setting name="stackTrace" label="Stack Trace" description="Record stack traces" contentType="jdk.jfr.Flag">true</setting>+ </event>++ <event name="jdk.SocketRead" label="Socket Read" description="Reading data from a socket">+ <setting name="enabled" label="Enabled" description="Record event" contentType="jdk.jfr.Flag">true</setting>+ <setting name="stackTrace" label="Stack Trace" description="Record stack traces" contentType="jdk.jfr.Flag">true</setting>+ <setting name="threshold" label="Threshold" description="Record event with duration above or equal to threshold" contentType="jdk.jfr.Timespan">10 ms</setting>+ </event>++ <event name="jdk.FileForce" label="File Force" description="Force updates to be written to file">+ <setting name="enabled" label="Enabled" description="Record event" contentType="jdk.jfr.Flag">true</setting>+ <setting name="stackTrace" label="Stack Trace" description="Record stack traces" contentType="jdk.jfr.Flag">true</setting>+ <setting name="threshold" label="Threshold" description="Record event with duration above or equal to threshold" contentType="jdk.jfr.Timespan">10 ms</setting>+ </event>++ <event name="jdk.SocketWrite" label="Socket Write" description="Writing data to a socket">+ <setting name="enabled" label="Enabled" description="Record event" contentType="jdk.jfr.Flag">true</setting>+ <setting name="stackTrace" label="Stack Trace" description="Record stack traces" contentType="jdk.jfr.Flag">true</setting>+ <setting name="threshold" label="Threshold" description="Record event with duration above or equal to threshold" contentType="jdk.jfr.Timespan">10 ms</setting>+ </event>++ <event name="jdk.JavaErrorThrow" label="Java Error" description="An object derived from java.lang.Error has been created. OutOfMemoryErrors are ignored">+ <setting name="enabled" label="Enabled" description="Record event" contentType="jdk.jfr.Flag">true</setting>+ <setting name="stackTrace" label="Stack Trace" description="Record stack traces" contentType="jdk.jfr.Flag">true</setting>+ <setting name="threshold" label="Threshold" description="Record event with duration above or equal to threshold" contentType="jdk.jfr.Timespan">0 ns</setting>+ </event>++ <event name="jdk.JavaMonitorInflate" label="Java Monitor Inflated">+ <setting name="enabled" label="Enabled" description="Record event" contentType="jdk.jfr.Flag">true</setting>+ <setting name="stackTrace" label="Stack Trace" description="Record stack traces" contentType="jdk.jfr.Flag">true</setting>+ <setting name="threshold" label="Threshold" description="Record event with duration above or equal to threshold" contentType="jdk.jfr.Timespan">10 ms</setting>+ </event>++ <event name="jdk.ObjectAllocationOutsideTLAB" label="Allocation outside TLAB" description="Allocation outside Thread Local Allocation Buffers">+ <setting name="enabled" label="Enabled" description="Record event" contentType="jdk.jfr.Flag">true</setting>+ <setting name="stackTrace" label="Stack Trace" description="Record stack traces" contentType="jdk.jfr.Flag">true</setting>+ </event>++ <event name="jdk.FileWrite" label="File Write" description="Writing data to a file">+ <setting name="enabled" label="Enabled" description="Record event" contentType="jdk.jfr.Flag">true</setting>+ <setting name="stackTrace" label="Stack Trace" description="Record stack traces" contentType="jdk.jfr.Flag">true</setting>+ <setting name="threshold" label="Threshold" description="Record event with duration above or equal to threshold" contentType="jdk.jfr.Timespan">10 ms</setting>+ </event>++ <event name="jdk.JavaMonitorWait" label="Java Monitor Wait" description="Waiting on a Java monitor">+ <setting name="enabled" label="Enabled" description="Record event" contentType="jdk.jfr.Flag">true</setting>+ <setting name="stackTrace" label="Stack Trace" description="Record stack traces" contentType="jdk.jfr.Flag">true</setting>+ <setting name="threshold" label="Threshold" description="Record event with duration above or equal to threshold" contentType="jdk.jfr.Timespan">10 ms</setting>+ </event>++ <event name="jdk.ObjectAllocationInNewTLAB" label="Allocation in new TLAB" description="Allocation in new Thread Local Allocation Buffer">+ <setting name="enabled" label="Enabled" description="Record event" contentType="jdk.jfr.Flag">true</setting>+ <setting name="stackTrace" label="Stack Trace" description="Record stack traces" contentType="jdk.jfr.Flag">true</setting>+ </event>++ <event name="jdk.FileRead" label="File Read" description="Reading data from a file">+ <setting name="enabled" label="Enabled" description="Record event" contentType="jdk.jfr.Flag">true</setting>+ <setting name="stackTrace" label="Stack Trace" description="Record stack traces" contentType="jdk.jfr.Flag">true</setting>+ <setting name="threshold" label="Threshold" description="Record event with duration above or equal to threshold" contentType="jdk.jfr.Timespan">10 ms</setting>+ </event>++ <event name="jdk.ThreadSleep" label="Java Thread Sleep">+ <setting name="enabled" label="Enabled" description="Record event" contentType="jdk.jfr.Flag">true</setting>+ <setting name="stackTrace" label="Stack Trace" description="Record stack traces" contentType="jdk.jfr.Flag">true</setting>+ <setting name="threshold" label="Threshold" description="Record event with duration above or equal to threshold" contentType="jdk.jfr.Timespan">10 ms</setting>+ </event>++ <event name="jdk.ThreadEnd" label="Java Thread End">+ <setting name="enabled" label="Enabled" description="Record event" contentType="jdk.jfr.Flag">true</setting>+ </event>++ </category>++ <category label="Operating System">++ <category label="Processor">++ <event name="jdk.CPULoad" label="CPU Load" description="OS CPU Load">+ <setting name="enabled" label="Enabled" description="Record event" contentType="jdk.jfr.Flag">true</setting>+ <setting name="period" label="Period" description="Record event at interval" contentType="jdk.jfr.Period">1000 ms</setting>+ </event>++ <event name="jdk.ThreadCPULoad" label="Thread CPU Load">+ <setting name="enabled" label="Enabled" description="Record event" contentType="jdk.jfr.Flag">true</setting>+ <setting name="period" label="Period" description="Record event at interval" contentType="jdk.jfr.Period">10 s</setting>+ </event>++ <event name="jdk.CPUInformation" label="CPU Information">+ <setting name="enabled" label="Enabled" description="Record event" contentType="jdk.jfr.Flag">true</setting>+ <setting name="period" label="Period" description="Record event at interval" contentType="jdk.jfr.Period">beginChunk</setting>+ </event>++ <event name="jdk.ThreadContextSwitchRate" label="Thread Context Switch Rate">+ <setting name="enabled" label="Enabled" description="Record event" contentType="jdk.jfr.Flag">true</setting>+ <setting name="period" label="Period" description="Record event at interval" contentType="jdk.jfr.Period">10 s</setting>+ </event>++ <event name="jdk.CPUTimeStampCounter" label="CPU Time Stamp Counter">+ <setting name="enabled" label="Enabled" description="Record event" contentType="jdk.jfr.Flag">true</setting>+ <setting name="period" label="Period" description="Record event at interval" contentType="jdk.jfr.Period">beginChunk</setting>+ </event>++ </category>++ <event name="jdk.OSInformation" label="OS Information">+ <setting name="enabled" label="Enabled" description="Record event" contentType="jdk.jfr.Flag">true</setting>+ <setting name="period" label="Period" description="Record event at interval" contentType="jdk.jfr.Period">beginChunk</setting>+ </event>++ <category label="Memory">++ <event name="jdk.PhysicalMemory" label="Physical Memory" description="OS Physical Memory">+ <setting name="enabled" label="Enabled" description="Record event" contentType="jdk.jfr.Flag">true</setting>+ <setting name="period" label="Period" description="Record event at interval" contentType="jdk.jfr.Period">everyChunk</setting>+ </event>++ </category>++ <event name="jdk.SystemProcess" label="System Process">+ <setting name="enabled" label="Enabled" description="Record event" contentType="jdk.jfr.Flag">true</setting>+ <setting name="period" label="Period" description="Record event at interval" contentType="jdk.jfr.Period">endChunk</setting>+ </event>++ <event name="jdk.VirtualizationInformation" label="Virtualization Information">+ <setting name="enabled" label="Enabled" description="Record event" contentType="jdk.jfr.Flag">true</setting>+ <setting name="period" label="Period" description="Record event at interval" contentType="jdk.jfr.Period">beginChunk</setting>+ </event>++ <event name="jdk.InitialEnvironmentVariable" label="Initial Environment Variable">+ <setting name="enabled" label="Enabled" description="Record event" contentType="jdk.jfr.Flag">true</setting>+ <setting name="period" label="Period" description="Record event at interval" contentType="jdk.jfr.Period">beginChunk</setting>+ </event>++ <category label="Network">++ <event name="jdk.NetworkUtilization" label="Network Utilization">+ <setting name="enabled" label="Enabled" description="Record event" contentType="jdk.jfr.Flag">true</setting>+ <setting name="period" label="Period" description="Record event at interval" contentType="jdk.jfr.Period">5 s</setting>+ </event>++ </category>++ </category>++ <category label="Flight Recorder">++ <event name="jdk.ActiveSetting" label="Recording Setting">+ <setting name="enabled" label="Enabled" description="Record event" contentType="jdk.jfr.Flag">true</setting>+ <setting name="stackTrace" label="Stack Trace" description="Record stack traces" contentType="jdk.jfr.Flag">false</setting>+ <setting name="threshold" label="Threshold" description="Record event with duration above or equal to threshold" contentType="jdk.jfr.Timespan">0 ns</setting>+ </event>++ <event name="jdk.ActiveRecording" label="Flight Recording">+ <setting name="enabled" label="Enabled" description="Record event" contentType="jdk.jfr.Flag">true</setting>+ <setting name="stackTrace" label="Stack Trace" description="Record stack traces" contentType="jdk.jfr.Flag">false</setting>+ <setting name="threshold" label="Threshold" description="Record event with duration above or equal to threshold" contentType="jdk.jfr.Timespan">0 ns</setting>+ </event>++ <event name="jdk.DumpReason" label="Recording Reason" description="Who requested the recording and why">+ <setting name="enabled" label="Enabled" description="Record event" contentType="jdk.jfr.Flag">true</setting>+ </event>++ <event name="jdk.DataLoss" label="Data Loss" description="Data could not be copied out from a buffer, typically because of contention">+ <setting name="enabled" label="Enabled" description="Record event" contentType="jdk.jfr.Flag">true</setting>+ </event>++ </category>++</configuration>diff --git a/solr/benchmark/jmh.sh b/solr/benchmark/jmh.shnew file mode 100755index 00000000000..16cbe334a7e--- /dev/null+++ b/solr/benchmark/jmh.sh@@ -0,0 +1,65 @@+#!/usr/bin/env bash+# Licensed to the Apache Software Foundation (ASF) under one or more+# contributor license agreements. See the NOTICE file distributed with+# this work for additional information regarding copyright ownership.+# The ASF licenses this file to You under the Apache License, Version 2.0+# (the "License"); you may not use this file except in compliance with+# the License. You may obtain a copy of the License at+#+# http://www.apache.org/licenses/LICENSE-2.0+#+# Unless required by applicable law or agreed to in writing, software+# distributed under the License is distributed on an "AS IS" BASIS,+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.+# See the License for the specific language governing permissions and+# limitations under the License.++set -e++base_dir=$(dirname "$0")++if [ "${base_dir}" == "." ]; then+ gradlew_dir="../.."+else+ echo "Benchmarks need to be run from the 'solr/benchmark' directory"+ exit+fi+++if [ -d "lib" ]+then+ echo "Using lib directory for classpath..."+ classpath="lib/*:build/classes/java/main"+else+ echo "Getting classpath from gradle..."+ # --no-daemon+ gradleCmd="${gradlew_dir}/gradlew"+ $gradleCmd -q -p ../../ jar+ echo "gradle build done"+ classpath=$($gradleCmd -q echoCp)+fi++# shellcheck disable=SC2145+echo "running JMH with args: $@"+++# -XX:+PreserveFramePointer is not necessary with async profiler and they claim can be up to 10% hit++# -XX:+UseStringDeduplication should experiment with this+# -XX:MaxMetaspaceExpansion=64M # and this note: Avoids triggering full GC when we just allocate a bit more metaspace, and metaspace automatically gets cleaned anyway+# -XX:+UnlockExperimentalVMOptions -XX:G1NewSizePercent=20 # and this note: Prevents G1 undermining young gen, which otherwise causes a cascade of issues++# -XX:-UseBiasedLocking - should be unreflective in recent JVMs and removed in the latest.++jvmArgs="-jvmArgs -Djmh.shutdownTimeout=5 -jvmArgs -Djmh.separateClassLoader=true -jvmArgs -XX:-UseBiasedLocking -jvmArgs -XX:+UnlockDiagnosticVMOptions -jvmArgs -XX:+DebugNonSafepoints -jvmArgs --add-opens=java.base/java.lang.reflect=ALL-UNNAMED"+gcArgs="-jvmArgs -XX:+UseG1GC -jvmArgs -XX:+ParallelRefProcEnabled"+loggingArgs="-jvmArgs -Dlog4jConfigurationFile=./log4j2.xml -jvmArgs -Dlog4j2.is.webapp=false -jvmArgs -Dlog4j2.garbagefreeThreadContextMap=true -jvmArgs -Dlog4j2.enableDirectEncoders=true -jvmArgs -Dlog4j2.enable.threadlocals=true"++logDir="work/solr-logs"++#set -x++# shellcheck disable=SC2086+exec java -cp "$classpath" --add-opens=java.base/java.io=ALL-UNNAMED -Djdk.module.illegalAccess.silent=true org.openjdk.jmh.Main $jvmArgs $loggingArgs $gcArgs -jvmArgs "-Dsolr.log.dir=$logDir" "$@"++echo "JMH benchmarks done"diff --git a/solr/benchmark/log4j2.xml b/solr/benchmark/log4j2.xmlnew file mode 100644index 00000000000..eba3495022c--- /dev/null+++ b/solr/benchmark/log4j2.xml@@ -0,0 +1,84 @@+<?xml version="1.0" encoding="UTF-8"?>+<!--+ Licensed to the Apache Software Foundation (ASF) under one or more+ contributor license agreements. See the NOTICE file distributed with+ this work for additional information regarding copyright ownership.+ The ASF licenses this file to You under the Apache License, Version 2.0+ (the "License"); you may not use this file except in compliance with+ the License. You may obtain a copy of the License at++ http://www.apache.org/licenses/LICENSE-2.0++ Unless required by applicable law or agreed to in writing, software+ distributed under the License is distributed on an "AS IS" BASIS,+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.+ See the License for the specific language governing permissions and+ limitations under the License.+ -->++<!-- Default production configuration is asnychronous logging -->+<Configuration>+ <Appenders>++ <Console name="STDOUT" target="SYSTEM_OUT">+ <PatternLayout>+ <Pattern>+ %maxLen{%d{yyyy-MM-dd HH:mm:ss.SSS} %-5p (%t) [%X{collection} %X{shard} %X{replica} %X{core}] %c{1.} %m%notEmpty{ =>%ex{short}}}{10240}%n+ </Pattern>+ </PatternLayout>+ </Console>++ <RollingRandomAccessFile+ name="MainLogFile"+ fileName="${sys:solr.log.dir}/${sys:solr.log.name}.log"+ filePattern="${sys:solr.log.dir}/${sys:solr.log.name}.log.%i" >+ <PatternLayout>+ <Pattern>+ %maxLen{%d{yyyy-MM-dd HH:mm:ss.SSS} %-5p (%t) [%X{collection} %X{shard} %X{replica} %X{core}] %c{1.} %m%notEmpty{ =>%ex{short}}}{10240}%n+ </Pattern>+ </PatternLayout>+ <Policies>+ <OnStartupTriggeringPolicy />+ <SizeBasedTriggeringPolicy size="32 MB"/>+ </Policies>+ <DefaultRolloverStrategy max="10"/>+ </RollingRandomAccessFile>++ <RollingRandomAccessFile+ name="SlowLogFile"+ fileName="${sys:solr.log.dir}/${sys:solr.log.name}_slow_requests.log"+ filePattern="${sys:solr.log.dir}/${sys:solr.log.name}_slow_requests.log.%i" >+ <PatternLayout>+ <Pattern>+ %maxLen{%d{yyyy-MM-dd HH:mm:ss.SSS} %-5p (%t) [%X{collection} %X{shard} %X{replica} %X{core}] %c{1.} %m%notEmpty{ =>%ex{short}}}{10240}%n+ </Pattern>+ </PatternLayout>+ <Policies>+ <OnStartupTriggeringPolicy />+ <SizeBasedTriggeringPolicy size="32 MB"/>+ </Policies>+ <DefaultRolloverStrategy max="10"/>+ </RollingRandomAccessFile>++ </Appenders>+ <Loggers>+ <!-- Use <AsyncLogger/<AsyncRoot and <Logger/<Root for asynchronous logging or synchonous logging respectively -->+ <AsyncLogger name="org.apache.hadoop" level="warn"/>+ <AsyncLogger name="org.apache.solr.update.LoggingInfoStream" level="off"/>+ <AsyncLogger name="org.apache.zookeeper" level="warn"/>+ <!-- HttpSolrCall adds markers denoting the handler class to allow fine grained control, metrics are+ very noisy so by default the metrics handler is turned off to see metrics logging set DENY to ACCEPT -->+ <AsyncLogger name="org.apache.solr.servlet.HttpSolrCall" level="info">+ <MarkerFilter marker="org.apache.solr.handler.admin.MetricsHandler" onMatch="DENY" onMismatch="ACCEPT"/>+ </AsyncLogger>+ <AsyncLogger name="org.apache.solr.core.SolrCore.SlowRequest" level="info" additivity="false">+ <AppenderRef ref="SlowLogFile"/>+ </AsyncLogger>++ <AsyncRoot level="info">+ <AppenderRef ref="MainLogFile"/>+ <!-- <AppenderRef ref="STDOUT"/> -->+ </AsyncRoot>+ </Loggers>+</Configuration>+diff --git a/solr/benchmark/src/java/org/apache/solr/bench/DocMaker.java b/solr/benchmark/src/java/org/apache/solr/bench/DocMaker.javanew file mode 100644index 00000000000..71d269149d2--- /dev/null+++ b/solr/benchmark/src/java/org/apache/solr/bench/DocMaker.java@@ -0,0 +1,299 @@+/*+ * Licensed to the Apache Software Foundation (ASF) under one or more+ * contributor license agreements. See the NOTICE file distributed with+ * this work for additional information regarding copyright ownership.+ * The ASF licenses this file to You under the Apache License, Version 2.0+ * (the "License"); you may not use this file except in compliance with+ * the License. You may obtain a copy of the License at+ *+ * http://www.apache.org/licenses/LICENSE-2.0+ *+ * Unless required by applicable law or agreed to in writing, software+ * distributed under the License is distributed on an "AS IS" BASIS,+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.+ * See the License for the specific language governing permissions and+ * limitations under the License.+ */+package org.apache.solr.bench;++import java.util.HashMap;+import java.util.Iterator;+import java.util.Map;+import java.util.Objects;+import java.util.Queue;+import java.util.Random;+import java.util.SplittableRandom;+import java.util.concurrent.ConcurrentLinkedQueue;+import java.util.concurrent.ExecutorService;+import java.util.concurrent.Executors;+import java.util.concurrent.TimeUnit;+import java.util.concurrent.atomic.AtomicInteger;+import org.apache.commons.lang3.RandomStringUtils;+import org.apache.commons.lang3.Validate;+import org.apache.lucene.util.RamUsageEstimator;+import org.apache.lucene.util.TestUtil;+import org.apache.solr.common.SolrInputDocument;+import org.apache.solr.common.util.SolrNamedThreadFactory;+import org.apache.solr.common.util.SuppressForbidden;++/**+ * A tool to generate controlled random data for a benchmark. {@link SolrInputDocument}s are created+ * based on supplied FieldDef definitions.+ *+ * <p>You can call getDocument to build and retrieve one {@link SolrInputDocument} at a time, or you+ * can call {@link #preGenerateDocs} to generate the given number of documents in RAM, and then+ * retrieve them via {@link #getGeneratedDocsIterator}.+ */+public class DocMaker {++ private Queue<SolrInputDocument> docs = new ConcurrentLinkedQueue<>();++ private final Map<String, FieldDef> fields = new HashMap<>();++ private static final AtomicInteger ID = new AtomicInteger();++ private ExecutorService executorService;++ public DocMaker() {}++ @SuppressForbidden(reason = "This module does not need to deal with logging context")+ public void preGenerateDocs(int numDocs, SplittableRandom random) throws InterruptedException {+ MiniClusterState.log("preGenerateDocs " + numDocs + " ...");+ docs.clear();+ executorService =+ Executors.newFixedThreadPool(+ Runtime.getRuntime().availableProcessors() + 1,+ new SolrNamedThreadFactory("SolrJMH DocMaker"));++ for (int i = 0; i < numDocs; i++) {+ executorService.submit(+ new Runnable() {+ SplittableRandom threadRandom = random.split();++ @Override+ public void run() {+ try {+ SolrInputDocument doc = DocMaker.this.getDocument(threadRandom);+ docs.add(doc);+ } catch (Exception e) {+ executorService.shutdownNow();+ throw new RuntimeException(e);+ }+ }+ });+ }++ executorService.shutdown();+ boolean result = executorService.awaitTermination(10, TimeUnit.MINUTES);+ if (!result) {+ throw new RuntimeException("Timeout waiting for doc adds to finish");+ }+ MiniClusterState.log(+ "done preGenerateDocs docs="+ + docs.size()+ + " ram="+ + RamUsageEstimator.humanReadableUnits(RamUsageEstimator.sizeOfObject(docs)));++ if (numDocs != docs.size()) {+ throw new IllegalStateException("numDocs != " + docs.size());+ }+ }++ public Iterator<SolrInputDocument> getGeneratedDocsIterator() {+ return docs.iterator();+ }++ public SolrInputDocument getDocument(SplittableRandom random) {+ SolrInputDocument doc = new SolrInputDocument();++ for (Map.Entry<String, FieldDef> entry : fields.entrySet()) {+ doc.addField(entry.getKey(), getValue(entry.getValue(), random));+ }++ return doc;+ }++ public void addField(String name, FieldDef.FieldDefBuilder builder) {+ fields.put(name, builder.build());+ }++ private Object getValue(FieldDef fieldDef, SplittableRandom threadRandom) {++ switch (fieldDef.getContent()) {+ case UNIQUE_INT:+ return ID.incrementAndGet();+ case INTEGER:+ if (fieldDef.getMaxCardinality() > 0) {+ long start = fieldDef.getCardinalityStart();+ long seed = nextLong(start, start + fieldDef.getMaxCardinality(), threadRandom.split());+ return nextInt(0, Integer.MAX_VALUE, new SplittableRandom(seed));+ }++ return threadRandom.nextInt(Integer.MAX_VALUE);+ case ALPHEBETIC:+ return getString(+ fieldDef, value -> getAlphabeticString(fieldDef, threadRandom), threadRandom);+ case UNICODE:+ return getString(fieldDef, value -> getUnicodeString(fieldDef, threadRandom), threadRandom);+ default:+ throw new UnsupportedOperationException(+ "Unsupported content type type=" + fieldDef.getContent());+ }+ }++ private String getString(+ FieldDef fieldDef, StringSupplier supplier, SplittableRandom threadRandom) {+ if (fieldDef.getNumTokens() > 1 || fieldDef.getMaxNumTokens() > 1) {+ StringBuilder sb =+ new StringBuilder(+ fieldDef.getNumTokens()+ * (Math.max(fieldDef.getLength(), fieldDef.getMaxLength()) + 1));+ SplittableRandom random = threadRandom.split();+ for (int i = 0;+ i+ < (fieldDef.getMaxNumTokens() > 1+ ? random.nextInt(1, fieldDef.getMaxNumTokens())+ : fieldDef.getNumTokens());+ i++) {+ if (i > 0) {+ sb.append(' ');+ }+ sb.append(supplier.getString(fieldDef));+ }+ return sb.toString();+ }+ return supplier.getString(fieldDef);+ }++ private String getUnicodeString(FieldDef fieldDef, SplittableRandom threadRandom) {+ try {+ if (fieldDef.getMaxCardinality() > 0) {+ long start = fieldDef.getCardinalityStart();+ long seed = nextLong(start, start + fieldDef.getMaxCardinality(), threadRandom.split());+ if (fieldDef.getLength() > -1) {+ return TestUtil.randomRealisticUnicodeString(+ new Random(seed), fieldDef.getLength(), fieldDef.getLength());+ } else {+ return TestUtil.randomRealisticUnicodeString(+ new Random(seed), 1, fieldDef.getMaxLength());+ }+ }++ if (fieldDef.getLength() > -1) {+ return TestUtil.randomRealisticUnicodeString(+ new Random(threadRandom.nextLong()), fieldDef.getLength(), fieldDef.getLength());+ } else {+ return TestUtil.randomRealisticUnicodeString(+ new Random(threadRandom.nextLong()), 1, fieldDef.getMaxLength());+ }+ } catch (Exception e) {+ throw new RuntimeException("Failed getting UnicodeString with FieldDef=" + fieldDef, e);+ }+ }++ private String getAlphabeticString(FieldDef fieldDef, SplittableRandom threadRandom) {+ try {+ if (fieldDef.getMaxCardinality() > 0) {+ long start = fieldDef.getCardinalityStart();+ long seed = nextLong(start, start + fieldDef.getMaxCardinality(), threadRandom.split());+ SplittableRandom random = new SplittableRandom(seed);+ if (fieldDef.getLength() > -1) {+ return RandomStringUtils.random(+ nextInt(fieldDef.getLength(), fieldDef.getLength(), random),+ 0,+ 0,+ true,+ false,+ null,+ new Random(seed));+ } else {+ return RandomStringUtils.random(+ nextInt(1, fieldDef.getMaxLength(), random),+ 0,+ 0,+ true,+ false,+ null,+ new Random(seed));+ }+ }++ SplittableRandom random = threadRandom.split();+ Random r = new Random(random.nextLong());+ if (fieldDef.getLength() > -1) {+ return RandomStringUtils.random(+ nextInt(fieldDef.getLength(), fieldDef.getLength(), random),+ 0,+ 0,+ true,+ false,+ null,+ r);+ } else {+ return RandomStringUtils.random(+ nextInt(1, fieldDef.getMaxLength(), random), 0, 0, true, false, null, r);+ }+ } catch (Exception e) {+ throw new RuntimeException("Failed getting AlphabeticString with FieldDef=" + fieldDef, e);+ }+ }++ public void clear() {+ docs.clear();+ }++ public enum Content {+ UNICODE,+ ALPHEBETIC,+ INTEGER,+ UNIQUE_INT+ }++ @Override+ public boolean equals(Object o) {+ if (this == o) return true;+ if (o == null || getClass() != o.getClass()) return false;+ DocMaker that = (DocMaker) o;+ return fields.equals(that.fields);+ }++ @Override+ public int hashCode() {+ return Objects.hash(fields);+ }++ public static int nextInt(+ final int startInclusive, final int endExclusive, SplittableRandom random) {+ Validate.isTrue(+ endExclusive >= startInclusive, "Start value must be smaller or equal to end value.");+ Validate.isTrue(startInclusive >= 0, "Both range values must be non-negative.");++ if (startInclusive == endExclusive) {+ return startInclusive;+ }++ return startInclusive + random.nextInt(endExclusive - startInclusive);+ }++ public static long nextLong(+ final long startInclusive, final long endExclusive, SplittableRandom random) {+ Validate.isTrue(+ endExclusive >= startInclusive, "Start value must be smaller or equal to end value.");+ Validate.isTrue(+ startInclusive >= 0,+ "Both range values must be non-negative startInclusive="+ + startInclusive+ + " endExclusive="+ + endExclusive);++ if (startInclusive == endExclusive) {+ return startInclusive;+ }++ return startInclusive + random.nextLong(endExclusive - startInclusive);+ }++ private interface StringSupplier {+ String getString(FieldDef value);+ }+}diff --git a/solr/benchmark/src/java/org/apache/solr/bench/FieldDef.java b/solr/benchmark/src/java/org/apache/solr/bench/FieldDef.javanew file mode 100644index 00000000000..933a75968d7--- /dev/null+++ b/solr/benchmark/src/java/org/apache/solr/bench/FieldDef.java@@ -0,0 +1,170 @@+/*+ * Licensed to the Apache Software Foundation (ASF) under one or more+ * contributor license agreements. See the NOTICE file distributed with+ * this work for additional information regarding copyright ownership.+ * The ASF licenses this file to You under the Apache License, Version 2.0+ * (the "License"); you may not use this file except in compliance with+ * the License. You may obtain a copy of the License at+ *+ * http://www.apache.org/licenses/LICENSE-2.0+ *+ * Unless required by applicable law or agreed to in writing, software+ * distributed under the License is distributed on an "AS IS" BASIS,+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.+ * See the License for the specific language governing permissions and+ * limitations under the License.+ */+package org.apache.solr.bench;++import java.util.SplittableRandom;+import org.apache.solr.common.SolrInputDocument;++/**+ * Provides the definition for a randomly generated field in a {@link SolrInputDocument} created by+ * a {@link DocMaker}.+ */+public class FieldDef {+ public static final int DEFAULT_MAX_LENGTH = 64;++ private DocMaker.Content content;+ private int numTokens = 1;+ private int maxNumTokens = -1;+ private int maxCardinality = -1;+ private int maxLength = -1;+ private int length = -1;+ private long cardinalityStart;++ public int getNumTokens() {+ return numTokens;+ }++ public int getMaxNumTokens() {+ return maxNumTokens;+ }++ public int getMaxCardinality() {+ return maxCardinality;+ }++ public long getCardinalityStart() {+ return cardinalityStart;+ }++ public int getMaxLength() {+ return maxLength;+ }++ public int getLength() {+ return length;+ }++ public DocMaker.Content getContent() {+ return content;+ }++ public static final class FieldDefBuilder {++ private DocMaker.Content content;+ private int numTokens = 1;+ private int maxNumTokens = -1;+ private int maxCardinality = -1;+ private int maxLength = -1;+ private int length = -1;+ private long cardinalityStart;++ private FieldDefBuilder() {}++ public static FieldDefBuilder aFieldDef() {+ return new FieldDefBuilder();+ }++ public FieldDefBuilder withContent(DocMaker.Content content) {+ this.content = content;+ return this;+ }++ public FieldDefBuilder withTokenCount(int numTokens) {+ if (numTokens > 1 && content == DocMaker.Content.UNIQUE_INT) {+ throw new UnsupportedOperationException(+ "UNIQUE_INT content type cannot be used with token count > 1");+ }+ if (maxCardinality > 1) {+ throw new UnsupportedOperationException(+ "tokenCount cannot be used with maxCardinality > 0");+ }+ this.numTokens = numTokens;+ return this;+ }++ public FieldDefBuilder withMaxTokenCount(int maxNumTokens) {+ if (numTokens > 1 && content == DocMaker.Content.UNIQUE_INT) {+ throw new UnsupportedOperationException(+ "UNIQUE_INT content type cannot be used with token count > 1");+ }+ if (maxCardinality > 1) {+ throw new UnsupportedOperationException(+ "maxNumTokens cannot be used with maxCardinality > 0");+ }+ this.maxNumTokens = maxNumTokens;+ return this;+ }++ public FieldDefBuilder withMaxCardinality(int maxCardinality, SplittableRandom random) {+ if (numTokens > 1) {+ throw new UnsupportedOperationException(+ "maxCardinality cannot be used with token count > 1");+ }+ this.maxCardinality = maxCardinality;+ this.cardinalityStart = random.nextLong(0, Long.MAX_VALUE - maxCardinality);+ return this;+ }++ public FieldDefBuilder withMaxLength(int maxLength) {+ if (length > -1) {+ throw new UnsupportedOperationException("maxLength cannot be used with length");+ }+ this.maxLength = maxLength;+ return this;+ }++ public FieldDefBuilder withLength(int length) {+ if (maxLength > -1) {+ throw new UnsupportedOperationException("length cannot be used with maxLength");+ }+ this.length = length;+ return this;+ }++ public FieldDef build() {+ FieldDef fieldDef = new FieldDef();+ fieldDef.numTokens = this.numTokens;+ fieldDef.maxNumTokens = this.maxNumTokens;+ fieldDef.content = this.content;+ fieldDef.maxCardinality = this.maxCardinality;+ fieldDef.maxLength = this.maxLength == -1 ? DEFAULT_MAX_LENGTH : this.maxLength;+ fieldDef.length = this.length;+ fieldDef.cardinalityStart = this.cardinalityStart;+ return fieldDef;+ }+ }++ @Override+ public String toString() {+ return "FieldDef{"+ + "content="+ + content+ + ", numTokens="+ + numTokens+ + ", maxNumTokens="+ + maxNumTokens+ + ", maxCardinality="+ + maxCardinality+ + ", maxLength="+ + maxLength+ + ", length="+ + length+ + ", cardinalityStart="+ + cardinalityStart+ + '}';+ }+}diff --git a/solr/benchmark/src/java/org/apache/solr/bench/MiniClusterState.java b/solr/benchmark/src/java/org/apache/solr/bench/MiniClusterState.javanew file mode 100755index 00000000000..4d2f4984a86--- /dev/null+++ b/solr/benchmark/src/java/org/apache/solr/bench/MiniClusterState.java@@ -0,0 +1,371 @@+/*+ * Licensed to the Apache Software Foundation (ASF) under one or more+ * contributor license agreements. See the NOTICE file distributed with+ * this work for additional information regarding copyright ownership.+ * The ASF licenses this file to You under the Apache License, Version 2.0+ * (the "License"); you may not use this file except in compliance with+ * the License. You may obtain a copy of the License at+ *+ * http://www.apache.org/licenses/LICENSE-2.0+ *+ * Unless required by applicable law or agreed to in writing, software+ * distributed under the License is distributed on an "AS IS" BASIS,+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.+ * See the License for the specific language governing permissions and+ * limitations under the License.+ */+package org.apache.solr.bench;++import static org.apache.commons.io.file.PathUtils.deleteDirectory;++import com.codahale.metrics.Meter;+import java.io.IOException;+import java.lang.management.ManagementFactory;+import java.nio.file.Files;+import java.nio.file.Path;+import java.nio.file.Paths;+import java.util.ArrayList;+import java.util.List;+import java.util.Random;+import java.util.SplittableRandom;+import java.util.concurrent.ExecutorService;+import java.util.concurrent.Executors;+import java.util.concurrent.ScheduledExecutorService;+import java.util.concurrent.TimeUnit;+import org.apache.commons.io.output.NullPrintStream;+import org.apache.solr.client.solrj.SolrClient;+import org.apache.solr.client.solrj.SolrQuery;+import org.apache.solr.client.solrj.embedded.JettySolrRunner;+import org.apache.solr.client.solrj.impl.Http2SolrClient;+import org.apache.solr.client.solrj.request.CollectionAdminRequest;+import org.apache.solr.client.solrj.request.QueryRequest;+import org.apache.solr.client.solrj.request.UpdateRequest;+import org.apache.solr.cloud.MiniSolrCloudCluster;+import org.apache.solr.common.SolrInputDocument;+import org.apache.solr.common.params.ModifiableSolrParams;+import org.apache.solr.common.util.IOUtils;+import org.apache.solr.common.util.NamedList;+import org.apache.solr.common.util.SolrNamedThreadFactory;+import org.apache.solr.common.util.SuppressForbidden;+import org.openjdk.jmh.annotations.Level;+import org.openjdk.jmh.annotations.Scope;+import org.openjdk.jmh.annotations.Setup;+import org.openjdk.jmh.annotations.State;+import org.openjdk.jmh.annotations.TearDown;+import org.openjdk.jmh.infra.BenchmarkParams;+import org.openjdk.jmh.infra.Control;++/** The base class for Solr JMH benchmarks that operate against a {@link MiniSolrCloudCluster}. */+public class MiniClusterState {++ public static final boolean DEBUG_OUTPUT = false;++ private static final long RANDOM_SEED = 6624420638116043983L;++ public static final int PROC_COUNT =+ ManagementFactory.getOperatingSystemMXBean().getAvailableProcessors();++ private static boolean quietLog = Boolean.getBoolean("quietLog");++ @SuppressForbidden(reason = "JMH uses std out for user output")+ public static void log(String value) {+ if (!quietLog) {+ System.out.println((value.equals("") ? "" : "--> ") + value);+ }+ }++ @State(Scope.Benchmark)+ public static class MiniClusterBenchState {++ boolean metricsEnabled = true;++ public List<String> nodes;+ MiniSolrCloudCluster cluster;+ public SolrClient client;++ int runCnt = 0;++ boolean createCollectionAndIndex = true;++ boolean deleteMiniCluster = true;++ Path baseDir;+ boolean allowClusterReuse = false;++ boolean isWarmup;++ private SplittableRandom random;++ @TearDown(Level.Iteration)+ public void tearDown(BenchmarkParams benchmarkParams) throws Exception {++ // dump Solr metrics+ Path metricsResults =+ Paths.get(+ "work/metrics-results",+ benchmarkParams.id(),+ String.valueOf(runCnt++),+ benchmarkParams.getBenchmark() + ".txt");+ if (!Files.exists(metricsResults.getParent())) {+ Files.createDirectories(metricsResults.getParent());+ }++ cluster.outputMetrics(+ metricsResults.getParent().toFile(), metricsResults.getFileName().toString());+ }++ @Setup(Level.Iteration)+ public void checkWarmUp(Control control) throws Exception {+ isWarmup = control.stopMeasurement;+ }++ @TearDown(Level.Trial)+ public void shutdownMiniCluster() throws Exception {+ if (DEBUG_OUTPUT) log("closing client and shutting down minicluster");+ IOUtils.closeQuietly(client);+ cluster.shutdown();+ }++ @Setup(Level.Trial)+ public void doSetup(BenchmarkParams benchmarkParams) throws Exception {++ MiniClusterState.log("");+ Path currentRelativePath = Paths.get("");+ String s = currentRelativePath.toAbsolutePath().toString();+ log("current relative path is: " + s);++ Long seed = Long.getLong("solr.bench.seed");++ if (seed == null) {+ seed = RANDOM_SEED;+ }++ log("benchmark random seed: " + seed);++ // set the seed used by ThreadLocalRandom+ System.setProperty("randomSeed", Long.toString(new Random(seed).nextLong()));++ this.random = new SplittableRandom(seed);++ System.setProperty("pkiHandlerPrivateKeyPath", "");+ System.setProperty("pkiHandlerPublicKeyPath", "");++ System.setProperty("solr.log.name", benchmarkParams.id());++ System.setProperty("solr.default.confdir", "../server/solr/configsets/_default");++ // not currently usable, but would enable JettySolrRunner's ill-conceived jetty.testMode and+ // allow using SSL++ // System.getProperty("jetty.testMode", "true");+ // SolrCloudTestCase.sslConfig = SolrTestCaseJ4.buildSSLConfig();++ String baseDirSysProp = System.getProperty("miniClusterBaseDir");+ if (baseDirSysProp != null) {+ deleteMiniCluster = false;+ baseDir = Paths.get(baseDirSysProp);+ if (Files.exists(baseDir)) {+ createCollectionAndIndex = false;+ allowClusterReuse = true;+ }+ } else {+ baseDir = Paths.get("work/mini-cluster");+ }++ System.setProperty("metricsEnabled", String.valueOf(metricsEnabled));+ }++ public void metricsEnabled(boolean metricsEnabled) {+ this.metricsEnabled = metricsEnabled;+ }++ public void startMiniCluster(int nodeCount) {+ log("starting mini cluster at base directory: " + baseDir.toAbsolutePath());++ if (!allowClusterReuse && Files.exists(baseDir)) {+ log("mini cluster base directory exists, removing ...");+ try {+ deleteDirectory(baseDir);+ } catch (IOException e) {+ throw new RuntimeException(e);+ }+ createCollectionAndIndex = true;+ } else if (Files.exists(baseDir)) {+ createCollectionAndIndex = false;+ deleteMiniCluster = false;+ }++ try {+ cluster =+ new MiniSolrCloudCluster.Builder(nodeCount, baseDir)+ .formatZkServer(false)+ .addConfig("conf", Paths.get("src/resources/configs/cloud-minimal/conf"))+ .configure();+ } catch (Exception e) {+ if (Files.exists(baseDir)) {+ try {+ deleteDirectory(baseDir);+ } catch (IOException ex) {+ e.addSuppressed(ex);+ }+ }+ throw new RuntimeException(e);+ }++ nodes = new ArrayList<>(nodeCount);+ List<JettySolrRunner> jetties = cluster.getJettySolrRunners();+ for (JettySolrRunner runner : jetties) {+ nodes.add(runner.getBaseUrl().toString());+ }++ client = new Http2SolrClient.Builder().build();++ log("done starting mini cluster");+ log("");+ }++ public SplittableRandom getRandom() {+ return random;+ }++ public void createCollection(String collection, int numShards, int numReplicas)+ throws Exception {+ if (createCollectionAndIndex) {+ try {++ CollectionAdminRequest.Create request =+ CollectionAdminRequest.createCollection(collection, "conf", numShards, numReplicas);+ request.setBasePath(nodes.get(random.nextInt(cluster.getJettySolrRunners().size())));++ client.request(request);++ cluster.waitForActiveCollection(+ collection, 15, TimeUnit.SECONDS, numShards, numShards * numReplicas);+ } catch (Exception e) {+ if (Files.exists(baseDir)) {+ deleteDirectory(baseDir);+ }+ throw e;+ }+ }+ }++ @SuppressForbidden(reason = "This module does not need to deal with logging context")+ public void index(String collection, DocMaker docMaker, int docCount) throws Exception {+ if (createCollectionAndIndex) {++ log("indexing data for benchmark...");+ Meter meter = new Meter();+ ExecutorService executorService =+ Executors.newFixedThreadPool(+ Runtime.getRuntime().availableProcessors(),+ new SolrNamedThreadFactory("SolrJMH Indexer Progress"));+ ScheduledExecutorService scheduledExecutor =+ Executors.newSingleThreadScheduledExecutor(+ new SolrNamedThreadFactory("SolrJMH Indexer"));+ scheduledExecutor.scheduleAtFixedRate(+ () -> {+ if (meter.getCount() == docCount) {+ scheduledExecutor.shutdown();+ } else {+ log(meter.getCount() + " docs at " + meter.getMeanRate() + " doc/s");+ }+ },+ 10,+ 10,+ TimeUnit.SECONDS);+ for (int i = 0; i < docCount; i++) {+ executorService.submit(+ new Runnable() {+ SplittableRandom threadRandom = random.split();++ @Override+ public void run() {+ UpdateRequest updateRequest = new UpdateRequest();+ updateRequest.setBasePath(+ nodes.get(threadRandom.nextInt(cluster.getJettySolrRunners().size())));+ SolrInputDocument doc = docMaker.getDocument(threadRandom);+ // log("add doc " + doc);+ updateRequest.add(doc);+ meter.mark();++ try {+ client.request(updateRequest, collection);+ } catch (Exception e) {+ throw new RuntimeException(e);+ }+ }+ });+ }++ log("done adding docs, waiting for executor to terminate...");++ executorService.shutdown();+ boolean result = executorService.awaitTermination(600, TimeUnit.MINUTES);++ scheduledExecutor.shutdown();++ if (!result) {+ throw new RuntimeException("Timeout waiting for doc adds to finish");+ }+ log("done indexing data for benchmark");++ log("committing data ...");+ UpdateRequest commitRequest = new UpdateRequest();+ commitRequest.setBasePath(nodes.get(random.nextInt(cluster.getJettySolrRunners().size())));+ commitRequest.setAction(UpdateRequest.ACTION.COMMIT, false, true);+ commitRequest.process(client, collection);+ log("done committing data");+ } else {+ cluster.waitForActiveCollection(collection, 15, TimeUnit.SECONDS);+ }++ QueryRequest queryRequest = new QueryRequest(new SolrQuery("q", "*:*", "rows", "1"));+ queryRequest.setBasePath(nodes.get(random.nextInt(cluster.getJettySolrRunners().size())));++ NamedList<Object> result = client.request(queryRequest, collection);++ if (DEBUG_OUTPUT) MiniClusterState.log("result: " + result);++ MiniClusterState.log("");++ MiniClusterState.log("Dump Core Info");+ dumpCoreInfo();+ }++ public void waitForMerges(String collection) throws Exception {+ forceMerge(collection, Integer.MAX_VALUE);+ }++ public void forceMerge(String collection, int maxMergeSegments) throws Exception {+ if (createCollectionAndIndex) {+ // we control segment count for a more informative benchmark *and* because background+ // merging would continue after+ // indexing and overlap with the benchmark+ if (maxMergeSegments == Integer.MAX_VALUE) {+ log("waiting for merges to finish...\n");+ } else {+ log("merging segments to " + maxMergeSegments + " segments ...\n");+ }++ UpdateRequest optimizeRequest = new UpdateRequest();+ optimizeRequest.setBasePath(+ nodes.get(random.nextInt(cluster.getJettySolrRunners().size())));+ optimizeRequest.setAction(UpdateRequest.ACTION.OPTIMIZE, false, true, maxMergeSegments);+ optimizeRequest.process(client, collection);+ }+ }++ @SuppressForbidden(reason = "JMH uses std out for user output")+ public void dumpCoreInfo() throws IOException {+ cluster.dumpCoreInfo(!quietLog ? System.out : new NullPrintStream());+ }+ }++ public static ModifiableSolrParams params(ModifiableSolrParams params, String... moreParams) {+ for (int i = 0; i < moreParams.length; i += 2) {+ params.add(moreParams[i], moreParams[i + 1]);+ }+ return params;+ }+}diff --git a/solr/benchmark/src/java/org/apache/solr/bench/index/CloudIndexing.java b/solr/benchmark/src/java/org/apache/solr/bench/index/CloudIndexing.javanew file mode 100755index 00000000000..bac84064c9e--- /dev/null+++ b/solr/benchmark/src/java/org/apache/solr/bench/index/CloudIndexing.java@@ -0,0 +1,177 @@+/*+ * Licensed to the Apache Software Foundation (ASF) under one or more+ * contributor license agreements. See the NOTICE file distributed with+ * this work for additional information regarding copyright ownership.+ * The ASF licenses this file to You under the Apache License, Version 2.0+ * (the "License"); you may not use this file except in compliance with+ * the License. You may obtain a copy of the License at+ *+ * http://www.apache.org/licenses/LICENSE-2.0+ *+ * Unless required by applicable law or agreed to in writing, software+ * distributed under the License is distributed on an "AS IS" BASIS,+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.+ * See the License for the specific language governing permissions and+ * limitations under the License.+ */+package org.apache.solr.bench.index;++import java.util.Iterator;+import java.util.SplittableRandom;+import java.util.concurrent.TimeUnit;+import org.apache.solr.bench.DocMaker;+import org.apache.solr.bench.FieldDef;+import org.apache.solr.bench.MiniClusterState;+import org.apache.solr.client.solrj.request.UpdateRequest;+import org.apache.solr.common.SolrInputDocument;+import org.openjdk.jmh.annotations.Benchmark;+import org.openjdk.jmh.annotations.BenchmarkMode;+import org.openjdk.jmh.annotations.Fork;+import org.openjdk.jmh.annotations.Level;+import org.openjdk.jmh.annotations.Measurement;+import org.openjdk.jmh.annotations.Mode;+import org.openjdk.jmh.annotations.OutputTimeUnit;+import org.openjdk.jmh.annotations.Param;+import org.openjdk.jmh.annotations.Scope;+import org.openjdk.jmh.annotations.Setup;+import org.openjdk.jmh.annotations.State;+import org.openjdk.jmh.annotations.TearDown;+import org.openjdk.jmh.annotations.Threads;+import org.openjdk.jmh.annotations.Timeout;+import org.openjdk.jmh.annotations.Warmup;++@BenchmarkMode(Mode.Throughput)+@OutputTimeUnit(TimeUnit.SECONDS)+@Threads(6)+@Warmup(time = 3, iterations = 5)+@Measurement(time = 15, iterations = 5)+@Fork(value = 1)+@Timeout(time = 60)+/** A benchmark to experiment with the performance of distributed indexing. */+public class CloudIndexing {++ @State(Scope.Benchmark)+ public static class BenchState {++ String collection = "testCollection";++ @Param("4")+ int nodeCount;++ @Param("5")+ int numShards;++ @Param({"1", "3", "9"})+ int numReplicas;++ @Param({"100000"})+ public int docCount;++ @Setup(Level.Iteration)+ public void doSetup(MiniClusterState.MiniClusterBenchState miniClusterState) throws Exception {++ miniClusterState.startMiniCluster(nodeCount);++ miniClusterState.createCollection(collection, numShards, numReplicas);+ }++ @TearDown(Level.Iteration)+ public void doTearDown(MiniClusterState.MiniClusterBenchState miniClusterState)+ throws Exception {+ miniClusterState.shutdownMiniCluster();+ }++ @State(Scope.Thread)+ public static class Docs {++ private DocMaker largeDocMaker;+ private Iterator<SolrInputDocument> largeDocIterator;++ private DocMaker smallDocMaker;+ private Iterator<SolrInputDocument> smallDocIterator;++ private SplittableRandom random;++ @Setup(Level.Trial)+ public void setupDoc(BenchState state) throws Exception {+ Long seed = Long.getLong("randomSeed");++ random = new SplittableRandom(seed);++ largeDocMaker = new DocMaker();+ largeDocMaker.addField(+ "id", FieldDef.FieldDefBuilder.aFieldDef().withContent(DocMaker.Content.UNIQUE_INT));+ largeDocMaker.addField(+ "text",+ FieldDef.FieldDefBuilder.aFieldDef()+ .withContent(DocMaker.Content.ALPHEBETIC)+ .withMaxLength(64)+ .withTokenCount(random.nextInt(350, 512)));++ largeDocMaker.preGenerateDocs(state.docCount, random);++ largeDocIterator = largeDocMaker.getGeneratedDocsIterator();++ smallDocMaker = new DocMaker();+ smallDocMaker.addField(+ "id", FieldDef.FieldDefBuilder.aFieldDef().withContent(DocMaker.Content.UNIQUE_INT));+ smallDocMaker.addField(+ "text",+ FieldDef.FieldDefBuilder.aFieldDef()+ .withContent(DocMaker.Content.ALPHEBETIC)+ .withMaxLength(32)+ .withTokenCount(1));++ smallDocMaker.preGenerateDocs(state.docCount, random);++ smallDocIterator = smallDocMaker.getGeneratedDocsIterator();+ }++ public SolrInputDocument getLargeDoc() {+ if (!largeDocIterator.hasNext()) {+ largeDocIterator = largeDocMaker.getGeneratedDocsIterator();+ }+ return largeDocIterator.next();+ }++ public SolrInputDocument getSmallDoc() {+ if (!smallDocIterator.hasNext()) {+ smallDocIterator = smallDocMaker.getGeneratedDocsIterator();+ }+ return smallDocIterator.next();+ }+ }+ }++ @Benchmark+ @Timeout(time = 300)+ public Object indexLargeDoc(+ MiniClusterState.MiniClusterBenchState miniClusterState,+ BenchState state,+ BenchState.Docs docState)+ throws Exception {+ UpdateRequest updateRequest = new UpdateRequest();+ updateRequest.setBasePath(miniClusterState.nodes.get(docState.random.nextInt(state.nodeCount)));+ SolrInputDocument doc = docState.getLargeDoc();++ updateRequest.add(doc);++ return miniClusterState.client.request(updateRequest, state.collection);+ }++ @Benchmark+ @Timeout(time = 300)+ public Object indexSmallDoc(+ MiniClusterState.MiniClusterBenchState miniClusterState,+ BenchState state,+ BenchState.Docs docState)+ throws Exception {+ UpdateRequest updateRequest = new UpdateRequest();+ updateRequest.setBasePath(miniClusterState.nodes.get(docState.random.nextInt(state.nodeCount)));+ SolrInputDocument doc = docState.getSmallDoc();++ updateRequest.add(doc);++ return miniClusterState.client.request(updateRequest, state.collection);+ }+}diff --git a/solr/benchmark/src/java/org/apache/solr/bench/index/package-info.java b/solr/benchmark/src/java/org/apache/solr/bench/index/package-info.javanew file mode 100644index 00000000000..a7f2757f8aa--- /dev/null+++ b/solr/benchmark/src/java/org/apache/solr/bench/index/package-info.java@@ -0,0 +1,19 @@+/*+ * Licensed to the Apache Software Foundation (ASF) under one or more+ * contributor license agreements. See the NOTICE file distributed with+ * this work for additional information regarding copyright ownership.+ * The ASF licenses this file to You under the Apache License, Version 2.0+ * (the "License"); you may not use this file except in compliance with+ * the License. You may obtain a copy of the License at+ *+ * http://www.apache.org/licenses/LICENSE-2.0+ *+ * Unless required by applicable law or agreed to in writing, software+ * distributed under the License is distributed on an "AS IS" BASIS,+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.+ * See the License for the specific language governing permissions and+ * limitations under the License.+ */++/** Solr JMH benchmarks focused on indexing */+package org.apache.solr.bench.index;diff --git a/solr/benchmark/src/java/org/apache/solr/bench/package-info.java b/solr/benchmark/src/java/org/apache/solr/bench/package-info.javanew file mode 100644index 00000000000..8295b57859c--- /dev/null+++ b/solr/benchmark/src/java/org/apache/solr/bench/package-info.java@@ -0,0 +1,19 @@+/*+ * Licensed to the Apache Software Foundation (ASF) under one or more+ * contributor license agreements. See the NOTICE file distributed with+ * this work for additional information regarding copyright ownership.+ * The ASF licenses this file to You under the Apache License, Version 2.0+ * (the "License"); you may not use this file except in compliance with+ * the License. You may obtain a copy of the License at+ *+ * http://www.apache.org/licenses/LICENSE-2.0+ *+ * Unless required by applicable law or agreed to in writing, software+ * distributed under the License is distributed on an "AS IS" BASIS,+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.+ * See the License for the specific language governing permissions and+ * limitations under the License.+ */++/** Support for running Solr benchmarks using OpenJDK's JMH */+package org.apache.solr.bench;diff --git a/solr/benchmark/src/java/org/apache/solr/bench/search/JsonFaceting.java b/solr/benchmark/src/java/org/apache/solr/bench/search/JsonFaceting.javanew file mode 100755index 00000000000..088c703e67a--- /dev/null+++ b/solr/benchmark/src/java/org/apache/solr/bench/search/JsonFaceting.java@@ -0,0 +1,217 @@+/*+ * Licensed to the Apache Software Foundation (ASF) under one or more+ * contributor license agreements. See the NOTICE file distributed with+ * this work for additional information regarding copyright ownership.+ * The ASF licenses this file to You under the Apache License, Version 2.0+ * (the "License"); you may not use this file except in compliance with+ * the License. You may obtain a copy of the License at+ *+ * http://www.apache.org/licenses/LICENSE-2.0+ *+ * Unless required by applicable law or agreed to in writing, software+ * distributed under the License is distributed on an "AS IS" BASIS,+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.+ * See the License for the specific language governing permissions and+ * limitations under the License.+ */+package org.apache.solr.bench.search;++import java.util.SplittableRandom;+import java.util.concurrent.TimeUnit;+import org.apache.solr.bench.DocMaker;+import org.apache.solr.bench.FieldDef;+import org.apache.solr.bench.MiniClusterState;+import org.apache.solr.client.solrj.request.QueryRequest;+import org.apache.solr.common.params.ModifiableSolrParams;+import org.apache.solr.common.util.NamedList;+import org.openjdk.jmh.annotations.Benchmark;+import org.openjdk.jmh.annotations.BenchmarkMode;+import org.openjdk.jmh.annotations.Fork;+import org.openjdk.jmh.annotations.Level;+import org.openjdk.jmh.annotations.Measurement;+import org.openjdk.jmh.annotations.Mode;+import org.openjdk.jmh.annotations.OutputTimeUnit;+import org.openjdk.jmh.annotations.Param;+import org.openjdk.jmh.annotations.Scope;+import org.openjdk.jmh.annotations.Setup;+import org.openjdk.jmh.annotations.State;+import org.openjdk.jmh.annotations.Threads;+import org.openjdk.jmh.annotations.Timeout;+import org.openjdk.jmh.annotations.Warmup;+import org.openjdk.jmh.infra.BenchmarkParams;++/** A benchmark to experiment with the performance of json faceting. */+@BenchmarkMode(Mode.Throughput)+@OutputTimeUnit(TimeUnit.SECONDS)+@Threads(1)+@Warmup(time = 5, iterations = 3)+@Measurement(time = 15, iterations = 5)+@Fork(value = 1)+@Timeout(time = 60)+public class JsonFaceting {++ @State(Scope.Benchmark)+ public static class BenchState {++ public String collection = "testCollection";++ @Param({"100000"})+ public int docCount;++ @Param("2")+ int nodeCount;++ @Param("1")+ int numReplicas;++ @Param("2")+ int numShards;++ // DV, // DocValues, collect into ordinal array+ // UIF, // UnInvertedField, collect into ordinal array+ // DVHASH, // DocValues, collect into hash+ // ENUM, // TermsEnum then intersect DocSet (stream-able)+ // STREAM, // presently equivalent to ENUM+ // SMART,+ // "dv"+ // "uif"+ // "dvhash">+ // "enum"+ // "stream"+ // "smart"+ @Param({"smart"})+ String fm;++ @Param({"5000"})+ int facetCard;++ @Param({"1000"})+ int facetCard2;++ private ModifiableSolrParams params;++ @Setup(Level.Trial)+ public void setup(+ BenchmarkParams benchmarkParams, MiniClusterState.MiniClusterBenchState miniClusterState)+ throws Exception {++ SplittableRandom random = miniClusterState.getRandom();++ System.setProperty("maxMergeAtOnce", "20");+ System.setProperty("segmentsPerTier", "20");++ miniClusterState.startMiniCluster(nodeCount);++ miniClusterState.createCollection(collection, numShards, numReplicas);++ // Define random documents+ DocMaker docMaker = new DocMaker();+ docMaker.addField(+ "id", FieldDef.FieldDefBuilder.aFieldDef().withContent(DocMaker.Content.UNIQUE_INT));+ docMaker.addField(+ "facet_s",+ FieldDef.FieldDefBuilder.aFieldDef()+ .withContent(DocMaker.Content.ALPHEBETIC)+ .withMaxLength(64)+ .withMaxCardinality(facetCard, random));+ docMaker.addField(+ "facet2_s",+ FieldDef.FieldDefBuilder.aFieldDef()+ .withContent(DocMaker.Content.ALPHEBETIC)+ .withMaxLength(16)+ .withMaxCardinality(facetCard, random));+ docMaker.addField(+ "facet3_s",+ FieldDef.FieldDefBuilder.aFieldDef()+ .withContent(DocMaker.Content.UNICODE)+ .withMaxLength(128)+ .withMaxCardinality(facetCard2, random));+ docMaker.addField(+ "text",+ FieldDef.FieldDefBuilder.aFieldDef()+ .withContent(DocMaker.Content.ALPHEBETIC)+ .withMaxLength(64)+ .withTokenCount(random.nextInt(350, 512)));+ docMaker.addField(+ "int_i", FieldDef.FieldDefBuilder.aFieldDef().withContent(DocMaker.Content.INTEGER));+ docMaker.addField(+ "int2_i",+ FieldDef.FieldDefBuilder.aFieldDef()+ .withContent(DocMaker.Content.INTEGER)+ .withMaxCardinality(facetCard2, random));+ docMaker.addField(+ "int3_i",+ FieldDef.FieldDefBuilder.aFieldDef()+ .withContent(DocMaker.Content.INTEGER)+ .withMaxCardinality(facetCard2, random));+ docMaker.addField(+ "int4_i",+ FieldDef.FieldDefBuilder.aFieldDef()+ .withContent(DocMaker.Content.INTEGER)+ .withMaxCardinality(facetCard2, random));++ miniClusterState.index(collection, docMaker, docCount);+ miniClusterState.forceMerge(collection, 15);++ params = new ModifiableSolrParams();++ MiniClusterState.params(+ params,+ "q",+ "*:*",+ "json.facet",+ "{f1:{method:'"+ + fm+ + "', type:terms, field:'facet_s', sort:'x desc', facet:{x:'min(int3_i)'} }"+ + " , f2:{method:'"+ + fm+ + "',, type:terms, field:'facet_s', sort:'x desc', facet:{x:'max(int3_i)'} } "+ + " , f3:{method:'"+ + fm+ + "', type:terms, field:'facet_s', sort:'x desc', facet:{x:'unique(facet2_s)'} } "+ + " , f4:{method:'"+ + fm+ + "', type:terms, field:'facet_s', sort:'x desc', facet:{x:'hll(facet2_s)'} } "+ + " , f5:{method:'"+ + fm+ + "', type:terms, field:'facet_s', sort:'x desc', facet:{x:'variance(int3_i)'} } "+ + " , f6:{type:terms, field:'int3_i', limit:1, sort:'x desc', facet:{x:'hll(int2_i)'} } "+ + " , f7:{type:terms, field:'facet_s', limit:2, sort:'x desc', facet:{x:'missing(int4_i)'} } "+ + " , f8:{type:terms, field:'facet_s', limit:2, sort:'x desc', facet:{x:'countvals(int4_i)'} } "+ + "}");++ // MiniClusterState.log("params: " + params + "\n");+ }++ @State(Scope.Thread)+ public static class ThreadState {++ private SplittableRandom random;++ @Setup(Level.Trial)+ public void setup() throws Exception {+ Long seed = Long.getLong("randomSeed");++ this.random = new SplittableRandom(seed);+ }+ }+ }++ @Benchmark+ @Timeout(time = 500, timeUnit = TimeUnit.SECONDS)+ public Object jsonFacet(+ MiniClusterState.MiniClusterBenchState miniClusterState,+ BenchState state,+ BenchState.ThreadState threadState)+ throws Exception {+ QueryRequest queryRequest = new QueryRequest(state.params);+ queryRequest.setBasePath(+ miniClusterState.nodes.get(threadState.random.nextInt(state.nodeCount)));++ NamedList<Object> result = miniClusterState.client.request(queryRequest, state.collection);++ // MiniClusterState.log("result: " + result);++ return result;+ }+}diff --git a/solr/benchmark/src/java/org/apache/solr/bench/search/package-info.java b/solr/benchmark/src/java/org/apache/solr/bench/search/package-info.javanew file mode 100644index 00000000000..a0d9133d087--- /dev/null+++ b/solr/benchmark/src/java/org/apache/solr/bench/search/package-info.java@@ -0,0 +1,19 @@+/*+ * Licensed to the Apache Software Foundation (ASF) under one or more+ * contributor license agreements. See the NOTICE file distributed with+ * this work for additional information regarding copyright ownership.+ * The ASF licenses this file to You under the Apache License, Version 2.0+ * (the "License"); you may not use this file except in compliance with+ * the License. You may obtain a copy of the License at+ *+ * http://www.apache.org/licenses/LICENSE-2.0+ *+ * Unless required by applicable law or agreed to in writing, software+ * distributed under the License is distributed on an "AS IS" BASIS,+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.+ * See the License for the specific language governing permissions and+ * limitations under the License.+ */++/** Solr JMH benchmarks focused on search */+package org.apache.solr.bench.search;diff --git a/solr/benchmark/src/resources/configs/cloud-minimal/conf/schema.xml b/solr/benchmark/src/resources/configs/cloud-minimal/conf/schema.xmlnew file mode 100644index 00000000000..727f5e2837f--- /dev/null+++ b/solr/benchmark/src/resources/configs/cloud-minimal/conf/schema.xml@@ -0,0 +1,37 @@+<?xml version="1.0" encoding="UTF-8" ?>+<!--+ Licensed to the Apache Software Foundation (ASF) under one or more+ contributor license agreements. See the NOTICE file distributed with+ this work for additional information regarding copyright ownership.+ The ASF licenses this file to You under the Apache License, Version 2.0+ (the "License"); you may not use this file except in compliance with+ the License. You may obtain a copy of the License at++ http://www.apache.org/licenses/LICENSE-2.0++ Unless required by applicable law or agreed to in writing, software+ distributed under the License is distributed on an "AS IS" BASIS,+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.+ See the License for the specific language governing permissions and+ limitations under the License.+-->+<schema name="minimal" version="1.1">+ <fieldType name="string" class="solr.StrField" docValues="true"/>+ <fieldType name="int" class="org.apache.solr.schema.IntPointField" docValues="true" omitNorms="true" positionIncrementGap="0"/>+ <fieldType name="long" class="org.apache.solr.schema.LongPointField" docValues="true" omitNorms="true" positionIncrementGap="0"/>+ <fieldType name="text" class="solr.TextField">+ <analyzer>+ <tokenizer class="solr.StandardTokenizerFactory"/>+ <filter class="solr.LowerCaseFilterFactory"/>+ </analyzer>+ </fieldType>++ <!-- for versioning -->+ <field name="_version_" type="long" indexed="true" stored="true"/>+ <field name="_root_" type="string" indexed="true" stored="true" multiValued="false" required="false"/>+ <field name="id" type="string" indexed="true" stored="true"/>+ <field name="text" type="text" indexed="true" stored="false"/>+ <dynamicField name="*_s" type="string" indexed="true" stored="true" />+ <dynamicField name="*_i" type="int" indexed="false" stored="false" />+ <uniqueKey>id</uniqueKey>+</schema>diff --git a/solr/benchmark/src/resources/configs/cloud-minimal/conf/solrconfig.xml b/solr/benchmark/src/resources/configs/cloud-minimal/conf/solrconfig.xmlnew file mode 100644index 00000000000..9ddb0721f01--- /dev/null+++ b/solr/benchmark/src/resources/configs/cloud-minimal/conf/solrconfig.xml@@ -0,0 +1,110 @@+<?xml version="1.0" ?>++<!--+ Licensed to the Apache Software Foundation (ASF) under one or more+ contributor license agreements. See the NOTICE file distributed with+ this work for additional information regarding copyright ownership.+ The ASF licenses this file to You under the Apache License, Version 2.0+ (the "License"); you may not use this file except in compliance with+ the License. You may obtain a copy of the License at++ http://www.apache.org/licenses/LICENSE-2.0++ Unless required by applicable law or agreed to in writing, software+ distributed under the License is distributed on an "AS IS" BASIS,+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.+ See the License for the specific language governing permissions and+ limitations under the License.+-->++<!-- Minimal solrconfig.xml with /select, /admin and /update only -->++<config>++ <dataDir>${solr.data.dir:}</dataDir>++ <directoryFactory name="DirectoryFactory"+ class="${solr.directoryFactory:solr.NRTCachingDirectoryFactory}"/>+ <schemaFactory class="ClassicIndexSchemaFactory"/>++ <luceneMatchVersion>${tests.luceneMatchVersion:LATEST}</luceneMatchVersion>++ <indexConfig>+ <mergePolicyFactory class="org.apache.solr.index.TieredMergePolicyFactory">+ <int name="maxMergeAtOnce">${maxMergeAtOnce:10}</int>+ <int name="segmentsPerTier">${segmentsPerTier:10}</int>+ <double name="noCFSRatio">${noCFSRatio:.1}</double>+ </mergePolicyFactory>++ <useCompoundFile>${useCompoundFile:true}</useCompoundFile>++ <ramBufferSizeMB>${solr.ramBufferSizeMB:100}</ramBufferSizeMB>+ <!-- <ramPerThreadHardLimitMB>60</ramPerThreadHardLimitMB> -->++ <!-- <mergeScheduler class="org.apache.lucene.index.ConcurrentMergeScheduler">+ <int name="maxThreadCount">6</int>+ <int name="maxMergeCount">8</int>+ <bool name="ioThrottle">false</bool>+ </mergeScheduler> -->++ <writeLockTimeout>1000</writeLockTimeout>+ <commitLockTimeout>10000</commitLockTimeout>++ <!-- this sys property is not set by SolrTestCaseJ4 because almost all tests should+ use the single process lockType for speed - but tests that explicitly need+ to vary the lockType canset it as needed.+ -->+ <lockType>${solr.lockType:single}</lockType>++ <infoStream>${solr.tests.infostream:false}</infoStream>++ </indexConfig>++ <updateHandler class="solr.DirectUpdateHandler2">+ <commitWithin>+ <softCommit>${solr.commitwithin.softcommit:true}</softCommit>+ </commitWithin>+ <autoCommit>+ <maxTime>${solr.autoCommit.maxTime:60000}</maxTime>+ </autoCommit>+ <updateLog class="${solr.ulog:solr.UpdateLog}" enable="${enable.update.log:true}"/>+ </updateHandler>++ <requestHandler name="/select" class="solr.SearchHandler">+ <lst name="defaults">+ <str name="echoParams">explicit</str>+ <str name="indent">true</str>+ <str name="df">text</str>+ </lst>++ </requestHandler>++ <query>+ <queryResultCache+ enabled ="${queryResultCache.enabled:false}"+ class="${queryResultCache.class:solr.LRUCache}"+ size="${queryResultCache.size:0}"+ initialSize="${queryResultCache.initialSize:0}"+ autowarmCount="${queryResultCache.autowarmCount:0}"/>+ <documentCache+ enabled ="${documentCache.enabled:false}"+ class="${documentCache.class:solr.LRUCache}"+ size="${documentCache.size:0}"+ initialSize="${documentCache.initialSize:0}"+ autowarmCount="${documentCache.autowarmCount:0}"/>+ <filterCache+ enabled ="${filterCache.enabled:false}"+ class="${filterCache.class:solr.LRUCache}"+ size="${filterCache.size:0}"+ initialSize="${filterCache.initialSize:0}"+ autowarmCount="${filterCache.autowarmCount:0}"/>+ <cache name="myPerSegmentCache"+ enabled ="${myPerSegmentCache.enabled:false}"+ class="${myPerSegmentCache.class:solr.LRUCache}"+ size="${myPerSegmentCache.size:0}"+ initialSize="${myPerSegmentCache.initialSize:0}"+ autowarmCount="${myPerSegmentCache.autowarmCount:0}"/>+ </query>++</config>+diff --git a/solr/benchmark/src/test/org/apache/solr/bench/DockMakerTest.java b/solr/benchmark/src/test/org/apache/solr/bench/DockMakerTest.javanew file mode 100644index 00000000000..5351fcef4ae--- /dev/null+++ b/solr/benchmark/src/test/org/apache/solr/bench/DockMakerTest.java@@ -0,0 +1,144 @@+/*+ * Licensed to the Apache Software Foundation (ASF) under one or more+ * contributor license agreements. See the NOTICE file distributed with+ * this work for additional information regarding copyright ownership.+ * The ASF licenses this file to You under the Apache License, Version 2.0+ * (the "License"); you may not use this file except in compliance with+ * the License. You may obtain a copy of the License at+ *+ * http://www.apache.org/licenses/LICENSE-2.0+ *+ * Unless required by applicable law or agreed to in writing, software+ * distributed under the License is distributed on an "AS IS" BASIS,+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.+ * See the License for the specific language governing permissions and+ * limitations under the License.+ */+package org.apache.solr.bench;++import java.util.HashSet;+import java.util.Random;+import java.util.Set;+import java.util.SplittableRandom;+import java.util.concurrent.ThreadLocalRandom;+import org.apache.solr.SolrTestCaseJ4;+import org.apache.solr.common.SolrInputDocument;+import org.apache.solr.common.SolrInputField;+import org.junit.Before;+import org.junit.Test;++public class DockMakerTest extends SolrTestCaseJ4 {++ @Before+ public void setup() {+ System.setProperty("randomSeed", Long.toString(new Random().nextLong()));+ }++ @Test+ public void testGenDoc() throws Exception {+ SplittableRandom random = new SplittableRandom();++ DocMaker docMaker = new DocMaker();+ docMaker.addField(+ "id", FieldDef.FieldDefBuilder.aFieldDef().withContent(DocMaker.Content.UNIQUE_INT));++ docMaker.addField(+ "facet_s",+ FieldDef.FieldDefBuilder.aFieldDef()+ .withContent(DocMaker.Content.ALPHEBETIC)+ .withMaxLength(64)+ .withMaxCardinality(5, random));+ docMaker.addField(+ "facet2_s",+ FieldDef.FieldDefBuilder.aFieldDef()+ .withContent(DocMaker.Content.ALPHEBETIC)+ .withMaxLength(16)+ .withMaxCardinality(100, random));+ docMaker.addField(+ "facet3_s",+ FieldDef.FieldDefBuilder.aFieldDef()+ .withContent(DocMaker.Content.UNICODE)+ .withMaxLength(128)+ .withMaxCardinality(12000, random));+ docMaker.addField(+ "text",+ FieldDef.FieldDefBuilder.aFieldDef()+ .withContent(DocMaker.Content.ALPHEBETIC)+ .withMaxLength(12)+ .withMaxTokenCount(ThreadLocalRandom.current().nextInt(512) + 1));+ docMaker.addField(+ "int_i", FieldDef.FieldDefBuilder.aFieldDef().withContent(DocMaker.Content.INTEGER));+ docMaker.addField(+ "int2_i",+ FieldDef.FieldDefBuilder.aFieldDef()+ .withContent(DocMaker.Content.INTEGER)+ .withMaxCardinality(500, random));++ for (int i = 0; i < 5; i++) {+ SolrInputDocument doc = docMaker.getDocument(random);+ // System.out.println("doc:\n" + doc);+ }+ }++ @Test+ public void testBasicCardinalityAlpha() throws Exception {+ DocMaker docMaker = new DocMaker();+ SplittableRandom random = new SplittableRandom();+ int cardinality = 2;+ docMaker.addField(+ "AlphaCard3",+ FieldDef.FieldDefBuilder.aFieldDef()+ .withContent(DocMaker.Content.ALPHEBETIC)+ .withMaxCardinality(cardinality, random));++ Set<String> values = new HashSet<>();+ for (int i = 0; i < 10; i++) {+ SolrInputDocument doc = docMaker.getDocument(random);+ SolrInputField field = doc.getField("AlphaCard3");+ values.add(field.getValue().toString());+ }+ assertEquals(values.toString(), cardinality, values.size());+ }++ @Test+ public void testBasicCardinalityUnicode() throws Exception {+ DocMaker docMaker = new DocMaker();+ SplittableRandom random = new SplittableRandom();+ int cardinality = 4;+ docMaker.addField(+ "UnicodeCard3",+ FieldDef.FieldDefBuilder.aFieldDef()+ .withContent(DocMaker.Content.UNICODE)+ .withMaxCardinality(cardinality, random));++ HashSet<Object> values = new HashSet<>();+ for (int i = 0; i < 20; i++) {+ SolrInputDocument doc = docMaker.getDocument(random);+ SolrInputField field = doc.getField("UnicodeCard3");+ // System.out.println("field=" + doc);+ values.add(field.getValue().toString());+ }++ assertEquals(values.toString(), cardinality, values.size());+ }++ @Test+ public void testBasicCardinalityInteger() throws Exception {+ SplittableRandom random = new SplittableRandom();+ DocMaker docMaker = new DocMaker();+ int cardinality = 3;+ docMaker.addField(+ "IntCard2",+ FieldDef.FieldDefBuilder.aFieldDef()+ .withContent(DocMaker.Content.INTEGER)+ .withMaxCardinality(cardinality, random));++ HashSet<Object> values = new HashSet<>();+ for (int i = 0; i < 30; i++) {+ SolrInputDocument doc = docMaker.getDocument(random);+ SolrInputField field = doc.getField("IntCard2");+ values.add(field.getValue().toString());+ }+ assertEquals(values.toString(), cardinality, values.size());+ }+}diff --git a/solr/contrib/jaegertracer-configurator/src/test/org/apache/solr/jaeger/TestJaegerConfigurator.java b/solr/contrib/jaegertracer-configurator/src/test/org/apache/solr/jaeger/TestJaegerConfigurator.javaindex e15595acd4d..011b138a05e 100644--- a/solr/contrib/jaegertracer-configurator/src/test/org/apache/solr/jaeger/TestJaegerConfigurator.java+++ b/solr/contrib/jaegertracer-configurator/src/test/org/apache/solr/jaeger/TestJaegerConfigurator.java@@ -25,7 +25,6 @@ import org.apache.solr.client.solrj.SolrQuery;import org.apache.solr.client.solrj.request.CollectionAdminRequest;import org.apache.solr.client.solrj.request.UpdateRequest;import org.apache.solr.cloud.MiniSolrCloudCluster;-import org.apache.solr.cloud.SolrCloudTestCase;import org.apache.solr.common.util.ExecutorUtil;import org.apache.solr.common.util.TimeSource;import org.apache.solr.util.TimeOut;@@ -52,7 +51,7 @@ public class TestJaegerConfigurator extends SolrTestCaseJ4 {@Testpublic void testInjected() throws Exception {MiniSolrCloudCluster cluster =- new SolrCloudTestCase.Builder(2, createTempDir())+ new MiniSolrCloudCluster.Builder(2, createTempDir()).addConfig("config", TEST_PATH().resolve("collection1").resolve("conf")).withSolrXml(getFile("solr/solr.xml").toPath()).build();diff --git a/solr/core/src/java/org/apache/solr/client/solrj/embedded/JettySolrRunner.java b/solr/core/src/java/org/apache/solr/client/solrj/embedded/JettySolrRunner.javaindex 5a7dcd5c817..5f481c48e92 100644--- a/solr/core/src/java/org/apache/solr/client/solrj/embedded/JettySolrRunner.java+++ b/solr/core/src/java/org/apache/solr/client/solrj/embedded/JettySolrRunner.java@@ -26,32 +26,49 @@ import javax.servlet.ServletResponse;import javax.servlet.http.HttpServlet;import javax.servlet.http.HttpServletRequest;import javax.servlet.http.HttpServletResponse;+import java.io.File;import java.io.IOException;+import java.io.PrintStream;import java.lang.invoke.MethodHandles;import java.net.BindException;import java.net.MalformedURLException;import java.net.URL;+import java.nio.charset.StandardCharsets;+import java.nio.file.Files;+import java.nio.file.Path;import java.util.ArrayList;import java.util.EnumSet;+import java.util.LinkedHashMap;import java.util.LinkedList;import java.util.List;import java.util.Locale;import java.util.Map;import java.util.Properties;import java.util.Random;+import java.util.Set;import java.util.concurrent.ExecutorService;import java.util.concurrent.TimeUnit;import java.util.concurrent.atomic.AtomicInteger;import java.util.concurrent.atomic.AtomicLong;+import com.codahale.metrics.ConsoleReporter;+import com.codahale.metrics.MetricFilter;+import com.codahale.metrics.MetricRegistry;+import org.apache.commons.io.output.NullPrintStream;import org.apache.lucene.util.Constants;import org.apache.solr.client.solrj.SolrClient;import org.apache.solr.client.solrj.cloud.SocketProxy;import org.apache.solr.client.solrj.impl.HttpSolrClient;import org.apache.solr.common.util.ExecutorUtil;+import org.apache.solr.common.util.NamedList;+import org.apache.solr.common.util.SimpleOrderedMap;import org.apache.solr.common.util.SolrNamedThreadFactory;import org.apache.solr.common.util.TimeSource;import org.apache.solr.core.CoreContainer;+import org.apache.solr.core.SolrCore;+import org.apache.solr.handler.admin.CoreAdminOperation;+import org.apache.solr.handler.admin.LukeRequestHandler;+import org.apache.solr.metrics.SolrMetricManager;import org.apache.solr.servlet.SolrDispatchFilter;import org.apache.solr.util.TimeOut;import org.eclipse.jetty.alpn.server.ALPNServerConnectionFactory;@@ -78,6 +95,7 @@ import org.eclipse.jetty.util.component.LifeCycle;import org.eclipse.jetty.util.ssl.SslContextFactory;import org.eclipse.jetty.util.thread.QueuedThreadPool;import org.eclipse.jetty.util.thread.ReservedThreadExecutor;+import org.noggit.JSONUtil;import org.slf4j.Logger;import org.slf4j.LoggerFactory;import org.slf4j.MDC;@@ -313,8 +331,7 @@ public class JettySolrRunner {if (config.onlyHttp1) {connector = new ServerConnector(server, new HttpConnectionFactory(configuration));} else {- connector = new ServerConnector(server, new HttpConnectionFactory(configuration),- new HTTP2CServerConnectionFactory(configuration));+ connector = new ServerConnector(server, new HttpConnectionFactory(configuration), new HTTP2CServerConnectionFactory(configuration));}}@@ -322,13 +339,15 @@ public class JettySolrRunner {connector.setPort(port);connector.setHost("127.0.0.1");connector.setIdleTimeout(THREAD_POOL_MAX_IDLE_TIME_MS);- connector.setStopTimeout(0);+server.setConnectors(new Connector[] {connector});server.setSessionIdManager(new DefaultSessionIdManager(server, new Random()));} else {HttpConfiguration configuration = new HttpConfiguration();- ServerConnector connector = new ServerConnector(server, new HttpConnectionFactory(configuration));+ ServerConnector connector = new ServerConnector(server, new HttpConnectionFactory(configuration), new HTTP2CServerConnectionFactory(configuration));+ connector.setReuseAddress(true);connector.setPort(port);+ connector.setHost("127.0.0.1");connector.setIdleTimeout(THREAD_POOL_MAX_IDLE_TIME_MS);server.setConnectors(new Connector[] {connector});}@@ -697,6 +716,52 @@ public class JettySolrRunner {}}+ public void outputMetrics(File outputDirectory, String fileName) throws IOException {+ if (getCoreContainer() != null) {++ if (outputDirectory != null) {+ Path outDir = outputDirectory.toPath();+ if (!Files.exists(outDir)) {+ Files.createDirectories(outDir);+ }+ }++ SolrMetricManager metricsManager = getCoreContainer().getMetricManager();++ Set<String> registryNames = metricsManager.registryNames();+ for (String registryName : registryNames) {+ MetricRegistry metricsRegisty = metricsManager.registry(registryName);+ try (PrintStream ps = outputDirectory == null ? new NullPrintStream() : new PrintStream(new File(outputDirectory, registryName + "_" + fileName), StandardCharsets.UTF_8)) {+ ConsoleReporter reporter = ConsoleReporter.forRegistry(metricsRegisty).+ convertRatesTo(TimeUnit.SECONDS).convertDurationsTo(TimeUnit.MILLISECONDS).filter(MetricFilter.ALL).outputTo(ps).build();+ reporter.report();+ }+ }++ } else {+ throw new IllegalStateException("No CoreContainer found");+ }+ }++ public void dumpCoresInfo(PrintStream pw) throws IOException {+ if (getCoreContainer() != null) {+ List<SolrCore> cores = getCoreContainer().getCores();+ for (SolrCore core : cores) {+ NamedList<Object> coreStatus = CoreAdminOperation.getCoreStatus(getCoreContainer(), core.getName(), false);+ core.withSearcher(solrIndexSearcher -> {+ SimpleOrderedMap<Object> lukeIndexInfo = LukeRequestHandler.getIndexInfo(solrIndexSearcher.getIndexReader());+ @SuppressWarnings({"unchecked", "rawtypes"})+ Map<String,Object> indexInfoMap = coreStatus.toMap(new LinkedHashMap<>());+ indexInfoMap.putAll(lukeIndexInfo.toMap(new LinkedHashMap<>()));+ pw.println(JSONUtil.toJSON(indexInfoMap, 2));++ pw.println("");+ return null;+ });+ }+ }+ }+/*** Returns the Local Port of the jetty Server.*diff --git a/solr/core/src/java/org/apache/solr/core/SolrCore.java b/solr/core/src/java/org/apache/solr/core/SolrCore.javaindex 84181873182..9275c9a02f4 100644--- a/solr/core/src/java/org/apache/solr/core/SolrCore.java+++ b/solr/core/src/java/org/apache/solr/core/SolrCore.java@@ -173,6 +173,7 @@ import org.apache.solr.util.plugin.PluginInfoInitialized;import org.apache.solr.util.plugin.SolrCoreAware;import org.apache.zookeeper.KeeperException;import org.apache.zookeeper.data.Stat;+import org.eclipse.jetty.io.RuntimeIOException;import org.slf4j.Logger;import org.slf4j.LoggerFactory;@@ -487,6 +488,14 @@ public final class SolrCore implements SolrInfoBean, Closeable {return size;}+ public int getSegmentCount() {+ try {+ return withSearcher( solrIndexSearcher -> solrIndexSearcher.getRawReader().getIndexCommit().getSegmentCount());+ } catch (IOException e) {+ throw new RuntimeIOException(e);+ }+ }+@Overridepublic String getName() {return name;@@ -1201,6 +1210,7 @@ public final class SolrCore implements SolrInfoBean, Closeable {parentContext.gauge(() -> getInstancePath().toString(), true, "instanceDir", Category.CORE.toString());parentContext.gauge(() -> isClosed() ? parentContext.nullString() : getIndexDir(), true, "indexDir", Category.CORE.toString());parentContext.gauge(() -> isClosed() ? parentContext.nullNumber() : getIndexSize(), true, "sizeInBytes", Category.INDEX.toString());+ parentContext.gauge(() -> isClosed() ? parentContext.nullNumber() : getSegmentCount(), true, "segments", Category.INDEX.toString());parentContext.gauge(() -> isClosed() ? parentContext.nullString() : NumberUtils.readableSize(getIndexSize()), true, "size", Category.INDEX.toString());final CloudDescriptor cd = getCoreDescriptor().getCloudDescriptor();diff --git a/solr/core/src/java/org/apache/solr/handler/admin/CoreAdminOperation.java b/solr/core/src/java/org/apache/solr/handler/admin/CoreAdminOperation.javaindex 1a966a8a656..7ac25a2ee30 100644--- a/solr/core/src/java/org/apache/solr/handler/admin/CoreAdminOperation.java+++ b/solr/core/src/java/org/apache/solr/handler/admin/CoreAdminOperation.java@@ -63,7 +63,7 @@ import static org.apache.solr.handler.admin.CoreAdminHandler.RUNNING;import static org.apache.solr.handler.admin.CoreAdminHandler.buildCoreParams;import static org.apache.solr.handler.admin.CoreAdminHandler.normalizePath;-enum CoreAdminOperation implements CoreAdminOp {+public enum CoreAdminOperation implements CoreAdminOp {CREATE_OP(CREATE, it -> {assert TestInjection.injectRandomDelayInCoreCreation();@@ -235,7 +235,8 @@ enum CoreAdminOperation implements CoreAdminOp {}SolrSnapshotMetaDataManager mgr = core.getSnapshotMetaDataManager();- NamedList<Object> result = new NamedList<>();+ @SuppressWarnings({"rawtypes"})+ NamedList result = new NamedList();for (String name : mgr.listSnapshots()) {Optional<SnapshotMetaData> metadata = mgr.getSnapshotMetaData(name);if ( metadata.isPresent() ) {@@ -274,7 +275,8 @@ enum CoreAdminOperation implements CoreAdminOp {* @return - a named list of key/value pairs from the core.* @throws IOException - LukeRequestHandler can throw an I/O exception*/- static NamedList<Object> getCoreStatus(CoreContainer cores, String cname, boolean isIndexInfoNeeded) throws IOException {+ @SuppressWarnings({"unchecked", "rawtypes"})+ public static NamedList<Object> getCoreStatus(CoreContainer cores, String cname, boolean isIndexInfoNeeded) throws IOException {NamedList<Object> info = new SimpleOrderedMap<>();if (cores.isCoreLoading(cname)) {diff --git a/solr/core/src/test/org/apache/solr/cloud/CloudExitableDirectoryReaderTest.java b/solr/core/src/test/org/apache/solr/cloud/CloudExitableDirectoryReaderTest.javaindex 3c757ad7bf9..8fa115742ba 100644--- a/solr/core/src/test/org/apache/solr/cloud/CloudExitableDirectoryReaderTest.java+++ b/solr/core/src/test/org/apache/solr/cloud/CloudExitableDirectoryReaderTest.java@@ -81,7 +81,7 @@ public class CloudExitableDirectoryReaderTest extends SolrCloudTestCase {@BeforeClasspublic static void setupCluster() throws Exception {// create one more node then shard, so that we also test the case of proxied requests.- Builder clusterBuilder = configureCluster(3)+ MiniSolrCloudCluster.Builder clusterBuilder = configureCluster(3).addConfig("conf", TEST_PATH().resolve("configsets").resolve("exitable-directory").resolve("conf"));clusterBuilder.withMetrics(true);clusterBuilderdiff --git a/solr/core/src/test/org/apache/solr/cloud/DistribDocExpirationUpdateProcessorTest.java b/solr/core/src/test/org/apache/solr/cloud/DistribDocExpirationUpdateProcessorTest.javaindex 284e76e01e8..dad34efdd2d 100644--- a/solr/core/src/test/org/apache/solr/cloud/DistribDocExpirationUpdateProcessorTest.java+++ b/solr/core/src/test/org/apache/solr/cloud/DistribDocExpirationUpdateProcessorTest.java@@ -87,7 +87,7 @@ public class DistribDocExpirationUpdateProcessorTest extends SolrCloudTestCase {public void setupCluster(boolean security) throws Exception {// we want at most one core per node to force lots of network traffic to try and tickle distributed bugs- final Builder b = configureCluster(4)+ final MiniSolrCloudCluster.Builder b = configureCluster(4).addConfig("conf", TEST_PATH().resolve("configsets").resolve("doc-expiry").resolve("conf"));COLLECTION = "expiring";diff --git a/solr/core/src/test/org/apache/solr/cloud/TestConfigSetsAPIZkFailure.java b/solr/core/src/test/org/apache/solr/cloud/TestConfigSetsAPIZkFailure.javaindex 54a67160202..562798af59d 100644--- a/solr/core/src/test/org/apache/solr/cloud/TestConfigSetsAPIZkFailure.java+++ b/solr/core/src/test/org/apache/solr/cloud/TestConfigSetsAPIZkFailure.java@@ -83,7 +83,7 @@ public class TestConfigSetsAPIZkFailure extends SolrTestCaseJ4 {zkTestServer.run();zkTestServer.setZKDatabase(new FailureDuringCopyZKDatabase(zkTestServer.getZKDatabase(), zkTestServer));solrCluster = new MiniSolrCloudCluster(1, testDir,- MiniSolrCloudCluster.DEFAULT_CLOUD_SOLR_XML, buildJettyConfig("/solr"), zkTestServer);+ MiniSolrCloudCluster.DEFAULT_CLOUD_SOLR_XML, buildJettyConfig("/solr"), zkTestServer, true);}@Overridediff --git a/solr/core/src/test/org/apache/solr/handler/component/ShardsAllowListTest.java b/solr/core/src/test/org/apache/solr/handler/component/ShardsAllowListTest.javaindex 0af837d149a..d0a0d70bd13 100644--- a/solr/core/src/test/org/apache/solr/handler/component/ShardsAllowListTest.java+++ b/solr/core/src/test/org/apache/solr/handler/component/ShardsAllowListTest.java@@ -32,7 +32,6 @@ import org.apache.solr.client.solrj.SolrServerException;import org.apache.solr.client.solrj.embedded.JettySolrRunner;import org.apache.solr.cloud.MiniSolrCloudCluster;import org.apache.solr.cloud.MultiSolrCloudTestCase;-import org.apache.solr.cloud.SolrCloudTestCase;import org.apache.solr.common.SolrException;import org.apache.solr.common.SolrInputDocument;import org.apache.solr.security.AllowListUrlChecker;@@ -79,7 +78,7 @@ public class ShardsAllowListTest extends MultiSolrCloudTestCase {@Overridepublic MiniSolrCloudCluster apply(String clusterId) {try {- final MiniSolrCloudCluster cluster = new SolrCloudTestCase.Builder(nodesPerCluster(clusterId),+ final MiniSolrCloudCluster cluster = new MiniSolrCloudCluster.Builder(nodesPerCluster(clusterId),createTempDir()).addConfig("conf", configset("cloud-dynamic")).withSolrXml(MiniSolrCloudCluster.DEFAULT_CLOUD_SOLR_XML.replace(diff --git a/solr/core/src/test/org/apache/solr/security/AuditLoggerIntegrationTest.java b/solr/core/src/test/org/apache/solr/security/AuditLoggerIntegrationTest.javaindex 1d538e2cb93..f44335a433a 100644--- a/solr/core/src/test/org/apache/solr/security/AuditLoggerIntegrationTest.java+++ b/solr/core/src/test/org/apache/solr/security/AuditLoggerIntegrationTest.java@@ -422,7 +422,7 @@ public class AuditLoggerIntegrationTest extends SolrCloudAuthTestCase {securityJson = securityJson.replace("_MUTERULES_", "[" + StringUtils.join(muteRules, ",") + "]");- MiniSolrCloudCluster myCluster = new Builder(NUM_SERVERS, createTempDir())+ MiniSolrCloudCluster myCluster = new MiniSolrCloudCluster.Builder(NUM_SERVERS, createTempDir()).withSecurityJson(securityJson).addConfig("conf1", TEST_PATH().resolve("configsets").resolve("cloud-minimal").resolve("conf")).build();diff --git a/solr/cp-deps.sh b/solr/cp-deps.shnew file mode 100644index 00000000000..21d083bd2f3--- /dev/null+++ b/solr/cp-deps.sh@@ -0,0 +1,22 @@+#!/usr/bin/env bash+# Licensed to the Apache Software Foundation (ASF) under one or more+# contributor license agreements. See the NOTICE file distributed with+# this work for additional information regarding copyright ownership.+# The ASF licenses this file to You under the Apache License, Version 2.0+# (the "License"); you may not use this file except in compliance with+# the License. You may obtain a copy of the License at+#+# http://www.apache.org/licenses/LICENSE-2.0+#+# Unless required by applicable law or agreed to in writing, software+# distributed under the License is distributed on an "AS IS" BASIS,+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.+# See the License for the specific language governing permissions and+# limitations under the License.++bash -c 'param="$(../../gradlew -q echoCp)";mkdir -p docker/lib;cp -r ${param//:/ } docker/lib'++#docker build -t btest .++#docker run -it --entrypoint "bash" --name markscontainer22 btest:latest+diff --git a/solr/licenses/jmh-core-1.32.jar.sha1 b/solr/licenses/jmh-core-1.32.jar.sha1new file mode 100644index 00000000000..23e527818c7--- /dev/null+++ b/solr/licenses/jmh-core-1.32.jar.sha1@@ -0,0 +1 @@+9a8b69ea08118fd4e5d30a152d37b7087ee4a720diff --git a/solr/licenses/jmh-core-LICENSE-COMPOUND.txt b/solr/licenses/jmh-core-LICENSE-COMPOUND.txtnew file mode 100644index 00000000000..4bd7ad7e424--- /dev/null+++ b/solr/licenses/jmh-core-LICENSE-COMPOUND.txt@@ -0,0 +1,3 @@+JMH is GPL licensed and part of the OpenJDK project. It is only used for developer benchmarks during development.++* DO NOT DISTRIBUTE! *diff --git a/solr/licenses/jmh-core-NOTICE.txt b/solr/licenses/jmh-core-NOTICE.txtnew file mode 100644index 00000000000..4bd7ad7e424--- /dev/null+++ b/solr/licenses/jmh-core-NOTICE.txt@@ -0,0 +1,3 @@+JMH is GPL licensed and part of the OpenJDK project. It is only used for developer benchmarks during development.++* DO NOT DISTRIBUTE! *diff --git a/solr/licenses/jopt-simple-4.6.jar.sha1 b/solr/licenses/jopt-simple-4.6.jar.sha1new file mode 100644index 00000000000..03f870758c7--- /dev/null+++ b/solr/licenses/jopt-simple-4.6.jar.sha1@@ -0,0 +1 @@+306816fb57cf94f108a43c95731b08934dcae15cdiff --git a/solr/licenses/jopt-simple-LICENSE-MIT.txt b/solr/licenses/jopt-simple-LICENSE-MIT.txtnew file mode 100644index 00000000000..f20b2442af5--- /dev/null+++ b/solr/licenses/jopt-simple-LICENSE-MIT.txt@@ -0,0 +1,24 @@+/*+ The MIT License++ Copyright (c) 2004-2021 Paul R. Holser, Jr.++ Permission is hereby granted, free of charge, to any person obtaining+ a copy of this software and associated documentation files (the+ "Software"), to deal in the Software without restriction, including+ without limitation the rights to use, copy, modify, merge, publish,+ distribute, sublicense, and/or sell copies of the Software, and to+ permit persons to whom the Software is furnished to do so, subject to+ the following conditions:++ The above copyright notice and this permission notice shall be+ included in all copies or substantial portions of the Software.++ THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,+ EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF+ MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND+ NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE+ LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION+ OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION+ WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.+*/\ No newline at end of filediff --git a/solr/test-framework/src/java/org/apache/solr/SolrTestCaseJ4.java b/solr/test-framework/src/java/org/apache/solr/SolrTestCaseJ4.javaindex 6d45c05afe6..1a786fbea63 100644--- a/solr/test-framework/src/java/org/apache/solr/SolrTestCaseJ4.java+++ b/solr/test-framework/src/java/org/apache/solr/SolrTestCaseJ4.java@@ -266,7 +266,7 @@ public abstract class SolrTestCaseJ4 extends SolrTestCase {// these are meant to be accessed sequentially, but are volatile just to ensure any test// thread will read the latest value- protected static volatile SSLTestConfig sslConfig;+ public static volatile SSLTestConfig sslConfig;@Rulepublic TestRule solrTestRules =diff --git a/solr/test-framework/src/java/org/apache/solr/cloud/MiniSolrCloudCluster.java b/solr/test-framework/src/java/org/apache/solr/cloud/MiniSolrCloudCluster.javaindex 13320b2138f..2f9d2703c85 100644--- a/solr/test-framework/src/java/org/apache/solr/cloud/MiniSolrCloudCluster.java+++ b/solr/test-framework/src/java/org/apache/solr/cloud/MiniSolrCloudCluster.java@@ -16,23 +16,15 @@*/package org.apache.solr.cloud;+import java.io.File;import java.io.IOException;+import java.io.PrintStream;import java.lang.invoke.MethodHandles;import java.nio.charset.Charset;import java.nio.charset.StandardCharsets;import java.nio.file.Files;import java.nio.file.Path;-import java.util.ArrayList;-import java.util.Arrays;-import java.util.Collection;-import java.util.Collections;-import java.util.List;-import java.util.Objects;-import java.util.Optional;-import java.util.Properties;-import java.util.Random;-import java.util.Set;-import java.util.SortedMap;+import java.util.*;import java.util.concurrent.Callable;import java.util.concurrent.CopyOnWriteArrayList;import java.util.concurrent.CountDownLatch;@@ -43,6 +35,7 @@ import java.util.concurrent.TimeUnit;import java.util.concurrent.TimeoutException;import java.util.concurrent.atomic.AtomicInteger;import java.util.concurrent.atomic.AtomicReference;+import java.util.function.Consumer;import javax.servlet.Filter;@@ -52,20 +45,12 @@ import org.apache.solr.client.solrj.embedded.JettyConfig;import org.apache.solr.client.solrj.embedded.JettySolrRunner;import org.apache.solr.client.solrj.embedded.SSLConfig;import org.apache.solr.client.solrj.impl.CloudSolrClient;-import org.apache.solr.client.solrj.impl.CloudSolrClient.Builder;import org.apache.solr.client.solrj.request.CollectionAdminRequest;import org.apache.solr.client.solrj.request.ConfigSetAdminRequest;import org.apache.solr.common.SolrException;import org.apache.solr.common.SolrException.ErrorCode;-import org.apache.solr.common.cloud.Aliases;-import org.apache.solr.common.cloud.CloudCollectionsListener;-import org.apache.solr.common.cloud.CollectionStatePredicate;-import org.apache.solr.common.cloud.DocCollection;-import org.apache.solr.common.cloud.Replica;-import org.apache.solr.common.cloud.Slice;-import org.apache.solr.common.cloud.SolrZkClient;-import org.apache.solr.common.cloud.ZkMaintenanceUtils;-import org.apache.solr.common.cloud.ZkStateReader;+import org.apache.solr.common.cloud.*;+import org.apache.solr.common.params.CollectionAdminParams;import org.apache.solr.common.util.ExecutorUtil;import org.apache.solr.common.util.IOUtils;import org.apache.solr.common.util.SolrNamedThreadFactory;@@ -155,7 +140,7 @@ public class MiniSolrCloudCluster {* @throws Exception if there was an error starting the cluster*/public MiniSolrCloudCluster(int numServers, Path baseDir, JettyConfig jettyConfig) throws Exception {- this(numServers, baseDir, DEFAULT_CLOUD_SOLR_XML, jettyConfig, null);+ this(numServers, baseDir, DEFAULT_CLOUD_SOLR_XML, jettyConfig, null, false);}/**@@ -212,7 +197,7 @@ public class MiniSolrCloudCluster {* @throws Exception if there was an error starting the cluster*/public MiniSolrCloudCluster(int numServers, Path baseDir, String solrXml, JettyConfig jettyConfig) throws Exception {- this(numServers, baseDir, solrXml, jettyConfig, null);+ this(numServers, baseDir, solrXml, jettyConfig, null, false);}/**@@ -227,8 +212,8 @@ public class MiniSolrCloudCluster {* @throws Exception if there was an error starting the cluster*/public MiniSolrCloudCluster(int numServers, Path baseDir, String solrXml, JettyConfig jettyConfig,- ZkTestServer zkTestServer) throws Exception {- this(numServers, baseDir, solrXml, jettyConfig, zkTestServer, Optional.empty());+ ZkTestServer zkTestServer, boolean formatZkServer) throws Exception {+ this(numServers, baseDir, solrXml, jettyConfig, zkTestServer, Optional.empty(), formatZkServer);}/**@@ -247,9 +232,9 @@ public class MiniSolrCloudCluster {* @throws Exception if there was an error starting the cluster*/MiniSolrCloudCluster(int numServers, Path baseDir, String solrXml, JettyConfig jettyConfig,- ZkTestServer zkTestServer, Optional<String> securityJson) throws Exception {+ ZkTestServer zkTestServer, Optional<String> securityJson, boolean formatZkServer) throws Exception {this(numServers, baseDir, solrXml, jettyConfig,- zkTestServer,securityJson, false);+ zkTestServer,securityJson, false, formatZkServer);}/*** Create a MiniSolrCloudCluster.@@ -268,7 +253,7 @@ public class MiniSolrCloudCluster {* @throws Exception if there was an error starting the cluster*/MiniSolrCloudCluster(int numServers, Path baseDir, String solrXml, JettyConfig jettyConfig,- ZkTestServer zkTestServer, Optional<String> securityJson, boolean trackJettyMetrics) throws Exception {+ ZkTestServer zkTestServer, Optional<String> securityJson, boolean trackJettyMetrics, boolean formatZkServer) throws Exception {Objects.requireNonNull(securityJson);this.baseDir = Objects.requireNonNull(baseDir);@@ -284,7 +269,7 @@ public class MiniSolrCloudCluster {Path zkDir = baseDir.resolve("zookeeper/server1/data");zkTestServer = new ZkTestServer(zkDir);try {- zkTestServer.run();+ zkTestServer.run(formatZkServer);} catch (Exception e) {log.error("Error starting Zk Test Server, trying again ...");zkTestServer.shutdown();@@ -295,12 +280,14 @@ public class MiniSolrCloudCluster {this.zkServer = zkTestServer;try (SolrZkClient zkClient = new SolrZkClient(zkServer.getZkHost(), AbstractZkTestCase.TIMEOUT)) {- zkClient.makePath("/solr/solr.xml", solrXml.getBytes(Charset.defaultCharset()), true);- if (jettyConfig.sslConfig != null && jettyConfig.sslConfig.isSSLMode()) {- zkClient.makePath("/solr" + ZkStateReader.CLUSTER_PROPS, "{'urlScheme':'https'}".getBytes(StandardCharsets.UTF_8), true);- }- if (securityJson.isPresent()) { // configure Solr security- zkClient.makePath("/solr/security.json", securityJson.get().getBytes(Charset.defaultCharset()), true);+ if (!zkClient.exists("/solr/solr.xml", true)) {+ zkClient.makePath("/solr/solr.xml", solrXml.getBytes(Charset.defaultCharset()), true);+ if (jettyConfig.sslConfig != null && jettyConfig.sslConfig.isSSLMode()) {+ zkClient.makePath("/solr" + ZkStateReader.CLUSTER_PROPS, "{'urlScheme':'https'}".getBytes(StandardCharsets.UTF_8), true);+ }+ if (securityJson.isPresent()) { // configure Solr security+ zkClient.makePath("/solr/security.json", securityJson.get().getBytes(Charset.defaultCharset()), true);+ }}}@@ -398,7 +385,9 @@ public class MiniSolrCloudCluster {private Path createInstancePath(String name) throws IOException {Path instancePath = baseDir.resolve(name);- Files.createDirectory(instancePath);+ if (!Files.exists(instancePath)) {+ Files.createDirectory(instancePath);+ }return instancePath;}@@ -659,7 +648,7 @@ public class MiniSolrCloudCluster {}protected CloudSolrClient buildSolrClient() {- return new Builder(Collections.singletonList(getZkServer().getZkAddress()), Optional.empty())+ return new CloudSolrClient.Builder(Collections.singletonList(getZkServer().getZkAddress()), Optional.empty()).withSocketTimeout(90000).withConnectionTimeout(15000).build(); // we choose 90 because we run in some harsh envs}@@ -788,6 +777,26 @@ public class MiniSolrCloudCluster {public void waitForActiveCollection(String collection, int shards, int totalReplicas) {waitForActiveCollection(collection, 30, TimeUnit.SECONDS, shards, totalReplicas);}++ public void waitForActiveCollection(String collection, long wait, TimeUnit unit) {+ log.info("waitForActiveCollection: {}", collection);+ CollectionStatePredicate predicate = expectedActive();++ AtomicReference<DocCollection> state = new AtomicReference<>();+ AtomicReference<Set<String>> liveNodesLastSeen = new AtomicReference<>();+ try {+ getSolrClient().waitForState(collection, wait, unit, (n, c) -> {+ state.set(c);+ liveNodesLastSeen.set(n);++ return predicate.matches(n, c);+ });+ } catch (TimeoutException | InterruptedException e) {+ throw new RuntimeException("Failed while waiting for active collection" + "\n" + e.getMessage() + "\nLive Nodes: " + Arrays.toString(liveNodesLastSeen.get().toArray())+ + "\nLast available state: " + state.get());+ }++ }public static CollectionStatePredicate expectedShardsAndActiveReplicas(int expectedShards, int expectedReplicas) {return (liveNodes, collectionState) -> {@@ -813,6 +822,24 @@ public class MiniSolrCloudCluster {};}+ public static CollectionStatePredicate expectedActive() {+ return (liveNodes, collectionState) -> {+ if (collectionState == null)+ return false;+++ for (Slice slice : collectionState) {+ for (Replica replica : slice) {+ if (!replica.isActive(liveNodes)) {+ return false;+ }+ }+ }++ return true;+ };+ }+public void waitForJettyToStop(JettySolrRunner runner) throws TimeoutException {String nodeName = runner.getNodeName();if (nodeName == null) {@@ -830,6 +857,18 @@ public class MiniSolrCloudCluster {throw new SolrException(ErrorCode.SERVER_ERROR, "interrupted", e);}}++ public void outputMetrics(File outputDirectory, String fileName) throws IOException {+ for (JettySolrRunner jetty : jettys) {+ jetty.outputMetrics(outputDirectory, fileName);+ }+ }++ public void dumpCoreInfo(PrintStream pw) throws IOException {+ for (JettySolrRunner jetty : jettys) {+ jetty.dumpCoresInfo(pw);+ }+ }/** @lucene.experimental */public static final class JettySolrRunnerWithMetrics extends JettySolrRunner {@@ -855,4 +894,246 @@ public class MiniSolrCloudCluster {}}++ private static class Config {+ final String name;+ final Path path;+ private Config(String name, Path path) {+ this.name = name;+ this.path = path;+ }+ }++ /**+ * Builder class for a MiniSolrCloudCluster+ */+ public static class Builder {++ private final int nodeCount;+ private final Path baseDir;+ private String solrxml = DEFAULT_CLOUD_SOLR_XML;+ private JettyConfig.Builder jettyConfigBuilder;+ private Optional<String> securityJson = Optional.empty();++ private List<Config> configs = new ArrayList<>();+ private Map<String, Object> clusterProperties = new HashMap<>();++ private boolean trackJettyMetrics;+ private boolean useDistributedCollectionConfigSetExecution;+ private boolean useDistributedClusterStateUpdate;+ private boolean formatZkServer = true;++ /**+ * Create a builder+ *+ * @param nodeCount the number of nodes in the cluster+ * @param baseDir a base directory for the cluster+ */+ public Builder(int nodeCount, Path baseDir) {+ this.nodeCount = nodeCount;+ this.baseDir = baseDir;++ jettyConfigBuilder = JettyConfig.builder().setContext("/solr");+ if (SolrTestCaseJ4.sslConfig != null) {+ jettyConfigBuilder = jettyConfigBuilder.withSSLConfig(SolrTestCaseJ4.sslConfig.buildServerSSLConfig());+ }+ }++ /**+ * Use a JettyConfig.Builder to configure the cluster's jetty servers+ */+ public Builder withJettyConfig(Consumer<JettyConfig.Builder> fun) {+ fun.accept(jettyConfigBuilder);+ return this;+ }++ /**+ * Use the provided string as solr.xml content+ */+ public Builder withSolrXml(String solrXml) {+ this.solrxml = solrXml;+ return this;+ }++ /**+ * Read solr.xml from the provided path+ */+ public Builder withSolrXml(Path solrXml) {+ try {+ this.solrxml = new String(Files.readAllBytes(solrXml), Charset.defaultCharset());+ } catch (IOException e) {+ throw new RuntimeException(e);+ }+ return this;+ }++ /**+ * Configure the specified security.json for the {@linkplain MiniSolrCloudCluster}+ *+ * @param securityJson The path specifying the security.json file+ * @return the instance of {@linkplain Builder}+ */+ public Builder withSecurityJson(Path securityJson) {+ try {+ this.securityJson = Optional.of(new String(Files.readAllBytes(securityJson), Charset.defaultCharset()));+ } catch (IOException e) {+ throw new RuntimeException(e);+ }+ return this;+ }++ /**+ * Configure the specified security.json for the {@linkplain MiniSolrCloudCluster}+ *+ * @param securityJson The string specifying the security.json configuration+ * @return the instance of {@linkplain Builder}+ */+ public Builder withSecurityJson(String securityJson) {+ this.securityJson = Optional.of(securityJson);+ return this;+ }++ /**+ * Upload a collection config before tests start+ *+ * @param configName the config name+ * @param configPath the path to the config files+ */+ public Builder addConfig(String configName, Path configPath) {+ this.configs.add(new Config(configName, configPath));+ return this;+ }++ /**+ * <p>This method makes the MiniSolrCloudCluster use the "other" Collection API execution strategy than it normally+ * would. When some test classes call this method (and some don't) we make sure that a run of multiple tests with a+ * single seed will exercise both code lines (distributed and Overseer based Collection API) so regressions can be+ * spotted faster.</p>+ *+ * <p> The real need is for a few tests covering reasonable use cases to call this method. If you're adding a new test,+ * you don't have to call it (but it's ok if you do).</p>+ */+ public Builder useOtherCollectionConfigSetExecution() {+ // Switch from Overseer to distributed Collection execution and vice versa+ useDistributedCollectionConfigSetExecution = !useDistributedCollectionConfigSetExecution;+ // Reverse distributed cluster state updates as well if possible (state can't be Overseer based if Collections API is distributed)+ useDistributedClusterStateUpdate = !useDistributedClusterStateUpdate || useDistributedCollectionConfigSetExecution;+ return this;+ }++ /**+ * Force the cluster Collection and config state API execution as well as the cluster state update strategy to be+ * either Overseer based or distributed. <b>This method can be useful when debugging tests</b> failing in only one+ * of the two modes to have all local runs exhibit the issue, as well obviously for tests that are not compatible+ * with one of the two modes.+ * <p>+ * If this method is not called, the strategy being used will be random if the configuration passed to the cluster+ * ({@code solr.xml} equivalent) contains a placeholder similar to:+ * <pre>+ * {@code+ * <solrcloud>+ * ....+ * <str name="distributedClusterStateUpdates">${solr.distributedClusterStateUpdates:false}</str>+ * <str name="distributedCollectionConfigSetExecution">${solr.distributedCollectionConfigSetExecution:false}</str>+ * ....+ * </solrcloud>+ * }</pre>+ * For an example of a configuration supporting this setting, see {@link MiniSolrCloudCluster#DEFAULT_CLOUD_SOLR_XML}.+ * When a test sets a different {@code solr.xml} config (using {@link #withSolrXml}), if the config does not contain+ * the placeholder, the strategy will be defined by the values assigned to {@code useDistributedClusterStateUpdates}+ * and {@code useDistributedCollectionConfigSetExecution} in {@link org.apache.solr.core.CloudConfig.CloudConfigBuilder}.+ *+ * @param distributedCollectionConfigSetApi When {@code true}, Collection and Config Set API commands are executed+ * in a distributed way by nodes. When {@code false}, they are executed by+ * Overseer.+ * @param distributedClusterStateUpdates When {@code true}, cluster state updates are handled in a distributed+ * way by nodes. When {@code false}, cluster state updates are handled by+ * Overseer.<p> If {@code distributedCollectionConfigSetApi} is {@code+ * true} then this parameter must be {@code true}.+ */+ public Builder withDistributedClusterStateUpdates(boolean distributedCollectionConfigSetApi, boolean distributedClusterStateUpdates) {+ useDistributedCollectionConfigSetExecution = distributedCollectionConfigSetApi;+ useDistributedClusterStateUpdate = distributedClusterStateUpdates;+ return this;+ }++ /**+ * Set a cluster property+ *+ * @param propertyName the property name+ * @param propertyValue the property value+ */+ public Builder withProperty(String propertyName, String propertyValue) {+ this.clusterProperties.put(propertyName, propertyValue);+ return this;+ }++ public Builder withMetrics(boolean trackJettyMetrics) {+ this.trackJettyMetrics = trackJettyMetrics;+ return this;+ }++ public Builder formatZkServer(boolean formatZkServer) {+ this.formatZkServer = formatZkServer;+ return this;+ }++ /**+ * Configure and run the {@link MiniSolrCloudCluster}+ *+ * @throws Exception if an error occurs on startup+ */+ public MiniSolrCloudCluster configure() throws Exception {+ return SolrCloudTestCase.cluster = build();+ }++ /**+ * Configure, run and return the {@link MiniSolrCloudCluster}+ *+ * @throws Exception if an error occurs on startup+ */+ public MiniSolrCloudCluster build() throws Exception {+ // Two lines below will have an impact on how the MiniSolrCloudCluster and therefore the test run if the config being+ // used in the test does have the appropriate placeholders. See for example DEFAULT_CLOUD_SOLR_XML in MiniSolrCloudCluster.+ // Hard coding values here will impact such tests.+ // To hard code behavior for tests not having these placeholders - and for SolrCloud as well for that matter! -+ // change the values assigned to useDistributedClusterStateUpdates and useDistributedCollectionConfigSetExecution in+ // org.apache.solr.core.CloudConfig.CloudConfigBuilder. Do not forget then to revert before commit!+ System.setProperty("solr.distributedCollectionConfigSetExecution", Boolean.toString(useDistributedCollectionConfigSetExecution));+ System.setProperty("solr.distributedClusterStateUpdates", Boolean.toString(useDistributedClusterStateUpdate));++ JettyConfig jettyConfig = jettyConfigBuilder.build();+ MiniSolrCloudCluster cluster = new MiniSolrCloudCluster(nodeCount, baseDir, solrxml, jettyConfig,+ null, securityJson, trackJettyMetrics, formatZkServer);+ CloudSolrClient client = cluster.getSolrClient();+ for (Config config : configs) {+ cluster.uploadConfigSet(config.path, config.name);+ }++ if (clusterProperties.size() > 0) {+ ClusterProperties props = new ClusterProperties(cluster.getSolrClient().getZkStateReader().getZkClient());+ for (Map.Entry<String, Object> entry : clusterProperties.entrySet()) {+ props.setClusterProperty(entry.getKey(), entry.getValue());+ }+ }+ return cluster;+ }++ public Builder withDefaultClusterProperty(String key, String value) {+ @SuppressWarnings({"unchecked"})+ HashMap<String, Object> defaults = (HashMap<String, Object>) this.clusterProperties.get(CollectionAdminParams.DEFAULTS);+ if (defaults == null) {+ defaults = new HashMap<>();+ this.clusterProperties.put(CollectionAdminParams.DEFAULTS, defaults);+ }+ @SuppressWarnings({"unchecked"})+ HashMap<String, Object> cluster = (HashMap<String, Object>) defaults.get(CollectionAdminParams.CLUSTER);+ if (cluster == null) {+ cluster = new HashMap<>();+ defaults.put(CollectionAdminParams.CLUSTER, cluster);+ }+ cluster.put(key, value);+ return this;+ }+ }}diff --git a/solr/test-framework/src/java/org/apache/solr/cloud/MultiSolrCloudTestCase.java b/solr/test-framework/src/java/org/apache/solr/cloud/MultiSolrCloudTestCase.javaindex b7e6bb649f5..3746f2dbc92 100644--- a/solr/test-framework/src/java/org/apache/solr/cloud/MultiSolrCloudTestCase.java+++ b/solr/test-framework/src/java/org/apache/solr/cloud/MultiSolrCloudTestCase.java@@ -46,8 +46,7 @@ public abstract class MultiSolrCloudTestCase extends SolrTestCaseJ4 {@Overridepublic MiniSolrCloudCluster apply(String clusterId) {try {- final MiniSolrCloudCluster cluster = new SolrCloudTestCase- .Builder(nodesPerCluster(clusterId), createTempDir())+ final MiniSolrCloudCluster cluster = new MiniSolrCloudCluster.Builder(nodesPerCluster(clusterId), createTempDir()).addConfig("conf", configset("cloud-dynamic")).build();return cluster;diff --git a/solr/test-framework/src/java/org/apache/solr/cloud/SolrCloudTestCase.java b/solr/test-framework/src/java/org/apache/solr/cloud/SolrCloudTestCase.javaindex 08cce5d0dda..42e7b0d92f1 100644--- a/solr/test-framework/src/java/org/apache/solr/cloud/SolrCloudTestCase.java+++ b/solr/test-framework/src/java/org/apache/solr/cloud/SolrCloudTestCase.java@@ -19,33 +19,25 @@ package org.apache.solr.cloud;import java.io.IOException;import java.lang.invoke.MethodHandles;-import java.nio.charset.Charset;-import java.nio.file.Files;-import java.nio.file.Path;import java.util.ArrayList;import java.util.Arrays;import java.util.Collections;import java.util.HashMap;import java.util.List;import java.util.Map;-import java.util.Optional;import java.util.Set;import java.util.concurrent.TimeUnit;import java.util.concurrent.atomic.AtomicReference;-import java.util.function.Consumer;import java.util.function.Predicate;import org.apache.lucene.util.LuceneTestCase;import org.apache.solr.SolrTestCaseJ4;import org.apache.solr.client.solrj.SolrRequest;import org.apache.solr.client.solrj.SolrServerException;-import org.apache.solr.client.solrj.embedded.JettyConfig;import org.apache.solr.client.solrj.embedded.JettySolrRunner;-import org.apache.solr.client.solrj.impl.CloudSolrClient;import org.apache.solr.client.solrj.impl.HttpSolrClient;import org.apache.solr.client.solrj.request.CoreAdminRequest;import org.apache.solr.client.solrj.request.CoreStatus;-import org.apache.solr.common.cloud.ClusterProperties;import org.apache.solr.common.cloud.CollectionStatePredicate;import org.apache.solr.common.cloud.DocCollection;import org.apache.solr.common.cloud.LiveNodesPredicate;@@ -53,7 +45,6 @@ import org.apache.solr.common.cloud.Replica;import org.apache.solr.common.cloud.Slice;import org.apache.solr.common.cloud.SolrZkClient;import org.apache.solr.common.cloud.ZkStateReader;-import org.apache.solr.common.params.CollectionAdminParams;import org.apache.solr.common.util.NamedList;import org.junit.AfterClass;import org.junit.Before;@@ -85,240 +76,6 @@ public class SolrCloudTestCase extends SolrTestCaseJ4 {public static final int DEFAULT_TIMEOUT = 45; // this is an important timeout for test stability - can't be too short- private static class Config {- final String name;- final Path path;- private Config(String name, Path path) {- this.name = name;- this.path = path;- }- }-- /**- * Builder class for a MiniSolrCloudCluster- */- public static class Builder {-- private final int nodeCount;- private final Path baseDir;- private String solrxml = MiniSolrCloudCluster.DEFAULT_CLOUD_SOLR_XML;- private JettyConfig.Builder jettyConfigBuilder = JettyConfig.builder().setContext("/solr").withSSLConfig(sslConfig.buildServerSSLConfig());- private Optional<String> securityJson = Optional.empty();-- private List<Config> configs = new ArrayList<>();- private Map<String, Object> clusterProperties = new HashMap<>();-- private boolean trackJettyMetrics;- private boolean useDistributedCollectionConfigSetExecution;- private boolean useDistributedClusterStateUpdate;-- /**- * Create a builder- *- * @param nodeCount the number of nodes in the cluster- * @param baseDir a base directory for the cluster- */- public Builder(int nodeCount, Path baseDir) {- this.nodeCount = nodeCount;- this.baseDir = baseDir;- // By default the MiniSolrCloudCluster being built will randomly (seed based) decide which collection API strategy- // to use (distributed or Overseer based) and which cluster update strategy to use (distributed if collection API- // is distributed, but Overseer based or distributed randomly chosen if Collection API is Overseer based)- this.useDistributedCollectionConfigSetExecution = LuceneTestCase.random().nextInt(2) == 0;- this.useDistributedClusterStateUpdate = useDistributedCollectionConfigSetExecution || LuceneTestCase.random().nextInt(2) == 0;- }-- /**- * Use a JettyConfig.Builder to configure the cluster's jetty servers- */- public Builder withJettyConfig(Consumer<JettyConfig.Builder> fun) {- fun.accept(jettyConfigBuilder);- return this;- }-- /**- * Use the provided string as solr.xml content- */- public Builder withSolrXml(String solrXml) {- this.solrxml = solrXml;- return this;- }-- /**- * Read solr.xml from the provided path- */- public Builder withSolrXml(Path solrXml) {- try {- this.solrxml = new String(Files.readAllBytes(solrXml), Charset.defaultCharset());- } catch (IOException e) {- throw new RuntimeException(e);- }- return this;- }-- /**- * Configure the specified security.json for the {@linkplain MiniSolrCloudCluster}- *- * @param securityJson The path specifying the security.json file- * @return the instance of {@linkplain Builder}- */- public Builder withSecurityJson(Path securityJson) {- try {- this.securityJson = Optional.of(new String(Files.readAllBytes(securityJson), Charset.defaultCharset()));- } catch (IOException e) {- throw new RuntimeException(e);- }- return this;- }-- /**- * Configure the specified security.json for the {@linkplain MiniSolrCloudCluster}- *- * @param securityJson The string specifying the security.json configuration- * @return the instance of {@linkplain Builder}- */- public Builder withSecurityJson(String securityJson) {- this.securityJson = Optional.of(securityJson);- return this;- }-- /**- * Upload a collection config before tests start- *- * @param configName the config name- * @param configPath the path to the config files- */- public Builder addConfig(String configName, Path configPath) {- this.configs.add(new Config(configName, configPath));- return this;- }-- /**- * This method makes the MiniSolrCloudCluster use the "other" Collection API execution strategy than it normally would.- * When some test classes call this method (and some don't) we make sure that a run of multiple tests with a single- * seed will exercise both code lines (distributed and Overseer based Collection API) so regressions can be spotted- * faster.<p>- *- * The real need is for a few tests covering reasonable use cases to call this method. If you're adding a new test,- * you don't have to call it (but it's ok if you do).- */- public Builder useOtherCollectionConfigSetExecution() {- // Switch from Overseer to distributed Collection execution and vice versa- useDistributedCollectionConfigSetExecution = !useDistributedCollectionConfigSetExecution;- // Reverse distributed cluster state updates as well if possible (state can't be Overseer based if Collections API is distributed)- useDistributedClusterStateUpdate = !useDistributedClusterStateUpdate || useDistributedCollectionConfigSetExecution;- return this;- }-- /**- * Force the cluster Collection and config state API execution as well as the cluster state update strategy to be- * either Overseer based or distributed. <b>This method can be useful when- * debugging tests</b> failing in only one of the two modes to have all local runs exhibit the issue, as well obviously for- * tests that are not compatible with one of the two modes.- * <p>- * If this method is not called, the strategy being used will be random if the configuration passed to the cluster- * ({@code solr.xml} equivalent) contains a placeholder similar to:- * <pre>- * {@code- * <solrcloud>- * ....- * <str name="distributedClusterStateUpdates">${solr.distributedClusterStateUpdates:false}</str>- * <str name="distributedCollectionConfigSetExecution">${solr.distributedCollectionConfigSetExecution:false}</str>- * ....- * </solrcloud>- * }</pre>- * For an example of a configuration supporting this setting, see {@link MiniSolrCloudCluster#DEFAULT_CLOUD_SOLR_XML}.- * When a test sets a different {@code solr.xml} config (using {@link #withSolrXml}), if the config does not contain- * the placeholder, the strategy will be defined by the values assigned to {@code useDistributedClusterStateUpdates}- * and {@code useDistributedCollectionConfigSetExecution} in {@link org.apache.solr.core.CloudConfig.CloudConfigBuilder}.- *- * @param distributedCollectionConfigSetApi When {@code true}, Collection and Config Set API commands are executed in- * a distributed way by nodes. When {@code false}, they are executed by Overseer.- * @param distributedClusterStateUpdates When {@code true}, cluster state updates are handled in a distributed way by nodes. When- * {@code false}, cluster state updates are handled by Overseer.<p>- * If {@code distributedCollectionConfigSetApi} is {@code true} then this parameter must be {@code true}.- */- public Builder withDistributedClusterStateUpdates(boolean distributedCollectionConfigSetApi, boolean distributedClusterStateUpdates) {- useDistributedCollectionConfigSetExecution = distributedCollectionConfigSetApi;- useDistributedClusterStateUpdate = distributedClusterStateUpdates;- return this;- }-- /**- * Set a cluster property- *- * @param propertyName the property name- * @param propertyValue the property value- */- public Builder withProperty(String propertyName, String propertyValue) {- this.clusterProperties.put(propertyName, propertyValue);- return this;- }-- public Builder withMetrics(boolean trackJettyMetrics) {- this.trackJettyMetrics = trackJettyMetrics;- return this;- }-- /**- * Configure and run the {@link MiniSolrCloudCluster}- *- * @throws Exception if an error occurs on startup- */- public MiniSolrCloudCluster configure() throws Exception {- return cluster = build();- }-- /**- * Configure, run and return the {@link MiniSolrCloudCluster}- *- * @throws Exception if an error occurs on startup- */- public MiniSolrCloudCluster build() throws Exception {- // Two lines below will have an impact on how the MiniSolrCloudCluster and therefore the test run if the config being- // used in the test does have the appropriate placeholders. See for example DEFAULT_CLOUD_SOLR_XML in MiniSolrCloudCluster.- // Hard coding values here will impact such tests.- // To hard code behavior for tests not having these placeholders - and for SolrCloud as well for that matter! -- // change the values assigned to useDistributedClusterStateUpdates and useDistributedCollectionConfigSetExecution in- // org.apache.solr.core.CloudConfig.CloudConfigBuilder. Do not forget then to revert before commit!- System.setProperty("solr.distributedCollectionConfigSetExecution", Boolean.toString(useDistributedCollectionConfigSetExecution));- System.setProperty("solr.distributedClusterStateUpdates", Boolean.toString(useDistributedClusterStateUpdate));-- JettyConfig jettyConfig = jettyConfigBuilder.build();- MiniSolrCloudCluster cluster = new MiniSolrCloudCluster(nodeCount, baseDir, solrxml, jettyConfig,- null, securityJson, trackJettyMetrics);- CloudSolrClient client = cluster.getSolrClient();- for (Config config : configs) {- cluster.uploadConfigSet(config.path, config.name);- }-- if (clusterProperties.size() > 0) {- ClusterProperties props = new ClusterProperties(cluster.getSolrClient().getZkStateReader().getZkClient());- for (Map.Entry<String, Object> entry : clusterProperties.entrySet()) {- props.setClusterProperty(entry.getKey(), entry.getValue());- }- }- return cluster;- }-- public Builder withDefaultClusterProperty(String key, String value) {- @SuppressWarnings({"unchecked"})- HashMap<String, Object> defaults = (HashMap<String, Object>) this.clusterProperties.get(CollectionAdminParams.DEFAULTS);- if (defaults == null) {- defaults = new HashMap<>();- this.clusterProperties.put(CollectionAdminParams.DEFAULTS, defaults);- }- @SuppressWarnings({"unchecked"})- HashMap<String, Object> cluster = (HashMap<String, Object>) defaults.get(CollectionAdminParams.CLUSTER);- if (cluster == null) {- cluster = new HashMap<>();- defaults.put(CollectionAdminParams.CLUSTER, cluster);- }- cluster.put(key, value);- return this;- }- }-/*** The cluster*/@@ -334,12 +91,18 @@ public class SolrCloudTestCase extends SolrTestCaseJ4 {/*** Call this to configure a cluster of n nodes.* <p>- * NB you must call {@link Builder#configure()} to start the cluster+ * NB you must call {@link MiniSolrCloudCluster.Builder#configure()} to start the cluster** @param nodeCount the number of nodes*/- protected static Builder configureCluster(int nodeCount) {- return new Builder(nodeCount, createTempDir());+ protected static MiniSolrCloudCluster.Builder configureCluster(int nodeCount) {+ // By default the MiniSolrCloudCluster being built will randomly (seed based) decide which collection API strategy+ // to use (distributed or Overseer based) and which cluster update strategy to use (distributed if collection API+ // is distributed, but Overseer based or distributed randomly chosen if Collection API is Overseer based)++ boolean useDistributedCollectionConfigSetExecution = LuceneTestCase.random().nextInt(2) == 0;+ boolean useDistributedClusterStateUpdate = useDistributedCollectionConfigSetExecution || LuceneTestCase.random().nextInt(2) == 0;+ return new MiniSolrCloudCluster.Builder(nodeCount, createTempDir()).withDistributedClusterStateUpdates(useDistributedCollectionConfigSetExecution, useDistributedClusterStateUpdate);}@AfterClassdiff --git a/versions.lock b/versions.lockindex c50a475d4b4..6f2d1991196 100644--- a/versions.lock+++ b/versions.lock@@ -93,6 +93,7 @@ junit:junit:4.13.1 (3 constraints: e324f01e)net.arnx:jsonic:1.2.7 (2 constraints: db10d4d1)net.hydromatic:eigenbase-properties:1.1.5 (1 constraints: 0905f835)net.jcip:jcip-annotations:1.0 (1 constraints: 560ff165)+net.sf.jopt-simple:jopt-simple:4.6 (1 constraints: 610a91b7)net.sourceforge.argparse4j:argparse4j:0.8.1 (1 constraints: 0b050436)net.thisptr:jackson-jq:0.0.8 (1 constraints: 0a05f335)org.antlr:antlr4-runtime:4.5.1-1 (2 constraints: 9115f6e5)@@ -105,7 +106,7 @@ org.apache.commons:commons-configuration2:2.1.1 (1 constraints: 0605f935)org.apache.commons:commons-csv:1.7 (1 constraints: ac04212c)org.apache.commons:commons-exec:1.3 (1 constraints: a8041d2c)org.apache.commons:commons-lang3:3.9 (3 constraints: 2b24bbb0)-org.apache.commons:commons-math3:3.6.1 (1 constraints: 0c050d36)+org.apache.commons:commons-math3:3.6.1 (2 constraints: 670ffb89)org.apache.commons:commons-text:1.6 (1 constraints: ab04202c)org.apache.curator:curator-client:2.13.0 (1 constraints: 3805383b)org.apache.curator:curator-framework:2.13.0 (1 constraints: 3805383b)@@ -221,6 +222,7 @@ org.gagravarr:vorbis-java-tika:0.8 (1 constraints: ac041f2c)org.hamcrest:hamcrest:2.2 (1 constraints: 5e104998)org.jdom:jdom2:2.0.6 (1 constraints: 0a05fb35)org.locationtech.spatial4j:spatial4j:0.7 (1 constraints: 58105398)+org.openjdk.jmh:jmh-core:1.32 (1 constraints: da04f730)org.ow2.asm:asm:7.2 (3 constraints: a325a35c)org.ow2.asm:asm-commons:7.2 (1 constraints: 6b0f7267)org.slf4j:jcl-over-slf4j:1.7.24 (1 constraints: 4005473b)diff --git a/versions.props b/versions.propsindex 2617e1ed145..fee08a8d473 100644--- a/versions.props+++ b/versions.props@@ -119,6 +119,7 @@ org.jsoup:jsoup=1.12.1org.locationtech.spatial4j:*=0.7org.mockito:mockito-core=3.8.0org.objenesis:objenesis=2.6+org.openjdk.jmh:*=1.32org.ow2.asm:*=7.2org.slf4j:*=1.7.24org.tallison:jmatio=1.5