Apache Lucene open-source search software
Testing
Examples below assume cwd at the gradlew script in the top directory of the project's checkout.
Generic test/ checkup commands
Run all unit tests:
gradlew test
Run all(*) verification tasks, including tests:
gradlew check
(*) This step may omit certain validation tasks (errorprone, for example) that are slow or require external resources. A pull request runs all those locally omitted tasks on the CI (jenkins, gh actions). For this reason, it's a good idea to run patches via the CI.
Run all verification tasks, excluding tests (-x is gradle's generic task exclusion mechanism):
gradlew check -x test
Run verification for a selected module only:
gradlew :lucene:core:check # By full gradle project path
gradlew -p lucene/core check # By folder designation + task
Randomization
To run tests with the given starting seed pass tests.seed property:
gradlew :lucene:misc:test -Ptests.seed=DEADBEEF
There are a lot of other test randomization properties available. To list them, their defaults and current values run
the buildOptions task against a project that has tests. For example:
gradlew -p lucene/core buildOptions
or list all options with:
gradlew allOptions
Filtering
Run tests of lucene-core module:
gradlew -p lucene/core test
Run a single test case (from a single module). Uses gradle's built-in filtering (https://docs.gradle.org/current/userguide/java_testing.html#test_filtering):
gradlew -p lucene/core test --tests TestDemo
Run all tests in a package:
gradlew -p lucene/core test --tests "org.apache.lucene.document.*"
Run all test classes/ methods that match this pattern:
gradlew -p lucene/core test --tests "*testFeatureMissing*"
Test groups
Tests can be filtered by an annotation they're marked with. Some test group annotations include: @AwaitsFix,
@Nightly
This uses filtering infrastructure on the runner (randomizedtesting), not gradle's built-in mechanisms (but it can be
combined with --tests). For example, run all lucene-core tests annotated as @Nightly:
gradlew -p lucene/core test -Ptests.filter=@Nightly
Test group filters can be combined into Boolean expressions:
gradlew -p lucene/core test "default and not(@awaitsfix or @nightly)"
Reiteration ("beasting")
Multiply each test case N times (this works by repeating the same test within the same JVM; it also works in IDEs):
gradlew -p lucene/core test --tests TestDemo -Ptests.iters=5
Tests tasks will be (by default) re-executed on each invocation because we pick a random global tests.seed and
because we want them to (force the test task to run). If you want to make the tests task obey up-to-date gradle
rules, fix the seed and force rerunning:
gradlew -p lucene/core test -Ptests.seed=deadbeef -Ptests.rerun=true
These properties can be set in your local gradle.properties. To force re-execution of tests, even for the same master
seed, apply cleanTest task:
gradlew -p lucene/core cleanTest test -Ptests.seed=deadbeef
The tests.iters option should be sufficient for individual test cases and is much faster than trying to duplicate
re-runs of the entire test suites. When it is absolutely needed to re-run an entire suite (because of randomization in
the static initialization, for example), you can do it by running the beast task with tests.dups option:
gradlew -p lucene/core beast -Ptests.dups=10 --tests TestPerFieldDocValuesFormat
Note the filter (--tests) used to narrow down test reiterations to a particular class. You can use any filter,
including no filter at all, but it rarely makes sense (will take ages). By default the test tasks generated by the
beast mode use a random starting seed for randomization. If you pass an explicit seed, this won't be the case (all
tasks will use exactly the same starting seed):
gradlew -p lucene/core beast -Ptests.dups=10 --tests TestPerFieldDocValuesFormat -Dtests.seed=deadbeef
Verbose mode and debugging
The tests.verbose mode switch enables standard streams from tests to be dumped directly to the console. Run your
verbose tests explicitly specifying the project and test task or a fully qualified task path. Example:
gradlew -p lucene/core test -Ptests.verbose=true --tests "TestDemo"
Larger heap size
By default tests run with a 512 MB max heap. But some tests (monster/nightly) need more heap. Use -Ptests.heapsize
for this:
gradlew -p lucene/core test --tests "Test2BFST" -Ptests.heapsize=32g
Run GUI tests headlessly with Xvfb (Linux only)
GUI test for Luke application launches a window, this might mess up your monitor depending on the display manager you are using. In that case, you can install Xvfb (X Virtual Frame Buffer) so that the test runs on the virtual display and does not open a real window.
# redhat-type OS
sudo yum install Xvfb
# ubuntu/debian-type OS
sudo apt install xvfb
# arch
sudo pacman -S xorg-server-xvfb
Profiling slow tests
The tests.profile mode switch turns on a sampling profiler during test execution, and prints a simple summary at the
end.
For example, top-10 histogram of methods (cpu samples):
gradlew -p lucene/core test -Ptests.profile=true
Alternatively, you can profile heap allocations instead:
gradlew -p lucene/core test -Ptests.profile=true -Ptests.profile.mode=heap
By default, results are computed (deduplicated) on just the method name, folding together all events from the same method. To drill down further, you can increase the stack size from the default of 1, to get a histogram of stacktraces instead:
gradlew -p lucene/core test -Ptests.profile=true -Ptests.profile.stacksize=8
For big methods, it can also be helpful to include line numbers for more granularity:
gradlew -p lucene/core test -Ptests.profile=true -Ptests.profile.linenumbers=true
Using these additional options will make the results more sparse, so it may be useful to increase the top-N count:
gradlew -p lucene/core test -Ptests.profile=true -Ptests.profile.count=100
Generating coverage reports
Running the coverage task (or setting the property tests.coverage to true) will run the tests with instrumentation
to record code coverage.
Example:
gradlew -p lucene/core coverage
open lucene/core/build/reports/jacoco/test/html/index.html
If you want to use test filtering to just check a particular test, specify the test task explicitly before
coverage:
gradlew -p lucene/core test --tests TestDemo coverage
open lucene/core/build/reports/jacoco/test/html/index.html
External data sets
Some tests may require external (and large) data sets. To see relevant tasks that download and extract these data files automatically, run the following:
gradlew tasks --group "Data set download"