Apache Lucene open-source search software
Regeneration
Lucene has a number of machine-generated resources - some of these are resource (binary) files, others are Java source files that are stored (and compiled) with the rest of Lucene source code.
If you're reading this, chances are that:
- you've hit a build check error that said you've modified a generated resource and some checksums are out of sync.
- you need to regenerate one (or more) of these resources because of third party library updates or other reasons.
In many cases hitting (1) means you'll have to do (2) so let's discuss these in order.
Checksum validation errors
LUCENE-9868 introduced a system of storing (and validating) checksums of generated files so that they are not accidentally modified. This checksum-checking system will fail the build with a message similar to this one:
Execution failed for task ':lucene:core:regenerateStandardTokenizerChecksumCheck'.
> Checksums mismatch for generated resources; you might have modified a generated resource (regenerate task: regenerateStandardTokenizer, checksum file: .../lucene/core/src/generated/checksums/regenerateStandardTokenizer.json).
Current checksums:
- lucene/core/src/java/org/apache/lucene/analysis/standard/StandardTokenizerImpl.java=f4d6311e03a97cb55301cfa55fab3338b1fdb725
Expected checksums:
- lucene/core/src/java/org/apache/lucene/analysis/standard/StandardTokenizerImpl.java=f4d6311e03a97cb55301cfa55fab3338b1fdb724
Input files for this task are:
- .../lucene/core/src/java/org/apache/lucene/analysis/standard/StandardTokenizerImpl.jflex
- .../lucene/gradle/regenerate/jflex/skeleton.disable.buffer.expansion.txt
Files generated by this task are:
- .../lucene/core/src/java/org/apache/lucene/analysis/standard/StandardTokenizerImpl.java
The message shows you which resources have mismatches on checksums (in this case StandardTokenizerImpl.java) but
also the module where the generated resource exists, the task name that should be used to regenerate this resource
and the inputs and outputs of that task:
:lucene:core:regenerateStandardTokenizer
To resolve the problem, try to git diff the changes that caused the build failure (to see why the checksums have
changed). Then decide if the change was accidental or intentional and either:
- revert any accidental changes made to inputs/outputs of generated resources,
- regenerate the derived resources, possibly saving new checksums.
If you decide to regenerate, just running the task hinted at in the error message should work fine, for example:
gradlew :lucene:core:regenerateStandardTokenizer
This regenerates all resources the task generateStandardTokenizer produces and updates the corresponding checksums.
If you want to bypass any of gradle's smart up-to-date checks and force regeneration, use the standard gradle's switch:
gradlew :lucene:core:regenerateStandardTokenizer --rerun-tasks
Resource regeneration
Lucene's "convention" task for regenerating all derived resources in a given module is called regenerate and you can
apply it to all Lucene modules at once by running:
gradlew regenerate
It is typically much wiser to limit the scope of regeneration to only the module you're working with though:
gradlew -p lucene/analysis/common regenerate
If you're interested in what specific generation tasks are available, see the task list for the generation group:
gradlew tasks --group regeneration
or limit the output to a particular module:
gradlew -p lucene/analysis/common tasks --group regeneration
which displays (at the moment of writing):
regenerate - Rerun any code or static data generation tasks.
regenerateClassicTokenizer - Regenerate ClassicTokenizerImpl.java
regenerateHTMLCharacterEntities - Regenerate HTMLCharacterEntities.jflex
regenerateHTMLStripCharFilter - Regenerate HTMLStripCharFilter.java
regenerateTlds - Regenerate top-level domain jflex macros and tests
regenerateUAX29URLEmailTokenizer - Regenerate UAX29URLEmailTokenizerImpl.java
regenerateUnicodeProps - Regenerate UnicodeProps.java using ICU
regenerateWikipediaTokenizer - Regenerate WikipediaTokenizerImpl.java
Resource checksums, incremental generation and advanced topics
Many resource generation tasks require specific tools (perl, python, bash shell) and resources that may not be
available on all platforms. In LUCENE-9868 we tried to make resource generation tasks "incremental" so that they only
run if their sources (or outputs) have changed. So if you run the generic regenerate task, many of the actual
regeneration sub-tasks will be "skipped" - you can see this if you run gradle with plain console, for example:
gradlew -p lucene/analysis/common regenerate --console=plain
...
> Task :lucene:analysis:common:regenerateUnicodeProps SKIPPED
> Task :lucene:analysis:common:regenerateUAX29URLEmailTokenizer SKIPPED
...
This shouldn't worry you at all - the internal tasks are skipped because the inputs and outputs of these task are up to date. If changes were detected, the task would be re-executed, followed by any necessary cleanup tasks (like code formatting).
Sometimes you may want to force the regeneration task to run, even if the checksums indicate nothing has changed. This may happen if you're paranoid or don't trust gradle (and there's no reason you should).
If you want to force-run the regeneration, use gradle's --rerun-tasks option:
gradlew regenerate --rerun-tasks
Scoping the call to a particular module will also work:
gradlew -p lucene/analysis/common regenerate --rerun-tasks
Scoping the call to a particular task will also work:
gradlew -p lucene/analysis/common regenerateUnicodeProps --rerun-tasks
Finally, if you do feel like force-regenerating everything, remember to exclude this monster (it takes quite a long time and requires significant heap memory)...
gradlew regenerate -x regenerateUAX29URLEmailTokenizer --rerun-tasks
and on Windows, exclude snowball regeneration (requires bash):
gradlew regenerate -x regenerateUAX29URLEmailTokenizer -x snowball --rerun-tasks