Apache Lucene open-source search software
Inline the gitinfo plugin from carrotsearch's buildinfra, use dependencychecks and build options directly. (#14794)
Dawid Weiss52fb6ebf8336
diff --git a/CHANGES-record.md b/CHANGES-record.mddeleted file mode 100644index 1172d73fe9b..00000000000--- a/CHANGES-record.md+++ /dev/null@@ -1,112 +0,0 @@-<!--- Licensed to the Apache Software Foundation (ASF) under one or more- contributor license agreements. See the NOTICE file distributed with- this work for additional information regarding copyright ownership.- The ASF licenses this file to You under the Apache License, Version 2.0- the "License"); you may not use this file except in compliance with- the License. You may obtain a copy of the License at-- http://www.apache.org/licenses/LICENSE-2.0-- Unless required by applicable law or agreed to in writing, software- distributed under the License is distributed on an "AS IS" BASIS,- WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.- See the License for the specific language governing permissions and- limitations under the License.- -->--This is a record of changes to the modernized gradle build.--### ```buildOptions``` plugin instead of ```propertyOrDefault``` method--All "propertyOrDefault*" methods have been removed and an external-plugin for configuring various build options is now used. This is-the plugin (blame me for any shortcomings):--https://github.com/carrotsearch/gradle-build-infra#plugin-comcarrotsearchgradlebuildinfrabuildoptionsbuildoptionsplugin--this plugin will source the value of a given build option from-several places, in order:-- system property (```-Dfoo=value```),-- gradle property (```-Pfoo=value```),-- environment variable (```foo=value ./gradlew ...```)-- a *non-versioned*, ```build-options.local.properties``` property file- (for your personal, local tweaks),-- a *versioned* ```build-options.properties``` property file.--It works much like before - you can override any build option-temporarily by passing, for example, ```-Ptests.verbose=true```. But you-can also make such changes locally persistent by placing them-in your ```build-options.local.properties``` file. Alternatively,-the generated ```gradle.properties``` is also supported since it declares-project properties (equivalent to ```-Pkey=value```). At some point-in the future, we may want to keep gradle.properties versioned though,-so it's probably a good idea to switch over to-```build-options.local.properties```.--The biggest gain from using build options is that you can now see-all declared options and their values, including their source (where their-value comes from). To see what I mean, try running these:-```-./gradlew :buildOptions-./gradlew -p lucene/core buildOptions-```--### Lucene 'base version' string moved--* Various Lucene's "version" properties are now full build options (as described above).-Their default values are in ```build-options.properties```. You can override-any of ```version.suffix```, ```version.base``` or ```version.release```-using any of the build option plugin's methods. If you'd like to bump-Lucene's base version, it now lives in the versioned ```build-options.properties```.--### Tweaks to properties and options--* ```runtime.java.home``` is a build option now but the support for-```RUNTIME_JAVA_HOME``` env. variable has been implemented for backward-compatibility.--* ```tests.neverUpToDate``` option is renamed ```tests.rerun```.-The value of ```true``` indicates test tasks will re-run even if-nothing has changed.--### Changes to project structure--* ```build-tools/missing-doclet``` is now a regular project within Lucene (not-an included composite project). This simplifies the build and ensures we apply-the same checks to this subproject as we do to everything else.--* all gradle build scripts are converted to convention plugins (in groovy) and-live under ```dev-tools/build-infra/src/main/groovy```. The long-term-plan is to move some or all of these groovy plugins to Java (so that the code is-easier to read and debug for non-groovians).--* ```tidy``` is now applied to groovy/gradle scripts (formatting is enforced).--### Security manager support--* parts of the build supporting security manager settings have been just-removed, without any replacement.--### Other notable changes--* The legacy ```precommit``` task has been removed; use gradle's ```check```.--* Removed dependency on jgit entirely. This is replaced by forking the system's git-in porcelain mode (which should be stable and portable). This logic is implemented-in [this plugin](https://github.com/carrotsearch/gradle-build-infra/?tab=readme-ov-file#plugin-comcarrotsearchgradlebuildinfraenvironmentgitinfoplugin).-An additional benefit is that all features of git should now work (including worktrees).--* Added support for owasp API keys in the form of ```validation.owasp.apikey``` build option. Owasp check is-still very, very slow. We should probably just drop it.--* I've changed the default on ```gradle.ge``` (Gradle Enterprise, develocity) to ```false```-on non-CI builds.--* I've tried to clean up lots and lots of gradle/groovy code related to eager initialization of-tasks as well as update deprecated APIs. This isn't always straightforward and I might have broken-some things...--### Fixes to existing issues--* ```gradlew clean check``` will work now (https://github.com/apache/lucene/issues/13567)diff --git a/build-tools/build-infra-shadow/build.gradle b/build-tools/build-infra-shadow/build.gradleindex 8b47d43a0c9..cc348fccf9c 100644--- a/build-tools/build-infra-shadow/build.gradle+++ b/build-tools/build-infra-shadow/build.gradle@@ -26,14 +26,18 @@ repositories {gradlePluginPortal()}+// Convert a plugin dependency to a regular dependency so that we can+// use [plugins] section in the top-level toml but declare regular+// project dependencies here.static Provider<String> plugin(Provider<PluginDependency> plugin) {return plugin.map {- if (it.pluginId == "com.carrotsearch.gradle.buildinfra") {- return "${it.pluginId}:${it.pluginId}.gradle.plugin:${it.version}".toString()- } else if (it.pluginId == "de.thetaphi.forbiddenapis") {+ if (it.pluginId == "de.thetaphi.forbiddenapis") {+ // Uwe's forbiddenapis is on Maven Central, directly.return "de.thetaphi:forbiddenapis:${it.version}".toString()} else {- return "${it.pluginId}:${it.pluginId.replaceFirst(".*\\.", "")}-plugin-gradle:${it.version}".toString()+ // maven artifact pattern for gradle's plugin repositories.+ return "${it.pluginId}:${it.pluginId}.gradle.plugin:${it.version}".toString()+ // return "${it.pluginId}:${it.pluginId.replaceFirst(".*\\.", "")}-plugin-gradle:${it.version}".toString()}}}@@ -44,7 +48,8 @@ dependencies {implementation deps.commons.codecimplementation deps.randomizedtesting.runner- implementation plugin(deps.plugins.carrotsearch.buildinfra)+ implementation plugin(deps.plugins.carrotsearch.buildopts)+ implementation plugin(deps.plugins.carrotsearch.dependencychecks)implementation plugin(deps.plugins.forbiddenapis)implementation plugin(deps.plugins.spotless)}diff --git a/build-tools/build-infra/build.gradle b/build-tools/build-infra/build.gradleindex 14a1c1350df..3b1ba01ebf0 100644--- a/build-tools/build-infra/build.gradle+++ b/build-tools/build-infra/build.gradle@@ -69,13 +69,15 @@ dependencies {implementation deps.flexmark.ext.autolinkimplementation deps.flexmark.ext.tables- implementation plugin(deps.plugins.carrotsearch.buildinfra)+ implementation plugin(deps.plugins.carrotsearch.buildopts)+ implementation plugin(deps.plugins.carrotsearch.dependencychecks)implementation plugin(deps.plugins.forbiddenapis)implementation plugin(deps.plugins.spotless)implementation plugin(deps.plugins.owasp.dependencycheck)implementation plugin(deps.plugins.undercouch.download)implementation plugin(deps.plugins.errorprone)implementation plugin(deps.plugins.jacocolog)+ implementation plugin(deps.plugins.versionCatalogUpdate)}def hasJavaFlightRecorder = ModuleLayer.boot().findModule('jdk.jfr').map{ otherModule ->diff --git a/build-tools/build-infra/src/main/groovy/lucene.java.jar-manifest.gradle b/build-tools/build-infra/src/main/groovy/lucene.java.jar-manifest.gradleindex 9c36d8f1c9a..b35504731f6 100644--- a/build-tools/build-infra/src/main/groovy/lucene.java.jar-manifest.gradle+++ b/build-tools/build-infra/src/main/groovy/lucene.java.jar-manifest.gradle@@ -15,7 +15,7 @@* limitations under the License.*/-import com.carrotsearch.gradle.buildinfra.environment.GitInfoExtension+import org.apache.lucene.gradle.plugins.gitinfo.GitInfoExtension// Configures artifact JARs (manifest entries)diff --git a/build-tools/build-infra/src/main/groovy/lucene.root-project.setup.gradle b/build-tools/build-infra/src/main/groovy/lucene.root-project.setup.gradleindex 674021deeb5..b03836f0da2 100644--- a/build-tools/build-infra/src/main/groovy/lucene.root-project.setup.gradle+++ b/build-tools/build-infra/src/main/groovy/lucene.root-project.setup.gradle@@ -17,7 +17,7 @@import com.carrotsearch.gradle.buildinfra.buildoptions.BuildOptionValueSourceimport com.carrotsearch.gradle.buildinfra.buildoptions.BuildOptionsPlugin-import com.carrotsearch.gradle.buildinfra.environment.GitInfoPlugin+import org.apache.lucene.gradle.plugins.gitinfo.GitInfoPluginimport com.carrotsearch.randomizedtesting.SeedUtilsimport java.time.ZonedDateTimediff --git a/build-tools/build-infra/src/main/java/org/apache/lucene/gradle/plugins/gitinfo/GitInfoExtension.java b/build-tools/build-infra/src/main/java/org/apache/lucene/gradle/plugins/gitinfo/GitInfoExtension.javanew file mode 100644index 00000000000..fde1c6c6a61--- /dev/null+++ b/build-tools/build-infra/src/main/java/org/apache/lucene/gradle/plugins/gitinfo/GitInfoExtension.java@@ -0,0 +1,9 @@+package org.apache.lucene.gradle.plugins.gitinfo;++import org.gradle.api.provider.MapProperty;++public abstract class GitInfoExtension {+ public static final String NAME = "gitinfo";++ public abstract MapProperty<String, String> getGitInfo();+}diff --git a/build-tools/build-infra/src/main/java/org/apache/lucene/gradle/plugins/gitinfo/GitInfoPlugin.java b/build-tools/build-infra/src/main/java/org/apache/lucene/gradle/plugins/gitinfo/GitInfoPlugin.javanew file mode 100644index 00000000000..d9a67b79fe8--- /dev/null+++ b/build-tools/build-infra/src/main/java/org/apache/lucene/gradle/plugins/gitinfo/GitInfoPlugin.java@@ -0,0 +1,28 @@+package org.apache.lucene.gradle.plugins.gitinfo;++import org.gradle.api.GradleException;+import org.gradle.api.Plugin;+import org.gradle.api.Project;++public class GitInfoPlugin implements Plugin<Project> {+ @Override+ public void apply(Project project) {+ if (project != project.getRootProject()) {+ throw new GradleException("This plugin is applicable to the rootProject only.");+ }++ var gitInfoProvider =+ project+ .getProviders()+ .of(+ GitInfoValueSource.class,+ spec -> {+ spec.getParameters().getRootProjectDir().set(project.getProjectDir());+ });++ var gitInfoExtension =+ project.getExtensions().create(GitInfoExtension.NAME, GitInfoExtension.class);++ gitInfoExtension.getGitInfo().value(gitInfoProvider).finalizeValueOnRead();+ }+}diff --git a/build-tools/build-infra/src/main/java/org/apache/lucene/gradle/plugins/gitinfo/GitInfoValueSource.java b/build-tools/build-infra/src/main/java/org/apache/lucene/gradle/plugins/gitinfo/GitInfoValueSource.javanew file mode 100644index 00000000000..824e2bc3001--- /dev/null+++ b/build-tools/build-infra/src/main/java/org/apache/lucene/gradle/plugins/gitinfo/GitInfoValueSource.java@@ -0,0 +1,98 @@+package org.apache.lucene.gradle.plugins.gitinfo;++import java.io.BufferedOutputStream;+import java.io.ByteArrayOutputStream;+import java.io.IOException;+import java.nio.charset.StandardCharsets;+import java.util.Map;+import java.util.TreeMap;+import java.util.regex.Matcher;+import java.util.regex.Pattern;+import javax.inject.Inject;+import org.apache.tools.ant.taskdefs.condition.Os;+import org.gradle.api.GradleException;+import org.gradle.api.file.DirectoryProperty;+import org.gradle.api.provider.ValueSource;+import org.gradle.api.provider.ValueSourceParameters;+import org.gradle.api.services.BuildServiceParameters;+import org.gradle.process.ExecOperations;+import org.jetbrains.annotations.Nullable;++public abstract class GitInfoValueSource+ implements ValueSource<Map<String, String>, GitInfoValueSource.Parameters> {+ public abstract static class Parameters implements BuildServiceParameters, ValueSourceParameters {+ public abstract DirectoryProperty getRootProjectDir();+ }++ @Inject+ public abstract ExecOperations getExecOps();++ @Override+ public @Nullable Map<String, String> obtain() {+ try (var baos = new ByteArrayOutputStream();+ var out = new BufferedOutputStream(baos)) {+ var result =+ getExecOps()+ .exec(+ spec -> {+ spec.setStandardOutput(out);+ spec.setErrorOutput(out);+ spec.setIgnoreExitValue(true);++ spec.setWorkingDir(+ getParameters().getRootProjectDir().getAsFile().get().getAbsolutePath());+ spec.setExecutable(Os.isFamily(Os.FAMILY_WINDOWS) ? "git.exe" : "git");+ spec.args("status", "--porcelain=v2", "--branch");+ });+ out.flush();++ // Just assume it's UTF-8. I don't know if this can break anyhow.+ String gitOutput = baos.toString(StandardCharsets.UTF_8);++ if (result.getExitValue() != 0) {+ // Something went wrong. Assume this isn't a git checkout and+ // add placeholder values.+ return Map.ofEntries(+ Map.entry("git.commit", "unknown"),+ Map.entry("git.commit-short", "unknown"),+ Map.entry("git.clean", "false"),+ Map.entry("git.changed-files", "not a checkout?"),+ Map.entry("git.error-log", gitOutput));+ }++ // Parse git-porcelain output.+ Map<String, String> properties = new TreeMap<>();++ Pattern headerLine = Pattern.compile("# (?<key>[^ ]+) (?<value>.+)");+ Pattern changeLine = Pattern.compile("(?<type>[12u?!]) (?<value>.+)");+ StringBuilder changedLines = new StringBuilder();+ for (var line : gitOutput.split("\\n")) {+ if (line.startsWith("#")) {+ Matcher matcher = headerLine.matcher(line);+ if (matcher.matches()) {+ String key = matcher.group("key");+ String value = matcher.group("value");+ properties.put(key, value);++ if (key.equals("branch.oid")) {+ properties.put("git.commit", value);+ properties.put("git.commit-short", value.substring(0, 8));+ }+ }+ } else if (changeLine.matcher(line).matches()) {+ changedLines.append(line);+ changedLines.append("\n");+ } else {+ // Omit other lines.+ }+ }++ properties.put("git.changed-files", changedLines.toString());+ properties.put("git.clean", changedLines.isEmpty() ? "true" : "false");++ return properties;+ } catch (IOException e) {+ throw new GradleException("Errors calling git to fetch local repository status.", e);+ }+ }+}diff --git a/gradle/libs.versions.toml b/gradle/libs.versions.tomlindex c2f729aa05a..e95b370427c 100644--- a/gradle/libs.versions.toml+++ b/gradle/libs.versions.toml@@ -96,7 +96,8 @@ zstd = { module = "com.github.luben:zstd-jni", version.ref = "zstd" }[plugins]benmanes-versions = "com.github.ben-manes.versions:0.52.0"-carrotsearch-buildinfra = "com.carrotsearch.gradle.buildinfra:0.0.12"+carrotsearch-buildopts = "com.carrotsearch.gradle.opts:0.2.0"+carrotsearch-dependencychecks = "com.carrotsearch.gradle.dependencychecks:0.1.0"errorprone = "net.ltgt.errorprone:4.2.0"forbiddenapis = "de.thetaphi.forbiddenapis:3.9"jacocolog = "org.barfuin.gradle.jacocolog:3.1.0"