Skip to content

Apache Lucene open-source search software

Inline the gitinfo plugin from carrotsearch's buildinfra, use dependencychecks and build options directly. (#14794)

Dawid Weiss52fb6ebf8336
diff --git a/CHANGES-record.md b/CHANGES-record.md
deleted file mode 100644
index 1172d73fe9b..00000000000
--- a/CHANGES-record.md
+++ /dev/null
@@ -1,112 +0,0 @@
-<!--
- Licensed to the Apache Software Foundation (ASF) under one or more
- contributor license agreements. See the NOTICE file distributed with
- this work for additional information regarding copyright ownership.
- The ASF licenses this file to You under the Apache License, Version 2.0
- the "License"); you may not use this file except in compliance with
- the License. You may obtain a copy of the License at
-
- http://www.apache.org/licenses/LICENSE-2.0
-
- Unless required by applicable law or agreed to in writing, software
- distributed under the License is distributed on an "AS IS" BASIS,
- WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- See the License for the specific language governing permissions and
- limitations under the License.
- -->
-
-This is a record of changes to the modernized gradle build.
-
-### ```buildOptions``` plugin instead of ```propertyOrDefault``` method
-
-All "propertyOrDefault*" methods have been removed and an external
-plugin for configuring various build options is now used. This is
-the plugin (blame me for any shortcomings):
-
-https://github.com/carrotsearch/gradle-build-infra#plugin-comcarrotsearchgradlebuildinfrabuildoptionsbuildoptionsplugin
-
-this plugin will source the value of a given build option from
-several places, in order:
-- system property (```-Dfoo=value```),
-- gradle property (```-Pfoo=value```),
-- environment variable (```foo=value ./gradlew ...```)
-- a *non-versioned*, ```build-options.local.properties``` property file
- (for your personal, local tweaks),
-- a *versioned* ```build-options.properties``` property file.
-
-It works much like before - you can override any build option
-temporarily by passing, for example, ```-Ptests.verbose=true```. But you
-can also make such changes locally persistent by placing them
-in your ```build-options.local.properties``` file. Alternatively,
-the generated ```gradle.properties``` is also supported since it declares
-project properties (equivalent to ```-Pkey=value```). At some point
-in the future, we may want to keep gradle.properties versioned though,
-so it's probably a good idea to switch over to
-```build-options.local.properties```.
-
-The biggest gain from using build options is that you can now see
-all declared options and their values, including their source (where their
-value comes from). To see what I mean, try running these:
-```
-./gradlew :buildOptions
-./gradlew -p lucene/core buildOptions
-```
-
-### Lucene 'base version' string moved
-
-* Various Lucene's "version" properties are now full build options (as described above).
-Their default values are in ```build-options.properties```. You can override
-any of ```version.suffix```, ```version.base``` or ```version.release```
-using any of the build option plugin's methods. If you'd like to bump
-Lucene's base version, it now lives in the versioned ```build-options.properties```.
-
-### Tweaks to properties and options
-
-* ```runtime.java.home``` is a build option now but the support for
-```RUNTIME_JAVA_HOME``` env. variable has been implemented for backward
-compatibility.
-
-* ```tests.neverUpToDate``` option is renamed ```tests.rerun```.
-The value of ```true``` indicates test tasks will re-run even if
-nothing has changed.
-
-### Changes to project structure
-
-* ```build-tools/missing-doclet``` is now a regular project within Lucene (not
-an included composite project). This simplifies the build and ensures we apply
-the same checks to this subproject as we do to everything else.
-
-* all gradle build scripts are converted to convention plugins (in groovy) and
-live under ```dev-tools/build-infra/src/main/groovy```. The long-term
-plan is to move some or all of these groovy plugins to Java (so that the code is
-easier to read and debug for non-groovians).
-
-* ```tidy``` is now applied to groovy/gradle scripts (formatting is enforced).
-
-### Security manager support
-
-* parts of the build supporting security manager settings have been just
-removed, without any replacement.
-
-### Other notable changes
-
-* The legacy ```precommit``` task has been removed; use gradle's ```check```.
-
-* Removed dependency on jgit entirely. This is replaced by forking the system's git
-in porcelain mode (which should be stable and portable). This logic is implemented
-in [this plugin](https://github.com/carrotsearch/gradle-build-infra/?tab=readme-ov-file#plugin-comcarrotsearchgradlebuildinfraenvironmentgitinfoplugin).
-An additional benefit is that all features of git should now work (including worktrees).
-
-* Added support for owasp API keys in the form of ```validation.owasp.apikey``` build option. Owasp check is
-still very, very slow. We should probably just drop it.
-
-* I've changed the default on ```gradle.ge``` (Gradle Enterprise, develocity) to ```false```
-on non-CI builds.
-
-* I've tried to clean up lots and lots of gradle/groovy code related to eager initialization of
-tasks as well as update deprecated APIs. This isn't always straightforward and I might have broken
-some things...
-
-### Fixes to existing issues
-
-* ```gradlew clean check``` will work now (https://github.com/apache/lucene/issues/13567)
diff --git a/build-tools/build-infra-shadow/build.gradle b/build-tools/build-infra-shadow/build.gradle
index 8b47d43a0c9..cc348fccf9c 100644
--- a/build-tools/build-infra-shadow/build.gradle
+++ b/build-tools/build-infra-shadow/build.gradle
@@ -26,14 +26,18 @@ repositories {
gradlePluginPortal()
}
+// Convert a plugin dependency to a regular dependency so that we can
+// use [plugins] section in the top-level toml but declare regular
+// project dependencies here.
static Provider<String> plugin(Provider<PluginDependency> plugin) {
return plugin.map {
- if (it.pluginId == "com.carrotsearch.gradle.buildinfra") {
- return "${it.pluginId}:${it.pluginId}.gradle.plugin:${it.version}".toString()
- } else if (it.pluginId == "de.thetaphi.forbiddenapis") {
+ if (it.pluginId == "de.thetaphi.forbiddenapis") {
+ // Uwe's forbiddenapis is on Maven Central, directly.
return "de.thetaphi:forbiddenapis:${it.version}".toString()
} else {
- return "${it.pluginId}:${it.pluginId.replaceFirst(".*\\.", "")}-plugin-gradle:${it.version}".toString()
+ // maven artifact pattern for gradle's plugin repositories.
+ return "${it.pluginId}:${it.pluginId}.gradle.plugin:${it.version}".toString()
+ // return "${it.pluginId}:${it.pluginId.replaceFirst(".*\\.", "")}-plugin-gradle:${it.version}".toString()
}
}
}
@@ -44,7 +48,8 @@ dependencies {
implementation deps.commons.codec
implementation deps.randomizedtesting.runner
- implementation plugin(deps.plugins.carrotsearch.buildinfra)
+ implementation plugin(deps.plugins.carrotsearch.buildopts)
+ implementation plugin(deps.plugins.carrotsearch.dependencychecks)
implementation plugin(deps.plugins.forbiddenapis)
implementation plugin(deps.plugins.spotless)
}
diff --git a/build-tools/build-infra/build.gradle b/build-tools/build-infra/build.gradle
index 14a1c1350df..3b1ba01ebf0 100644
--- a/build-tools/build-infra/build.gradle
+++ b/build-tools/build-infra/build.gradle
@@ -69,13 +69,15 @@ dependencies {
implementation deps.flexmark.ext.autolink
implementation deps.flexmark.ext.tables
- implementation plugin(deps.plugins.carrotsearch.buildinfra)
+ implementation plugin(deps.plugins.carrotsearch.buildopts)
+ implementation plugin(deps.plugins.carrotsearch.dependencychecks)
implementation plugin(deps.plugins.forbiddenapis)
implementation plugin(deps.plugins.spotless)
implementation plugin(deps.plugins.owasp.dependencycheck)
implementation plugin(deps.plugins.undercouch.download)
implementation plugin(deps.plugins.errorprone)
implementation plugin(deps.plugins.jacocolog)
+ implementation plugin(deps.plugins.versionCatalogUpdate)
}
def hasJavaFlightRecorder = ModuleLayer.boot().findModule('jdk.jfr').map{ otherModule ->
diff --git a/build-tools/build-infra/src/main/groovy/lucene.java.jar-manifest.gradle b/build-tools/build-infra/src/main/groovy/lucene.java.jar-manifest.gradle
index 9c36d8f1c9a..b35504731f6 100644
--- a/build-tools/build-infra/src/main/groovy/lucene.java.jar-manifest.gradle
+++ b/build-tools/build-infra/src/main/groovy/lucene.java.jar-manifest.gradle
@@ -15,7 +15,7 @@
* limitations under the License.
*/
-import com.carrotsearch.gradle.buildinfra.environment.GitInfoExtension
+import org.apache.lucene.gradle.plugins.gitinfo.GitInfoExtension
// Configures artifact JARs (manifest entries)
diff --git a/build-tools/build-infra/src/main/groovy/lucene.root-project.setup.gradle b/build-tools/build-infra/src/main/groovy/lucene.root-project.setup.gradle
index 674021deeb5..b03836f0da2 100644
--- a/build-tools/build-infra/src/main/groovy/lucene.root-project.setup.gradle
+++ b/build-tools/build-infra/src/main/groovy/lucene.root-project.setup.gradle
@@ -17,7 +17,7 @@
import com.carrotsearch.gradle.buildinfra.buildoptions.BuildOptionValueSource
import com.carrotsearch.gradle.buildinfra.buildoptions.BuildOptionsPlugin
-import com.carrotsearch.gradle.buildinfra.environment.GitInfoPlugin
+import org.apache.lucene.gradle.plugins.gitinfo.GitInfoPlugin
import com.carrotsearch.randomizedtesting.SeedUtils
import java.time.ZonedDateTime
diff --git a/build-tools/build-infra/src/main/java/org/apache/lucene/gradle/plugins/gitinfo/GitInfoExtension.java b/build-tools/build-infra/src/main/java/org/apache/lucene/gradle/plugins/gitinfo/GitInfoExtension.java
new file mode 100644
index 00000000000..fde1c6c6a61
--- /dev/null
+++ b/build-tools/build-infra/src/main/java/org/apache/lucene/gradle/plugins/gitinfo/GitInfoExtension.java
@@ -0,0 +1,9 @@
+package org.apache.lucene.gradle.plugins.gitinfo;
+
+import org.gradle.api.provider.MapProperty;
+
+public abstract class GitInfoExtension {
+ public static final String NAME = "gitinfo";
+
+ public abstract MapProperty<String, String> getGitInfo();
+}
diff --git a/build-tools/build-infra/src/main/java/org/apache/lucene/gradle/plugins/gitinfo/GitInfoPlugin.java b/build-tools/build-infra/src/main/java/org/apache/lucene/gradle/plugins/gitinfo/GitInfoPlugin.java
new file mode 100644
index 00000000000..d9a67b79fe8
--- /dev/null
+++ b/build-tools/build-infra/src/main/java/org/apache/lucene/gradle/plugins/gitinfo/GitInfoPlugin.java
@@ -0,0 +1,28 @@
+package org.apache.lucene.gradle.plugins.gitinfo;
+
+import org.gradle.api.GradleException;
+import org.gradle.api.Plugin;
+import org.gradle.api.Project;
+
+public class GitInfoPlugin implements Plugin<Project> {
+ @Override
+ public void apply(Project project) {
+ if (project != project.getRootProject()) {
+ throw new GradleException("This plugin is applicable to the rootProject only.");
+ }
+
+ var gitInfoProvider =
+ project
+ .getProviders()
+ .of(
+ GitInfoValueSource.class,
+ spec -> {
+ spec.getParameters().getRootProjectDir().set(project.getProjectDir());
+ });
+
+ var gitInfoExtension =
+ project.getExtensions().create(GitInfoExtension.NAME, GitInfoExtension.class);
+
+ gitInfoExtension.getGitInfo().value(gitInfoProvider).finalizeValueOnRead();
+ }
+}
diff --git a/build-tools/build-infra/src/main/java/org/apache/lucene/gradle/plugins/gitinfo/GitInfoValueSource.java b/build-tools/build-infra/src/main/java/org/apache/lucene/gradle/plugins/gitinfo/GitInfoValueSource.java
new file mode 100644
index 00000000000..824e2bc3001
--- /dev/null
+++ b/build-tools/build-infra/src/main/java/org/apache/lucene/gradle/plugins/gitinfo/GitInfoValueSource.java
@@ -0,0 +1,98 @@
+package org.apache.lucene.gradle.plugins.gitinfo;
+
+import java.io.BufferedOutputStream;
+import java.io.ByteArrayOutputStream;
+import java.io.IOException;
+import java.nio.charset.StandardCharsets;
+import java.util.Map;
+import java.util.TreeMap;
+import java.util.regex.Matcher;
+import java.util.regex.Pattern;
+import javax.inject.Inject;
+import org.apache.tools.ant.taskdefs.condition.Os;
+import org.gradle.api.GradleException;
+import org.gradle.api.file.DirectoryProperty;
+import org.gradle.api.provider.ValueSource;
+import org.gradle.api.provider.ValueSourceParameters;
+import org.gradle.api.services.BuildServiceParameters;
+import org.gradle.process.ExecOperations;
+import org.jetbrains.annotations.Nullable;
+
+public abstract class GitInfoValueSource
+ implements ValueSource<Map<String, String>, GitInfoValueSource.Parameters> {
+ public abstract static class Parameters implements BuildServiceParameters, ValueSourceParameters {
+ public abstract DirectoryProperty getRootProjectDir();
+ }
+
+ @Inject
+ public abstract ExecOperations getExecOps();
+
+ @Override
+ public @Nullable Map<String, String> obtain() {
+ try (var baos = new ByteArrayOutputStream();
+ var out = new BufferedOutputStream(baos)) {
+ var result =
+ getExecOps()
+ .exec(
+ spec -> {
+ spec.setStandardOutput(out);
+ spec.setErrorOutput(out);
+ spec.setIgnoreExitValue(true);
+
+ spec.setWorkingDir(
+ getParameters().getRootProjectDir().getAsFile().get().getAbsolutePath());
+ spec.setExecutable(Os.isFamily(Os.FAMILY_WINDOWS) ? "git.exe" : "git");
+ spec.args("status", "--porcelain=v2", "--branch");
+ });
+ out.flush();
+
+ // Just assume it's UTF-8. I don't know if this can break anyhow.
+ String gitOutput = baos.toString(StandardCharsets.UTF_8);
+
+ if (result.getExitValue() != 0) {
+ // Something went wrong. Assume this isn't a git checkout and
+ // add placeholder values.
+ return Map.ofEntries(
+ Map.entry("git.commit", "unknown"),
+ Map.entry("git.commit-short", "unknown"),
+ Map.entry("git.clean", "false"),
+ Map.entry("git.changed-files", "not a checkout?"),
+ Map.entry("git.error-log", gitOutput));
+ }
+
+ // Parse git-porcelain output.
+ Map<String, String> properties = new TreeMap<>();
+
+ Pattern headerLine = Pattern.compile("# (?<key>[^ ]+) (?<value>.+)");
+ Pattern changeLine = Pattern.compile("(?<type>[12u?!]) (?<value>.+)");
+ StringBuilder changedLines = new StringBuilder();
+ for (var line : gitOutput.split("\\n")) {
+ if (line.startsWith("#")) {
+ Matcher matcher = headerLine.matcher(line);
+ if (matcher.matches()) {
+ String key = matcher.group("key");
+ String value = matcher.group("value");
+ properties.put(key, value);
+
+ if (key.equals("branch.oid")) {
+ properties.put("git.commit", value);
+ properties.put("git.commit-short", value.substring(0, 8));
+ }
+ }
+ } else if (changeLine.matcher(line).matches()) {
+ changedLines.append(line);
+ changedLines.append("\n");
+ } else {
+ // Omit other lines.
+ }
+ }
+
+ properties.put("git.changed-files", changedLines.toString());
+ properties.put("git.clean", changedLines.isEmpty() ? "true" : "false");
+
+ return properties;
+ } catch (IOException e) {
+ throw new GradleException("Errors calling git to fetch local repository status.", e);
+ }
+ }
+}
diff --git a/gradle/libs.versions.toml b/gradle/libs.versions.toml
index c2f729aa05a..e95b370427c 100644
--- a/gradle/libs.versions.toml
+++ b/gradle/libs.versions.toml
@@ -96,7 +96,8 @@ zstd = { module = "com.github.luben:zstd-jni", version.ref = "zstd" }
[plugins]
benmanes-versions = "com.github.ben-manes.versions:0.52.0"
-carrotsearch-buildinfra = "com.carrotsearch.gradle.buildinfra:0.0.12"
+carrotsearch-buildopts = "com.carrotsearch.gradle.opts:0.2.0"
+carrotsearch-dependencychecks = "com.carrotsearch.gradle.dependencychecks:0.1.0"
errorprone = "net.ltgt.errorprone:4.2.0"
forbiddenapis = "de.thetaphi.forbiddenapis:3.9"
jacocolog = "org.barfuin.gradle.jacocolog:3.1.0"