Skip to content

Apache Lucene open-source search software

Relax Lucene Index Upgrade Policy to Allow Safe Upgrades Across Multiple Major Versions #13797 (#15012)

Mark Robert Miller2d8f0cb5289b
diff --git a/dev-tools/scripts/releaseWizard.yaml b/dev-tools/scripts/releaseWizard.yaml
index a96664d9732..a273d3cef45 100644
--- a/dev-tools/scripts/releaseWizard.yaml
+++ b/dev-tools/scripts/releaseWizard.yaml
@@ -300,6 +300,42 @@ groups:
user_input: !UserInput
prompt: Enter end date of feature freeze (YYYY-MM-DD)
name: feature_freeze_date
+ - !Todo
+ id: evaluate_min_supported_major
+ title: Evaluate MIN_SUPPORTED_MAJOR policy
+ types:
+ - major
+ description: |
+ For major releases, evaluate whether MIN_SUPPORTED_MAJOR in Version.java should be bumped.
+ This constant controls the relaxed index upgrade policy - when format-breaking changes occur,
+ it should be updated to the new minimum supported version.
+
+ **Evaluation Checklist:**
+
+ 1. **Review index format changes**: Have any format-breaking changes been introduced since the current MIN_SUPPORTED_MAJOR?
+ - Codec version changes that break backward compatibility
+ - SegmentInfo format changes
+ - New mandatory index structures
+ - Changes to core file formats
+
+ 2. **Check compatibility span**:
+ - Current MIN_SUPPORTED_MAJOR: Review Version.java
+ - Proposed new value (if changes occurred): Should match the oldest version that can still be opened
+
+ 3. **Update if needed**:
+ - Modify Version.MIN_SUPPORTED_MAJOR constant in lucene/core/src/java/org/apache/lucene/util/Version.java
+ - Update test expectations in backward compatibility tests
+ - Document the change in CHANGES.txt with clear reasoning
+
+ 4. **Validate the change**:
+ - Run backward compatibility tests: `./gradlew -p lucene/backward-codecs test`
+ - Ensure error messages are clear and actionable
+ - Verify no unintended version restrictions
+ **Note**: If no format-breaking changes occurred, leave MIN_SUPPORTED_MAJOR unchanged to
+ maintain the widest possible upgrade span for users.
+ links:
+ - https://github.com/apache/lucene/blob/main/lucene/core/src/java/org/apache/lucene/util/Version.java
+ - https://github.com/apache/lucene/issues/13797
- !TodoGroup
id: branching_versions
title: Create branch (if needed) and update versions
diff --git a/lucene/CHANGES.txt b/lucene/CHANGES.txt
index 43b64c2977a..e7964da56bb 100644
--- a/lucene/CHANGES.txt
+++ b/lucene/CHANGES.txt
@@ -7,6 +7,14 @@ http://s.apache.org/luceneversions
API Changes
---------------------
+* GITHUB#13797: Relaxed Lucene Index Upgrade Policy to Allow Safe Upgrades Across Multiple Major Versions.
+ The MIN_SUPPORTED_MAJOR constant in Version.java is now manually maintained instead of auto-computed as
+ LATEST.major-1. This enables users to perform their one index upgrade across multiple major version numbers
+ when no format breaks occur (previously limited to exactly one major version). The constant is set to 10 for
+ Lucene 11.0.0 and will only be bumped when actual incompatible format changes are introduced. This changes
+ reindexing frequency from "every major release" to "only when format breaks occur" while maintaining the
+ fundamental "one upgrade per index lifetime" limitation. See MIGRATE.md for upgrade instructions. (Mark Miller)
+
* GITHUB#11023: Removing deprecated parameters from CheckIndex. (Jakub Slowinski)
* GITHUB#14165: TieredMergePolicy's maxMergeAtOnce parameter was removed. (Adrien Grand)
diff --git a/lucene/MIGRATE.md b/lucene/MIGRATE.md
index dcc1cef2547..f70d852199d 100644
--- a/lucene/MIGRATE.md
+++ b/lucene/MIGRATE.md
@@ -19,6 +19,62 @@
## Migration from Lucene 10.x to Lucene 11.0
+### Relaxed Index Upgrade Policy (GITHUB#13797)
+
+Starting with Lucene 11.0.0, the index upgrade policy has been relaxed to allow safe upgrades across multiple major version numbers without reindexing when no format breaks occur.
+
+#### Key Changes
+
+- `Version.MIN_SUPPORTED_MAJOR` is now manually maintained instead of auto-computed as `LATEST.major-1`
+- Set to 10 for Lucene 11.0.0, allowing indexes created with Lucene 10.x to be opened directly
+- Will only be bumped when actual incompatible format changes are introduced
+
+#### Two-Tier Version Policy
+
+1. **Index opening policy**: An index can be opened if its creation version >= `MIN_SUPPORTED_MAJOR`
+2. **Codec reader policy**: Segments can only be read directly if written by current or previous major version number
+
+#### Upgrade Scenarios
+
+**Scenario 1: No format breaks (wider upgrade span)**
+- Index created with Lucene 10.x can be opened directly in Lucene 11.x, 12.x, 13.x, 14.x (as long as MIN_SUPPORTED_MAJOR stays ≤ 10)
+- Simply open the index with the new version; segments will be upgraded gradually through normal merging
+- Optional: Call `forceMerge()` or use `UpgradeIndexMergePolicy` to upgrade segment formats immediately
+- **Important**: You still only get one upgrade per index lifetime. Once MIN_SUPPORTED_MAJOR is bumped above 10, the index becomes unopenable and must be reindexed.
+
+**Scenario 2: Format breaks occur**
+- If a major version introduces incompatible format changes, `MIN_SUPPORTED_MAJOR` will be bumped
+- Indexes created before the new minimum will throw `IndexFormatTooOldException`
+- Full reindexing is required for such indexes
+
+**Scenario 3: After using your upgrade**
+- Index created with Lucene 10.x, successfully opened with Lucene 14.x
+- The index's creation version is still 10 (this never changes)
+- When Lucene 15+ bumps MIN_SUPPORTED_MAJOR above 10, this index becomes unopenable
+- Must reindex to continue using newer Lucene versions
+
+#### Upgrade Example
+
+```java
+// Opening an index created with Lucene 10.x in Lucene 11.x+
+try (Directory dir = FSDirectory.open(indexPath)) {
+ // This will now succeed (if MIN_SUPPORTED_MAJOR <= 10)
+ try (DirectoryReader reader = DirectoryReader.open(dir)) {
+ // Index can be read normally
+ }
+ // Optional: Upgrade segment formats
+ try (IndexWriter writer = new IndexWriter(dir, config)) {
+ writer.forceMerge(1); // Rewrites all segments to latest format
+ }
+}
+```
+
+#### Error Handling
+
+Enhanced error messages will clearly indicate:
+- Whether the index creation version is below `MIN_SUPPORTED_MAJOR` (reindex required)
+- Whether segments are too old to read directly (sequential upgrade required)
+
### TieredMergePolicy#setMaxMergeAtOnce removed
This parameter has no replacement, TieredMergePolicy no longer bounds the
diff --git a/lucene/backward-codecs/src/test/org/apache/lucene/backward_index/BackwardsCompatibilityTestBase.java b/lucene/backward-codecs/src/test/org/apache/lucene/backward_index/BackwardsCompatibilityTestBase.java
index edbed96be75..ff06ce63257 100644
--- a/lucene/backward-codecs/src/test/org/apache/lucene/backward_index/BackwardsCompatibilityTestBase.java
+++ b/lucene/backward-codecs/src/test/org/apache/lucene/backward_index/BackwardsCompatibilityTestBase.java
@@ -160,7 +160,7 @@ public abstract class BackwardsCompatibilityTestBase extends LuceneTestCase {
private static Version getLatestPreviousMajorVersion() {
Version lastPrevMajorVersion = null;
for (Version v : getAllCurrentVersions()) {
- if (v.major == Version.LATEST.major - 1
+ if (v.major == Version.MIN_SUPPORTED_MAJOR
&& (lastPrevMajorVersion == null || v.onOrAfter(lastPrevMajorVersion))) {
lastPrevMajorVersion = v;
}
diff --git a/lucene/backward-codecs/src/test/org/apache/lucene/backward_index/TestBasicBackwardsCompatibility.java b/lucene/backward-codecs/src/test/org/apache/lucene/backward_index/TestBasicBackwardsCompatibility.java
index a93c5c7617a..77dc6d2412c 100644
--- a/lucene/backward-codecs/src/test/org/apache/lucene/backward_index/TestBasicBackwardsCompatibility.java
+++ b/lucene/backward-codecs/src/test/org/apache/lucene/backward_index/TestBasicBackwardsCompatibility.java
@@ -863,7 +863,10 @@ public class TestBasicBackwardsCompatibility extends BackwardsCompatibilityTestB
() -> StandardDirectoryReader.open(commit, Version.LATEST.major, null));
assertTrue(
ex.getMessage()
- .contains("only supports reading from version " + Version.LATEST.major + " upwards."));
+ .contains(
+ "This Lucene version only supports indexes created with major version "
+ + Version.LATEST.major
+ + " or later"));
// now open with allowed min version
StandardDirectoryReader.open(commit, Version.MIN_SUPPORTED_MAJOR, null).close();
}
diff --git a/lucene/backward-codecs/src/test/org/apache/lucene/backward_index/TestMinSupportedMajorBackwardsCompatibility.java b/lucene/backward-codecs/src/test/org/apache/lucene/backward_index/TestMinSupportedMajorBackwardsCompatibility.java
new file mode 100644
index 00000000000..82246624445
--- /dev/null
+++ b/lucene/backward-codecs/src/test/org/apache/lucene/backward_index/TestMinSupportedMajorBackwardsCompatibility.java
@@ -0,0 +1,214 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.lucene.backward_index;
+
+import static org.apache.lucene.util.Version.MIN_SUPPORTED_MAJOR;
+
+import com.carrotsearch.randomizedtesting.annotations.ParametersFactory;
+import java.io.IOException;
+import java.util.List;
+import java.util.stream.Collectors;
+import org.apache.lucene.document.Document;
+import org.apache.lucene.document.Field;
+import org.apache.lucene.document.TextField;
+import org.apache.lucene.index.DirectoryReader;
+import org.apache.lucene.index.IndexFormatTooOldException;
+import org.apache.lucene.index.IndexWriter;
+import org.apache.lucene.index.IndexWriterConfig;
+import org.apache.lucene.index.SegmentInfos;
+import org.apache.lucene.index.Term;
+import org.apache.lucene.search.IndexSearcher;
+import org.apache.lucene.search.TermQuery;
+import org.apache.lucene.search.TopDocs;
+import org.apache.lucene.store.Directory;
+import org.apache.lucene.tests.analysis.MockAnalyzer;
+import org.apache.lucene.tests.util.TestUtil;
+import org.apache.lucene.util.Version;
+
+/**
+ * Tests for the MIN_SUPPORTED_MAJOR backwards compatibility policy. Verifies that indexes created
+ * with major version numbers >= MIN_SUPPORTED_MAJOR can be opened, while older indexes throw
+ * IndexFormatTooOldException.
+ *
+ * <p>This test uses the precreated index archives to validate the policy against real historical
+ * indexes. The test automatically adapts when MIN_SUPPORTED_MAJOR changes by using the existing
+ * backwards compatibility testing infrastructure.
+ */
+public class TestMinSupportedMajorBackwardsCompatibility extends BackwardsCompatibilityTestBase {
+ static final String INDEX_NAME = "index";
+ static final String SUFFIX = "-nocfs";
+
+ public TestMinSupportedMajorBackwardsCompatibility(Version version, String pattern) {
+ super(version, pattern);
+ }
+
+ @Override
+ protected void createIndex(Directory directory) throws IOException {
+ IndexWriterConfig conf =
+ new IndexWriterConfig(new MockAnalyzer(random()))
+ .setUseCompoundFile(false)
+ .setCodec(TestUtil.getDefaultCodec());
+ try (IndexWriter writer = new IndexWriter(directory, conf)) {
+ Document doc = new Document();
+ doc.add(new TextField("content", "test document for version compatibility", Field.Store.YES));
+ writer.addDocument(doc);
+ writer.commit();
+ }
+ }
+
+ /**
+ * Uses the standard allVersion() method to dynamically test all available versions. The
+ * supportsVersion() override controls which versions are actually tested.
+ */
+ @ParametersFactory(argumentFormatting = "Lucene-Version:%1$s; Pattern: %2$s")
+ public static Iterable<Object[]> testVersionsFactory() {
+ return allVersion(INDEX_NAME, SUFFIX);
+ }
+
+ @Override
+ protected boolean supportsVersion(Version version) {
+ // Test only versions that should be supported according to MIN_SUPPORTED_MAJOR
+ return version.major >= MIN_SUPPORTED_MAJOR;
+ }
+
+ /**
+ * Test that indexes created with versions >= MIN_SUPPORTED_MAJOR can be opened and searched. This
+ * validates that the relaxed policy correctly allows older but supported indexes.
+ */
+ public void testSupportedVersionCanBeOpened() throws IOException {
+ // The base class setup ensures we only get here for supported versions
+ assertTrue("Version should be supported", version.major >= MIN_SUPPORTED_MAJOR);
+
+ // This should succeed for any version >= MIN_SUPPORTED_MAJOR
+ try (DirectoryReader reader = DirectoryReader.open(directory)) {
+ // Basic validation that the index can be read
+ assertTrue("Index should be readable", reader.numDocs() >= 0);
+
+ // Verify we can read the SegmentInfos
+ SegmentInfos sis = SegmentInfos.readLatestCommit(directory);
+ assertTrue(
+ "Creation version should be >= MIN_SUPPORTED_MAJOR",
+ sis.getIndexCreatedVersionMajor() >= MIN_SUPPORTED_MAJOR);
+
+ // Test basic search functionality if there are documents
+ if (reader.numDocs() > 0) {
+ IndexSearcher searcher = new IndexSearcher(reader);
+ TopDocs hits = searcher.search(new TermQuery(new Term("content", "test")), 10);
+ // Note: we can't guarantee the specific content, but should be able to search
+ assertNotNull("Search should return results", hits);
+ }
+ }
+ }
+
+ /**
+ * Test unsupported versions by dynamically finding versions below MIN_SUPPORTED_MAJOR. This
+ * validates that both DirectoryReader and IndexWriter properly throw IndexFormatTooOldException
+ * for versions below MIN_SUPPORTED_MAJOR.
+ */
+ public void testUnsupportedVersionsThrowException() throws IOException {
+ // Find all versions that should be unsupported
+ List<Version> allVersions = getAllCurrentReleasedVersionsAndCurrent();
+ List<Version> unsupportedVersions =
+ allVersions.stream()
+ .filter(v -> v.major < MIN_SUPPORTED_MAJOR)
+ .collect(Collectors.toList());
+
+ if (unsupportedVersions.isEmpty()) {
+ // If there are no unsupported versions (e.g., MIN_SUPPORTED_MAJOR is very low),
+ // we can't test this behavior
+ return;
+ }
+
+ // Test each unsupported version
+ for (Version unsupportedVersion : unsupportedVersions) {
+ String indexName =
+ String.format(
+ java.util.Locale.ROOT, createPattern(INDEX_NAME, SUFFIX), unsupportedVersion);
+ java.io.InputStream resource =
+ TestAncientIndicesCompatibility.class.getResourceAsStream(indexName);
+
+ if (resource == null) {
+ // Skip versions that don't have archives (this is expected for some older versions)
+ continue;
+ }
+
+ java.nio.file.Path tempDir = createTempDir("unsupported-" + unsupportedVersion);
+ TestUtil.unzip(resource, tempDir);
+
+ try (org.apache.lucene.store.Directory dir = newFSDirectory(tempDir)) {
+ // Test DirectoryReader fails for unsupported versions
+ IndexFormatTooOldException readerException =
+ expectThrows(
+ IndexFormatTooOldException.class,
+ () -> {
+ DirectoryReader.open(dir);
+ });
+
+ // Verify the exception message contains useful information
+ String readerMessage = readerException.getMessage();
+ assertNotNull(
+ "Reader exception message should not be null for version " + unsupportedVersion,
+ readerMessage);
+ assertTrue(
+ "Reader exception message should contain version information for "
+ + unsupportedVersion
+ + ": "
+ + readerMessage,
+ readerMessage.length() > 0);
+
+ // Test IndexWriter creation also fails appropriately
+ IndexWriterConfig config = new IndexWriterConfig();
+ IndexFormatTooOldException writerException =
+ expectThrows(
+ IndexFormatTooOldException.class,
+ () -> {
+ new IndexWriter(dir, config);
+ });
+
+ String writerMessage = writerException.getMessage();
+ assertNotNull(
+ "Writer exception message should not be null for version " + unsupportedVersion,
+ writerMessage);
+ assertTrue(
+ "Writer exception message should contain version information for "
+ + unsupportedVersion
+ + ": "
+ + writerMessage,
+ writerMessage.length() > 0);
+ }
+ }
+ }
+
+ /**
+ * Test that validates the constant MIN_SUPPORTED_MAJOR has the expected value. This test will
+ * fail if someone accidentally changes the constant without considering the implications.
+ */
+ public void testMinSupportedMajorConstantValue() {
+ // This test validates that MIN_SUPPORTED_MAJOR is set to the expected value
+ // If this test fails, it likely means the constant was changed and the
+ // implications need to be considered (tests updated, documentation updated, etc.)
+ assertEquals(
+ "MIN_SUPPORTED_MAJOR should be 10 for the relaxed upgrade policy", 10, MIN_SUPPORTED_MAJOR);
+
+ // Additional validation: MIN_SUPPORTED_MAJOR should be <= current major
+ assertTrue(
+ "MIN_SUPPORTED_MAJOR should not exceed current major version",
+ MIN_SUPPORTED_MAJOR <= Version.LATEST.major);
+
+ assertTrue("MIN_SUPPORTED_MAJOR should be positive", MIN_SUPPORTED_MAJOR > 0);
+ }
+}
diff --git a/lucene/core/src/java/org/apache/lucene/index/SegmentInfos.java b/lucene/core/src/java/org/apache/lucene/index/SegmentInfos.java
index 9d87edc6e60..204bd89b350 100644
--- a/lucene/core/src/java/org/apache/lucene/index/SegmentInfos.java
+++ b/lucene/core/src/java/org/apache/lucene/index/SegmentInfos.java
@@ -349,14 +349,19 @@ public final class SegmentInfos implements Cloneable, Iterable<SegmentCommitInfo
if (indexCreatedVersion < minSupportedMajorVersion) {
throw new IndexFormatTooOldException(
input,
- "This index was initially created with Lucene "
+ "Index created with Lucene "
+ indexCreatedVersion
- + ".x while the current version is "
+ + ".x is not supported by Lucene "
+ Version.LATEST
- + " and Lucene only supports reading"
- + (minSupportedMajorVersion == Version.MIN_SUPPORTED_MAJOR
- ? " the current and previous major versions"
- : " from version " + minSupportedMajorVersion + " upwards"));
+ + ". This Lucene version only supports indexes created with major version "
+ + minSupportedMajorVersion
+ + " or later (found: "
+ + indexCreatedVersion
+ + ", minimum: "
+ + minSupportedMajorVersion
+ + "). To resolve this issue: (1) Re-index your data using Lucene "
+ + Version.LATEST.major
+ + ".x, or (2) Use an older Lucene version that supports your index format.");
}
SegmentInfos infos = new SegmentInfos(indexCreatedVersion);
diff --git a/lucene/core/src/java/org/apache/lucene/util/Version.java b/lucene/core/src/java/org/apache/lucene/util/Version.java
index c572911f970..c500ee49218 100644
--- a/lucene/core/src/java/org/apache/lucene/util/Version.java
+++ b/lucene/core/src/java/org/apache/lucene/util/Version.java
@@ -108,10 +108,57 @@ public final class Version {
@Deprecated public static final Version LUCENE_CURRENT = LATEST;
/**
- * Constant for the minimal supported major version of an index. This version is defined by the
- * version that initially created the index.
+ * Constant for the minimal supported major version number of an index. This version is defined by
+ * the major version number that initially created the index.
+ *
+ * <p>This constant is manually controlled and should only be bumped when format changes make it
+ * impossible to safely read older indexes. Examples include:
+ *
+ * <ul>
+ * <li>Lossy encoding changes (e.g., norms format changes that cannot be recovered)
+ * <li>Index-level format changes that prevent reading (e.g., segments_N file format)
+ * <li>Critical corruption bugs that make older indexes potentially invalid
+ * </ul>
+ *
+ * <p>This constant should NOT be bumped automatically with major version number releases. The
+ * goal is to allow users to upgrade across multiple major version numbers when safe to do so.
+ *
+ * <p><b>Two-tier version policy:</b>
+ *
+ * <ul>
+ * <li><b>Index opening policy:</b> An index can be opened if its {@code
+ * indexCreatedVersionMajor} is >= this constant, regardless of how many major version
+ * numbers have been released since.
+ * <li><b>Codec reader policy:</b> Segment codecs are only shipped for the current major version
+ * number and the immediately previous major version number. When no format breaks occur
+ * between consecutive major version numbers, the previous major version number reader can
+ * read segments from older major version numbers that use the same format.
+ * </ul>
+ *
+ * <p><b>When to bump this constant:</b>
+ *
+ * <ul>
+ * <li>When introducing an incompatible on-disk format change
+ * <li>When fixing a critical bug that makes older indexes potentially corrupt
+ * <li>When the maintenance burden of supporting older creation versions becomes too high
+ * </ul>
+ *
+ * <p><b>How to bump this constant:</b>
+ *
+ * <ol>
+ * <li>Set the value to the last major version number before the breaking change
+ * <li>Update tests to reflect the new minimum
+ * <li>Update CHANGES.txt and MIGRATE.md with upgrade instructions
+ * <li>Ensure IndexFormatTooOldException messages reference the new minimum
+ * </ol>
+ *
+ * <p><b>Example:</b> If a breaking change is introduced in version 15.0.0, set this constant to
+ * 14, which will prevent indexes created with major version numbers 13 and earlier from being
+ * opened.
+ *
+ * @since 11.0.0
*/
- public static final int MIN_SUPPORTED_MAJOR = Version.LATEST.major - 1;
+ public static final int MIN_SUPPORTED_MAJOR = 10;
/**
* @see #getPackageImplementationVersion()