Apache Lucene open-source search software
Relax Lucene Index Upgrade Policy to Allow Safe Upgrades Across Multiple Major Versions #13797 (#15012)
Mark Robert Miller2d8f0cb5289b
diff --git a/dev-tools/scripts/releaseWizard.yaml b/dev-tools/scripts/releaseWizard.yamlindex a96664d9732..a273d3cef45 100644--- a/dev-tools/scripts/releaseWizard.yaml+++ b/dev-tools/scripts/releaseWizard.yaml@@ -300,6 +300,42 @@ groups:user_input: !UserInputprompt: Enter end date of feature freeze (YYYY-MM-DD)name: feature_freeze_date+ - !Todo+ id: evaluate_min_supported_major+ title: Evaluate MIN_SUPPORTED_MAJOR policy+ types:+ - major+ description: |+ For major releases, evaluate whether MIN_SUPPORTED_MAJOR in Version.java should be bumped.+ This constant controls the relaxed index upgrade policy - when format-breaking changes occur,+ it should be updated to the new minimum supported version.++ **Evaluation Checklist:**++ 1. **Review index format changes**: Have any format-breaking changes been introduced since the current MIN_SUPPORTED_MAJOR?+ - Codec version changes that break backward compatibility+ - SegmentInfo format changes+ - New mandatory index structures+ - Changes to core file formats++ 2. **Check compatibility span**:+ - Current MIN_SUPPORTED_MAJOR: Review Version.java+ - Proposed new value (if changes occurred): Should match the oldest version that can still be opened++ 3. **Update if needed**:+ - Modify Version.MIN_SUPPORTED_MAJOR constant in lucene/core/src/java/org/apache/lucene/util/Version.java+ - Update test expectations in backward compatibility tests+ - Document the change in CHANGES.txt with clear reasoning++ 4. **Validate the change**:+ - Run backward compatibility tests: `./gradlew -p lucene/backward-codecs test`+ - Ensure error messages are clear and actionable+ - Verify no unintended version restrictions+ **Note**: If no format-breaking changes occurred, leave MIN_SUPPORTED_MAJOR unchanged to+ maintain the widest possible upgrade span for users.+ links:+ - https://github.com/apache/lucene/blob/main/lucene/core/src/java/org/apache/lucene/util/Version.java+ - https://github.com/apache/lucene/issues/13797- !TodoGroupid: branching_versionstitle: Create branch (if needed) and update versionsdiff --git a/lucene/CHANGES.txt b/lucene/CHANGES.txtindex 43b64c2977a..e7964da56bb 100644--- a/lucene/CHANGES.txt+++ b/lucene/CHANGES.txt@@ -7,6 +7,14 @@ http://s.apache.org/luceneversionsAPI Changes---------------------+* GITHUB#13797: Relaxed Lucene Index Upgrade Policy to Allow Safe Upgrades Across Multiple Major Versions.+ The MIN_SUPPORTED_MAJOR constant in Version.java is now manually maintained instead of auto-computed as+ LATEST.major-1. This enables users to perform their one index upgrade across multiple major version numbers+ when no format breaks occur (previously limited to exactly one major version). The constant is set to 10 for+ Lucene 11.0.0 and will only be bumped when actual incompatible format changes are introduced. This changes+ reindexing frequency from "every major release" to "only when format breaks occur" while maintaining the+ fundamental "one upgrade per index lifetime" limitation. See MIGRATE.md for upgrade instructions. (Mark Miller)+* GITHUB#11023: Removing deprecated parameters from CheckIndex. (Jakub Slowinski)* GITHUB#14165: TieredMergePolicy's maxMergeAtOnce parameter was removed. (Adrien Grand)diff --git a/lucene/MIGRATE.md b/lucene/MIGRATE.mdindex dcc1cef2547..f70d852199d 100644--- a/lucene/MIGRATE.md+++ b/lucene/MIGRATE.md@@ -19,6 +19,62 @@## Migration from Lucene 10.x to Lucene 11.0+### Relaxed Index Upgrade Policy (GITHUB#13797)++Starting with Lucene 11.0.0, the index upgrade policy has been relaxed to allow safe upgrades across multiple major version numbers without reindexing when no format breaks occur.++#### Key Changes++- `Version.MIN_SUPPORTED_MAJOR` is now manually maintained instead of auto-computed as `LATEST.major-1`+- Set to 10 for Lucene 11.0.0, allowing indexes created with Lucene 10.x to be opened directly+- Will only be bumped when actual incompatible format changes are introduced++#### Two-Tier Version Policy++1. **Index opening policy**: An index can be opened if its creation version >= `MIN_SUPPORTED_MAJOR`+2. **Codec reader policy**: Segments can only be read directly if written by current or previous major version number++#### Upgrade Scenarios++**Scenario 1: No format breaks (wider upgrade span)**+- Index created with Lucene 10.x can be opened directly in Lucene 11.x, 12.x, 13.x, 14.x (as long as MIN_SUPPORTED_MAJOR stays ≤ 10)+- Simply open the index with the new version; segments will be upgraded gradually through normal merging+- Optional: Call `forceMerge()` or use `UpgradeIndexMergePolicy` to upgrade segment formats immediately+- **Important**: You still only get one upgrade per index lifetime. Once MIN_SUPPORTED_MAJOR is bumped above 10, the index becomes unopenable and must be reindexed.++**Scenario 2: Format breaks occur**+- If a major version introduces incompatible format changes, `MIN_SUPPORTED_MAJOR` will be bumped+- Indexes created before the new minimum will throw `IndexFormatTooOldException`+- Full reindexing is required for such indexes++**Scenario 3: After using your upgrade**+- Index created with Lucene 10.x, successfully opened with Lucene 14.x+- The index's creation version is still 10 (this never changes)+- When Lucene 15+ bumps MIN_SUPPORTED_MAJOR above 10, this index becomes unopenable+- Must reindex to continue using newer Lucene versions++#### Upgrade Example++```java+// Opening an index created with Lucene 10.x in Lucene 11.x++try (Directory dir = FSDirectory.open(indexPath)) {+ // This will now succeed (if MIN_SUPPORTED_MAJOR <= 10)+ try (DirectoryReader reader = DirectoryReader.open(dir)) {+ // Index can be read normally+ }+ // Optional: Upgrade segment formats+ try (IndexWriter writer = new IndexWriter(dir, config)) {+ writer.forceMerge(1); // Rewrites all segments to latest format+ }+}+```++#### Error Handling++Enhanced error messages will clearly indicate:+- Whether the index creation version is below `MIN_SUPPORTED_MAJOR` (reindex required)+- Whether segments are too old to read directly (sequential upgrade required)+### TieredMergePolicy#setMaxMergeAtOnce removedThis parameter has no replacement, TieredMergePolicy no longer bounds thediff --git a/lucene/backward-codecs/src/test/org/apache/lucene/backward_index/BackwardsCompatibilityTestBase.java b/lucene/backward-codecs/src/test/org/apache/lucene/backward_index/BackwardsCompatibilityTestBase.javaindex edbed96be75..ff06ce63257 100644--- a/lucene/backward-codecs/src/test/org/apache/lucene/backward_index/BackwardsCompatibilityTestBase.java+++ b/lucene/backward-codecs/src/test/org/apache/lucene/backward_index/BackwardsCompatibilityTestBase.java@@ -160,7 +160,7 @@ public abstract class BackwardsCompatibilityTestBase extends LuceneTestCase {private static Version getLatestPreviousMajorVersion() {Version lastPrevMajorVersion = null;for (Version v : getAllCurrentVersions()) {- if (v.major == Version.LATEST.major - 1+ if (v.major == Version.MIN_SUPPORTED_MAJOR&& (lastPrevMajorVersion == null || v.onOrAfter(lastPrevMajorVersion))) {lastPrevMajorVersion = v;}diff --git a/lucene/backward-codecs/src/test/org/apache/lucene/backward_index/TestBasicBackwardsCompatibility.java b/lucene/backward-codecs/src/test/org/apache/lucene/backward_index/TestBasicBackwardsCompatibility.javaindex a93c5c7617a..77dc6d2412c 100644--- a/lucene/backward-codecs/src/test/org/apache/lucene/backward_index/TestBasicBackwardsCompatibility.java+++ b/lucene/backward-codecs/src/test/org/apache/lucene/backward_index/TestBasicBackwardsCompatibility.java@@ -863,7 +863,10 @@ public class TestBasicBackwardsCompatibility extends BackwardsCompatibilityTestB() -> StandardDirectoryReader.open(commit, Version.LATEST.major, null));assertTrue(ex.getMessage()- .contains("only supports reading from version " + Version.LATEST.major + " upwards."));+ .contains(+ "This Lucene version only supports indexes created with major version "+ + Version.LATEST.major+ + " or later"));// now open with allowed min versionStandardDirectoryReader.open(commit, Version.MIN_SUPPORTED_MAJOR, null).close();}diff --git a/lucene/backward-codecs/src/test/org/apache/lucene/backward_index/TestMinSupportedMajorBackwardsCompatibility.java b/lucene/backward-codecs/src/test/org/apache/lucene/backward_index/TestMinSupportedMajorBackwardsCompatibility.javanew file mode 100644index 00000000000..82246624445--- /dev/null+++ b/lucene/backward-codecs/src/test/org/apache/lucene/backward_index/TestMinSupportedMajorBackwardsCompatibility.java@@ -0,0 +1,214 @@+/*+ * Licensed to the Apache Software Foundation (ASF) under one or more+ * contributor license agreements. See the NOTICE file distributed with+ * this work for additional information regarding copyright ownership.+ * The ASF licenses this file to You under the Apache License, Version 2.0+ * (the "License"); you may not use this file except in compliance with+ * the License. You may obtain a copy of the License at+ *+ * http://www.apache.org/licenses/LICENSE-2.0+ *+ * Unless required by applicable law or agreed to in writing, software+ * distributed under the License is distributed on an "AS IS" BASIS,+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.+ * See the License for the specific language governing permissions and+ * limitations under the License.+ */+package org.apache.lucene.backward_index;++import static org.apache.lucene.util.Version.MIN_SUPPORTED_MAJOR;++import com.carrotsearch.randomizedtesting.annotations.ParametersFactory;+import java.io.IOException;+import java.util.List;+import java.util.stream.Collectors;+import org.apache.lucene.document.Document;+import org.apache.lucene.document.Field;+import org.apache.lucene.document.TextField;+import org.apache.lucene.index.DirectoryReader;+import org.apache.lucene.index.IndexFormatTooOldException;+import org.apache.lucene.index.IndexWriter;+import org.apache.lucene.index.IndexWriterConfig;+import org.apache.lucene.index.SegmentInfos;+import org.apache.lucene.index.Term;+import org.apache.lucene.search.IndexSearcher;+import org.apache.lucene.search.TermQuery;+import org.apache.lucene.search.TopDocs;+import org.apache.lucene.store.Directory;+import org.apache.lucene.tests.analysis.MockAnalyzer;+import org.apache.lucene.tests.util.TestUtil;+import org.apache.lucene.util.Version;++/**+ * Tests for the MIN_SUPPORTED_MAJOR backwards compatibility policy. Verifies that indexes created+ * with major version numbers >= MIN_SUPPORTED_MAJOR can be opened, while older indexes throw+ * IndexFormatTooOldException.+ *+ * <p>This test uses the precreated index archives to validate the policy against real historical+ * indexes. The test automatically adapts when MIN_SUPPORTED_MAJOR changes by using the existing+ * backwards compatibility testing infrastructure.+ */+public class TestMinSupportedMajorBackwardsCompatibility extends BackwardsCompatibilityTestBase {+ static final String INDEX_NAME = "index";+ static final String SUFFIX = "-nocfs";++ public TestMinSupportedMajorBackwardsCompatibility(Version version, String pattern) {+ super(version, pattern);+ }++ @Override+ protected void createIndex(Directory directory) throws IOException {+ IndexWriterConfig conf =+ new IndexWriterConfig(new MockAnalyzer(random()))+ .setUseCompoundFile(false)+ .setCodec(TestUtil.getDefaultCodec());+ try (IndexWriter writer = new IndexWriter(directory, conf)) {+ Document doc = new Document();+ doc.add(new TextField("content", "test document for version compatibility", Field.Store.YES));+ writer.addDocument(doc);+ writer.commit();+ }+ }++ /**+ * Uses the standard allVersion() method to dynamically test all available versions. The+ * supportsVersion() override controls which versions are actually tested.+ */+ @ParametersFactory(argumentFormatting = "Lucene-Version:%1$s; Pattern: %2$s")+ public static Iterable<Object[]> testVersionsFactory() {+ return allVersion(INDEX_NAME, SUFFIX);+ }++ @Override+ protected boolean supportsVersion(Version version) {+ // Test only versions that should be supported according to MIN_SUPPORTED_MAJOR+ return version.major >= MIN_SUPPORTED_MAJOR;+ }++ /**+ * Test that indexes created with versions >= MIN_SUPPORTED_MAJOR can be opened and searched. This+ * validates that the relaxed policy correctly allows older but supported indexes.+ */+ public void testSupportedVersionCanBeOpened() throws IOException {+ // The base class setup ensures we only get here for supported versions+ assertTrue("Version should be supported", version.major >= MIN_SUPPORTED_MAJOR);++ // This should succeed for any version >= MIN_SUPPORTED_MAJOR+ try (DirectoryReader reader = DirectoryReader.open(directory)) {+ // Basic validation that the index can be read+ assertTrue("Index should be readable", reader.numDocs() >= 0);++ // Verify we can read the SegmentInfos+ SegmentInfos sis = SegmentInfos.readLatestCommit(directory);+ assertTrue(+ "Creation version should be >= MIN_SUPPORTED_MAJOR",+ sis.getIndexCreatedVersionMajor() >= MIN_SUPPORTED_MAJOR);++ // Test basic search functionality if there are documents+ if (reader.numDocs() > 0) {+ IndexSearcher searcher = new IndexSearcher(reader);+ TopDocs hits = searcher.search(new TermQuery(new Term("content", "test")), 10);+ // Note: we can't guarantee the specific content, but should be able to search+ assertNotNull("Search should return results", hits);+ }+ }+ }++ /**+ * Test unsupported versions by dynamically finding versions below MIN_SUPPORTED_MAJOR. This+ * validates that both DirectoryReader and IndexWriter properly throw IndexFormatTooOldException+ * for versions below MIN_SUPPORTED_MAJOR.+ */+ public void testUnsupportedVersionsThrowException() throws IOException {+ // Find all versions that should be unsupported+ List<Version> allVersions = getAllCurrentReleasedVersionsAndCurrent();+ List<Version> unsupportedVersions =+ allVersions.stream()+ .filter(v -> v.major < MIN_SUPPORTED_MAJOR)+ .collect(Collectors.toList());++ if (unsupportedVersions.isEmpty()) {+ // If there are no unsupported versions (e.g., MIN_SUPPORTED_MAJOR is very low),+ // we can't test this behavior+ return;+ }++ // Test each unsupported version+ for (Version unsupportedVersion : unsupportedVersions) {+ String indexName =+ String.format(+ java.util.Locale.ROOT, createPattern(INDEX_NAME, SUFFIX), unsupportedVersion);+ java.io.InputStream resource =+ TestAncientIndicesCompatibility.class.getResourceAsStream(indexName);++ if (resource == null) {+ // Skip versions that don't have archives (this is expected for some older versions)+ continue;+ }++ java.nio.file.Path tempDir = createTempDir("unsupported-" + unsupportedVersion);+ TestUtil.unzip(resource, tempDir);++ try (org.apache.lucene.store.Directory dir = newFSDirectory(tempDir)) {+ // Test DirectoryReader fails for unsupported versions+ IndexFormatTooOldException readerException =+ expectThrows(+ IndexFormatTooOldException.class,+ () -> {+ DirectoryReader.open(dir);+ });++ // Verify the exception message contains useful information+ String readerMessage = readerException.getMessage();+ assertNotNull(+ "Reader exception message should not be null for version " + unsupportedVersion,+ readerMessage);+ assertTrue(+ "Reader exception message should contain version information for "+ + unsupportedVersion+ + ": "+ + readerMessage,+ readerMessage.length() > 0);++ // Test IndexWriter creation also fails appropriately+ IndexWriterConfig config = new IndexWriterConfig();+ IndexFormatTooOldException writerException =+ expectThrows(+ IndexFormatTooOldException.class,+ () -> {+ new IndexWriter(dir, config);+ });++ String writerMessage = writerException.getMessage();+ assertNotNull(+ "Writer exception message should not be null for version " + unsupportedVersion,+ writerMessage);+ assertTrue(+ "Writer exception message should contain version information for "+ + unsupportedVersion+ + ": "+ + writerMessage,+ writerMessage.length() > 0);+ }+ }+ }++ /**+ * Test that validates the constant MIN_SUPPORTED_MAJOR has the expected value. This test will+ * fail if someone accidentally changes the constant without considering the implications.+ */+ public void testMinSupportedMajorConstantValue() {+ // This test validates that MIN_SUPPORTED_MAJOR is set to the expected value+ // If this test fails, it likely means the constant was changed and the+ // implications need to be considered (tests updated, documentation updated, etc.)+ assertEquals(+ "MIN_SUPPORTED_MAJOR should be 10 for the relaxed upgrade policy", 10, MIN_SUPPORTED_MAJOR);++ // Additional validation: MIN_SUPPORTED_MAJOR should be <= current major+ assertTrue(+ "MIN_SUPPORTED_MAJOR should not exceed current major version",+ MIN_SUPPORTED_MAJOR <= Version.LATEST.major);++ assertTrue("MIN_SUPPORTED_MAJOR should be positive", MIN_SUPPORTED_MAJOR > 0);+ }+}diff --git a/lucene/core/src/java/org/apache/lucene/index/SegmentInfos.java b/lucene/core/src/java/org/apache/lucene/index/SegmentInfos.javaindex 9d87edc6e60..204bd89b350 100644--- a/lucene/core/src/java/org/apache/lucene/index/SegmentInfos.java+++ b/lucene/core/src/java/org/apache/lucene/index/SegmentInfos.java@@ -349,14 +349,19 @@ public final class SegmentInfos implements Cloneable, Iterable<SegmentCommitInfoif (indexCreatedVersion < minSupportedMajorVersion) {throw new IndexFormatTooOldException(input,- "This index was initially created with Lucene "+ "Index created with Lucene "+ indexCreatedVersion- + ".x while the current version is "+ + ".x is not supported by Lucene "+ Version.LATEST- + " and Lucene only supports reading"- + (minSupportedMajorVersion == Version.MIN_SUPPORTED_MAJOR- ? " the current and previous major versions"- : " from version " + minSupportedMajorVersion + " upwards"));+ + ". This Lucene version only supports indexes created with major version "+ + minSupportedMajorVersion+ + " or later (found: "+ + indexCreatedVersion+ + ", minimum: "+ + minSupportedMajorVersion+ + "). To resolve this issue: (1) Re-index your data using Lucene "+ + Version.LATEST.major+ + ".x, or (2) Use an older Lucene version that supports your index format.");}SegmentInfos infos = new SegmentInfos(indexCreatedVersion);diff --git a/lucene/core/src/java/org/apache/lucene/util/Version.java b/lucene/core/src/java/org/apache/lucene/util/Version.javaindex c572911f970..c500ee49218 100644--- a/lucene/core/src/java/org/apache/lucene/util/Version.java+++ b/lucene/core/src/java/org/apache/lucene/util/Version.java@@ -108,10 +108,57 @@ public final class Version {@Deprecated public static final Version LUCENE_CURRENT = LATEST;/**- * Constant for the minimal supported major version of an index. This version is defined by the- * version that initially created the index.+ * Constant for the minimal supported major version number of an index. This version is defined by+ * the major version number that initially created the index.+ *+ * <p>This constant is manually controlled and should only be bumped when format changes make it+ * impossible to safely read older indexes. Examples include:+ *+ * <ul>+ * <li>Lossy encoding changes (e.g., norms format changes that cannot be recovered)+ * <li>Index-level format changes that prevent reading (e.g., segments_N file format)+ * <li>Critical corruption bugs that make older indexes potentially invalid+ * </ul>+ *+ * <p>This constant should NOT be bumped automatically with major version number releases. The+ * goal is to allow users to upgrade across multiple major version numbers when safe to do so.+ *+ * <p><b>Two-tier version policy:</b>+ *+ * <ul>+ * <li><b>Index opening policy:</b> An index can be opened if its {@code+ * indexCreatedVersionMajor} is >= this constant, regardless of how many major version+ * numbers have been released since.+ * <li><b>Codec reader policy:</b> Segment codecs are only shipped for the current major version+ * number and the immediately previous major version number. When no format breaks occur+ * between consecutive major version numbers, the previous major version number reader can+ * read segments from older major version numbers that use the same format.+ * </ul>+ *+ * <p><b>When to bump this constant:</b>+ *+ * <ul>+ * <li>When introducing an incompatible on-disk format change+ * <li>When fixing a critical bug that makes older indexes potentially corrupt+ * <li>When the maintenance burden of supporting older creation versions becomes too high+ * </ul>+ *+ * <p><b>How to bump this constant:</b>+ *+ * <ol>+ * <li>Set the value to the last major version number before the breaking change+ * <li>Update tests to reflect the new minimum+ * <li>Update CHANGES.txt and MIGRATE.md with upgrade instructions+ * <li>Ensure IndexFormatTooOldException messages reference the new minimum+ * </ol>+ *+ * <p><b>Example:</b> If a breaking change is introduced in version 15.0.0, set this constant to+ * 14, which will prevent indexes created with major version numbers 13 and earlier from being+ * opened.+ *+ * @since 11.0.0*/- public static final int MIN_SUPPORTED_MAJOR = Version.LATEST.major - 1;+ public static final int MIN_SUPPORTED_MAJOR = 10;/*** @see #getPackageImplementationVersion()