feat: Upgrade ES indexes to version 8 - EXO-88232 - #1756
Merged
Conversation
This change aims to force the recreation of indices which was created previously using ES 7 to be reindexed using ES 8 in order to prepare the transition to ES version 9 which isn't compatible only with indices V8.
|
boubaker
added a commit
to Meeds-io/commons
that referenced
this pull request
Aug 26, 2026
…n alias - EXO-88232 (#769) ## Problem While upgrading `notes_v1 → notes_v2` (Meeds-io/notes#1756), the upgrade failed with: ``` ERROR | An error occurred while upgrading index notes_v1 java.lang.NullPointerException: Cannot invoke "java.util.Set.size()" because the return value of "java.util.Map.get(Object)" is null at ...ElasticIndexingOperationProcessor$ReindexESType.run(ElasticIndexingOperationProcessor.java:792) ``` Notes declares **two** connectors on the same `index_alias` (`WikiPageIndexingServiceConnector` and `NoteVersionLanguageIndexingServiceConnector`). One `ReindexESType` task is submitted per connector. The first task to finish saw `size() > 1`, did nothing, then its `finally` removed the **whole alias entry** from `indexUpgrading` → the second task NPE'd. Net effect: the alias was never switched to the new index and the old index was never deleted, on every alias shared by ≥ 2 connectors. ## Fix - Track completion **per connector** (`markConnectorUpgraded`); switch the alias and delete the old index once, after the last connector of that alias finishes. - On failure, release only the finished connector's slot; drop the alias entry when nothing is pending anymore. - Send the ES-side `_reindex` **once per alias/pipeline** instead of once per connector (it was duplicated for notes). - Logs name the connector; the pipeline-failure WARN now carries the exception. - `setExecutors(ExecutorService)` / `isIndexUpgrading(alias)` added for tests. ## Tests Two new tests in `ElasticOperationProcessorTest` with two connectors on `notes_alias`, run on a same-thread executor: - alias switch / old-index delete / `_reindex` each happen exactly once, after both connectors; - after only the first connector finishes, the alias is untouched and the upgrade is still pending. **Mutation-verified**: with the previous `size() > 1` + whole-alias `remove` logic put back, both tests fail; with the fix, the full `commons-search` suite passes. ## Classification N2 — shared indexing engine (cross-domain seam), no ACL/schema/REST surface touched. Knowledge: none — behaviour fix in the indexing engine, no domain-doc claim changes. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
boubaker
added a commit
to Meeds-io/commons
that referenced
this pull request
Aug 26, 2026
…n alias - EXO-88232 (#769) ## Problem While upgrading `notes_v1 → notes_v2` (Meeds-io/notes#1756), the upgrade failed with: ``` ERROR | An error occurred while upgrading index notes_v1 java.lang.NullPointerException: Cannot invoke "java.util.Set.size()" because the return value of "java.util.Map.get(Object)" is null at ...ElasticIndexingOperationProcessor$ReindexESType.run(ElasticIndexingOperationProcessor.java:792) ``` Notes declares **two** connectors on the same `index_alias` (`WikiPageIndexingServiceConnector` and `NoteVersionLanguageIndexingServiceConnector`). One `ReindexESType` task is submitted per connector. The first task to finish saw `size() > 1`, did nothing, then its `finally` removed the **whole alias entry** from `indexUpgrading` → the second task NPE'd. Net effect: the alias was never switched to the new index and the old index was never deleted, on every alias shared by ≥ 2 connectors. ## Fix - Track completion **per connector** (`markConnectorUpgraded`); switch the alias and delete the old index once, after the last connector of that alias finishes. - On failure, release only the finished connector's slot; drop the alias entry when nothing is pending anymore. - Send the ES-side `_reindex` **once per alias/pipeline** instead of once per connector (it was duplicated for notes). - Logs name the connector; the pipeline-failure WARN now carries the exception. - `setExecutors(ExecutorService)` / `isIndexUpgrading(alias)` added for tests. ## Tests Two new tests in `ElasticOperationProcessorTest` with two connectors on `notes_alias`, run on a same-thread executor: - alias switch / old-index delete / `_reindex` each happen exactly once, after both connectors; - after only the first connector finishes, the alias is untouched and the upgrade is still pending. **Mutation-verified**: with the previous `size() > 1` + whole-alias `remove` logic put back, both tests fail; with the fix, the full `commons-search` suite passes. ## Classification N2 — shared indexing engine (cross-domain seam), no ACL/schema/REST surface touched. Knowledge: none — behaviour fix in the indexing engine, no domain-doc claim changes. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
exo-swf
pushed a commit
that referenced
this pull request
Aug 26, 2026
boubaker
added a commit
to Meeds-io/commons
that referenced
this pull request
Aug 27, 2026
…n alias - EXO-88232 (#771) ## Problem While upgrading `notes_v1 → notes_v2` (Meeds-io/notes#1756), the upgrade failed with: ``` ERROR | An error occurred while upgrading index notes_v1 java.lang.NullPointerException: Cannot invoke "java.util.Set.size()" because the return value of "java.util.Map.get(Object)" is null at ...ElasticIndexingOperationProcessor$ReindexESType.run(ElasticIndexingOperationProcessor.java:792) ``` Notes declares **two** connectors on the same `index_alias` (`WikiPageIndexingServiceConnector` and `NoteVersionLanguageIndexingServiceConnector`). One `ReindexESType` task is submitted per connector. The first task to finish saw `size() > 1`, did nothing, then its `finally` removed the **whole alias entry** from `indexUpgrading` → the second task NPE'd. Net effect: the alias was never switched to the new index and the old index was never deleted, on every alias shared by ≥ 2 connectors. ## Fix - Track completion **per connector** (`markConnectorUpgraded`); switch the alias and delete the old index once, after the last connector of that alias finishes. - On failure, release only the finished connector's slot; drop the alias entry when nothing is pending anymore. - Send the ES-side `_reindex` **once per alias/pipeline** instead of once per connector (it was duplicated for notes). - Logs name the connector; the pipeline-failure WARN now carries the exception. - `setExecutors(ExecutorService)` / `isIndexUpgrading(alias)` added for tests. ## Tests Two new tests in `ElasticOperationProcessorTest` with two connectors on `notes_alias`, run on a same-thread executor: - alias switch / old-index delete / `_reindex` each happen exactly once, after both connectors; - after only the first connector finishes, the alias is untouched and the upgrade is still pending. **Mutation-verified**: with the previous `size() > 1` + whole-alias `remove` logic put back, both tests fail; with the fix, the full `commons-search` suite passes. ## Classification N2 — shared indexing engine (cross-domain seam), no ACL/schema/REST surface touched. Knowledge: none — behaviour fix in the indexing engine, no domain-doc claim changes. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
boubaker
added a commit
to Meeds-io/commons
that referenced
this pull request
Aug 27, 2026
…n alias - EXO-88232 (#770) ## Problem While upgrading `notes_v1 → notes_v2` (Meeds-io/notes#1756), the upgrade failed with: ``` ERROR | An error occurred while upgrading index notes_v1 java.lang.NullPointerException: Cannot invoke "java.util.Set.size()" because the return value of "java.util.Map.get(Object)" is null at ...ElasticIndexingOperationProcessor$ReindexESType.run(ElasticIndexingOperationProcessor.java:792) ``` Notes declares **two** connectors on the same `index_alias` (`WikiPageIndexingServiceConnector` and `NoteVersionLanguageIndexingServiceConnector`). One `ReindexESType` task is submitted per connector. The first task to finish saw `size() > 1`, did nothing, then its `finally` removed the **whole alias entry** from `indexUpgrading` → the second task NPE'd. Net effect: the alias was never switched to the new index and the old index was never deleted, on every alias shared by ≥ 2 connectors. ## Fix - Track completion **per connector** (`markConnectorUpgraded`); switch the alias and delete the old index once, after the last connector of that alias finishes. - On failure, release only the finished connector's slot; drop the alias entry when nothing is pending anymore. - Send the ES-side `_reindex` **once per alias/pipeline** instead of once per connector (it was duplicated for notes). - Logs name the connector; the pipeline-failure WARN now carries the exception. - `setExecutors(ExecutorService)` / `isIndexUpgrading(alias)` added for tests. ## Tests Two new tests in `ElasticOperationProcessorTest` with two connectors on `notes_alias`, run on a same-thread executor: - alias switch / old-index delete / `_reindex` each happen exactly once, after both connectors; - after only the first connector finishes, the alias is untouched and the upgrade is still pending. **Mutation-verified**: with the previous `size() > 1` + whole-alias `remove` logic put back, both tests fail; with the fix, the full `commons-search` suite passes. ## Classification N2 — shared indexing engine (cross-domain seam), no ACL/schema/REST surface touched. Knowledge: none — behaviour fix in the indexing engine, no domain-doc claim changes. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This change aims to force the recreation of indices which was created previously using ES 7 to be reindexed using ES 8 in order to prepare the transition to ES version 9 which isn't compatible only with indices V8.