feat: Graph rag mvp by aotenjou · Pull Request #876 · apache/geaflow

aotenjou · 2026-10-08T16:13:44Z

Summary

This PR implements the GraphRAG hybrid retrieval MVP. It provides a unified retrieval API and combines BM25, vector, and graph recall channels through weighted Reciprocal Rank Fusion (RRF).

The implementation covers the complete MVP path from artifact construction and request validation to channel retrieval, result fusion, and service response.

#860 #863 #864

MVP Scope

Unified retrieval request and response API.
Request validation, supported modes, and structured error handling.
Lucene-based BM25 artifact construction and keyword retrieval.
Offline vector artifact construction and similarity retrieval.
Graph artifact/provider integration with entity anchor resolution.
Weighted RRF fusion across BM25, vector, and graph candidates.
topK, candidate budget, timeout, and maximum candidate controls.
Artifact identity, version, and readiness validation.
Basic ingestion state machine and artifact registration.
Local Memory Server integration.
End-to-end hybrid retrieval test coverage.

API and Domain Model

Retrieval request/response serialization.
Retrieval modes and channel configuration.
Candidate, score, source-channel, and metadata models.
Validation errors and structured failure responses.

Artifact and Index Layer

BM25/Lucene index builder.
Offline vector index builder.
Graph artifact identity and version handling.
Artifact readiness and missing-artifact validation.

Recall Channel Layer

BM25/keyword retriever.
Vector retriever.
Graph retriever.
Entity anchor resolver.
Normalized channel candidate output.

Fusion Layer

Weighted Reciprocal Rank Fusion.
Candidate merging and deduplication.
Channel weighting.
Basic handling of empty or unavailable channels.

Service and Orchestration Layer

Retrieval service orchestration.
Channel execution and result fusion.
Timeout and candidate-budget enforcement.
Handling for missing graphs, unavailable indexes, and unsupported options.
Basic retrieval status and latency logging.

Ingestion and Metadata Layer

Import state machine.
Artifact registration and identity matching.
In-memory metadata and graph provider implementations.
MVP graph/index version consistency checks.

MVP Test Coverage

The MVP test suite contains 114 tests and is divided into component tests, service tests, end-to-end retrieval tests, and Memory Server integration tests.

Test Breakdown

Area	Test Classes	Tests	Coverage
Graph and index artifacts	`GraphArtifactTest`, `ArtifactIdentityTest`	6	Artifact identity, version matching, graph metadata, and invalid artifact handling
BM25 indexing	`LuceneBm25IndexBuilderTest`	4	Temporary Lucene index creation, document indexing, keyword lookup, and index validation
Vector indexing	`OfflineVectorIndexBuilderTest`	5	Offline vector artifact creation, vector dimensions, similarity lookup, and invalid input handling
Retrieval API	`RetrievalApiJsonTest`, `RetrievalPropertiesTest`, `RetrievalRequestValidatorTest`	29	JSON serialization, default properties, supported modes, invalid options, and request validation
Graph recall	`GraphRetrieverTest`, `EntityAnchorResolverTest`	5	Entity anchor resolution, graph candidate retrieval, empty results, and graph traversal boundaries
Result fusion	`WeightedRrfFusionTest`	3	Weighted RRF ranking, duplicate candidate merging, and empty-channel behavior
Domain and metadata	`RetrievalDomainModelTest`, `MetadataModelTest`, `InMemoryMetadataStoreTest`	22	Domain object construction, metadata validation, artifact registration, and lookup behavior
Ingestion	`ImportStateMachineTest`, `IngestionPipelineTest`	5	Import state transitions, invalid transitions, artifact registration, and ingestion failure handling
Retrieval service	`RetrievalServiceTest`	5	Service orchestration, successful retrieval, missing graph, unavailable index, and unsupported options
End-to-end MVP flow	`GraphRagMvpEndToEndTest`	28	Complete BM25/vector/graph retrieval flow, fusion, ranking, version checks, candidate limits, and failure responses
Memory Server	`MemoryServerTest`	2	Server startup, graph initialization, keyword retrieval, and HTTP service integration

End-to-End Fixture

GraphRagMvpEndToEndTest builds a deterministic local fixture without external services:

BM25 and vector artifacts are created in temporary directories.
Graph records are registered through the in-memory RetrievalFixtureRegistry.
All artifacts use the same mvp-graph/v1 graph/index version.
The fixture contains three chunks:

Chunk	Document	Text	Fixed Vector
`c1`	`doc-1`	`Confucius taught ethics and philosophy`	`[1.0, 0.0]`
`c2`	`doc-2`	`Astronomy studies distant stars`	`[0.0, 1.0]`
`c3`	`doc-3`	`Confucius and astronomy are studied`	`[0.8, 0.2]`

The fixed documents and vectors make ranking assertions deterministic. The graph fixture provides the entity records required to exercise anchor resolution and graph recall.

Assertions Covered

The tests verify:

Correct routing for keyword, vector, graph, and hybrid retrieval modes.
Stable top-k ordering and source-channel metadata.
Weighted RRF merging and duplicate elimination.
Artifact and graph/index version consistency.
Candidate budget and maximum candidate enforcement.
Timeout and unsupported-option validation.
Correct error codes for missing graphs and unavailable indexes.
Empty-result behavior for valid but unmatched queries.
Successful service startup and graph initialization.
End-to-end propagation of retrieval results and status metadata.

Gaps Compared with the Full Design

The following production capabilities are outside the scope of this MVP:

Production graph construction, persistence, and incremental synchronization.
Online embedding generation, model serving, and embedding version governance.
Full multi-hop graph reasoning, path constraints, and graph-aware reranking.
Adaptive fusion based on query intent, feedback, or online quality signals.
Persistent metadata catalog, tenant isolation, authorization, and full artifact lifecycle management.
Distributed serving, high availability, rate limiting, and production observability.
Large-scale evaluation covering Recall@K, latency, throughput, and load-testing baselines.

aotenjou · 2026-10-09T02:58:45Z

@yaozhongq

Copilot

Warning

Copilot couldn't run its full agentic review because it didn't start before the timeout. Make sure your repository has a runner available, or add a copilot-code-review.yml file specifying one with the runs-on attribute. See the docs for more details.

Copilot review overview

0 open findings

What changed in this PR

Implements the GraphRAG hybrid retrieval MVP by introducing a unified retrieval request/response API, offline ingestion + artifact/version validation, and hybrid recall across BM25/vector/graph with weighted RRF fusion (plus Memory Server readiness/keyword endpoint updates).

Changes:

Added offline ingestion, artifact identity/versioning, and metadata lifecycle models with JSON boundaries.
Implemented recall channels (BM25/vector/graph), weighted RRF fusion, and trace validation for hybrid responses.
Expanded and stabilized test coverage (Python ingest tooling + Java retrieval/ingest/index/service tests) and updated server configuration/readiness behavior.

File	Description
tools/graphrag/tests/test_source_loader.py	Adds unit tests for deterministic source loading, checksum validation, and size limits.
tools/graphrag/tests/test_pipeline.py	Adds tests for restartable ingestion pipeline behavior and failure handling.
tools/graphrag/tests/test_normalize.py	Adds tests for text normalization and deterministic chunking offsets/overlap.
tools/graphrag/tests/test_entity_graph.py	Adds tests for deterministic entity/edge IDs and provenance capture.
tools/graphrag/manifests/hotpotqa-distractor.json	Adds dataset manifest scaffold for offline ingestion.
tools/graphrag/manifests/2wikimultihopqa-dev.json	Adds dataset manifest scaffold for offline ingestion.
tools/graphrag/ingest/sources/__init__.py	Exposes source loader public API.
tools/graphrag/ingest/pipeline.py	Adds restartable offline ingestion orchestration that publishes READY metadata atomically.
tools/graphrag/ingest/normalize.py	Adds canonical normalization + deterministic chunking implementation.
tools/graphrag/ingest/__init__.py	Declares ingestion package.
tools/graphrag/__init__.py	Declares GraphRAG tooling package.
tools/__init__.py	Declares tooling top-level package.
geaflow-ai/src/test/resources/retrieval/api/response-empty.json	Updates retrieval response fixture to include requestId in trace.
geaflow-ai/src/test/java/org/apache/geaflow/ai/retrieval/support/RetrievalTestFixture.java	Adds deterministic graph fixture used by retrieval tests.
geaflow-ai/src/test/java/org/apache/geaflow/ai/retrieval/support/HttpTestResponse.java	Adds detached HTTP test response helper.
geaflow-ai/src/test/java/org/apache/geaflow/ai/retrieval/support/HttpTestClient.java	Adds HTTP test client that safely closes OkHttp responses.
geaflow-ai/src/test/java/org/apache/geaflow/ai/retrieval/service/RetrievalServiceTest.java	Adds keyword retrieval service tests including readiness/error cases.
geaflow-ai/src/test/java/org/apache/geaflow/ai/retrieval/metadata/MetadataModelTest.java	Adds contract tests for metadata models + JSON boundary behavior.
geaflow-ai/src/test/java/org/apache/geaflow/ai/retrieval/metadata/InMemoryMetadataStoreTest.java	Adds lifecycle/readiness/CAS/retry tests for metadata store behavior.
geaflow-ai/src/test/java/org/apache/geaflow/ai/retrieval/ingest/ImportStateMachineTest.java	Adds state machine transition tests for ingestion attempts.
geaflow-ai/src/test/java/org/apache/geaflow/ai/retrieval/index/vector/OfflineVectorIndexBuilderTest.java	Adds tests for vector artifact validation, cleanup, and path traversal prevention.
geaflow-ai/src/test/java/org/apache/geaflow/ai/retrieval/index/bm25/LuceneBm25IndexBuilderTest.java	Adds tests for BM25 artifact determinism, validation, cleanup, and path traversal prevention.
geaflow-ai/src/test/java/org/apache/geaflow/ai/retrieval/index/ArtifactIdentityTest.java	Adds tests for safe version/path resolution.
geaflow-ai/src/test/java/org/apache/geaflow/ai/retrieval/fusion/WeightedRrfFusionTest.java	Adds tests for weighted RRF scoring, ties, and edge conditions.
geaflow-ai/src/test/java/org/apache/geaflow/ai/retrieval/channel/graph/GraphRetrieverTest.java	Adds tests for graph recall bounding (edge cap, deadlines, candidate budget).
geaflow-ai/src/test/java/org/apache/geaflow/ai/retrieval/channel/graph/EntityAnchorResolverTest.java	Adds tests for deterministic anchor resolution and deadline behavior.
geaflow-ai/src/test/java/org/apache/geaflow/ai/retrieval/api/RetrievalRequestValidatorTest.java	Updates/extends request validation tests for hybrid/vector + requestId validation.
geaflow-ai/src/test/java/org/apache/geaflow/ai/operator/SearchStoreTest.java	Adds regression test ensuring SearchStore honors requested topN.
geaflow-ai/src/test/java/org/apache/geaflow/ai/GraphMemoryKeywordSearchTest.java	Adds tests for bounded keyword search maxCandidates + timeout mapping.
geaflow-ai/src/test/java/org/apache/geaflow/ai/GeaFlowMemoryServerTest.java	Adds tests for requestId normalization/validation behavior.
geaflow-ai/src/main/resources/application.yml	Makes server port configurable via GEAFLOW_SERVER_PORT env var.
geaflow-ai/src/main/java/org/apache/geaflow/ai/service/ServerMemoryCache.java	Makes cache concurrent + adds graph versioning/readiness reporting for keyword retrieval.
geaflow-ai/src/main/java/org/apache/geaflow/ai/retrieval/service/RetrievalService.java	Adds request validation + bounded keyword retrieval orchestration and trace construction.
geaflow-ai/src/main/java/org/apache/geaflow/ai/retrieval/service/RetrievalResponseValidator.java	Strengthens success response validation, especially for hybrid trace fields.
geaflow-ai/src/main/java/org/apache/geaflow/ai/retrieval/service/RetrievalRequestValidator.java	Adds mode parsing, vector validation, request metadata validation, and hybrid constraints.
geaflow-ai/src/main/java/org/apache/geaflow/ai/retrieval/service/RetrievalMetrics.java	Adds process-local retrieval counters and snapshot API.
geaflow-ai/src/main/java/org/apache/geaflow/ai/retrieval/service/RetrievalBudgetValidator.java	Adjusts budget validation behavior (removes “all values required” gate).
geaflow-ai/src/main/java/org/apache/geaflow/ai/retrieval/service/EvidenceMapper.java	Maps keyword recall hits to stable Evidence records with per-stage scores.
geaflow-ai/src/main/java/org/apache/geaflow/ai/retrieval/model/evidence/GraphEvidenceTrace.java	Adds per-candidate graph provenance model.
geaflow-ai/src/main/java/org/apache/geaflow/ai/retrieval/model/evidence/Evidence.java	Extends Evidence with graphTraces + finalScore getters/serialization fields.
geaflow-ai/src/main/java/org/apache/geaflow/ai/retrieval/metadata/QualityCounters.java	Adds ingestion quality counters model with validation + aliases.
geaflow-ai/src/main/java/org/apache/geaflow/ai/retrieval/metadata/MetadataValidation.java	Adds shared validation helpers for metadata models.
geaflow-ai/src/main/java/org/apache/geaflow/ai/retrieval/metadata/MetadataStore.java	Introduces storage-neutral metadata lifecycle interface.
geaflow-ai/src/main/java/org/apache/geaflow/ai/retrieval/metadata/MetadataJson.java	Adds constructor-validated metadata JSON (unknown fields ignored).
geaflow-ai/src/main/java/org/apache/geaflow/ai/retrieval/metadata/MetadataException.java	Adds typed metadata exception codes for lifecycle failures.
geaflow-ai/src/main/java/org/apache/geaflow/ai/retrieval/metadata/IndexBuildMetadata.java	Adds index build metadata bound to graph identity/version.
geaflow-ai/src/main/java/org/apache/geaflow/ai/retrieval/metadata/ImportState.java	Adds import attempt state enum.
geaflow-ai/src/main/java/org/apache/geaflow/ai/retrieval/metadata/ImportMetadata.java	Adds immutable attempt snapshot incl. readiness reasons and failure typing.
geaflow-ai/src/main/java/org/apache/geaflow/ai/retrieval/metadata/GraphBuildMetadata.java	Adds graph build metadata incl. required index names for publication.
geaflow-ai/src/main/java/org/apache/geaflow/ai/retrieval/metadata/DatasetManifest.java	Adds immutable dataset manifest model with reproducibility fields.
geaflow-ai/src/main/java/org/apache/geaflow/ai/retrieval/metadata/ChunkingConfiguration.java	Adds chunking policy model with validation.
geaflow-ai/src/main/java/org/apache/geaflow/ai/retrieval/ingest/package-info.java	Documents ingestion extension points and publication rules.
geaflow-ai/src/main/java/org/apache/geaflow/ai/retrieval/ingest/SourceStreamProvider.java	Adds source stream provider interface for checksum verification.
geaflow-ai/src/main/java/org/apache/geaflow/ai/retrieval/ingest/SourceLoader.java	Adds canonical source loader interface.
geaflow-ai/src/main/java/org/apache/geaflow/ai/retrieval/ingest/MetadataStorePublisher.java	Implements metadata publisher on top of MetadataStore with CAS publication.
geaflow-ai/src/main/java/org/apache/geaflow/ai/retrieval/ingest/MetadataPublisher.java	Adds metadata publication interface.
geaflow-ai/src/main/java/org/apache/geaflow/ai/retrieval/ingest/IngestionPipeline.java	Adds deterministic orchestration for source->chunks->graph/index->publish with cleanup.
geaflow-ai/src/main/java/org/apache/geaflow/ai/retrieval/ingest/IngestionContext.java	Adds ingestion context carrying manifest/version + collected counters.
geaflow-ai/src/main/java/org/apache/geaflow/ai/retrieval/ingest/ImportStateMachine.java	Adds monotonic import attempt transition validation.
geaflow-ai/src/main/java/org/apache/geaflow/ai/retrieval/ingest/GraphWriter.java	Adds graph writer interface.
geaflow-ai/src/main/java/org/apache/geaflow/ai/retrieval/ingest/GraphArtifact.java	Adds closeable graph artifact interface.
geaflow-ai/src/main/java/org/apache/geaflow/ai/retrieval/ingest/ExtractionResult.java	Adds container for extracted entities/edges with immutability checks.
geaflow-ai/src/main/java/org/apache/geaflow/ai/retrieval/ingest/EntityExtractor.java	Adds entity extraction interface.
geaflow-ai/src/main/java/org/apache/geaflow/ai/retrieval/ingest/DocumentNormalizer.java	Adds document normalizer interface.
geaflow-ai/src/main/java/org/apache/geaflow/ai/retrieval/index/vector/VectorHit.java	Adds deterministic vector hit model.
geaflow-ai/src/main/java/org/apache/geaflow/ai/retrieval/index/package-info.java	Documents index builder contracts and publication constraints.
geaflow-ai/src/main/java/org/apache/geaflow/ai/retrieval/index/graph/OfflineGraphArtifactBuilder.java	Builds deterministic graph artifact with identity validation and atomic publish.
geaflow-ai/src/main/java/org/apache/geaflow/ai/retrieval/index/bm25/Bm25Hit.java	Adds deterministic BM25 hit model.
geaflow-ai/src/main/java/org/apache/geaflow/ai/retrieval/index/VectorIndexBuilder.java	Adds vector index builder contract.
geaflow-ai/src/main/java/org/apache/geaflow/ai/retrieval/index/IndexArtifact.java	Adds closeable index artifact contract.
geaflow-ai/src/main/java/org/apache/geaflow/ai/retrieval/index/ChannelSearchResult.java	Adds channel result container with counters and stop reason.
geaflow-ai/src/main/java/org/apache/geaflow/ai/retrieval/index/Bm25IndexBuilder.java	Adds BM25 builder contract.
geaflow-ai/src/main/java/org/apache/geaflow/ai/retrieval/index/ArtifactIdentity.java	Adds artifact identity + safe versioned path resolution and chunk fingerprinting.
geaflow-ai/src/main/java/org/apache/geaflow/ai/retrieval/fusion/WeightedRrfFusion.java	Implements deterministic weighted RRF sorting and fused scoring assignment.
geaflow-ai/src/main/java/org/apache/geaflow/ai/retrieval/fusion/WeightedRrfConfig.java	Adds validated fusion configuration (weights + rank constant).
geaflow-ai/src/main/java/org/apache/geaflow/ai/retrieval/execution/RecallStopReason.java	Adds typed stop reasons used in trace/status reporting.
geaflow-ai/src/main/java/org/apache/geaflow/ai/retrieval/execution/RecallStageStatus.java	Adds typed per-channel stage statuses.
geaflow-ai/src/main/java/org/apache/geaflow/ai/retrieval/execution/RecallPlan.java	Adds deterministic per-channel candidate budget allocation for hybrid and single-channel modes.
geaflow-ai/src/main/java/org/apache/geaflow/ai/retrieval/config/RetrievalProperties.java	Hardens retrieval configuration validation with clearer erroring for unsupported defaults.
geaflow-ai/src/main/java/org/apache/geaflow/ai/retrieval/codec/RetrievalModelJson.java	Extends Evidence decoding to include graphTraces + finalScore.
geaflow-ai/src/main/java/org/apache/geaflow/ai/retrieval/channel/vector/VectorRetriever.java	Adds vector recall channel wrapper.
geaflow-ai/src/main/java/org/apache/geaflow/ai/retrieval/channel/graph/InMemoryGraphNeighborProvider.java	Adds deterministic in-memory neighbor provider for fixtures/artifacts.
geaflow-ai/src/main/java/org/apache/geaflow/ai/retrieval/channel/graph/GraphNeighborProvider.java	Adds graph neighbor provider interface for graph recall boundary.
geaflow-ai/src/main/java/org/apache/geaflow/ai/retrieval/channel/graph/EntityAnchorResolver.java	Adds deterministic canonical/alias anchor resolution for graph recall.
geaflow-ai/src/main/java/org/apache/geaflow/ai/retrieval/channel/bm25/Bm25Retriever.java	Adds BM25 recall channel wrapper.
geaflow-ai/src/main/java/org/apache/geaflow/ai/retrieval/api/package-info.java	Updates v1 API documentation to cover offline/hybrid core facade semantics.
geaflow-ai/src/main/java/org/apache/geaflow/ai/retrieval/api/model/RetrievalRequest.java	Extends request with requestId, version constraints, vector metadata, and partial results toggle.
geaflow-ai/src/main/java/org/apache/geaflow/ai/retrieval/api/model/RetrievalMode.java	Adds modes (BM25/VECTOR/GRAPH/HYBRID) and KEYWORD alias canonicalization.
geaflow-ai/src/main/java/org/apache/geaflow/ai/retrieval/api/model/RetrievalCommand.java	Extends validated command to carry queryVector.
geaflow-ai/src/main/java/org/apache/geaflow/ai/operator/SessionOperator.java	Fixes SearchStore lifecycle by finishing writes before searching and closing safely.
geaflow-ai/src/main/java/org/apache/geaflow/ai/operator/SearchStore.java	Adds searchDoc(topN) overload + ensures resources are closed and writer finalized.
geaflow-ai/src/main/java/org/apache/geaflow/ai/operator/GraphSearchStore.java	Adds scored result support and bounded search (topK/maxCandidates/deadline).
geaflow-ai/src/main/java/org/apache/geaflow/ai/index/EntityAttributeIndexStore.java	Adds readiness check + throws if used before initialization.
geaflow-ai/src/main/java/org/apache/geaflow/ai/GraphMemoryServer.java	Adds bounded session-free keyword search used by the HTTP retrieval service.
geaflow-ai/README-week1-retrieval.md	Documents HTTP keyword retrieval usage and readiness/metrics endpoints.
geaflow-ai/README-hybrid-recall.md	Documents session-free core hybrid recall facade and artifact/trace semantics.

🧠 Review effort: Lite

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

yaozhongq · 2026-10-09T12:35:45Z

+            closeAll(graphArtifacts, failure);
+            graphArtifacts.clear();
+            try {
+                metadataPublisher.fail(context, failure);


This catch block also handles failures from metadataPublisher.begin(). If another invocation is already importing the same GraphVersion through the same publisher, begin() rejects the duplicate, but fail() then removes the existing attempt and marks the original import FAILED. A rejected duplicate therefore prevents the active import from completing. Please only call fail() after this invocation successfully begins an attempt, or use an attempt-specific ownership token. A regression test should verify that rejecting a duplicate leaves the original attempt able to publish.
2.

yaozhongq · 2026-10-09T12:36:03Z

+                    published.toString(), true), published);
+            }
+            try {
+                Files.move(staging, published, java.nio.file.StandardCopyOption.ATOMIC_MOVE);


The existence check and atomic move are separate operations. Two builders targeting the same version can both observe that the destination is absent, after which ATOMIC_MOVE can replace the first builder's published file on Unix filesystems. This bypasses validateArtifact() and allows different vector contents to be published under the same version. Please serialize publication for each target or use a publication mechanism that cannot overwrite an existing artifact, then validate the winner's contents when another builder loses the race. OfflineGraphArtifactBuilder uses the same pattern.
3.

yaozhongq · 2026-10-09T12:36:19Z

+        ctx.headerSet("X-Request-Id", requestId);
+        ctx.contentType("application/json; charset=utf-8");
+        try {
+            String body = RetrievalApiJson.toJson(


The service validates requests against the injected RetrievalProperties, but this toJson() overload creates default properties and validates the response against those defaults. For example, with retrieval.max-top-k=200, a request using topK=150 and maxCandidates=200 succeeds in the service but fails serialization against the default maxTopK=100, returning HTTP 400 INVALID_REQUEST. The service has also already recorded a successful retrieval. Please pass the same configured properties to toJson(response, properties()) and add an HTTP regression test for limits raised above the defaults.

Copilot

Warning

Copilot couldn't run its full agentic review because it didn't start before the timeout. Make sure your repository has a runner available, or add a copilot-code-review.yml file specifying one with the runs-on attribute. See the docs for more details.

Copilot review overview

2 open findings

A full in-memory index is rebuilt on every searchKeyword call (scan vertices → compute vectors →… · New
close() is no longer idempotent: directory.close() / analyzer.close() are called… · New

🧠 Review effort: Lite

+    /** Executes bounded keyword retrieval without creating a session. */
+    public List<ScoredGraphEntity> searchKeyword(String query, int topK, int maxCandidates,
+                                                 long deadlineNanos) {
+        if (graphAccessors.isEmpty()) {
+            throw new IllegalStateException("No graph accessor available");
+        }
+        for (IndexStore indexStore : indexStores) {
+            if (indexStore instanceof EntityAttributeIndexStore) {
+                EntityAttributeIndexStore keywordIndex = (EntityAttributeIndexStore) indexStore;
+                if (!keywordIndex.isInitialized()) {
+                    throw new IllegalStateException("Keyword index is not initialized");
+                }
+                GraphAccessor accessor = graphAccessors.get(0);
+                GraphSearchStore store = new GraphSearchStore();
+                try {
+                    java.util.Iterator<GraphVertex> vertices = accessor.scanVertex();
+                    int candidates = 0;
+                    while (vertices.hasNext()) {
+                        if (candidates >= maxCandidates || System.nanoTime() > deadlineNanos) {
+                            if (System.nanoTime() > deadlineNanos) {
+                                throw new org.apache.geaflow.ai.retrieval.api.model.RetrievalException(
+                                    org.apache.geaflow.ai.retrieval.api.model.RetrievalErrorCode.RETRIEVAL_TIMEOUT,
+                                    "retrieval deadline exceeded");
+                            }
+                            break;
+                        }
+                        candidates++;
+                        GraphVertex vertex = vertices.next();
+                        List<org.apache.geaflow.ai.index.vector.IVector> vectors =
+                            keywordIndex.getEntityIndex(vertex);
+                        if (vectors != null && !vectors.isEmpty()) {
+                            store.indexVertex(vertex, vectors);
+                        }
+                    }
+                    if (System.nanoTime() > deadlineNanos) {
+                        throw new org.apache.geaflow.ai.retrieval.api.model.RetrievalException(
+                            org.apache.geaflow.ai.retrieval.api.model.RetrievalErrorCode.RETRIEVAL_TIMEOUT,
+                            "retrieval deadline exceeded");
+                    }
+                    store.finishWriting();
+                    return store.searchScored(query, accessor, topK, maxCandidates, deadlineNanos);
+                } finally {
+                    store.close();
+                }
+            }
+        }


    public void close() throws IOException {
-        if (writeStats) {
-            writer.close();
-            writeStats = false;
-        }
+        finishWriting();
        if (readStats) {
            reader.close();
            readStats = false;
        }
+        directory.close();
+        analyzer.close();
+    }


Copilot

🟡 Changes recommended

Artifact identity, dataset loading, trace correctness, citation provenance, budget validation, memory usage, and endpoint security contain unresolved issues.

9 open findings

🧠 Review effort: Balanced

+    @Mapping("/api/v1/retrievals")
+    public String retrieve(org.noear.solon.core.handle.Context ctx, @Body String input) {


+                    names = sorted(name for name in archive.namelist()
+                                   if name.endswith((".json", ".jsonl", ".json.gz", ".jsonl.gz")))
+                    if len(archive.namelist()) > max_zip_entries:
+                        raise SourceError("ZIP source %s has too many entries" % path)
+                    if len(names) != 1:
+                        raise SourceError("ZIP source %s must contain exactly one JSON data file" % path)


+                            builder.graphTraces.add(new GraphEvidenceTrace(command.getQuery(),
+                                plan.getGraphVersion() == null ? "fixture" : plan.getGraphVersion(),
+                                hit.getAnchor().getEntity().getEntityId(), hit.getAnchor().getMatchType(),
+                                hit.getAnchor().getConfidence(), hit.getPath(),
+                                hit.getSourceChunkIds()));


+            SourceRef source = new SourceRef(chunk.getDocumentId(), chunk.getDocumentId(), chunk.getStartOffset(), chunk.getEndOffset());
+            return new Evidence(id, kind, chunk.getText(), Collections.singletonList(chunk), entities, paths,
+                Collections.singletonList(source), scores, fused, null, 1, graphTraces);


+            digest = hashlib.sha256(chunk_text.encode("utf-8")).hexdigest()
+            chunk_id = hashlib.sha256((document["document_id"] + "\0" + str(index) + "\0" + digest)
+                                      .encode("utf-8")).hexdigest()
+            result.append({
+                "chunk_id": chunk_id,
+                "document_id": document["document_id"],
+                "chunk_index": index,
+                "start_offset": _utf16_length(text[:actual_start]),
+                "end_offset": _utf16_length(text[:actual_end]),
+                "token_estimate": (len(chunk_text) + config.token_characters - 1)
+                    // config.token_characters,
+                "text": chunk_text,
+                "policy_version": CHUNKING_VERSION,
+                "text_hash": digest,


+    version = hashlib.sha256((manifest.dataset + "\0" + manifest.split + "\0" + manifest.sha256
+                             + "\0" + manifest.parser_version + "\0" + repr(chunking))
+                             .encode("utf-8")).hexdigest()[:16]


+        source_records = list(loader.load(manifest))
+        normalized = list(documents(source_records))
+        normalized_chunks = list(chunks(normalized, chunking))


…stic

aotenjou · 2026-10-10T12:45:56Z

Implemented and pushed fixes for all 12 review findings:

Added ownership tokens for ingestion attempts; rejected duplicate imports can no longer fail the active owner.
Added concurrent publication protection for BM25, Vector, and Graph artifacts.
Prevented artifact overwrites and validated equivalent or conflicting winners.
Reused the configured retrieval limits during response serialization.
Cached the Graph keyword search index and invalidated it on graph updates.
Made search-store and graph-store cleanup idempotent.
Added loopback-only defaults and Bearer-token authentication for remote access.
Protected legacy /graph/* and /query/* endpoints under remote authentication.
Fixed Graph trace queryId to use the request ID instead of the raw query.
Propagated real sourceUri values into citations and artifact fingerprints.
Added chunking-policy parameters to chunk identity and dataset release to artifact identity.
Updated the 2Wiki loader to select the requested split from multi-file ZIP archives.
Reworked Python ingestion to use JSONL spill files and SQLite-backed graph merging.
Added a regression test proving that a rejected duplicate import leaves the original owner able to publish READY.

aotenjou · 2026-10-10T12:46:21Z

@yaozhongq

aotenjou added 24 commits October 8, 2026 21:46

feat(ai): define retrieval api contract and configuration

7d2207f

fix(ai): harden retrieval api validation

c928d66

feat(ai): add versioned ingestion metadata and import state model

9818d52

test(ai): add deterministic retrieval fixture and HTTP contract baseline

79a8451

chore(ai): preserve week1 retrieval baseline

07bbd25

feat(graphrag): add verified dataset source loader

abbad8b

chore(graphrag): remove generated Python caches

ceb101d

fix(graphrag): harden source verification and streaming

8ebfc8b

chore(graphrag): remove generated Python caches

a4350af

fix(graphrag): verify staged downloads and strict JSON arrays

87f1889

feat(graphrag): add deterministic document normalization and chunking

3400599

fix(graphrag): preserve source identity and Java-compatible offsets

bee429d

feat(graphrag): build deterministic entity graph artifacts

b825c9f

fix(graphrag): preserve dataset evidence triples

a125601

fix(graphrag): make graph publication deterministic and staged

c4157df

feat(ai): add deterministic Lucene BM25 index builder

985fba0

feat(ai): add offline versioned vector index builder

deea988

feat(graphrag): complete week2 ingestion and readiness pipeline

9a0d2c3

fix(graphrag): harden ingestion and index artifact validation

ace2021

test(ai): align index fixtures with chunking contract

e2f59fe

feat(graphrag): publish validated retrieval artifacts

c9e9d02

feat(ai): add budgeted hybrid recall with graph channel

d9394f8

feat(ai): extend retrieval contract for hybrid recall

0e3b79d

docs(ai): document fixed hybrid recall MVP

98cd90f

yaozhongq requested review from yaozhongq and a balanced review from Copilot October 9, 2026 12:17

Copilot AI reviewed Oct 9, 2026

View reviewed changes

yaozhongq reviewed Oct 9, 2026

View reviewed changes

yaozhongq requested a balanced review from Copilot October 9, 2026 12:37

Copilot stopped reviewing on behalf of yaozhongq due to an error October 9, 2026 12:38
An unexpected error occurred. For more details, see the detailed logs in GitHub Actions.

Copilot AI reviewed Oct 9, 2026

View reviewed changes

Copilot stopped reviewing on behalf of yaozhongq due to an error October 9, 2026 13:10
An unexpected error occurred. For more details, see the detailed logs in GitHub Actions.

yaozhongq requested a balanced review from Copilot October 10, 2026 02:22

Copilot started reviewing on behalf of yaozhongq October 10, 2026 02:23 View session

Copilot AI reviewed Oct 10, 2026

View reviewed changes

E2ern1ty mentioned this pull request Oct 10, 2026

fix(ci): remove duplicate geaflow-cluster dependency in geaflow-mcp pom #879

Merged

2 of 3 tasks

aotenjou added 5 commits October 10, 2026 20:30

fix(ai): make ingestion and artifact publication exclusive

3676d43

fix(ai): secure and cache memory retrieval service

4218914

fix(ai): preserve retrieval provenance and trace identity

1dc145a

fix(graphrag): make dataset ingestion identities and spills determini…

a93c597

…stic

docs(ai): update MVP review verification

e025eb3

Navigation Menu

Repository navigation

feat: Graph rag mvp - #876

feat: Graph rag mvp#876
aotenjou wants to merge 29 commits into
apache:masterfrom
aotenjou:graphRAG-mvp

aotenjou commented Oct 8, 2026 •

edited

Loading

Uh oh!

aotenjou commented Oct 9, 2026

Uh oh!

Copilot AI left a comment

Uh oh!

yaozhongq Oct 9, 2026

Uh oh!

yaozhongq Oct 9, 2026

Uh oh!

yaozhongq Oct 9, 2026

Uh oh!

Copilot AI left a comment

Uh oh!

Copilot AI left a comment

Uh oh!

aotenjou commented Oct 10, 2026

Uh oh!

aotenjou commented Oct 10, 2026

Uh oh!

This branch has not been deployed

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

		@Mapping("/api/v1/retrievals")
		public String retrieve(org.noear.solon.core.handle.Context ctx, @Body String input) {

Conversation

aotenjou commented Oct 8, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

MVP Scope

API and Domain Model

Artifact and Index Layer

Recall Channel Layer

Fusion Layer

Service and Orchestration Layer

Ingestion and Metadata Layer

MVP Test Coverage

Test Breakdown

End-to-End Fixture

Assertions Covered

Gaps Compared with the Full Design

Uh oh!

aotenjou commented Oct 9, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Copilot review overview

Uh oh!

yaozhongq Oct 9, 2026

Choose a reason for hiding this comment

Uh oh!

yaozhongq Oct 9, 2026

Choose a reason for hiding this comment

Uh oh!

yaozhongq Oct 9, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Copilot review overview

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

🟡 Changes recommended

Uh oh!

aotenjou commented Oct 10, 2026

Uh oh!

aotenjou commented Oct 10, 2026

Uh oh!

This branch has not been deployed

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

aotenjou commented Oct 8, 2026 •

edited

Loading