[Feature] Track incremental computation with row binlog, Table Stream, and MTMV

[morningman]

Description

This issue tracks the end-to-end incremental computation capability in Apache Doris.

The implementation is being built in three layers:

OLAP table DML
    -> row binlog: durable row-level changes, before images, operation type, LSN/TSO
    -> Table Stream: change-query semantics, per-partition consumption offsets, snapshots and reset
    -> MTMV incremental maintenance: derive and transactionally apply only the affected result changes

This tracker replaces the stale umbrella issue #57921 and incorporates the row-binlog foundation originally tracked by #61956.

Goals

• Provide a durable row-level change log for Doris internal tables.
• Expose deterministic incremental consumption through Table Stream.
• Provide snapshot reads aligned with stream consumption points for joins and recovery.
• Integrate Table Stream with MTMV planning and refresh execution for row-level incremental maintenance.
• Preserve correctness across replicas, compaction, schema changes, concurrent consumers, failures, retries, and restart/replay.
• Add complete unit, regression, compatibility, and failure-recovery coverage.
• Document supported table models, SQL syntax, semantics, limitations, and operational guidance.

Scope boundary

This tracker covers the internal row-binlog and Table Stream stack used by incremental computation, plus its MTMV integration. External CDC Streaming Jobs and general MTMV partition refresh or query-rewrite changes are out of scope unless they directly modify this stack.

The initial repository review covered 2026-04-09 through 2026-07-09. PR #61382 predates that window but is included because it is the Table Stream metadata and DDL foundation.

Progress

Row binlog foundation

• [Feature] add binlog meta module (row type) (1/3) #62058 — define row-binlog metadata and schema.
• [Feature] support binlog<row> read and write (row type) (2/3) #63110 — implement row-binlog read and write paths.
• [Feature] support binlog replica schedule and compaction (row type) (3/3) #63643 — implement replica clone, scheduling, and compaction.
• [fix](core): fix cumu policy npe when do binlog schedule #64249 — initialize cumulative policy safely during binlog scheduling.
• [fix](binlog): fix GroupRowsetWriterTest coredump due to incomplete binlog meta setting #64545 — fix incomplete binlog metadata in the grouped rowset writer path.
• [fix](binlog compaction): cumu policy may be null #65032 — prevent a null cumulative policy when binlog compaction runs first.
• [Enhancement] (binlog<row>) move write lsn on sink, ensure lsn same on all replica #64133 — allocate LSNs at the sink and keep LSN/TSO semantics consistent across replicas.
• [refactor](storage) Add a block transform chain before segment writers #64674 — factor row-binlog derivation and historical-value lookup into the segment-writer transform chain.
• [fix](binlog) Support hidden key columns in row binlog #65076 — include hidden key columns while excluding hidden non-key internal columns.

Table Stream foundation

• [feat](dynamic table) support stream part 1: stream meta data & basic DDL #61382 — add stream metadata, DDL, and information-schema visibility.
• [feature](dynamic table) support stream part 2: stream query & comsuption infrastructure #62453 — add stream query and transactional consumption infrastructure.
• [feat](dynamic table) support table stream part 3: docking olap table stream with binlog #63850 — connect OLAP Table Stream to row binlog and add DETAIL, APPEND_ONLY, and MIN_DELTA scan modes.
• [feat](dynamic table) support table stream part 4: support snapshot read #64776 — add stream snapshot/reset reads, table incremental reads, and time-travel snapshot support.
• [draft](test) Add Table Stream Exception cases #65274 — expand Table Stream negative and exception-path regression coverage.
• [Tracking][Binlog] Binlog feature umbrella — all sub-issues tracked here #65265 — track the consolidated binlog and Table Stream bug and test backlog.

MTMV incremental maintenance

No PR implementing the row-level MTMV integration was identified as of 2026-07-09. The existing PartitionIncrementMaintainer supports partition-aware MTMV refresh analysis; it is not the row-level incremental-computation runtime tracked here.

• Define supported MTMV query shapes and eligibility/fallback rules.
• Derive delta plans from Table Streams, including snapshot sides for joins.
• Define incremental semantics for projection, filter, join, union, aggregation, and supported expressions.
• Integrate stream creation and lifecycle with MTMV creation, alteration, pause/resume, and drop.
• Execute result changes and stream-offset advancement atomically.
• Support full-refresh bootstrap/reset and fallback when incremental maintenance is not valid.
• Define retry, idempotency, concurrent refresh, failover, and EditLog replay behavior.
• Add observability for refresh mode, consumed offsets, lag, fallback reason, and failures.
• Add FE and BE unit tests plus end-to-end regression tests for correctness and recovery.
• Add user and operator documentation.

Correctness and compatibility checklist

• DUP and supported UNIQUE KEY MOW table semantics are documented and tested.
• Full update, partial update, delete, sequence column, hidden key, and schema-change paths are covered.
• LSN/TSO ordering is deterministic across replicas and remains valid after clone and compaction.
• Concurrent stream consumption either commits data and offsets together or fails without advancing offsets.
• Snapshot, reset, and incremental reads agree at partition boundaries and after restart.
• Binlog retention cannot silently invalidate an active stream or MTMV; stale state is observable and recoverable.
• Unsupported plans fail clearly or use a documented full-refresh fallback.
• Cloud-mode support or exclusion is explicit.
• Upgrade, downgrade, rolling restart, backup/restore, and replication behavior is defined before general availability.

Overlapping or superseded PRs found during review

These PRs are relevant to the same row-binlog failure paths but are not separate remaining deliverables:

• [fix](be) Fix binlog compaction success log #64137 — closed; its compaction-policy crash path is covered by [fix](core): fix cumu policy npe when do binlog schedule #64249.
• [fix](binlog) Fix missing binlog column index when converting TTabletSchema to TabletSchemaPB #64484 and [fix](binlog) Fix nullable type mismatch for timestamp column in _fill_binlog_columns #64495 — overlap with the metadata and nullable-column fixes merged in [fix](binlog): fix GroupRowsetWriterTest coredump due to incomplete binlog meta setting #64545.

Exit criteria

• Row binlog and Table Stream semantics are stable and documented.
• At least one supported MTMV query class refreshes incrementally end to end with atomic offset advancement.
• Correctness, failure recovery, replay, compaction, and schema evolution tests pass in the supported deployment modes.
• Unsupported cases have explicit diagnostics and a documented fallback.
• Remaining limitations and follow-up work are tracked by linked issues or PRs.