Skip to content

Support multi-stage late materialization for DeltaMerge TableScan #11003

Description

@windtalker

Enhancement

Support multi-stage late materialization for TiFlash DeltaMerge TableScan when a query has both pushed-down TableScan filters and residual Selection filters.

Tracking implementation PR: #10999

Motivation

For wide table scans, the current read path can still read many output columns for rows that will later be removed by residual filters. This is inefficient when the residual filter only depends on a small subset of columns and has good selectivity.

Multi-stage late materialization should reduce unnecessary column reads by splitting scan work into multiple stages:

  • Stage 0 reads columns needed by the pushed-down TableScan filter and applies rough-set / bitmap filtering.
  • Stage 1 reads only residual-filter columns for candidate rows and evaluates the residual filter.
  • The final stage reads the remaining output columns only for rows that pass previous stages.

Proposal

Add a guarded DeltaMerge read path that can split TableScan columns into stage0 filter columns, stage1 residual-filter columns, and final remaining columns. The implementation should preserve existing query semantics, including extra casts after TableScan, virtual-column handling, generated-column restrictions, remote read behavior, and runtime execution summaries.

Acceptance Criteria

  • The optimization is controlled by dt_enable_multi_stage_late_materialization.
  • The optimization is only enabled for supported paths and is disabled for unsafe cases such as generated columns, keep-order scans, fast scan, force-pushdown-all-filters, and stage1 filters referencing virtual _tidb_tid.
  • Query results match the existing non-MSLM path.
  • TableScan and Selection actRows remain correct for local, remote-only, and mixed local/remote execution.
  • Unit tests cover stage splitting, residual filter evaluation, direct/late final reads, extra casts, hidden commit_ts / virtual column edge cases, and runtime row reporting.

Metadata

Metadata

Assignees

No one assigned

    Labels

    affects-8.5This bug affects the 8.5.x(LTS) versions.type/bugThe issue is confirmed as a bug.

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions