Bug Report
Please answer these questions before submitting your issue. Thanks!
1. Minimal reproduce step (Required)
1, run tpch
"go-tpc ch run -D tpcc --host tc-tidb.ha-test-tiflash-tps-8198192-1-725 -P4000 --warehouses 2000 -T 32 --acThreads 1 --queries q16 --ignore-error '2013,1213,1205,8022,8028,9004,9007,1062,1105' --time 36000m --user root --interval '10s'
2, kill one of wn
2. What did you expect to see? (Required)
no panic
3. What did you see instead (Required)
tiflash wn crash repeatedly after kill one of tiflash wn
{"stream":"stdout","log":"[2026/07/26 04:04:18.286 +08:00] [FATAL] [Exception.cpp:98] ["Code: 11000, e.displayText() = DB::Exception: Open S3 file for read fail after retries when init file, key=s1191/data/ks_1_t_2737/dmf_2814/meta: file_page_id=2849 meta_version=0: while restoreSegment, segment_id=823 ident=keyspace=1 table_id=2737, e.what() = DB::Exception, Stack trace:\n\n\n 0x562242ac5a11\tDB::Exception::Exception<std::__1::basic_string_view<char, std::__1::char_traits>&, std::__1::basic_string<char, std::__1::char_traits, std::__1::allocator>&>(int, std::__1::basic_string<char, std::__1::char_traits, std::__1::allocator> const&, std::__1::basic_string_view<char, std::__1::char_traits>&, std::__1::basic_string<char, std::__1::char_traits, std::__1::allocator>&) [tiflash+146831889]\n \tdbms/src/Common/StackTrace.cpp:25\n 0x562242abea6c\tDB::S3::S3RandomAccessFile::initialize(std::__1::basic_string_view<char, std::__1::char_traits>) [tiflash+146803308]\n \tdbms/src/Storages/S3/S3RandomAccessFile.cpp:564\n 0x562242abd6a2\tDB::S3::S3RandomAccessFile::S3RandomAccessFile(std::__1::shared_ptrDB::S3::TiFlashS3Client, std::__1::basic_string<char, std::__1::char_traits, std::__1::allocator> const&, std::__1::shared_ptrDB::DM::ScanContext const&) [tiflash+146798242]\n \tdbms/src/Storages/S3/S3RandomAccessFile.cpp:90\n 0x562242ac5ee0\tDB::S3::createFromNormalFile(std::__1::basic_string<char, std::__1::char_traits, std::__1::allocator> const&, std::__1::optional, std::__1::optional<std::__1::shared_ptrDB::DM::ScanContext>) [tiflash+146833120]\n \t/usr/local/bin/../include/c++/v1/__memory/construct_at.h:41\n 0x562242ac5b7e\tDB::S3::S3RandomAccessFile::create(std::__1::basic_string<char, std::__1::char_traits, std::__1::allocator> const&) [tiflash+146832254]\n \tdbms/src/Storages/S3/S3RandomAccessFile.cpp:623\n 0x562241fed3d1\tDB::FileProvider::newRandomAccessFile(std::__1::basic_string<char, std::__1::char_traits, std::__1::allocator> const&, DB::EncryptionPath const&, std::__1::shared_ptrDB::ReadLimiter const&, int) const [tiflash+135459793]\n \tdbms/src/IO/FileProvider/FileProvider.cpp:43\n 0x562241ff492d\tDB::ReadBufferFromRandomAccessFileBuilder::build(std::__1::shared_ptrDB::FileProvider const&, std::__1::basic_string<char, std::__1::char_traits, std::__1::allocator> const&, DB::EncryptionPath const&, unsigned long, std::__1::shared_ptrDB::ReadLimiter const&, int, char*, unsigned long) [tiflash+135489837]\n \tdbms/src/IO/FileProvider/ReadBufferFromRandomAccessFileBuilder.cpp:60\n 0x562241c95bf2\tDB::DM::DMFileMetaV2::read(std::__1::shared_ptrDB::FileProvider const&, DB::DM::DMFileMeta::ReadMode const&) [tiflash+131955698]\n \tdbms/src/Storages/DeltaMerge/File/DMFileMeta.h:295\n 0x562241c630c9\tDB::DM::DMFile::restore(std::__1::shared_ptrDB::FileProvider const&, unsigned long, unsigned long, std::__1::basic_string<char, std::__1::char_traits, std::__1::allocator> const&, DB::DM::DMFileMeta::ReadMode const&, unsigned long, unsigned int) [tiflash+131748041]\n \tdbms/src/Storages/DeltaMerge/File/DMFile.cpp:0\n 0x5622428855a7\tDB::DM::Remote::S3PreparedDMFileToken::restore(DB::DM::DMFileMeta::ReadMode, unsigned long) [tiflash+144471463]\n \tdbms/src/Storages/DeltaMerge/Remote/DataStore/DataStoreS3.cpp:285\n 0x562241b94726\tDB::DM::restoreDMFileFromRemoteDataSource(DB::DM::DMContext const&, std::__1::shared_ptrDB::DM::Remote::IDataStore, unsigned long, unsigned long) [tiflash+130901798]\n \tdbms/src/Storages/DeltaMerge/RestoreDMFile.cpp:51\n 0x562241b8c702\tDB::DM::StableValueSpace::restore(DB::DM::DMContext&, DB::ReadBuffer&, unsigned long) [tiflash+130868994]\n \tdbms/src/Storages/DeltaMerge/StableValueSpace.cpp:185\n 0x562241b8c5a6\tDB::DM::StableValueSpace::restore(DB::DM::DMContext&, unsigned long) [tiflash+130868646]\n \tdbms/src/Storages/DeltaMerge/StableValueSpace.cpp:171\n 0x562241ad3afe\tDB::DM::Segment::restoreSegment(std::__1::shared_ptrDB::Logger const&, DB::DM::DMContext&, unsigned long) [tiflash+130112254]\n \tdbms/src/Storages/DeltaMerge/Segment.cpp:447\n 0x562241a43f6f\tDB::DM::DeltaMergeStore::DeltaMergeStore(DB::Context&, bool, std::__1::basic_string<char, std::__1::char_traits, std::__1::allocator> const&, std::__1::basic_string<char, std::__1::char_traits, std::__1::allocator> const&, unsigned int, long, long, bool, std::__1::vector<DB::DM::ColumnDefine, std::__1::allocatorDB::DM::ColumnDefine> const&, DB::DM::ColumnDefine const&, bool, unsigned long, std::__1::shared_ptr<std::__1::vector<DB::DM::LocalIndexInfo, std::__1::allocatorDB::DM::LocalIndexInfo>>, DB::DM::DeltaMergeStore::Settings const&, DB::ThreadPoolImpl<DB::ThreadFromGlobalPoolImpl>) [tiflash+129523567]\n \tdbms/src/Storages/DeltaMerge/DeltaMergeStore.cpp:347\n 0x562241a46901\tDB::DM::DeltaMergeStore::create(DB::Context&, bool, std::__1::basic_string<char, std::__1::char_traits, std::__1::allocator> const&, std::__1::basic_string<char, std::__1::char_traits, std::__1::allocator> const&, unsigned int, long, long, bool, std::__1::vector<DB::DM::ColumnDefine, std::__1::allocatorDB::DM::ColumnDefine> const&, DB::DM::ColumnDefine const&, bool, unsigned long, std::__1::shared_ptr<std::__1::vector<DB::DM::LocalIndexInfo, std::__1::allocatorDB::DM::LocalIndexInfo>>, DB::DM::DeltaMergeStore::Settings const&, DB::ThreadPoolImpl<DB::ThreadFromGlobalPoolImpl>) [tiflash+129534209]\n \tdbms/src/Storages/DeltaMerge/DeltaMergeStore.cpp:385\n 0x5622429dad85\tDB::StorageDeltaMerge::getAndMaybeInitStore(DB::ThreadPoolImpl<DB::ThreadFromGlobalPoolImpl>*) [tiflash+145870213]\n \tdbms/src/Storages/StorageDeltaMerge.cpp:1866\n 0x5622429e2b85\tDB::StorageDeltaMerge::getSchemaSnapshotAndBlockForDecoding(DB::TableDoubleLockHolder const&, bool, bool) [tiflash+145902469]\n \tdbms/src/Storages/StorageDeltaMerge.cpp:1254\n 0x562243f40366\tbool DB::atomicReadWrite<std::__1::vector<DB::RegionDataReadInfo, std::__1::allocatorDB::RegionDataReadInfo>>(DB::AtomicReadWriteCtx&, std::__1::shared_ptrDB::Region const&, std::__1::vector<DB::RegionDataReadInfo, std::__1::allocatorDB::RegionDataReadInfo>&, bool) [tiflash+168305510]\n \tdbms/src/Storages/KVStore/Decode/PartitionStreams.cpp:160\n 0x562243f3d393\tDB::RegionTable::writeCommittedByRegion(DB::Context&, std::__1::shared_ptrDB::Region const&, std::__1::vector<DB::RegionDataReadInfo, std::__1::allocatorDB::RegionDataReadInfo>&, std::__1::shared_ptrDB::Logger const&, bool) [tiflash+168293267]\n \tdbms/src/Storages/KVStore/Decode/PartitionStreams.cpp:239\n 0x562243ea5283\tDB::Region::handleWriteRaftCmd(DB::WriteCmdsView const&, unsigned long, unsigned long, DB::TMTContext&) [tiflash+167670403]\n \tdbms/src/Storages/KVStore/MultiRaft/RaftCommands.cpp:514\n 0x562243eac997\tDB::KVStore::handleWriteRaftCmdInner(DB::WriteCmdsView const&, unsigned long, unsigned long, unsigned long, DB::TMTContext&, std::__1::optionalDB::DM::RaftWriteResult&) [tiflash+167700887]\n \tdbms/src/Storages/KVStore/MultiRaft/RaftCommandsKVS.cpp:62\n 0x562243eac804\tDB::KVStore::handleWriteRaftCmd(DB::WriteCmdsView const&, unsigned long, unsigned long, unsigned long, DB::TMTContext&) [tiflash+167700484]\n \tdbms/src/Storages/KVStore/MultiRaft/RaftCommandsKVS.cpp:41\n 0x562243e364c6\tHandleWriteRaftCmd [tiflash+167216326]\n \tdbms/src/Storages/KVStore/FFI/ProxyFFI.cpp:100\n 0x7f732474936d\tengine_store_ffi::core::forward_raft::command::$LT$impl$u20$engine_store_ffi..core..forwarder..ProxyForwarder$LT$T$C$ER$GT$$GT$::post_exec_query::hf1bd6705fd54a9a5 [libtiflash_proxy.so+28439405]\n 0x7f732542bdcc\traftstore::store::fsm::apply::ApplyDelegate$LT$EK$GT$::process_raft_cmd::hccf4673cefb05378 [libtiflash_proxy.so+41950668]\n 0x7f7325433733\traftstore::store::fsm::apply::ApplyDelegate$LT$EK$GT$::handle_raft_committed_entries::h99122f5e14aa60c4 [libtiflash_proxy.so+41981747]\n 0x7f732540c749\traftstore::store::fsm::apply::ApplyFsm$LT$EK$GT$::handle_apply::h5ebdbdc48ab0c9d5 [libtiflash_proxy.so+41822025]\n 0x7f7325410fa3\traftstore::store::fsm::apply::ApplyFsm$LT$EK$GT$::handle_tasks::h297e23ede1b5b1f0 [libtiflash_proxy.so+41840547]\n 0x7f7324774ae5\t$LT$raftstore..store..fsm..apply..ApplyPoller$LT$EK$GT$$u20$as$u20$batch_system..batch..PollHandler$LT$raftstore..store..fsm..apply..ApplyFsm$LT$EK$GT$$C$raftstore..store..fsm..apply..ControlFsm$GT$$GT$::handle_normal::h94499c390d84dbe7 [libtiflash_proxy.so+28617445]\n 0x7f73246e9e65\tbatch_system::batch::Poller$LT$N$C$C$C$Handler$GT$::poll::h3cfc6addfa4de40c [libtiflash_proxy.so+28048997]\n 0x7f73247ac845\tstd::sys_common::backtrace::__rust_begin_short_backtrace::h20f7ac87f05251c9 [libtiflash_proxy.so+28846149]"] [source="EngineStoreApplyRes DB::HandleWriteRaftCmd(const EngineStoreServerWrap *, WriteCmdsView, RaftCmdHeader)"] [thread_id=604]\n","pod":"tc-tiflash-0","container":"errorlog","namespace":"ha-test-tiflash-tps-8198192-1-725","time":"2026-07-25T20:04:18.523735662Z"}
4. What is your TiFlash version? (Required)
/tiflash/tiflash version
TiFlash
Release Version: v9.0.0-beta.2.pre-201-g555bb7cc2
Edition: Enterprise
Git Commit Hash: 555bb7c
Git Branch: HEAD
UTC Build Time: 2026-07-23 03:30:40
Enable Features: jemalloc sm4(GmSSL) mem-profiling avx2 avx512 unwind thinlto next-gen hnsw.l2=skylake hnsw.cosine=skylake vec.l2=skylake vec.cos=skylake
Profile: RELWITHDEBINFO
Compiler: clang++ 17.0.6
Raft Proxy
Git Commit Hash: 45187f537fd50287eb46d067960d00d7adbdb11e
Git Commit Branch: HEAD
UTC Build Time: ""
Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27)
Storage Engine: tiflash
Prometheus Prefix: tiflash_proxy_
Profile: release
Enable Features: "raftstore-proxy/external-jemalloc" portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure
2026-07-26T03:53:40.604+0800
Bug Report
Please answer these questions before submitting your issue. Thanks!
1. Minimal reproduce step (Required)
1, run tpch
"go-tpc ch run -D tpcc --host tc-tidb.ha-test-tiflash-tps-8198192-1-725 -P4000 --warehouses 2000 -T 32 --acThreads 1 --queries q16 --ignore-error '2013,1213,1205,8022,8028,9004,9007,1062,1105' --time 36000m --user root --interval '10s'
2, kill one of wn
2. What did you expect to see? (Required)
no panic
3. What did you see instead (Required)
tiflash wn crash repeatedly after kill one of tiflash wn
{"stream":"stdout","log":"[2026/07/26 04:04:18.286 +08:00] [FATAL] [Exception.cpp:98] ["Code: 11000, e.displayText() = DB::Exception: Open S3 file for read fail after retries when init file, key=s1191/data/ks_1_t_2737/dmf_2814/meta: file_page_id=2849 meta_version=0: while restoreSegment, segment_id=823 ident=keyspace=1 table_id=2737, e.what() = DB::Exception, Stack trace:\n\n\n 0x562242ac5a11\tDB::Exception::Exception<std::__1::basic_string_view<char, std::__1::char_traits>&, std::__1::basic_string<char, std::__1::char_traits, std::__1::allocator>&>(int, std::__1::basic_string<char, std::__1::char_traits, std::__1::allocator> const&, std::__1::basic_string_view<char, std::__1::char_traits>&, std::__1::basic_string<char, std::__1::char_traits, std::__1::allocator>&) [tiflash+146831889]\n \tdbms/src/Common/StackTrace.cpp:25\n 0x562242abea6c\tDB::S3::S3RandomAccessFile::initialize(std::__1::basic_string_view<char, std::__1::char_traits>) [tiflash+146803308]\n \tdbms/src/Storages/S3/S3RandomAccessFile.cpp:564\n 0x562242abd6a2\tDB::S3::S3RandomAccessFile::S3RandomAccessFile(std::__1::shared_ptrDB::S3::TiFlashS3Client, std::__1::basic_string<char, std::__1::char_traits, std::__1::allocator> const&, std::__1::shared_ptrDB::DM::ScanContext const&) [tiflash+146798242]\n \tdbms/src/Storages/S3/S3RandomAccessFile.cpp:90\n 0x562242ac5ee0\tDB::S3::createFromNormalFile(std::__1::basic_string<char, std::__1::char_traits, std::__1::allocator> const&, std::__1::optional, std::__1::optional<std::__1::shared_ptrDB::DM::ScanContext>) [tiflash+146833120]\n \t/usr/local/bin/../include/c++/v1/__memory/construct_at.h:41\n 0x562242ac5b7e\tDB::S3::S3RandomAccessFile::create(std::__1::basic_string<char, std::__1::char_traits, std::__1::allocator> const&) [tiflash+146832254]\n \tdbms/src/Storages/S3/S3RandomAccessFile.cpp:623\n 0x562241fed3d1\tDB::FileProvider::newRandomAccessFile(std::__1::basic_string<char, std::__1::char_traits, std::__1::allocator> const&, DB::EncryptionPath const&, std::__1::shared_ptrDB::ReadLimiter const&, int) const [tiflash+135459793]\n \tdbms/src/IO/FileProvider/FileProvider.cpp:43\n 0x562241ff492d\tDB::ReadBufferFromRandomAccessFileBuilder::build(std::__1::shared_ptrDB::FileProvider const&, std::__1::basic_string<char, std::__1::char_traits, std::__1::allocator> const&, DB::EncryptionPath const&, unsigned long, std::__1::shared_ptrDB::ReadLimiter const&, int, char*, unsigned long) [tiflash+135489837]\n \tdbms/src/IO/FileProvider/ReadBufferFromRandomAccessFileBuilder.cpp:60\n 0x562241c95bf2\tDB::DM::DMFileMetaV2::read(std::__1::shared_ptrDB::FileProvider const&, DB::DM::DMFileMeta::ReadMode const&) [tiflash+131955698]\n \tdbms/src/Storages/DeltaMerge/File/DMFileMeta.h:295\n 0x562241c630c9\tDB::DM::DMFile::restore(std::__1::shared_ptrDB::FileProvider const&, unsigned long, unsigned long, std::__1::basic_string<char, std::__1::char_traits, std::__1::allocator> const&, DB::DM::DMFileMeta::ReadMode const&, unsigned long, unsigned int) [tiflash+131748041]\n \tdbms/src/Storages/DeltaMerge/File/DMFile.cpp:0\n 0x5622428855a7\tDB::DM::Remote::S3PreparedDMFileToken::restore(DB::DM::DMFileMeta::ReadMode, unsigned long) [tiflash+144471463]\n \tdbms/src/Storages/DeltaMerge/Remote/DataStore/DataStoreS3.cpp:285\n 0x562241b94726\tDB::DM::restoreDMFileFromRemoteDataSource(DB::DM::DMContext const&, std::__1::shared_ptrDB::DM::Remote::IDataStore, unsigned long, unsigned long) [tiflash+130901798]\n \tdbms/src/Storages/DeltaMerge/RestoreDMFile.cpp:51\n 0x562241b8c702\tDB::DM::StableValueSpace::restore(DB::DM::DMContext&, DB::ReadBuffer&, unsigned long) [tiflash+130868994]\n \tdbms/src/Storages/DeltaMerge/StableValueSpace.cpp:185\n 0x562241b8c5a6\tDB::DM::StableValueSpace::restore(DB::DM::DMContext&, unsigned long) [tiflash+130868646]\n \tdbms/src/Storages/DeltaMerge/StableValueSpace.cpp:171\n 0x562241ad3afe\tDB::DM::Segment::restoreSegment(std::__1::shared_ptrDB::Logger const&, DB::DM::DMContext&, unsigned long) [tiflash+130112254]\n \tdbms/src/Storages/DeltaMerge/Segment.cpp:447\n 0x562241a43f6f\tDB::DM::DeltaMergeStore::DeltaMergeStore(DB::Context&, bool, std::__1::basic_string<char, std::__1::char_traits, std::__1::allocator> const&, std::__1::basic_string<char, std::__1::char_traits, std::__1::allocator> const&, unsigned int, long, long, bool, std::__1::vector<DB::DM::ColumnDefine, std::__1::allocatorDB::DM::ColumnDefine> const&, DB::DM::ColumnDefine const&, bool, unsigned long, std::__1::shared_ptr<std::__1::vector<DB::DM::LocalIndexInfo, std::__1::allocatorDB::DM::LocalIndexInfo>>, DB::DM::DeltaMergeStore::Settings const&, DB::ThreadPoolImpl<DB::ThreadFromGlobalPoolImpl>) [tiflash+129523567]\n \tdbms/src/Storages/DeltaMerge/DeltaMergeStore.cpp:347\n 0x562241a46901\tDB::DM::DeltaMergeStore::create(DB::Context&, bool, std::__1::basic_string<char, std::__1::char_traits, std::__1::allocator> const&, std::__1::basic_string<char, std::__1::char_traits, std::__1::allocator> const&, unsigned int, long, long, bool, std::__1::vector<DB::DM::ColumnDefine, std::__1::allocatorDB::DM::ColumnDefine> const&, DB::DM::ColumnDefine const&, bool, unsigned long, std::__1::shared_ptr<std::__1::vector<DB::DM::LocalIndexInfo, std::__1::allocatorDB::DM::LocalIndexInfo>>, DB::DM::DeltaMergeStore::Settings const&, DB::ThreadPoolImpl<DB::ThreadFromGlobalPoolImpl>) [tiflash+129534209]\n \tdbms/src/Storages/DeltaMerge/DeltaMergeStore.cpp:385\n 0x5622429dad85\tDB::StorageDeltaMerge::getAndMaybeInitStore(DB::ThreadPoolImpl<DB::ThreadFromGlobalPoolImpl>*) [tiflash+145870213]\n \tdbms/src/Storages/StorageDeltaMerge.cpp:1866\n 0x5622429e2b85\tDB::StorageDeltaMerge::getSchemaSnapshotAndBlockForDecoding(DB::TableDoubleLockHolder const&, bool, bool) [tiflash+145902469]\n \tdbms/src/Storages/StorageDeltaMerge.cpp:1254\n 0x562243f40366\tbool DB::atomicReadWrite<std::__1::vector<DB::RegionDataReadInfo, std::__1::allocatorDB::RegionDataReadInfo>>(DB::AtomicReadWriteCtx&, std::__1::shared_ptrDB::Region const&, std::__1::vector<DB::RegionDataReadInfo, std::__1::allocatorDB::RegionDataReadInfo>&, bool) [tiflash+168305510]\n \tdbms/src/Storages/KVStore/Decode/PartitionStreams.cpp:160\n 0x562243f3d393\tDB::RegionTable::writeCommittedByRegion(DB::Context&, std::__1::shared_ptrDB::Region const&, std::__1::vector<DB::RegionDataReadInfo, std::__1::allocatorDB::RegionDataReadInfo>&, std::__1::shared_ptrDB::Logger const&, bool) [tiflash+168293267]\n \tdbms/src/Storages/KVStore/Decode/PartitionStreams.cpp:239\n 0x562243ea5283\tDB::Region::handleWriteRaftCmd(DB::WriteCmdsView const&, unsigned long, unsigned long, DB::TMTContext&) [tiflash+167670403]\n \tdbms/src/Storages/KVStore/MultiRaft/RaftCommands.cpp:514\n 0x562243eac997\tDB::KVStore::handleWriteRaftCmdInner(DB::WriteCmdsView const&, unsigned long, unsigned long, unsigned long, DB::TMTContext&, std::__1::optionalDB::DM::RaftWriteResult&) [tiflash+167700887]\n \tdbms/src/Storages/KVStore/MultiRaft/RaftCommandsKVS.cpp:62\n 0x562243eac804\tDB::KVStore::handleWriteRaftCmd(DB::WriteCmdsView const&, unsigned long, unsigned long, unsigned long, DB::TMTContext&) [tiflash+167700484]\n \tdbms/src/Storages/KVStore/MultiRaft/RaftCommandsKVS.cpp:41\n 0x562243e364c6\tHandleWriteRaftCmd [tiflash+167216326]\n \tdbms/src/Storages/KVStore/FFI/ProxyFFI.cpp:100\n 0x7f732474936d\tengine_store_ffi::core::forward_raft::command::$LT$impl$u20$engine_store_ffi..core..forwarder..ProxyForwarder$LT$T$C$ER$GT$$GT$::post_exec_query::hf1bd6705fd54a9a5 [libtiflash_proxy.so+28439405]\n 0x7f732542bdcc\traftstore::store::fsm::apply::ApplyDelegate$LT$EK$GT$::process_raft_cmd::hccf4673cefb05378 [libtiflash_proxy.so+41950668]\n 0x7f7325433733\traftstore::store::fsm::apply::ApplyDelegate$LT$EK$GT$::handle_raft_committed_entries::h99122f5e14aa60c4 [libtiflash_proxy.so+41981747]\n 0x7f732540c749\traftstore::store::fsm::apply::ApplyFsm$LT$EK$GT$::handle_apply::h5ebdbdc48ab0c9d5 [libtiflash_proxy.so+41822025]\n 0x7f7325410fa3\traftstore::store::fsm::apply::ApplyFsm$LT$EK$GT$::handle_tasks::h297e23ede1b5b1f0 [libtiflash_proxy.so+41840547]\n 0x7f7324774ae5\t$LT$raftstore..store..fsm..apply..ApplyPoller$LT$EK$GT$$u20$as$u20$batch_system..batch..PollHandler$LT$raftstore..store..fsm..apply..ApplyFsm$LT$EK$GT$$C$raftstore..store..fsm..apply..ControlFsm$GT$$GT$::handle_normal::h94499c390d84dbe7 [libtiflash_proxy.so+28617445]\n 0x7f73246e9e65\tbatch_system::batch::Poller$LT$N$C$C$C$Handler$GT$::poll::h3cfc6addfa4de40c [libtiflash_proxy.so+28048997]\n 0x7f73247ac845\tstd::sys_common::backtrace::__rust_begin_short_backtrace::h20f7ac87f05251c9 [libtiflash_proxy.so+28846149]"] [source="EngineStoreApplyRes DB::HandleWriteRaftCmd(const EngineStoreServerWrap *, WriteCmdsView, RaftCmdHeader)"] [thread_id=604]\n","pod":"tc-tiflash-0","container":"errorlog","namespace":"ha-test-tiflash-tps-8198192-1-725","time":"2026-07-25T20:04:18.523735662Z"}
4. What is your TiFlash version? (Required)
/tiflash/tiflash version
TiFlash
Release Version: v9.0.0-beta.2.pre-201-g555bb7cc2
Edition: Enterprise
Git Commit Hash: 555bb7c
Git Branch: HEAD
UTC Build Time: 2026-07-23 03:30:40
Enable Features: jemalloc sm4(GmSSL) mem-profiling avx2 avx512 unwind thinlto next-gen hnsw.l2=skylake hnsw.cosine=skylake vec.l2=skylake vec.cos=skylake
Profile: RELWITHDEBINFO
Compiler: clang++ 17.0.6
Raft Proxy
Git Commit Hash: 45187f537fd50287eb46d067960d00d7adbdb11e
Git Commit Branch: HEAD
UTC Build Time: ""
Rust Version: rustc 1.77.0-nightly (89e2160c4 2023-12-27)
Storage Engine: tiflash
Prometheus Prefix: tiflash_proxy_
Profile: release
Enable Features: "raftstore-proxy/external-jemalloc" portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure
2026-07-26T03:53:40.604+0800