Skip to content

Conversation

@jacktengg
Copy link
Contributor

What problem does this PR solve?

Issue Number: close #xxx

Related PR: #xxx

Problem Summary:

Release note

None

Check List (For Author)

  • Test

    • Regression test
    • Unit Test
    • Manual test (add detailed scripts or steps below)
    • No need to test or manual test. Explain why:
      • This is a refactor/code format and no logic has been changed.
      • Previous test can cover this change.
      • No code files have been changed.
      • Other reason
  • Behavior changed:

    • No.
    • Yes.
  • Does this need documentation?

    • No.
    • Yes.

Check List (For Reviewer who merge this PR)

  • Confirm the release note
  • Confirm test cases
  • Confirm document
  • Add branch pick label

@hello-stephen
Copy link
Contributor

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@jacktengg
Copy link
Contributor Author

run buildall

Copy link

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This pull request aims to improve the performance of casting strings to decimal types by optimizing the parsing logic in StringParser::string_to_decimal. The optimization removes the intermediate std::vector<unsigned char> digits buffer and processes characters directly from the input string.

Changes:

  • Optimized string-to-decimal parsing by eliminating intermediate digit buffer and processing characters directly
  • Added performance test infrastructure and test cases for various decimal types
  • Refactored variable naming for clarity in the parsing logic

Reviewed changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated 17 comments.

File Description
be/src/util/string_parser.cpp Core optimization - removed vector allocation and processes input string directly with updated parsing logic
be/test/vec/function/cast/cast_to_decimal128_perf.cpp Added performance test methods and helper functions (mostly commented out)

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment on lines +191 to +194
auto total_significant_digit_count = i - ((found_dot && int_part_count > 0) ? 1 : 0);
if (result_int_part_digit_count > total_significant_digit_count) {
int_part_number *= get_scale_multiplier<T>(result_int_part_digit_count -
total_significant_digit_count);
Copy link

Copilot AI Jan 18, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The variable name 'total_significant_digit_count' is misleading. Based on the calculation 'i - ((found_dot && int_part_count > 0) ? 1 : 0)', this represents the total count of characters processed (including digits and potentially a decimal point), not just significant digits. Leading zeros were already skipped earlier in the function. A more accurate name would be 'total_digit_char_count' or 'digits_processed_count' to clarify what this value represents.

Suggested change
auto total_significant_digit_count = i - ((found_dot && int_part_count > 0) ? 1 : 0);
if (result_int_part_digit_count > total_significant_digit_count) {
int_part_number *= get_scale_multiplier<T>(result_int_part_digit_count -
total_significant_digit_count);
auto total_digit_char_count = i - ((found_dot && int_part_count > 0) ? 1 : 0);
if (result_int_part_digit_count > total_digit_char_count) {
int_part_number *= get_scale_multiplier<T>(result_int_part_digit_count -
total_digit_char_count);

Copilot uses AI. Check for mistakes.
}

TEST_F(FunctionCastToDecimalPerfTest, test_string_to_decimal32v3_perf) {
std::string test_data_file = "/mnt/disk2/tengjianping/cast_perf/decimal_test_data_9_4.txt";
Copy link

Copilot AI Jan 18, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The test methods contain hardcoded absolute file paths (e.g., "/mnt/disk2/tengjianping/cast_perf/decimal_test_data_9_4.txt"). These paths are machine-specific and will fail on other development or CI environments. Consider using relative paths from the project root, or environment variables, or generate test data programmatically within the test itself.

Copilot uses AI. Check for mistakes.
}

TEST_F(FunctionCastToDecimalPerfTest, test_string_to_decimalv2_perf) {
std::string test_data_file = "/mnt/disk2/tengjianping/cast_perf/decimalv2_test_data_27_9.txt";
Copy link

Copilot AI Jan 18, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The test methods contain hardcoded absolute file paths (e.g., "/mnt/disk2/tengjianping/cast_perf/decimalv2_test_data_27_9.txt"). These paths are machine-specific and will fail on other development or CI environments. Consider using relative paths from the project root, or environment variables, or generate test data programmatically within the test itself.

Copilot uses AI. Check for mistakes.
// Comparison test: String vs Direct decimal parsing
TEST_F(FunctionCastToDecimalPerfTest, test_string_parse_comparison) {
std::vector<std::string> file_contents;
read_file_to_vector_optimized("/mnt/disk2/tengjianping/cast_perf/decimal_test_data_38_19.txt",
Copy link

Copilot AI Jan 18, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The test methods contain hardcoded absolute file paths (e.g., "/mnt/disk2/tengjianping/cast_perf/decimal_test_data_38_19.txt"). These paths are machine-specific and will fail on other development or CI environments. Consider using relative paths from the project root, or environment variables, or generate test data programmatically within the test itself.

Copilot uses AI. Check for mistakes.
}
// parse exponent if any
int64_t exponent = 0;
auto end_digit_index = i;
Copy link

Copilot AI Jan 18, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The variable name 'end_digit_index' is misleading. This variable stores the index where digit parsing stopped (where 'e'/'E' was found or end of string), but it may include the position of a decimal point '.'. A more accurate name would be 'digit_section_end_index' or 'parse_end_index' to clarify that it marks the end of the numeric section before the exponent.

Suggested change
auto end_digit_index = i;
auto parse_end_index = i;

Copilot uses AI. Check for mistakes.
}
if (digit_index != actual_int_part_count) {
int_part_number *= get_scale_multiplier<T>(actual_int_part_count - digit_index);
auto total_significant_digit_count = i - ((found_dot && int_part_count > 0) ? 1 : 0);
Copy link

Copilot AI Jan 18, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Incorrect variable used in calculation. At this point in the code, the variable 'i' has been modified during exponent parsing (see lines 111, 118, 126) and now points to the end of the string or past the exponent section. The calculation should use 'end_digit_index' instead of 'i', which correctly stores the position where the digit section ended (before the exponent). Using 'i' here will result in incorrect calculation of the total significant digit count.

Suggested change
auto total_significant_digit_count = i - ((found_dot && int_part_count > 0) ? 1 : 0);
auto total_significant_digit_count =
end_digit_index - ((found_dot && int_part_count > 0) ? 1 : 0);

Copilot uses AI. Check for mistakes.
Comment on lines 207 to 322
/*
TEST_F(FunctionCastToDecimalPerfTest, test_string_to_decimal128_tmp) {
{
// std::string s("1234567890.123456");
// std::string s("0.0000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000e2147483647");
// std::string s("1234567890.123456e7");
// std::string s("1.5");
// std::string s("+0000.004e1");
// std::string s("+00099999999999999999999999999999999999.904e1");
std::string s("+000900000000000000000000000000000000000.004e1");
int type_precision = 38;
int type_scale = 1;
StringParser::ParseResult result = StringParser::PARSE_FAILURE;
auto value = StringParser::string_to_decimal<PrimitiveType::TYPE_DECIMAL128I>(
s.c_str(), s.size(), type_precision, type_scale, &result);
std::cout << "Parsed decimal128 value: " << value << ", ParseResult: " << result
<< std::endl;
}
}
TEST_F(FunctionCastToDecimalPerfTest, test_string_to_decimal128v3_perf) {
std::string test_data_file = "/mnt/disk2/tengjianping/cast_perf/decimal_test_data_38_19.txt";
size_t orig_line_count = 10000000; // 10 million lines
// Create string column and load data from file
auto dt_from = std::make_shared<DataTypeString>();
auto column_orig = dt_from->create_column();

{
MutableColumns columns;
columns.push_back(column_orig->get_ptr());
DataTypeSerDeSPtrs serde = {dt_from->get_serde()};
load_columns_data_from_file(columns, serde, ';', {0}, test_data_file);
EXPECT_EQ(column_orig->size(), orig_line_count);
}

int test_rounds = 10;
int64_t total_duration_ns = 0;
for (int i = 0; i < test_rounds; ++i) {
std::cout << "\n--- Test Round " << (i + 1) << " ---\n";
total_duration_ns += perf_test_string_to_decimal<PrimitiveType::TYPE_DECIMAL128I>(
column_orig, orig_line_count, 38, 19, false);
}
auto avg_duration_ns = total_duration_ns / test_rounds;
std::cout << fmt::format("\nAverage time over {} rounds: {}\n", test_rounds,
PrettyPrinter::print(avg_duration_ns, TUnit::TIME_NS));
}

TEST_F(FunctionCastToDecimalPerfTest, test_string_to_decimal128v3_nullable_perf) {
std::string test_data_file = "/mnt/disk2/tengjianping/cast_perf/decimal_test_data_38_19.txt";
size_t orig_line_count = 10000000; // 10 million lines
// Create string column and load data from file
auto dt_from = std::make_shared<DataTypeString>();
auto column_orig = dt_from->create_column();

{
MutableColumns columns;
columns.push_back(column_orig->get_ptr());
DataTypeSerDeSPtrs serde = {dt_from->get_serde()};
load_columns_data_from_file(columns, serde, ';', {0}, test_data_file);
EXPECT_EQ(column_orig->size(), orig_line_count);
}

int test_rounds = 10;
int64_t total_duration_ns = 0;
for (int i = 0; i < test_rounds; ++i) {
std::cout << "\n--- Test Round " << (i + 1) << " ---\n";
total_duration_ns += perf_test_string_to_decimal<PrimitiveType::TYPE_DECIMAL128I>(
column_orig, orig_line_count, 38, 19, true);
}
auto avg_duration_ns = total_duration_ns / test_rounds;
std::cout << fmt::format("\nAverage time over {} rounds: {}\n", test_rounds,
PrettyPrinter::print(avg_duration_ns, TUnit::TIME_NS));
}

TEST_F(FunctionCastToDecimalPerfTest, test_string_to_decimal64v3_perf) {
std::string test_data_file = "/mnt/disk2/tengjianping/cast_perf/decimal_test_data_18_9.txt";
size_t orig_line_count = 10000000;
perf_test_string_to_decimal<PrimitiveType::TYPE_DECIMAL64>(test_data_file, orig_line_count, 18,
9, false);
}

TEST_F(FunctionCastToDecimalPerfTest, test_string_to_decimal32v3_perf) {
std::string test_data_file = "/mnt/disk2/tengjianping/cast_perf/decimal_test_data_9_4.txt";
size_t orig_line_count = 10000000;
perf_test_string_to_decimal<PrimitiveType::TYPE_DECIMAL32>(test_data_file, orig_line_count, 9,
4, false);
}

TEST_F(FunctionCastToDecimalPerfTest, test_string_to_decimalv2_perf) {
std::string test_data_file = "/mnt/disk2/tengjianping/cast_perf/decimalv2_test_data_27_9.txt";
size_t orig_line_count = 10000000;
perf_test_string_to_decimal<PrimitiveType::TYPE_DECIMALV2>(test_data_file, orig_line_count, 27,
9, false);
}

TEST_F(FunctionCastToDecimalPerfTest, test_string_to_decimal128v3_strict_mode_perf) {
std::string test_data_file = "/mnt/disk2/tengjianping/cast_perf/decimal_test_data_38_19.txt";
size_t orig_line_count = 10000000;

std::cout << "\n=== Testing with strict_mode=false ===\n";
perf_test_string_to_decimal_strict<PrimitiveType::TYPE_DECIMAL128I>(
test_data_file, orig_line_count, 38, 19, false, false);

std::cout << "\n=== Testing with strict_mode=true ===\n";
perf_test_string_to_decimal_strict<PrimitiveType::TYPE_DECIMAL128I>(
test_data_file, orig_line_count, 38, 19, false, true);
}

// Comparison test: String vs Direct decimal parsing
TEST_F(FunctionCastToDecimalPerfTest, test_string_parse_comparison) {
std::vector<std::string> file_contents;
read_file_to_vector_optimized("/mnt/disk2/tengjianping/cast_perf/decimal_test_data_38_19.txt",
file_contents);

size_t line_count = file_contents.size();
std::cout << "Read " << line_count << " lines" << std::endl;

// Test 1: Direct StringParser
int64_t duration_ns_parser = 0;
StringParser::ParseResult result;
{
SCOPED_RAW_TIMER(&duration_ns_parser);
for (size_t i = 0; i != line_count; ++i) {
auto decimal_value = StringParser::string_to_decimal<PrimitiveType::TYPE_DECIMAL128I>(
file_contents[i].c_str(), file_contents[i].size(), 38, 19, &result);
(void)decimal_value;
}
}
std::cout << "Direct StringParser time: "
<< PrettyPrinter::print(duration_ns_parser, TUnit::TIME_NS) << "\n";

// Test 2: Cast function (already tested in other tests)
std::cout << "\nFor cast function performance, see test_string_to_decimal128v3_perf\n";
}
*/
Copy link

Copilot AI Jan 18, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

These test cases are commented out with a block comment (/* */). If these tests are no longer needed, they should be removed entirely. If they're intended for future use or debugging, consider using a feature flag or test attribute to disable them instead of commenting them out, which makes the codebase harder to maintain.

Copilot uses AI. Check for mistakes.
int max_index = std::min(found_dot ? (result_int_part_digit_count +
((int_part_count > 0 && exponent > 0) ? 1 : 0))
: result_int_part_digit_count,
end_digit_index);
Copy link

Copilot AI Jan 18, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The check for 'std::numeric_limits::min()' seems suspicious. The condition checks if 'max_index == std::numeric_limits::min()', which would only occur if both 'result_int_part_digit_count' and 'end_digit_index' are at their minimum values due to overflow in the ternary operation. This appears to be defensive coding for an edge case, but the logic is unclear. Consider adding a comment explaining when this condition would be true and why setting max_index to end_digit_index is the correct behavior in that case.

Suggested change
end_digit_index);
end_digit_index);
// Defensive check: under normal circumstances max_index is never INT_MIN, because
// result_int_part_digit_count is clamped to the range of int above and end_digit_index
// is derived from the parsed string length. If max_index were ever INT_MIN (e.g. due to
// unexpected overflow or undefined behavior in intermediate computations), fall back to
// using end_digit_index as a safe upper bound for digit iteration.

Copilot uses AI. Check for mistakes.
#include "vec/columns/column_vector.h"
#include "vec/columns/common_column_test.h"
#include "vec/core/field.h"
#include "vec/json/json_parser.h"
Copy link

Copilot AI Jan 18, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The include "vec/json/json_parser.h" is not used anywhere in this file. Remove unused includes to keep the code clean and reduce compilation dependencies.

Suggested change
#include "vec/json/json_parser.h"

Copilot uses AI. Check for mistakes.
Comment on lines +283 to +279
perf_test_string_to_decimal<PrimitiveType::TYPE_DECIMAL64>(test_data_file, orig_line_count, 18,
9, false);
}

TEST_F(FunctionCastToDecimalPerfTest, test_string_to_decimal32v3_perf) {
std::string test_data_file = "/mnt/disk2/tengjianping/cast_perf/decimal_test_data_9_4.txt";
size_t orig_line_count = 10000000;
perf_test_string_to_decimal<PrimitiveType::TYPE_DECIMAL32>(test_data_file, orig_line_count, 9,
4, false);
}

TEST_F(FunctionCastToDecimalPerfTest, test_string_to_decimalv2_perf) {
std::string test_data_file = "/mnt/disk2/tengjianping/cast_perf/decimalv2_test_data_27_9.txt";
size_t orig_line_count = 10000000;
perf_test_string_to_decimal<PrimitiveType::TYPE_DECIMALV2>(test_data_file, orig_line_count, 27,
Copy link

Copilot AI Jan 18, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Type mismatch in function call. The method 'perf_test_string_to_decimal' declared on line 81 expects a 'MutableColumnPtr' as the first parameter, but here it's being called with 'test_data_file' which is a 'std::string'. This code will fail to compile. Either the method signature needs an overload that accepts a string filename, or the test needs to load the data first into a column like the other test cases do.

Suggested change
perf_test_string_to_decimal<PrimitiveType::TYPE_DECIMAL64>(test_data_file, orig_line_count, 18,
9, false);
}
TEST_F(FunctionCastToDecimalPerfTest, test_string_to_decimal32v3_perf) {
std::string test_data_file = "/mnt/disk2/tengjianping/cast_perf/decimal_test_data_9_4.txt";
size_t orig_line_count = 10000000;
perf_test_string_to_decimal<PrimitiveType::TYPE_DECIMAL32>(test_data_file, orig_line_count, 9,
4, false);
}
TEST_F(FunctionCastToDecimalPerfTest, test_string_to_decimalv2_perf) {
std::string test_data_file = "/mnt/disk2/tengjianping/cast_perf/decimalv2_test_data_27_9.txt";
size_t orig_line_count = 10000000;
perf_test_string_to_decimal<PrimitiveType::TYPE_DECIMALV2>(test_data_file, orig_line_count, 27,
// Create string column and load data from file
auto dt_from = std::make_shared<DataTypeString>();
auto column_orig = dt_from->create_column();
{
MutableColumns columns;
columns.push_back(column_orig->get_ptr());
DataTypeSerDeSPtrs serde = {dt_from->get_serde()};
load_columns_data_from_file(columns, serde, ';', {0}, test_data_file);
EXPECT_EQ(column_orig->size(), orig_line_count);
}
perf_test_string_to_decimal<PrimitiveType::TYPE_DECIMAL64>(column_orig, orig_line_count, 18,
9, false);
}
TEST_F(FunctionCastToDecimalPerfTest, test_string_to_decimal32v3_perf) {
std::string test_data_file = "/mnt/disk2/tengjianping/cast_perf/decimal_test_data_9_4.txt";
size_t orig_line_count = 10000000;
// Create string column and load data from file
auto dt_from = std::make_shared<DataTypeString>();
auto column_orig = dt_from->create_column();
{
MutableColumns columns;
columns.push_back(column_orig->get_ptr());
DataTypeSerDeSPtrs serde = {dt_from->get_serde()};
load_columns_data_from_file(columns, serde, ';', {0}, test_data_file);
EXPECT_EQ(column_orig->size(), orig_line_count);
}
perf_test_string_to_decimal<PrimitiveType::TYPE_DECIMAL32>(column_orig, orig_line_count, 9,
4, false);
}
TEST_F(FunctionCastToDecimalPerfTest, test_string_to_decimalv2_perf) {
std::string test_data_file = "/mnt/disk2/tengjianping/cast_perf/decimalv2_test_data_27_9.txt";
size_t orig_line_count = 10000000;
// Create string column and load data from file
auto dt_from = std::make_shared<DataTypeString>();
auto column_orig = dt_from->create_column();
{
MutableColumns columns;
columns.push_back(column_orig->get_ptr());
DataTypeSerDeSPtrs serde = {dt_from->get_serde()};
load_columns_data_from_file(columns, serde, ';', {0}, test_data_file);
EXPECT_EQ(column_orig->size(), orig_line_count);
}
perf_test_string_to_decimal<PrimitiveType::TYPE_DECIMALV2>(column_orig, orig_line_count, 27,

Copilot uses AI. Check for mistakes.
@zclllyybb
Copy link
Contributor

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 32120 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit e959fbc57871c9bb3a642d32790cae1a2c330c51, data reload: false

------ Round 1 ----------------------------------
q1	17622	4231	4052	4052
q2	2105	349	242	242
q3	10125	1294	729	729
q4	10222	892	308	308
q5	7513	2094	1870	1870
q6	196	170	139	139
q7	946	788	643	643
q8	9263	1427	1229	1229
q9	4839	4562	4733	4562
q10	6738	1804	1399	1399
q11	506	295	282	282
q12	654	729	603	603
q13	17767	3902	3118	3118
q14	287	288	279	279
q15	594	509	516	509
q16	670	673	627	627
q17	657	807	562	562
q18	6714	6353	7002	6353
q19	1294	1085	673	673
q20	439	374	246	246
q21	3283	2698	2683	2683
q22	1164	1142	1012	1012
Total cold run time: 103598 ms
Total hot run time: 32120 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4528	4255	4399	4255
q2	341	409	329	329
q3	2263	2707	2377	2377
q4	1454	1823	1411	1411
q5	4493	4464	4353	4353
q6	220	173	130	130
q7	2071	1910	1786	1786
q8	2560	2590	2385	2385
q9	7222	7159	7140	7140
q10	2559	2654	2350	2350
q11	540	497	466	466
q12	686	757	611	611
q13	3709	4067	3116	3116
q14	273	287	262	262
q15	522	478	480	478
q16	630	635	595	595
q17	1098	1244	1247	1244
q18	7590	7374	7136	7136
q19	885	800	794	794
q20	1889	1962	1781	1781
q21	4534	4276	4150	4150
q22	1093	1043	986	986
Total cold run time: 51160 ms
Total hot run time: 48135 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 174390 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit e959fbc57871c9bb3a642d32790cae1a2c330c51, data reload: false

query5	4407	603	493	493
query6	330	244	227	227
query7	4219	454	263	263
query8	339	266	258	258
query9	8710	2917	2915	2915
query10	531	379	338	338
query11	15198	15200	14844	14844
query12	180	118	113	113
query13	1245	479	389	389
query14	6458	3106	2815	2815
query14_1	2664	2615	2645	2615
query15	190	187	172	172
query16	986	484	461	461
query17	1066	640	536	536
query18	2569	417	318	318
query19	216	218	193	193
query20	120	118	110	110
query21	212	138	112	112
query22	4046	3994	3920	3920
query23	16031	15641	15466	15466
query23_1	15630	15479	15399	15399
query24	7084	1602	1178	1178
query24_1	1153	1175	1188	1175
query25	499	425	401	401
query26	1257	266	162	162
query27	2775	450	286	286
query28	4558	2192	2166	2166
query29	752	546	448	448
query30	310	243	207	207
query31	802	633	588	588
query32	87	76	79	76
query33	553	342	313	313
query34	919	867	542	542
query35	728	754	681	681
query36	846	873	821	821
query37	133	95	84	84
query38	2737	2756	2654	2654
query39	763	743	745	743
query39_1	717	715	696	696
query40	219	140	125	125
query41	76	67	67	67
query42	108	103	111	103
query43	488	478	403	403
query44	1354	766	755	755
query45	186	193	183	183
query46	840	958	574	574
query47	1421	1431	1350	1350
query48	328	327	245	245
query49	617	438	347	347
query50	622	272	205	205
query51	3774	3796	3836	3796
query52	103	109	98	98
query53	300	331	278	278
query54	331	289	266	266
query55	83	80	78	78
query56	303	300	326	300
query57	1050	996	954	954
query58	294	272	264	264
query59	2022	2207	2035	2035
query60	351	341	316	316
query61	182	173	174	173
query62	414	351	315	315
query63	301	276	267	267
query64	4942	1309	946	946
query65	3726	3742	3692	3692
query66	1423	416	304	304
query67	15700	15568	15616	15568
query68	2626	1125	747	747
query69	444	360	330	330
query70	950	946	943	943
query71	314	306	294	294
query72	5389	3251	3233	3233
query73	605	723	308	308
query74	8744	8864	8563	8563
query75	2747	2807	2468	2468
query76	2393	1085	644	644
query77	353	385	325	325
query78	9781	9926	9182	9182
query79	1063	907	584	584
query80	1317	582	490	490
query81	544	263	235	235
query82	1342	144	111	111
query83	336	266	242	242
query84	251	118	92	92
query85	945	484	419	419
query86	412	296	329	296
query87	2835	2889	2722	2722
query88	3486	2598	2577	2577
query89	400	350	335	335
query90	1957	172	164	164
query91	168	151	134	134
query92	74	77	71	71
query93	958	915	558	558
query94	633	327	297	297
query95	591	332	373	332
query96	652	525	227	227
query97	2349	2404	2354	2354
query98	226	201	202	201
query99	582	601	508	508
Total cold run time: 248779 ms
Total hot run time: 174390 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 27.05 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit e959fbc57871c9bb3a642d32790cae1a2c330c51, data reload: false

query1	0.06	0.05	0.05
query2	0.10	0.05	0.04
query3	0.26	0.08	0.09
query4	1.60	0.11	0.12
query5	0.28	0.26	0.25
query6	1.16	0.66	0.65
query7	0.03	0.03	0.03
query8	0.05	0.05	0.04
query9	0.58	0.50	0.50
query10	0.55	0.55	0.53
query11	0.14	0.09	0.10
query12	0.14	0.11	0.10
query13	0.61	0.58	0.58
query14	0.96	0.94	0.94
query15	0.78	0.77	0.80
query16	0.39	0.40	0.40
query17	1.05	1.05	1.01
query18	0.24	0.21	0.21
query19	1.84	1.79	1.89
query20	0.02	0.01	0.02
query21	15.43	0.26	0.14
query22	5.34	0.05	0.05
query23	16.04	0.29	0.10
query24	0.92	0.77	0.62
query25	0.08	0.10	0.13
query26	0.14	0.13	0.12
query27	0.09	0.06	0.07
query28	4.32	1.10	0.89
query29	12.55	3.84	3.14
query30	0.28	0.14	0.11
query31	2.81	0.63	0.39
query32	3.23	0.56	0.45
query33	3.06	3.03	3.08
query34	16.04	5.12	4.43
query35	4.52	4.47	4.41
query36	0.66	0.50	0.49
query37	0.10	0.06	0.07
query38	0.07	0.04	0.04
query39	0.04	0.02	0.02
query40	0.17	0.15	0.13
query41	0.09	0.03	0.04
query42	0.04	0.03	0.03
query43	0.04	0.04	0.03
Total cold run time: 96.9 s
Total hot run time: 27.05 s

@doris-robot
Copy link

BE UT Coverage Report

Increment line coverage 97.56% (40/41) 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 52.43% (19053/36337)
Line Coverage 35.72% (176439/493953)
Region Coverage 32.27% (136627/423438)
Branch Coverage 33.16% (59070/178136)

@hello-stephen
Copy link
Contributor

BE Regression && UT Coverage Report

Increment line coverage 97.56% (40/41) 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 71.41% (25434/35616)
Line Coverage 53.95% (266181/493395)
Region Coverage 51.70% (221246/427941)
Branch Coverage 53.00% (94817/178897)

@jacktengg jacktengg force-pushed the 260105-improve-decimal-cast branch from e959fbc to 47ead66 Compare January 22, 2026 14:27
@jacktengg jacktengg changed the title [improvement](cast) improve performance of casting string to decimal [improvement](cast) improve performance of casting string to decimal and int Jan 22, 2026
@jacktengg jacktengg force-pushed the 260105-improve-decimal-cast branch from 47ead66 to 81180ec Compare January 22, 2026 14:31
@jacktengg
Copy link
Contributor Author

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 31759 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 81180ec4e9e8e689bbd7337d117251dcd8bd3f20, data reload: false

------ Round 1 ----------------------------------
q1	17632	4787	4559	4559
q2	2038	304	193	193
q3	10234	1432	725	725
q4	10205	812	304	304
q5	7509	2175	1886	1886
q6	188	175	141	141
q7	932	730	594	594
q8	9283	1461	1149	1149
q9	5049	4613	4710	4613
q10	6821	1695	1278	1278
q11	522	299	299	299
q12	399	409	226	226
q13	17817	3828	3079	3079
q14	249	232	211	211
q15	618	531	535	531
q16	644	664	589	589
q17	665	821	511	511
q18	6757	6648	7047	6648
q19	1445	1001	671	671
q20	425	364	249	249
q21	3055	2302	2223	2223
q22	1107	1080	1100	1080
Total cold run time: 103594 ms
Total hot run time: 31759 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4982	5188	4940	4940
q2	342	416	336	336
q3	2376	2848	2424	2424
q4	1428	1859	1410	1410
q5	4415	4450	4421	4421
q6	228	170	124	124
q7	2080	1988	1841	1841
q8	2679	2386	2355	2355
q9	7256	7237	7220	7220
q10	2553	2657	2297	2297
q11	545	478	445	445
q12	695	752	638	638
q13	3575	4057	3087	3087
q14	270	288	271	271
q15	536	494	498	494
q16	614	665	604	604
q17	1089	1307	1268	1268
q18	7534	7234	7258	7234
q19	823	822	804	804
q20	1884	1961	1831	1831
q21	4496	4282	4195	4195
q22	1047	1058	985	985
Total cold run time: 51447 ms
Total hot run time: 49224 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 173183 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 81180ec4e9e8e689bbd7337d117251dcd8bd3f20, data reload: false

query5	4394	637	485	485
query6	330	212	200	200
query7	4220	474	270	270
query8	359	260	258	258
query9	8700	2909	2879	2879
query10	442	329	299	299
query11	15223	15069	14942	14942
query12	175	119	115	115
query13	1259	474	365	365
query14	5751	3054	2818	2818
query14_1	2698	2655	2699	2655
query15	212	200	175	175
query16	970	444	456	444
query17	1092	682	565	565
query18	2424	430	333	333
query19	192	176	155	155
query20	124	119	115	115
query21	220	144	120	120
query22	4153	4142	4293	4142
query23	16105	15522	15321	15321
query23_1	15432	15422	15432	15422
query24	7154	1534	1185	1185
query24_1	1166	1167	1177	1167
query25	532	453	413	413
query26	1258	264	155	155
query27	2781	453	281	281
query28	4597	2175	2153	2153
query29	792	544	462	462
query30	313	239	206	206
query31	759	632	564	564
query32	92	78	73	73
query33	526	385	330	330
query34	940	962	545	545
query35	719	766	678	678
query36	902	896	845	845
query37	140	101	94	94
query38	2743	2667	2651	2651
query39	793	746	735	735
query39_1	715	716	722	716
query40	232	137	130	130
query41	84	68	67	67
query42	98	92	98	92
query43	458	489	427	427
query44	1359	756	755	755
query45	190	185	180	180
query46	843	959	624	624
query47	1385	1491	1457	1457
query48	311	320	232	232
query49	604	441	339	339
query50	713	272	210	210
query51	3916	3797	3817	3797
query52	92	94	87	87
query53	211	237	162	162
query54	283	254	239	239
query55	79	74	74	74
query56	287	303	304	303
query57	997	1041	974	974
query58	267	257	259	257
query59	2085	2084	2081	2081
query60	337	329	301	301
query61	146	139	142	139
query62	397	353	316	316
query63	189	163	156	156
query64	4918	1147	824	824
query65	3832	3758	3788	3758
query66	1452	404	319	319
query67	15466	15500	15402	15402
query68	2411	1115	715	715
query69	383	308	287	287
query70	963	1012	903	903
query71	302	284	265	265
query72	5304	3357	3242	3242
query73	600	732	320	320
query74	8724	8770	8498	8498
query75	2256	2381	1911	1911
query76	2275	1055	659	659
query77	352	383	295	295
query78	9745	9956	9107	9107
query79	1060	906	575	575
query80	1305	518	429	429
query81	528	271	235	235
query82	1011	152	119	119
query83	327	261	242	242
query84	258	117	101	101
query85	869	458	409	409
query86	430	337	290	290
query87	2861	2862	2742	2742
query88	3504	2594	2559	2559
query89	306	270	243	243
query90	2036	172	151	151
query91	159	157	129	129
query92	71	71	70	70
query93	1107	1034	641	641
query94	634	331	307	307
query95	585	388	314	314
query96	648	498	227	227
query97	2362	2372	2306	2306
query98	208	207	206	206
query99	599	575	497	497
Total cold run time: 245467 ms
Total hot run time: 173183 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 26.88 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 81180ec4e9e8e689bbd7337d117251dcd8bd3f20, data reload: false

query1	0.06	0.05	0.05
query2	0.10	0.05	0.05
query3	0.26	0.09	0.09
query4	1.60	0.12	0.11
query5	0.28	0.26	0.27
query6	1.15	0.66	0.66
query7	0.04	0.03	0.03
query8	0.06	0.04	0.05
query9	0.56	0.49	0.49
query10	0.56	0.55	0.54
query11	0.15	0.10	0.10
query12	0.14	0.10	0.11
query13	0.60	0.59	0.59
query14	0.97	0.95	0.92
query15	0.80	0.78	0.79
query16	0.39	0.42	0.40
query17	0.99	1.07	1.01
query18	0.22	0.21	0.22
query19	1.97	1.88	1.81
query20	0.02	0.01	0.02
query21	15.43	0.24	0.14
query22	4.92	0.06	0.05
query23	15.85	0.29	0.11
query24	1.48	0.36	0.57
query25	0.08	0.08	0.08
query26	0.14	0.13	0.14
query27	0.08	0.06	0.05
query28	4.17	1.07	0.89
query29	12.52	3.94	3.16
query30	0.31	0.17	0.13
query31	2.82	0.60	0.38
query32	3.24	0.54	0.46
query33	3.07	2.99	3.02
query34	15.81	5.07	4.42
query35	4.50	4.41	4.47
query36	0.65	0.50	0.49
query37	0.11	0.07	0.07
query38	0.08	0.05	0.04
query39	0.05	0.04	0.03
query40	0.17	0.15	0.14
query41	0.10	0.04	0.03
query42	0.05	0.04	0.03
query43	0.06	0.04	0.05
Total cold run time: 96.61 s
Total hot run time: 26.88 s

zclllyybb
zclllyybb previously approved these changes Jan 25, 2026
Copy link
Contributor

@zclllyybb zclllyybb left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Jan 25, 2026
@github-actions
Copy link
Contributor

PR approved by at least one committer and no changes requested.

@github-actions
Copy link
Contributor

PR approved by anyone and no changes requested.

@jacktengg
Copy link
Contributor Author

run buildall

@github-actions github-actions bot removed the approved Indicates a PR has been approved by one committer. label Jan 26, 2026
@doris-robot
Copy link

TPC-H: Total hot run time: 32410 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 070404164c2f81fe763ed228618d40f99e4dc78e, data reload: false

------ Round 1 ----------------------------------
q1	17643	5207	5056	5056
q2	2000	331	192	192
q3	10219	1330	732	732
q4	10208	819	312	312
q5	7566	2231	1856	1856
q6	236	183	148	148
q7	905	722	609	609
q8	9270	1421	1151	1151
q9	5518	4895	4745	4745
q10	6859	1952	1575	1575
q11	532	294	281	281
q12	390	373	221	221
q13	17794	4055	3253	3253
q14	239	240	212	212
q15	916	813	820	813
q16	666	677	623	623
q17	645	841	465	465
q18	7044	6495	6472	6472
q19	1469	1003	624	624
q20	399	344	229	229
q21	2694	1938	1873	1873
q22	1068	1032	968	968
Total cold run time: 104280 ms
Total hot run time: 32410 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5420	5302	5319	5302
q2	267	333	262	262
q3	2171	2658	2291	2291
q4	1332	1752	1313	1313
q5	4315	4224	4628	4224
q6	262	205	158	158
q7	2203	2010	1800	1800
q8	2606	2424	2456	2424
q9	7412	7572	7713	7572
q10	2870	2993	2688	2688
q11	546	474	448	448
q12	662	712	651	651
q13	3948	4314	3734	3734
q14	305	318	306	306
q15	888	856	792	792
q16	687	746	694	694
q17	1202	1423	1351	1351
q18	8330	7806	8170	7806
q19	871	817	853	817
q20	2063	2288	1960	1960
q21	4467	4256	3985	3985
q22	1063	1029	963	963
Total cold run time: 53890 ms
Total hot run time: 51541 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 28.47 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 070404164c2f81fe763ed228618d40f99e4dc78e, data reload: false

query1	0.05	0.04	0.05
query2	0.09	0.05	0.05
query3	0.24	0.08	0.08
query4	1.61	0.12	0.11
query5	0.28	0.25	0.25
query6	1.17	0.67	0.67
query7	0.03	0.03	0.03
query8	0.05	0.04	0.04
query9	0.57	0.51	0.48
query10	0.56	0.53	0.56
query11	0.15	0.09	0.10
query12	0.14	0.10	0.11
query13	0.64	0.62	0.62
query14	1.08	1.08	1.04
query15	0.88	0.86	0.87
query16	0.40	0.40	0.40
query17	1.08	1.12	1.09
query18	0.23	0.21	0.21
query19	2.02	1.97	2.06
query20	0.02	0.01	0.01
query21	15.39	0.27	0.14
query22	5.22	0.04	0.04
query23	16.20	0.28	0.10
query24	1.67	0.67	0.51
query25	0.11	0.08	0.06
query26	0.13	0.14	0.13
query27	0.08	0.05	0.07
query28	4.53	1.14	0.96
query29	12.58	3.90	3.13
query30	0.28	0.13	0.12
query31	2.82	0.63	0.41
query32	3.22	0.60	0.49
query33	3.21	3.25	3.32
query34	16.72	5.39	4.74
query35	4.80	4.77	4.80
query36	0.66	0.50	0.50
query37	0.11	0.07	0.07
query38	0.06	0.04	0.04
query39	0.05	0.03	0.03
query40	0.19	0.17	0.16
query41	0.08	0.03	0.03
query42	0.04	0.03	0.03
query43	0.04	0.04	0.04
Total cold run time: 99.48 s
Total hot run time: 28.47 s

@jacktengg jacktengg force-pushed the 260105-improve-decimal-cast branch from 0704041 to 46d6893 Compare January 26, 2026 15:04
@jacktengg
Copy link
Contributor Author

run buildall

1 similar comment
@jacktengg
Copy link
Contributor Author

run buildall

@jacktengg jacktengg force-pushed the 260105-improve-decimal-cast branch from 870e534 to f24df0d Compare January 26, 2026 15:50
@jacktengg
Copy link
Contributor Author

run buildall

// and update the len to the new length, which does not include
// trailing whitespaces
template <typename T>
inline const char* skip_trailing_whitespaces(const char* s, T& len) {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

can it be abstracted by skip_qualified_char?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm afraid not, seems skip_qualified_char can only skip chars from begining, skip_trailing_whitespaces skips white spaces from end.

@doris-robot
Copy link

TPC-H: Total hot run time: 32659 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit f24df0d0f7cd8c3af91a3190e34bf94b4825d29c, data reload: false

------ Round 1 ----------------------------------
q1	17707	5216	5045	5045
q2	2055	314	192	192
q3	10190	1334	799	799
q4	10211	915	324	324
q5	7738	2175	1924	1924
q6	254	186	150	150
q7	883	729	627	627
q8	9269	1372	1061	1061
q9	5473	4817	4931	4817
q10	6814	1965	1584	1584
q11	510	293	280	280
q12	349	375	223	223
q13	17796	4068	3284	3284
q14	239	245	217	217
q15	914	833	810	810
q16	676	667	623	623
q17	639	838	474	474
q18	6920	6452	6476	6452
q19	1562	992	639	639
q20	398	353	233	233
q21	2630	2057	1951	1951
q22	1058	1009	950	950
Total cold run time: 104285 ms
Total hot run time: 32659 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5411	5317	5351	5317
q2	259	342	269	269
q3	2170	2690	2266	2266
q4	1347	1781	1287	1287
q5	4280	4155	4889	4155
q6	243	200	155	155
q7	2239	2015	1838	1838
q8	2555	2449	2404	2404
q9	7453	7697	7579	7579
q10	2878	3013	2644	2644
q11	553	479	450	450
q12	707	771	634	634
q13	3985	4462	3613	3613
q14	291	320	303	303
q15	918	849	811	811
q16	702	724	728	724
q17	1251	1388	1346	1346
q18	8295	7823	7757	7757
q19	842	872	833	833
q20	2106	2177	1942	1942
q21	4538	4235	4132	4132
q22	1054	1060	966	966
Total cold run time: 54077 ms
Total hot run time: 51425 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 28.25 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit f24df0d0f7cd8c3af91a3190e34bf94b4825d29c, data reload: false

query1	0.06	0.04	0.05
query2	0.10	0.04	0.04
query3	0.26	0.08	0.08
query4	1.61	0.11	0.12
query5	0.28	0.25	0.26
query6	1.16	0.68	0.68
query7	0.03	0.02	0.03
query8	0.06	0.04	0.04
query9	0.57	0.50	0.48
query10	0.56	0.54	0.54
query11	0.14	0.09	0.10
query12	0.13	0.10	0.10
query13	0.64	0.62	0.62
query14	1.07	1.06	1.06
query15	0.88	0.86	0.86
query16	0.39	0.39	0.39
query17	1.10	1.14	1.11
query18	0.23	0.20	0.21
query19	2.08	2.05	2.02
query20	0.02	0.01	0.02
query21	15.44	0.26	0.15
query22	5.30	0.06	0.05
query23	16.29	0.28	0.10
query24	2.73	0.27	0.29
query25	0.09	0.11	0.09
query26	0.13	0.14	0.14
query27	0.10	0.05	0.05
query28	3.79	1.15	0.96
query29	12.56	3.93	3.12
query30	0.27	0.13	0.11
query31	2.82	0.65	0.40
query32	3.24	0.60	0.49
query33	3.29	3.22	3.23
query34	16.18	5.37	4.75
query35	4.82	4.75	4.80
query36	0.67	0.50	0.49
query37	0.11	0.06	0.06
query38	0.08	0.04	0.04
query39	0.04	0.03	0.03
query40	0.18	0.16	0.15
query41	0.09	0.03	0.04
query42	0.05	0.03	0.03
query43	0.05	0.04	0.03
Total cold run time: 99.69 s
Total hot run time: 28.25 s

@hello-stephen
Copy link
Contributor

BE UT Coverage Report

Increment line coverage 98.77% (80/81) 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 52.48% (19144/36481)
Line Coverage 35.87% (177963/496174)
Region Coverage 32.30% (137410/425455)
Branch Coverage 33.26% (59536/178993)

@hello-stephen
Copy link
Contributor

BE Regression && UT Coverage Report

Increment line coverage 98.77% (80/81) 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 71.43% (25541/35758)
Line Coverage 54.04% (267829/495630)
Region Coverage 51.62% (221930/429960)
Branch Coverage 53.08% (95417/179761)

@jacktengg
Copy link
Contributor Author

run p0

@github-actions
Copy link
Contributor

PR approved by at least one committer and no changes requested.

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Jan 27, 2026
@yiguolei yiguolei merged commit c108263 into apache:master Jan 27, 2026
30 of 34 checks passed
github-actions bot pushed a commit that referenced this pull request Jan 27, 2026
…and int (#60004)

### What problem does this PR solve?

Issue Number: close #xxx

Related PR: #xxx

Problem Summary:

### Release note

None

### Check List (For Author)

- Test <!-- At least one of them must be included. -->
    - [ ] Regression test
    - [ ] Unit Test
    - [ ] Manual test (add detailed scripts or steps below)
    - [ ] No need to test or manual test. Explain why:
- [ ] This is a refactor/code format and no logic has been changed.
        - [ ] Previous test can cover this change.
        - [ ] No code files have been changed.
        - [ ] Other reason <!-- Add your reason?  -->

- Behavior changed:
    - [ ] No.
    - [ ] Yes. <!-- Explain the behavior change -->

- Does this need documentation?
    - [ ] No.
- [ ] Yes. <!-- Add document PR link here. eg:
apache/doris-website#1214 -->

### Check List (For Reviewer who merge this PR)

- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label <!-- Add branch pick label that this PR
should merge into -->
yiguolei pushed a commit that referenced this pull request Jan 28, 2026
…and int (#60004) (#60159)

### What problem does this PR solve?

Issue Number: close #xxx

Pick #60004

Problem Summary:

### Release note

None

### Check List (For Author)

- Test <!-- At least one of them must be included. -->
    - [ ] Regression test
    - [ ] Unit Test
    - [ ] Manual test (add detailed scripts or steps below)
    - [ ] No need to test or manual test. Explain why:
- [ ] This is a refactor/code format and no logic has been changed.
        - [ ] Previous test can cover this change.
        - [ ] No code files have been changed.
        - [ ] Other reason <!-- Add your reason?  -->

- Behavior changed:
    - [ ] No.
    - [ ] Yes. <!-- Explain the behavior change -->

- Does this need documentation?
    - [ ] No.
- [ ] Yes. <!-- Add document PR link here. eg:
apache/doris-website#1214 -->

### Check List (For Reviewer who merge this PR)

- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label <!-- Add branch pick label that this PR
should merge into -->
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

approved Indicates a PR has been approved by one committer. dev/4.0.3-merged reviewed

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants