Skip to content

Conversation

@cambyzju
Copy link
Contributor

What problem does this PR solve?

When there are large of columns(such as 3W columns) in table, setColumnPositionMapping function will be the main performance bottleneck.

this pr opt it from $O(N \times M)$ to $O(N + M)$

Clipboard_Screenshot_1768746220

Release note

None

Check List (For Author)

  • Test

    • Regression test
    • Unit Test
    • Manual test (add detailed scripts or steps below)
    • No need to test or manual test. Explain why:
      • This is a refactor/code format and no logic has been changed.
      • Previous test can cover this change.
      • No code files have been changed.
      • Other reason
  • Behavior changed:

    • No.
    • Yes.
  • Does this need documentation?

    • No.
    • Yes.

Check List (For Reviewer who merge this PR)

  • Confirm the release note
  • Confirm test cases
  • Confirm document
  • Add branch pick label

@Thearas
Copy link
Contributor

Thearas commented Jan 18, 2026

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@cambyzju
Copy link
Contributor Author

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 32032 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 1966db5712fb95c0ef33e078b40188d279ff6ea2, data reload: false

------ Round 1 ----------------------------------
q1	17631	4278	4042	4042
q2	2041	364	255	255
q3	10133	1280	710	710
q4	10206	811	305	305
q5	7543	2061	1820	1820
q6	188	170	137	137
q7	897	775	667	667
q8	9292	1393	1228	1228
q9	4893	4559	4667	4559
q10	6718	1782	1403	1403
q11	533	298	266	266
q12	696	740	566	566
q13	17791	3807	3112	3112
q14	295	298	272	272
q15	586	503	507	503
q16	680	700	629	629
q17	650	792	487	487
q18	6617	6496	6715	6496
q19	1362	1099	678	678
q20	443	393	252	252
q21	3214	2595	2623	2595
q22	1165	1110	1050	1050
Total cold run time: 103574 ms
Total hot run time: 32032 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4440	4264	4358	4264
q2	339	430	340	340
q3	2296	2856	2465	2465
q4	1506	1899	1493	1493
q5	4532	4325	4270	4270
q6	211	169	130	130
q7	2027	2028	1705	1705
q8	2769	2378	2370	2370
q9	7324	7419	7138	7138
q10	2478	2681	2271	2271
q11	534	475	454	454
q12	681	741	611	611
q13	3609	4010	3100	3100
q14	276	281	254	254
q15	519	489	481	481
q16	614	648	610	610
q17	1091	1228	1306	1228
q18	7532	7331	7209	7209
q19	840	787	795	787
q20	1880	2017	1792	1792
q21	4537	4238	4108	4108
q22	1063	1031	1000	1000
Total cold run time: 51098 ms
Total hot run time: 48080 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 174521 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 1966db5712fb95c0ef33e078b40188d279ff6ea2, data reload: false

query5	5014	610	489	489
query6	331	221	215	215
query7	4221	452	269	269
query8	328	242	233	233
query9	8751	2890	2907	2890
query10	556	376	314	314
query11	15358	15298	14915	14915
query12	188	111	114	111
query13	1239	502	381	381
query14	6481	3036	2832	2832
query14_1	2691	2636	2705	2636
query15	221	194	174	174
query16	1009	488	484	484
query17	1059	658	566	566
query18	2703	430	343	343
query19	227	217	201	201
query20	126	116	115	115
query21	221	140	119	119
query22	4235	4056	4230	4056
query23	16075	15671	15337	15337
query23_1	15562	15596	15421	15421
query24	7124	1578	1169	1169
query24_1	1178	1194	1191	1191
query25	544	452	430	430
query26	1245	264	151	151
query27	2744	443	278	278
query28	4566	2190	2191	2190
query29	748	516	409	409
query30	309	237	205	205
query31	819	628	577	577
query32	79	74	69	69
query33	533	346	310	310
query34	911	879	529	529
query35	727	748	678	678
query36	878	914	835	835
query37	133	100	84	84
query38	2700	2721	2666	2666
query39	770	745	736	736
query39_1	744	717	703	703
query40	208	134	118	118
query41	67	62	65	62
query42	111	105	103	103
query43	440	428	414	414
query44	1320	750	746	746
query45	182	184	180	180
query46	818	934	566	566
query47	1430	1483	1333	1333
query48	323	318	247	247
query49	607	453	351	351
query50	627	283	208	208
query51	3892	3790	3867	3790
query52	101	105	94	94
query53	285	324	279	279
query54	287	265	259	259
query55	80	78	78	78
query56	305	291	311	291
query57	1051	962	888	888
query58	268	257	269	257
query59	1992	2117	1968	1968
query60	337	338	317	317
query61	150	146	146	146
query62	380	363	311	311
query63	300	263	267	263
query64	4824	1237	954	954
query65	3843	3837	3775	3775
query66	1388	424	307	307
query67	15639	15557	15396	15396
query68	2430	1095	783	783
query69	424	367	323	323
query70	989	871	917	871
query71	330	312	302	302
query72	5292	3313	3364	3313
query73	617	720	316	316
query74	8821	8825	8658	8658
query75	2763	2835	2471	2471
query76	2304	1058	648	648
query77	367	391	315	315
query78	9772	9993	9159	9159
query79	1077	924	600	600
query80	1465	591	537	537
query81	520	270	232	232
query82	1352	146	112	112
query83	362	250	244	244
query84	250	122	94	94
query85	1020	484	430	430
query86	389	302	289	289
query87	2922	2842	2778	2778
query88	3467	2584	2561	2561
query89	405	344	322	322
query90	1928	173	164	164
query91	166	168	133	133
query92	76	76	73	73
query93	941	891	553	553
query94	617	313	303	303
query95	599	398	320	320
query96	633	504	232	232
query97	2349	2432	2328	2328
query98	212	206	202	202
query99	597	588	545	545
Total cold run time: 249173 ms
Total hot run time: 174521 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 26.89 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 1966db5712fb95c0ef33e078b40188d279ff6ea2, data reload: false

query1	0.05	0.05	0.05
query2	0.10	0.05	0.04
query3	0.26	0.09	0.09
query4	1.60	0.12	0.11
query5	0.28	0.26	0.26
query6	1.15	0.67	0.66
query7	0.03	0.02	0.02
query8	0.05	0.04	0.04
query9	0.57	0.50	0.50
query10	0.54	0.56	0.56
query11	0.14	0.10	0.10
query12	0.15	0.11	0.12
query13	0.61	0.59	0.59
query14	0.95	0.95	0.93
query15	0.80	0.78	0.78
query16	0.38	0.43	0.39
query17	1.02	1.02	1.04
query18	0.22	0.21	0.21
query19	1.94	1.90	1.84
query20	0.02	0.03	0.01
query21	15.47	0.29	0.15
query22	4.98	0.06	0.05
query23	15.75	0.28	0.10
query24	1.58	0.23	0.25
query25	0.10	0.05	0.05
query26	0.14	0.14	0.14
query27	0.08	0.09	0.08
query28	3.60	1.09	0.88
query29	12.54	3.93	3.18
query30	0.28	0.14	0.12
query31	2.81	0.63	0.40
query32	3.25	0.56	0.47
query33	3.02	3.05	3.03
query34	16.55	5.09	4.46
query35	4.53	4.43	4.47
query36	0.65	0.50	0.48
query37	0.10	0.07	0.06
query38	0.07	0.04	0.03
query39	0.04	0.03	0.02
query40	0.16	0.14	0.13
query41	0.09	0.04	0.03
query42	0.04	0.03	0.03
query43	0.04	0.04	0.03
Total cold run time: 96.73 s
Total hot run time: 26.89 s

@hello-stephen
Copy link
Contributor

FE UT Coverage Report

Increment line coverage 0.00% (0/6) 🎉
Increment coverage report
Complete coverage report

@cambyzju
Copy link
Contributor Author

run p0

@cambyzju
Copy link
Contributor Author

run cloud_p0

@cambyzju
Copy link
Contributor Author

run external

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants