Custom column sorting during optimization for Iceberg tables

[owenmonn]

Hi Amoro team, I have a question about the sorting capabilities during table optimization.

Current Behavior

I found that optimization only sorts data by:

• Partition columns (primary sorting)
• Primary key columns (secondary sorting, only for Mixed-Iceberg tables)

amoro/amoro-format-iceberg/src/main/java/org/apache/amoro/io/writer/BaseTaskWriter.java

Lines 97 to 108 in c1702c2

	protected DataWriterKey buildWriterKey(T row) {
	StructLike structLike = asStructLike(row);
	partitionKey.partition(structLike);
	DataTreeNode node;
	if (primaryKey != null) {
	primaryKey.primaryKey(structLike);
	node = primaryKey.treeNode(mask);
	} else {
	node = DataTreeNode.ROOT;
	}
	return new DataWriterKey(partitionKey, node, DataFileType.BASE_FILE);
	}

Issue

Since regular Iceberg tables don't have primary keys, they can only be sorted by partition columns during optimization. This limits the optimization effectiveness for queries that filter on non-partition columns.

Questions

1. Is my understanding correct that custom column-based sorting during optimization is not currently supported for Iceberg tables?
2. Are there any plans to support user-specified column sorting during optimization? I noticed issue [Feature]: Support table sorting process #1360 was closed when Spark optimizer was added, but it seems the Spark optimizer doesn't provide this functionality either.

Thank you for your time and consideration.

[zhoujinsong]

Hi, I have reopened #1360.

Yes, Amoro has not supported sorting in the optimization process.
Sorting requires the external optimizing process, and some developers are working on this.

[owenmonn]

@zhoujinsong

Hi, thank you for the update.
I’m interested in following the progress on this feature.

Is there a way to track the current development status?

Do you expect this feature to be included in the upcoming release, or is it planned for a later one?

Thanks again for your time and support.