REST: Configure shared storage credential refresh endpoints

[zhangxinyao88]

Fixes #17810.

Scheduled storage credential refresh can fail when FileIO receives REST catalog properties without a resolved credentials URI. Resolve the table credentials endpoint in RESTSessionCatalog when vended credentials are present, and pass it through a shared rest.credentials.endpoint property. AWS and GCP prefer this endpoint and retain their legacy refresh properties as fallbacks.

Testing

• Full TestRESTCatalog suite, including endpoint configuration with and without storage credentials.
• AWS/GCP credential configuration, endpoint precedence, legacy compatibility, and consecutive scheduled refresh tests.
• Formatting checks for core, AWS, and GCP.

---

AI Disclosure

• Model: GPT-6
• Platform/Tool: Codex
• Human Oversight: unreviewed
• Prompt Summary: Address review feedback by configuring a shared storage credential refresh endpoint in RESTSessionCatalog, retaining legacy compatibility, and adding regression coverage.

[uros-b]

The fix looks correct and minimal. It resolves the property that actually carries the server-supplied relative value against the operator-trusted catalog uri, mirroring the existing AwsClientProperties resolution exactly, with both putIfAbsent branches covered and no new credential-exposure surface. Thank you @zhangxinyao88, please ping the expert maintainers for further review!

[zhangxinyao88]

The fix looks correct and minimal. It resolves the property that actually carries the server-supplied relative value against the operator-trusted catalog uri, mirroring the existing AwsClientProperties resolution exactly, with both putIfAbsent branches covered and no new credential-exposure surface. Thank you @zhangxinyao88, please ping the expert maintainers for further review!

Thanks @uros-b! @nastra, would you mind taking a look when you get a chance? This is a small fix around resolving the relative storage credential refresh endpoint in S3FileIO.

[nastra]

we are doing this already in

iceberg/aws/src/main/java/org/apache/iceberg/aws/AwsClientProperties.java

Lines 120 to 122 in 48da548

	this.refreshCredentialsEndpoint =
	RESTUtil.resolveEndpoint(
	properties.get(CatalogProperties.URI), properties.get(REFRESH_CREDENTIALS_ENDPOINT));

Why do we need to do it here again?

[zhangxinyao88]

You're right, thanks. AwsClientProperties handles the initial client setup, but the scheduled refresh creates VendedCredentialsProvider from the raw FileIO properties. I moved the resolution into the provider so both paths use it, and added a relative URI test in 9480069.

[danielcweeks]

I don't feel like this is the right place to configure this. It seems like a better place is in the RESTSessionCatalog when we process the storage credentials and configure the FileIO. This isn't specific to S3FileIO, so finding the right place up the stack would make more sense.

[nastra]

I guess we would have the same issue with GCSFileIO when its refreshStorageCredentials() is being called, right? Maybe we should fix both places then

[zhangxinyao88]

Yes, GCS had the same gap. Its refresh path also uses raw properties, so I added the same resolution in OAuth2RefreshCredentialsHandler, with a regression test in 9480069.

[zhangxinyao88]

@nastra @danielcweeks, could one of you take a look and merge if everything looks good? Nastra has approved it, CI passes, and it merges cleanly with current main.

[danielcweeks]

I think this would be unnecessary if we do the resolution in the RESTSessionCatalog during configuration of the File IO.

[danielcweeks]

@zhangxinyao88 It feels like this resolution logic is being pushed down too far. There's nothing specific to the FileIO and can be handled when the IO is constructed. We should also only configure the refresh if there are actually credentials (e.g. if using remote signing, we don't need the refresh path).

Looking a little closer, I now see that each of the FileIOs are using a different property to identify the refresh URI, which I don't think is the right way to handle this. There really should only be on property which resolves to the ../{table}/credentials endpoint.

I would recommend that we create a single property for the refresh endpoint and resolve it when we create the FileIO in the RESTSessionCatalog. That would then flow down to the FileIO and we can resolve to that property (we can deprecate the FileIO properties or fallback to those for backward compatibility). I think this messiness is largely due to how the refresh evolved.

[danielcweeks]

I'm not sure why this is a cloud property specific field. It seems like this should be the same for all IO implementations.

[zhangxinyao88]

I would recommend that we create a single property for the refresh endpoint and resolve it when we create the FileIO in the RESTSessionCatalog.

Moved this into RESTSessionCatalog: it sets rest.credentials.endpoint only when storage credentials are present. AWS and GCP now use that endpoint, with the existing properties retained as fallbacks; tests cover both paths and the case without storage credentials.