Skip to content

feat(io): Vended credential refresh for S3#3351

Open
gabeiglio wants to merge 2 commits into
apache:mainfrom
gabeiglio:vended-credential-refresh
Open

feat(io): Vended credential refresh for S3#3351
gabeiglio wants to merge 2 commits into
apache:mainfrom
gabeiglio:vended-credential-refresh

Conversation

@gabeiglio
Copy link
Copy Markdown
Contributor

@gabeiglio gabeiglio commented May 11, 2026

Rationale for this change

When doing long writes to a table (like ML training) the credentials vended with the last load table can be expired if the commit takes too long.

Here as same as Java, we proactively refresh the s3 vended credentials if they are 5 minutes away or less from expiration.

Are these changes tested?

Yes, added tests to tests/io/test_io.py to test all the logic for when its needed to call the vended credentials endpoint from the file io

Are there any user-facing changes?

Yes, users can now add the following properties to allow automatic vended credential refresh to take place

client.refresh-credentials-endpoint -> endpoint to where to call IRC
client.refresh-credentials-enabled -> Refresh credential enabled set to "true"
uri -> the catalog uri

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant