Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add an optional expirationTimestamp field for pre signed urls #312

Merged
merged 17 commits into from
May 23, 2023
Merged

Add an optional expirationTimestamp field for pre signed urls #312

merged 17 commits into from
May 23, 2023

Conversation

linzhou-db
Copy link
Collaborator

@linzhou-db linzhou-db commented May 16, 2023 •
edited
Loading

  • In delta sharing sever, add expirationTimestamp for each action in the response
  • In delta sharing client, update refresh function to return the min expiration timestamp across all urls.
  • In delta sharing client, refresh based on the expiration timestamp
  • In DeltaSharingSource, updated the refresh function to refresh the sortedFetchedFiles, so it doesn't have to do an extra local refresh -- delegate the refresh to CachedTableManager entirely.

@linzhou-db linzhou-db self-assigned this May 17, 2023
chakankardb
chakankardb reviewed May 18, 2023
View reviewed changes
server/src/main/scala/io/delta/sharing/server/model.scala Show resolved Hide resolved
server/src/main/scala/io/delta/standalone/internal/DeltaSharedTableLoader.scala Outdated Show resolved Hide resolved
spark/src/main/scala/io/delta/sharing/spark/DeltaSharingProfileProvider.scala Outdated Show resolved Hide resolved
spark/src/main/scala/io/delta/sharing/spark/DeltaSharingSource.scala Show resolved Hide resolved
spark/src/main/scala/io/delta/sharing/spark/DeltaSharingSource.scala Outdated Show resolved Hide resolved
spark/src/main/scala/io/delta/sharing/spark/RemoteDeltaCDFRelation.scala Outdated Show resolved Hide resolved
spark/src/main/scala/org/apache/spark/delta/sharing/PreSignedUrlCache.scala Outdated Show resolved Hide resolved
spark/src/main/scala/org/apache/spark/delta/sharing/PreSignedUrlCache.scala Outdated Show resolved Hide resolved
spark/src/main/scala/io/delta/sharing/spark/DeltaSharingSource.scala Show resolved Hide resolved
spark/src/main/scala/io/delta/sharing/spark/DeltaSharingSource.scala Outdated Show resolved Hide resolved
@zsxwing zsxwing changed the title Add expiration_time for each action Add an optional expirationTimestamp field for pre signed urls May 18, 2023
@linzhou-db linzhou-db requested a review from zsxwing May 18, 2023 06:57
@chakankardb
Copy link
Collaborator

@zsxwing Can you also please take another look?

@linzhou-db linzhou-db requested a review from zsxwing May 20, 2023 09:28
@linzhou-db
Copy link
Collaborator Author

Manually tested in #313

zsxwing
zsxwing approved these changes May 23, 2023
View reviewed changes
Copy link
Member

@zsxwing zsxwing left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@linzhou-db linzhou-db merged commit 44bab62 into main May 23, 2023
@linzhou-db linzhou-db deleted the SC-130341 branch July 25, 2023 07:10
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@zsxwing zsxwing zsxwing approved these changes

@chakankardb chakankardb chakankardb approved these changes

Assignees

@linzhou-db linzhou-db

Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@linzhou-db @chakankardb @zsxwing