-
Notifications
You must be signed in to change notification settings - Fork 150
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Update objects timestamps on cache hit #542
Update objects timestamps on cache hit #542
Conversation
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Hi, thanks for the PR.
Object: object, | ||
} | ||
|
||
_, err := c.mcore.ComposeObject(context.Background(), dst, src) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I see that the copying is done server side- nice. Are there any downsides to having this always on?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Good question - I don't see any downsides in terms of performance or storage usage but I'm running self hosted s3 storage. As I understand AWS s3 pricing documentation there is some cost for coping object - you pay the S3 charges for storage in the selected destination S3 storage classes, for the primary copy, for replication PUT request
. So making this turn on by default could potentially generate costs for people using AWS so maybe it's better to leave it as opt-in?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
OK, let's make it opt-in to be on the safe side.
…it is trying to copy an object to itself without changing the object’s metadata` error
Thanks for the contribution- merged. |
In this PR I would like to introduce option to leverage S3 storage lifecycle to automatically remove not used objects after specified amount of time.
We can configure S3 bucket to remove objects after certain amount of time from its modification date. To update modification date, on each cache hit we need to copy object in place - that's limitation of s3. We can now pass
s3.update_timestamps
option in config to use it.