Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[SPIKE] s3 lifecycle rules for backups #906

Open
wants to merge 13 commits into
base: develop
Choose a base branch
from

Conversation

BasilMawejje
Copy link

PT Story:

PT URL: https://www.pivotaltracker.com/story/show/173667123

Changes proposed in this pull request:

  1. Spike using aws sdk to transition backups in s3 to different storage classes e.g transition from S3 Standard to Standard IA after 30 days, then transition from standard IA to Glacier after 90 days and finally transition from Glacier to Deep Archive after 180 days.
  2. Spike also sets expirations for file storage in S3. For example we can explicitly set a rule to expire objects after 365 days.
  3. Requirements will be discussed further to make sense of this spike. I made a number of assumptions while I wrote the lifecycle method.

Screenshots (Optional):

N/A

Ready for review:

@

Ps:

  • No tests added because I don't have the .env file to setup the test database.

app/models/conditions_response/backup.rb Outdated Show resolved Hide resolved
app/models/conditions_response/backup.rb Outdated Show resolved Hide resolved
spec/models/conditions_response/backup_spec.rb Outdated Show resolved Hide resolved
@BasilMawejje BasilMawejje force-pushed the spike/backup-s3-lifecycle-rules branch 2 times, most recently from 16f12dd to b542219 Compare August 25, 2021 10:57
@BasilMawejje BasilMawejje force-pushed the spike/backup-s3-lifecycle-rules branch from 6226a43 to 016ec4e Compare August 26, 2021 09:48
@BasilMawejje BasilMawejje changed the title WIP: spike for s3 lifecycle rules for backups Spike for s3 lifecycle rules for backups Aug 26, 2021
Modify method signature comment to reflect storage_rules as a list
@BasilMawejje BasilMawejje force-pushed the spike/backup-s3-lifecycle-rules branch from 5fe838c to 3101aed Compare August 31, 2021 12:22
@BasilMawejje BasilMawejje force-pushed the spike/backup-s3-lifecycle-rules branch from 96fb4b3 to 631f7e0 Compare September 3, 2021 06:08
@BasilMawejje BasilMawejje force-pushed the spike/backup-s3-lifecycle-rules branch from dcd8f44 to 2cd76e3 Compare October 7, 2021 14:05
@BasilMawejje BasilMawejje changed the title Spike for s3 lifecycle rules for backups [SPIKE] s3 lifecycle rules for backups Oct 7, 2021
@BasilMawejje BasilMawejje force-pushed the spike/backup-s3-lifecycle-rules branch from 2cd76e3 to 5dfe439 Compare October 7, 2021 14:33
@BasilMawejje BasilMawejje marked this pull request as ready for review October 7, 2021 16:42
@weedySeaDragon
Copy link
Collaborator

weedySeaDragon commented Oct 8, 2021

I put up a wiki page that has the backups schedules: https://github.com/AgileVentures/shf-project/wiki/Backups-on-AWS

Other Misc. thoughts:

  • will want to read the configuration from a file (ex: number of days to keep for different types of backups, storage class). Perhaps this should be the XML that AWS uses (ex: those shown on https://docs.aws.amazon.com/AmazonS3/latest/userguide/lifecycle-configuration-examples.html)
  • default zone should be Stockholm
  • should we also be setting lifecycle rules on buckets? (Would this be easier? Should we do this instead of setting lifecycles rules on files or in addition to?
  • do we want to keep backups in 2 different zones? If so, which backups and which additional zone? (maybe monthly and yearly? maybe additional zone should be in the US?) what will the cost be?

@BasilMawejje BasilMawejje force-pushed the spike/backup-s3-lifecycle-rules branch 3 times, most recently from 202c0fa to 12025ca Compare October 11, 2021 15:29
@BasilMawejje BasilMawejje force-pushed the spike/backup-s3-lifecycle-rules branch from 12025ca to d4e3fee Compare October 11, 2021 15:31
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants