Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Migrate to a new S3 bucket #1089

Merged
merged 1 commit into from
Oct 6, 2020
Merged

Migrate to a new S3 bucket #1089

merged 1 commit into from
Oct 6, 2020

Conversation

edmorley
Copy link
Member

@edmorley edmorley commented Oct 6, 2020

Since:

  • We want the S3 bucket to be owned by a different AWS account and it's not possible to transfer ownership of an existing bucket.
  • In the future we want to rebuild some of the Python runtime archives (for example to improve the sqlite3 handling, or to tweak the compile flags used), and it will be easier to reason about the change if we can guarantee only recent buildpack versions are using the assets rather than several year old unmaintained forks.

The assets were synced from the old bucket using (minus the --dryrun):

aws s3 sync s3://lang-python s3://heroku-buildpack-python \
  --dryrun \
  --metadata-directive REPLACE \
  --exclude "*" \
  --include 'common/*' \
  --include 'heroku-*/runtimes/*' \
  --include 'heroku-*/libraries/vendor/gdal.tar.gz' \
  --include 'heroku-*/libraries/vendor/geos.tar.gz' \
  --include 'heroku-*/libraries/vendor/proj.tar.gz' \
  --exclude 'common/pip-20.0.2-py2.py3-none-any.whl' \
  --exclude '*/runtimes/*-opt.tar.gz' \
  --exclude '*/runtimes/sqlite-free/*'

The files that were --excluded are those that are no longer used, or test assets that were not officially released.

The Cedar-14 assets were not migrated since it's EOL next month.

The old S3 bucket will be left untouched for the foreseeable future (ie: we won't be deleting it), since builds using older versions of this buildpack (either due to pinning to a tag or via a fork) will still be using assets from it.

Closes W-8060097.

@edmorley edmorley self-assigned this Oct 6, 2020
Since:
* We want the S3 bucket to be owned by a different AWS account and it's
  not possible to transfer ownership of an existing bucket.
* In the future we want to rebuild some of the Python runtime archives
  (for example to improve the sqlite3 handling, or to tweak the compile
  flags used), and it will be easier to reason about the change if we
  can guarantee only recent buildpack versions are using the assets
  rather than several year old unmaintained forks.

The assets were synced from the old bucket using (minus the `--dryrun`):

```
aws s3 sync s3://lang-python s3://heroku-buildpack-python \
  --dryrun \
  --metadata-directive REPLACE \
  --exclude "*" \
  --include 'common/*' \
  --include 'heroku-*/runtimes/*' \
  --include 'heroku-*/libraries/vendor/gdal.tar.gz' \
  --include 'heroku-*/libraries/vendor/geos.tar.gz' \
  --include 'heroku-*/libraries/vendor/proj.tar.gz' \
  --exclude 'common/pip-20.0.2-py2.py3-none-any.whl' \
  --exclude '*/runtimes/*-opt.tar.gz' \
  --exclude '*/runtimes/sqlite-free/*'
```

The files that were `--exclude`d are those that are no longer used,
or test assets that were not officially released.

The Cedar-14 assets were not migrated since it's EOL next month.

The old S3 bucket will be left untouched for the foreseeable future
(ie: we won't be deleting it), since builds using older versions of this
buildpack (either due to pinning to a tag or via a fork) will still be
using assets from it.

Closes W-8060097.
@edmorley edmorley marked this pull request as ready for review October 6, 2020 06:47
@edmorley edmorley requested a review from a team as a code owner October 6, 2020 06:47
@edmorley edmorley merged commit b250300 into main Oct 6, 2020
@edmorley edmorley deleted the migrate-s3-bucket branch October 6, 2020 08:23
scorphus added a commit to scorphus/platforms that referenced this pull request Oct 15, 2020
heroku-buildpack-python migrated to a new S3 bucket 9 days ago. Here are
the details: heroku/heroku-buildpack-python#1089
scorphus added a commit to scorphus/platforms that referenced this pull request Oct 15, 2020
heroku-buildpack-python migrated to a new S3 bucket 9 days ago. Here are
the details: heroku/heroku-buildpack-python#1089
cezarsa pushed a commit to tsuru/platforms that referenced this pull request Oct 15, 2020
heroku-buildpack-python migrated to a new S3 bucket 9 days ago. Here are
the details: heroku/heroku-buildpack-python#1089
dryan pushed a commit to dryan/heroku-buildpack-python that referenced this pull request Nov 19, 2020
Since:
* We want the S3 bucket to be owned by a different AWS account and it's
  not possible to transfer ownership of an existing bucket.
* In the future we want to rebuild some of the Python runtime archives
  (for example to improve the sqlite3 handling, or to tweak the compile
  flags used), and it will be easier to reason about the change if we
  can guarantee only recent buildpack versions are using the assets
  rather than several year old unmaintained forks.

The assets were synced from the old bucket using (minus the `--dryrun`):

```
aws s3 sync s3://lang-python s3://heroku-buildpack-python \
  --dryrun \
  --metadata-directive REPLACE \
  --exclude "*" \
  --include 'common/*' \
  --include 'heroku-*/runtimes/*' \
  --include 'heroku-*/libraries/vendor/gdal.tar.gz' \
  --include 'heroku-*/libraries/vendor/geos.tar.gz' \
  --include 'heroku-*/libraries/vendor/proj.tar.gz' \
  --exclude 'common/pip-20.0.2-py2.py3-none-any.whl' \
  --exclude '*/runtimes/*-opt.tar.gz' \
  --exclude '*/runtimes/sqlite-free/*'
```

The files that were `--exclude`d are those that are no longer used,
or test assets that were not officially released.

The Cedar-14 assets were not migrated since it's EOL next month.

The old S3 bucket will be left untouched for the foreseeable future
(ie: we won't be deleting it), since builds using older versions of this
buildpack (either due to pinning to a tag or via a fork) will still be
using assets from it.

Closes @W-8060097@.
joaodaher pushed a commit to edukorg/tsuru-platforms that referenced this pull request Mar 1, 2021
heroku-buildpack-python migrated to a new S3 bucket 9 days ago. Here are
the details: heroku/heroku-buildpack-python#1089
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants