Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Upload Test Remote Extensions #4792

Merged
merged 13 commits into from
Jul 26, 2023
Merged

Upload Test Remote Extensions #4792

merged 13 commits into from
Jul 26, 2023

Conversation

awestover
Copy link
Contributor

@awestover awestover commented Jul 24, 2023

Problem

We need some real extensions in S3 to accurately test the code for handling remote extensions.
In this PR we just upload two extensions (anon and kq_imcx), which is enough for testing purposes for now. In addition to creating and uploading the extension archives, we must generate a file ext_index.json which specifies important metadata about the extensions. See below.

Summary of Changes

The layout of the S3 bucket is as follows:

5615610098 // this is an extension build number
├── v14
│   ├── extensions
│   │   ├── anon.tar.zst
│   │   └── embedding.tar.zst
│   └── ext_index.json
└── v15
    ├── extensions
    │   ├── anon.tar.zst
    │   └── embedding.tar.zst
    └── ext_index.json
5615261079
├── v14
│   ├── extensions
│   │   └── anon.tar.zst
│   └── ext_index.json
└── v15
    ├── extensions
    │   └── anon.tar.zst
    └── ext_index.json
5623261088
├── v14
│   ├── extensions
│   │   └── embedding.tar.zst
│   └── ext_index.json
└── v15
    ├── extensions
    │   └── embedding.tar.zst
    └── ext_index.json

ext_index.json stores the control files and location of extension archives.

More specifically, here is an example ext_index.json

{
  "embedding": {
    "control_data": {
      "embedding.control": "comment = 'hnsw index' \ndefault_version = '0.1.0' \nmodule_pathname = '$libdir/embedding' \nrelocatable = true \ntrusted = true"
    },
    "archive_path": "5623261088/v15/extensions/embedding.tar.zst"
  },
  "anon": {
    "control_data": {
      "anon.control": "# PostgreSQL Anonymizer (anon) extension \ncomment = 'Data anonymization tools' \ndefault_version = '1.1.0' \ndirectory='extension/anon' \nrelocatable = false \nrequires = 'pgcrypto' \nsuperuser = false \nmodule_pathname = '$libdir/anon' \ntrusted = true \n"
    },
    "archive_path": "5615261079/v15/extensions/anon.tar.zst"
  }
}

…rate an ext_index.json containing metadata about the extensions
@awestover
Copy link
Contributor Author

Note: this is safe to merge, it has already been tested in #4764
This is just a squashed version of #4764 . I made it primarily because I didn't want to mess up the git history on @bayandin 's branch

@github-actions
Copy link

github-actions bot commented Jul 24, 2023

1240 tests run: 1187 passed, 0 failed, 53 skipped (full report)


Flaky tests (1)

Postgres 15

  • test_metrics_normal_work: release

.github/workflows/build_and_test.yml Show resolved Hide resolved
Dockerfile.compute-node Show resolved Hide resolved
Dockerfile.compute-node Outdated Show resolved Hide resolved
Dockerfile.compute-node Outdated Show resolved Hide resolved
Dockerfile.compute-node Outdated Show resolved Hide resolved
scripts/combine_control_files.py Outdated Show resolved Hide resolved
scripts/combine_control_files.py Outdated Show resolved Hide resolved
scripts/combine_control_files.py Outdated Show resolved Hide resolved
scripts/combine_control_files.py Outdated Show resolved Hide resolved
scripts/combine_control_files.py Show resolved Hide resolved
lubennikovaav and others added 2 commits July 25, 2023 14:50
Co-authored-by: Alexander Bayandin <alexander@neon.tech>
Co-authored-by: Alexander Bayandin <alexander@neon.tech>
Dockerfile.compute-node Outdated Show resolved Hide resolved
@lubennikovaav lubennikovaav merged commit 5f8fd64 into main Jul 26, 2023
@lubennikovaav lubennikovaav deleted the alek/new-extension-format branch July 26, 2023 12:24
problame added a commit that referenced this pull request Aug 1, 2023
commit

	commit 5f8fd64
	Author: Alek Westover <alek.westover@gmail.com>
	Date:   Wed Jul 26 08:24:03 2023 -0400

	    Upload Test Remote Extensions (#4792)

switched to using the release tag instead of `latest`, but,
the `promote-images` job only uploads `latest` to the prod ECR.

The switch to using release tag was good in principle, but,
reverting that part to make the release pipeine work.

Note that a proper fix should abandon use of `:latest` tag
at all: currently, if a `main` pipeline runs concurrently
with a `release` pipeline, the `release` pipeline may end
up using the `main` pipeline's images.
problame added a commit that referenced this pull request Aug 1, 2023
commit

	commit 5f8fd64
	Author: Alek Westover <alek.westover@gmail.com>
	Date:   Wed Jul 26 08:24:03 2023 -0400

	    Upload Test Remote Extensions (#4792)

switched to using the release tag instead of `latest`, but,
the `promote-images` job only uploads `latest` to the prod ECR.

The switch to using release tag was good in principle, but,
reverting that part to make the release pipeine work.

Note that a proper fix should abandon use of `:latest` tag
at all: currently, if a `main` pipeline runs concurrently
with a `release` pipeline, the `release` pipeline may end
up using the `main` pipeline's images.
problame added a commit that referenced this pull request Aug 22, 2023
commit

	commit 5f8fd64
	Author: Alek Westover <alek.westover@gmail.com>
	Date:   Wed Jul 26 08:24:03 2023 -0400

	    Upload Test Remote Extensions (#4792)

switched to using the release tag instead of `latest`, but,
the `promote-images` job only uploads `latest` to the prod ECR.

The switch to using release tag was good in principle, but,
reverting that part to make the release pipeine work.

Note that a proper fix should abandon use of `:latest` tag
at all: currently, if a `main` pipeline runs concurrently
with a `release` pipeline, the `release` pipeline may end
up using the `main` pipeline's images.

(cherry picked from release branch commit 4204960 PR #4861)
problame added a commit that referenced this pull request Aug 22, 2023
This is cherry-picked-then-improved version of release branch commit
4204960 PR #4861)

The commit

	commit 5f8fd64
	Author: Alek Westover <alek.westover@gmail.com>
	Date:   Wed Jul 26 08:24:03 2023 -0400

	    Upload Test Remote Extensions (#4792)

switched to using the release tag instead of `latest`, but,
the `promote-images` job only uploads `latest` to the prod ECR.

The switch to using release tag was good in principle, but,
it broke the release pipeline. So, switch release pipeline
back to using `latest`.

Note that a proper fix should abandon use of `:latest` tag
at all: currently, if a `main` pipeline runs concurrently
with a `release` pipeline, the `release` pipeline may end
up using the `main` pipeline's images.

---------

Co-authored-by: Alexander Bayandin <alexander@neon.tech>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants