Spike: Investigate options for hosting software dependencies into airgap/egress-limited/controlled environments

[zachariahmiller]

Is your feature request related to a problem? Please describe.

This is a proof of concept of different ways we can manage dependencies for software development in an airgapped environment focused on using open source tools. Scope of this issue includes just hosting/using the packages (#54 covers transfer). It could involve writing a custom tool to facilitate this. It could also involve collating multiple lightweight OSS tools together or relying on an existing monolithic OSS software repository tool. Each of the following options should be evaluate based on the criteria for success defined below.

Output of this ticket should be working proof of concept and a followup ADR.

As there are so many different programming languages, this POC will focus on only a few, common ones:

1. golang
2. javascript/typescript/npm
3. python/pypi

Describe the solution you'd like

Any solution that will be picked should be the following:

1. Performant to the end user (and scalable)
2. Sustainable
3. Have little to no impact of provenance, interaction with the repositories with package managers, etc
4. Positive UX
5. Hosted in kubernetes (for the end user interface

[Racer159]

This could be descoped to hosting first

[Racer159]

Rescoped

[marshall007]

@zachariahmiller FYI I dug up this proposal I had at GitLab around a "universal artifact registry" based on OCI as the storage backend: https://gitlab.com/groups/gitlab-org/-/epics/7056

This is basically just a slightly more opinionated take on "S3 as the storage backend". We probably have to implement the package manager APIs on top of storage in any case, but picking OCI as the store has the big advantages of:

1. content-addressable, so you get de-duplication of blobs across all package types for free
2. we can use OCI artifacts/references to associate attestations like build/source provenance, scan results, etc, regardless of what is supported by the upstream package manager
3. we can use off-the-shelf registries like zot and specifically it's pull-through cache and sync/mirroring features as a general transfer solution for all package types