Spack Buildcache Views

[kwryankrattiger]

Motivation

The Spack CI team is working on reducing the total number of packages built in CI by restructuring the stacks in a way that daisy chains stacks with increasing specificity using the new feature include-concrete. The benefit of this is common dependency can be built once and then reused to build the root level specs in the downstream stacks.

flowchart
  E[e4s];
  O[e4s-oneapi]
  C[common-x86_64_v3];
  B[build-systems];
  E ----> C;
  O ----> C;
  B ----> C;

Loading

This introduces an new wrinkle into how build caches are used. The common-x86_64_v3 is its own stack, and produces its own binary cache and mirror containing packages that are also used by the e4s and e4s-oneapi and build-systems stacks. When the downstream stacks use the common build cache, they do not build the common dependencies and therefore do not push the binaries to their build cache. This currently creates "holes" in the downstream build caches for all of the dependency packages built by the common stack.

Solutions

Build caches generated by the Spack CI build farm should continue to work as if they were complete and without "holes" to the extent that is it possible to do so. We do not want to have to add the common stack mirror to use the e4s binary cache, the e4s binary cache should be complete, or look complete, to consumers.

If nothing is done, properly registering the mirror for e4s would require adding all of the common mirrors containing dependencies used by e4s.

spack mirror add common-x86 s3://spack-binaries/develop/common-x86_64_v3
spack mirror add e4s s3://spack-binaries/develop/e4s

This discussion is about finding solutions, referred to here as "views", to allow adding a single mirror that contains valid references to all of the binaries used in the stack, similar to how things are done now.

spack mirror add e4s s3://spack-binaries/develop/e4s

To achieve this there have been a few ideas with varying pros and cons.

Embedded Cache URL

One option is to embed the build cache URL of the binary object in the spec.json. This would require the spec.json to be duplicated across mirrors using something similar to the current sync script.

Pros:

• spec.json files are location agnostic and can reference the build cache provenance from anywhere.
• The index.json effectively functions the same as it does now.

Cons:

• The original build cache may change the hash of the stored binary, invalidating the contents in the spec.json.
• Validating if a spec.json is correct or can be pruned or has been pruned is much more difficult.

View Redirect

This method is similar to the the previous in the sense that it points to binaries in another build cache, but unlike the previous it does not duplicate the spec.json files or index.json files of the build cache. In this method it composes a new build cache as a list of references to build caches containing binaries. This is effectively implicitly adding multiple mirrors for the price of one.

An example of a possible implementation would be to have a static mirror index.json for e4s containing redirects only:
s3://spack-binaries/develop/e4s/view.json

{
  "urls": ["s3://spack-binaries/develop/e4s", "s3://spack-binaries/develop/common-x86_64_v3"]
}

Pros:

• Migration to this format is trivial as it only requires minor changes to spack to check for and read a view.json in a build cache before reading the index.json.
• De-duplicate binaries between the stack and top level mirrors

Cons:

• Somewhat more complicated requiring resolving possible cycles in build cache references
• Less transparent to the user where binaries are coming from

View index.json

This method of views completely drops the idea of multiple build caches in favor of curated views into a build cache. The basic idea is each view file is an index.json for a build cache only containing the specs associated with that view. These indices would be generated using a either list of concrete specs from spack.lock files or a list of spec.json files directly. The view index would then be stored under the build cache url and referenced using a URL extension.

An example of a possible implementation:

The build cache would look like this.

+-- build_cache
|    +-- views
|    |    +-- common-x86_64_v3.json
|    |    +-- data-and-vis-sdk.json
|    |    +-- e4s.json
|    |    +-- e4s-oneapi.json
|    +-- index.json (normal index.json, not required)
|    +-- ...

and then the mirror configuration using the view would look something like this, using the query view=e4s to point that the index file build_cache/views/e4s.json instead of the default index build_cache/index.json

mirrors:
  e4s: s3://spack-binaries/develop?view=e4s

Pros:

• This method is transparent about where binaries are coming from and only handles curration.
• Views are part of the build cache so should be easier to keep in sync with the state of the build cache.
• De-duplicate binaries between the stack and top level mirrors

Cons:

• The top level index can become very large depending on how many views are created.
• Using a query in the URL string is a bit awkward is not a real URL that can be invoked. Adding a ?view=e4s to an s3 url breaks aws s3 ls for example.

The requirements for this method require the content addressed tarballs to be implemented. Using a single build cache with the possibility of multiple sources for the same binary falls into the race condition when pushing binaries for the same spec with difference content hashes. This is already on the road map of work to do, so I don't view it as a "con".

Summary

These are just the methods I have come up with so far talking with others on the topic. More ideas are more than welcome.

[kwryankrattiger]

@haampie had the idea for an interface to do

spack buildcache update-index --from-env ...

Adding interfaces to update the build cache index from spack.lock files will be very useful for creating the view indices. It may also be helpful to have the ability to directly append to an existing index without having to process all of the spec.json files over again.