Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

pkg/cache: support creating a json cache directly from a model.Model #1111

Closed
wants to merge 1 commit into from
Closed

pkg/cache: support creating a json cache directly from a model.Model #1111

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Jump to
Jump to file
Failed to load files.
Diff view
Diff view
55 changes: 38 additions & 17 deletions pkg/cache/json.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -11,10 +11,12 @@ import (
"path/filepath"
"strings"

"k8s.io/apimachinery/pkg/util/sets"

"github.com/operator-framework/operator-registry/alpha/declcfg"
"github.com/operator-framework/operator-registry/alpha/model"
"github.com/operator-framework/operator-registry/pkg/api"
"github.com/operator-framework/operator-registry/pkg/registry"
"k8s.io/apimachinery/pkg/util/sets"
)

var _ Cache = &JSON{}
Expand Down Expand Up @@ -135,6 +137,17 @@ func NewJSON(baseDir string) *JSON {
return &JSON{baseDir: baseDir}
}

func LoadJSONFromModel(baseDir string, m model.Model) (*JSON, error) {
c := NewJSON(baseDir)
if err := c.buildFromModel(m); err != nil {
return nil, err
}
if err := c.Load(); err != nil {
return nil, err
}
return c, nil
}

const (
jsonDigestFile = "digest"
jsonDir = "cache"
Expand Down Expand Up @@ -179,6 +192,30 @@ func (q *JSON) computeDigest(fbcFsys fs.FS) (string, error) {
}

func (q *JSON) Build(ctx context.Context, fbcFsys fs.FS) error {
fbc, err := declcfg.LoadFS(ctx, fbcFsys)
if err != nil {
return err
}
fbcModel, err := declcfg.ConvertToModel(*fbc)
if err != nil {
return err
}

if err := q.buildFromModel(fbcModel); err != nil {
return err
}

digest, err := q.computeDigest(fbcFsys)
if err != nil {
return err
}
if err := os.WriteFile(filepath.Join(q.baseDir, jsonDigestFile), []byte(digest), jsonCacheModeFile); err != nil {
return err
}
Comment on lines +212 to +214
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe we should create an empty digest file in the case that we're loading from a model?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Would this be done in LoadJSONFromModel?

Would you ever do a integrity check when you're loaded directly from a model?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I mean from my perspective I'd call LoadJSONFromModel with a temporary directory and throw the directory away when I am done.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Would you ever do a integrity check when you're loaded directly from a model?

I don't think so. The digest is a combination of the filesystem contents of the FBC and the cache, so if either change, then the cache is invalid. In this case, we have no FBC filesystem, so the integrity check doesn't make sense.

I mean from my perspective I'd call LoadJSONFromModel with a temporary directory and throw the directory away when I am done.

Yes, that would be the expected use of this function. But I'm not happy with the API because it never makes sense to do anything other than that. In which case, why ask the caller to pass a directory in? Ideally, the temporary directory would be an implementation detail not exposed in the API. But then how does the caller clean up when they're done? The Cache API design expects the cache to be re-usable and saved for later, so there is no Close() or Cleanup() method in the cache API or in the JSON cache struct.

Maybe need a higher level abstraction that uses a cache under the hood?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Well, the "magic" happens in the packagesFromModel function so maybe the easier approach is to expose that publicly and then call the functions on the packageIndex? This seems to sidestep the caching mechanism entirely if I am not mistaken.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Some of those functions accept a cache though (e.g.

operator-registry/pkg/cache/pkgs.go

Line 62 in 6c60284

func (pkgs packageIndex) GetBundleForChannel(ctx context.Context, c Cache, pkgName string, channelName string) (*api.Bundle, error) {
)

So right now, packageIndex is very much coupled to the cache.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah.. I see that now. My mistake.

return nil
}

func (q *JSON) buildFromModel(fbcModel model.Model) error {
// ensure that generated cache is available to all future users
oldUmask := umask(000)
defer umask(oldUmask)
Expand All @@ -190,15 +227,6 @@ func (q *JSON) Build(ctx context.Context, fbcFsys fs.FS) error {
return fmt.Errorf("ensure clean base directory: %v", err)
}

fbc, err := declcfg.LoadFS(ctx, fbcFsys)
if err != nil {
return err
}
fbcModel, err := declcfg.ConvertToModel(*fbc)
if err != nil {
return err
}

pkgs, err := packagesFromModel(fbcModel)
if err != nil {
return err
Expand Down Expand Up @@ -232,13 +260,6 @@ func (q *JSON) Build(ctx context.Context, fbcFsys fs.FS) error {
}
}
}
digest, err := q.computeDigest(fbcFsys)
if err != nil {
return err
}
if err := os.WriteFile(filepath.Join(q.baseDir, jsonDigestFile), []byte(digest), jsonCacheModeFile); err != nil {
return err
}
return nil
}

Expand Down