-
Notifications
You must be signed in to change notification settings - Fork 26
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Update TUF client to support options and add LiveTrustedRoot #41
Conversation
2235c6a
to
0e74150
Compare
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Super excited to see the spec implemented!
pkg/tuf/client.go
Outdated
func checkEmbedded(tufRootURL string, fileJSONStore *filejsonstore.FileJSONStore) (json.RawMessage, error) { | ||
embeddedRootPath := path.Join("repository", tufRootURL, RootTUFPath) | ||
// Refresh forces a refresh of the underlying TUF client. | ||
// As the tuf client does not support multiple refreshes during its |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Out of scope, is this something that's being tracked in go-tuf-metadata?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This is a deliberate choice to maintain consistency with the reference implementation (python-tuf) which does not support multiple Refresh invocations: https://github.com/theupdateframework/python-tuf/blob/f711997a08cbb558fc8ab91406a846bbe4883d1a/tuf/ngclient/updater.py#L115
It has been discussed in this issue: rdimitrov/go-tuf-metadata#86
I don't think this is a blocker for go-tuf, as in the majority of cases I would reckon a TUF client is a short lived client, so multiple calls to Refresh()
would not be needed.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I can chime in there, but I think the issue will come up if TUF is used by a server rather than client. Not a blocker for Sigstore since the services don't depend on TUF though.
pkg/tuf/client.go
Outdated
} | ||
return c.up.Refresh() |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
How is concurrency handled with calling Refresh
in paralell? By the underlying TUF repo?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
No, this should be documented as a method unsafe for concurrent use. For concurrency it appears the LiveTrustedRoot
implements required serialization via a Mutex
.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I'll add an issue for that in go-tuf-metadata so we can work on it 👍
fyi, with slsa-verifier, we are planning to use the npmjs delegation, pending this sigstore-go PR and in |
TrustedRoot: tr, | ||
mu: sync.RWMutex{}, | ||
} | ||
ticker := time.NewTicker(time.Hour * 24) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This should be configurable if metadata expires more frequently than 24 hours.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
That makes sense to me, however since tuf.Options.CacheValidity
is using days as the unit, I wonder if we should also change that? If that config value is >= 1, then this ticker will have no effect if configured <= 24 hours, as it will just pull the cached data.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
To confirm, cache validity ignores whether the metadata is still valid?
Maybe it should be max(24 hours, CacheValidity)? If cache validity is set, just refresh once the cache expires, otherwise refresh every 24 hours?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thinking more on this, I think it's simpler to leave this as-is and let the client initialization handle the check for cache validity. Worst case, it just means an unnecessary reinitialization of the TUF client from disk.
The |
Move has begun: theupdateframework/go-tuf#583 |
Signed-off-by: Cody Soyland <codysoyland@github.com>
Signed-off-by: Fredrik Skogman <kommendorkapten@github.com>
Signed-off-by: Fredrik Skogman <kommendorkapten@github.com>
Signed-off-by: Fredrik Skogman <kommendorkapten@github.com>
Signed-off-by: Fredrik Skogman <kommendorkapten@github.com>
Signed-off-by: Fredrik Skogman <kommendorkapten@github.com>
Signed-off-by: Fredrik Skogman <kommendorkapten@github.com>
Signed-off-by: Fredrik Skogman <kommendorkapten@github.com>
Signed-off-by: Fredrik Skogman <kommendorkapten@github.com>
Signed-off-by: Fredrik Skogman <kommendorkapten@github.com>
b036ee1
to
8297aeb
Compare
Signed-off-by: Fredrik Skogman <kommendorkapten@github.com>
go-tuf is now updated with the new |
Co-authored-by: Hayden B <hblauzvern@google.com> Signed-off-by: Cody Soyland <codysoyland@github.com>
Signed-off-by: Cody Soyland <codysoyland@github.com>
Signed-off-by: Cody Soyland <codysoyland@github.com>
Signed-off-by: Cody Soyland <codysoyland@github.com>
I added a basic test to initialize a TUF repo and client in 96326fa. I was hoping to use the Assuming I'm on the right path, I can continue to add more tests based on this to validate the caching behavior. /cc @kommendorkapten |
Signed-off-by: Fredrik Skogman <kommendorkapten@github.com>
Signed-off-by: Fredrik Skogman <kommendorkapten@github.com>
Signed-off-by: Cody Soyland <codysoyland@github.com>
88a2c04
to
72edefd
Compare
Signed-off-by: Cody Soyland <codysoyland@github.com>
Signed-off-by: Cody Soyland <codysoyland@github.com>
Signed-off-by: Cody Soyland <codysoyland@github.com>
Signed-off-by: Cody Soyland <codysoyland@github.com>
It is impossible to reach the timestamp checks in loadMetadata, as the preceeding code to load metadata and verify it will force an online refresh anyway, so at this point, the cache has already been updated. Setting RemoteTargetsURL is not necessary as go-tuf will set that correctly by default. Signed-off-by: Cody Soyland <codysoyland@github.com>
I added tests for the It turns out that writing some tests has already paid off as I was able to identify a few bugs and dead code. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Nice cody!
@haydentherapper Have some time to give this a review?
Yep reviewing now! |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Just a few questions! This looks really good, nice work on the tests.
// Based on that metadata we take a decision if a full TUF | ||
// refresh should be done or not. As so, the tmpCfg is only needed | ||
// here and not in future invocations. | ||
tmpCfg.UnsafeLocalMode = true |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
To confirm, this means that if the timestamp along with all other metadata is not expired, then we will not make any network connections, correct?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Correct.
TrustedRoot: tr, | ||
mu: sync.RWMutex{}, | ||
} | ||
ticker := time.NewTicker(time.Hour * 24) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
To confirm, cache validity ignores whether the metadata is still valid?
Maybe it should be max(24 hours, CacheValidity)? If cache validity is set, just refresh once the cache expires, otherwise refresh every 24 hours?
pkg/tuf/options.go
Outdated
// Fall back to using a TUF repository in the temp location | ||
home = os.TempDir() | ||
} | ||
opts.CacheValidity = 1 |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Should we have a cache by default? I think cache is expected to be used in the case of airgapped environments, so should we have this be 0 and just rely on timestamp expiration for when to fetch metadata by default?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Back in the summer of 2022 when we where discussing this, 1 day was proposed https://docs.google.com/document/d/1QWBvpwYxOy9njAmd8vpizNQpPti9rd5ugVhji0r3T4c/edit#heading=h.s9pcbj5k0o1z
In retrospect, I think that the tuf package maybe should set the cache validity to 0 (i.e no cache) but each client should set the value to the best in their domain. This way we minimize any possible footguns (i.e secure by default). Even if a single is quick, I could still see possible issues being opened when the client (e.g. cosign) does not immediately get the new root.
I can change this. cc @codysoyland
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
How does this interact with the target cache files? I don’t believe it does, correct? As in, the client will still cache targets on disk regardless of this configuration.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Reading this comment again:
Should we have a cache by default? I think cache is expected to be used in the case of airgapped environments, so should we have this be 0 and just rely on timestamp expiration for when to fetch metadata by default?
This is not really how it works. If caching is not used, the client will always try to get a new timestamp and root, regardless if the local timestamp is expired or not (this is according to the TUF spec).
The cache allows us to control if this (full TUF online refresh) should be performed on every invocation, or only once per day. But I'm still thinking that maybe having the cache off by default is a better option, as a full TUF update is not that expensive, if the client is up to date, it's two egress HTTP requests (one for root and one for timestamp). So the cache is not really with arigapped, it's more a way to speed up invocations where e.g. frequent cosign verify is performed in a short period of time.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
How does this interact with the target cache files? I don’t believe it does, correct? As in, the client will still cache targets on disk regardless of this configuration.
Exactly, it's only the metadata that is cached in non-standard way to avoid a full metadata refresh against the remote TUF repository.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Since we're using the
UnsafeLocalMode
updater for the initial update, won't cache validity only matter if the timestamp is expired?
No, timestamp expiration is always respected. As the timestamp may expire at any time, or a new root (or target) can be published, so the client always looks for that on every invocation. This is the behaviour that's controlled via CacheValidity
Also, should cache validity be checked in GetTarget too? Otherwise GetTargetInfo forces a refresh if metadata is expired.
Yes, if the metadata is expired, this will happen. If this is in an airgapped environment this would fail, but for that, I would think a better approach is to just provide the trusted_root.json
instead of using TUF, as we won't get any real security benefit from it. And also, I would think that the normal scenario is that a client does:
- Create a TUF client
- Get
trusted_root.json
target
and the two calls happens during initialization, pretty close to each other in time. And once thetrusted_root.json
is retrieved, it's used for the rest of the program's execution?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Ah, I took another pass over the code and now see what I was missing. In the current Sigstore TUF client, we have it set up where if the timestamp is not expired, no additional metadata is fetched. With the current implementation, if you wanted to replicate that behavior, you would set CacheValidity
to be greater than the timestamp expiration - Then it would not refresh until the timestamp expires, which will then force a refresh.
Have you considered having this behavior controlled via a boolean rather than a time period? The current implementation does give finer control though. One idea to improve UX, maybe a NO_CACHE and MAX_CACHE constant, set to 0 and MAX_INT respectively? Then when setting up the config, you can choose to a) follow the TUF spec with NO_CACHE, b) only refresh when absolutely necessary with MAX_CACHE, or c) fine-tune based on needs.
I might also add a comment making this explicit that CacheValidity
is not for airgapped environments, since it still does respect timestamp expiration.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Have you considered having this behavior controlled via a boolean rather than a time period?
I can't speak for @kommendorkapten, but in my opinion, the CacheValidity
flag works well for systems that perform a lot of verifications, without having to manually bump the cache daily, and avoids the risk of missing new metadata if the repo's timestamp interval is long. However, I could see the argument of changing it to hours or time.Duration, or adding a few constants as you suggested.
I might also add a comment making this explicit that CacheValidity is not for airgapped environments, since it still does respect timestamp expiration.
What is your opinion on the correct behavior for airgapped environments? Would you suggest having an option to disregard an expired timestamp? That is certainly possible, but I would probably want to slap the Insecure
prefix in front of that option. 😆
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
CacheValidity flag works well for systems that perform a lot of verifications, without having to manually bump the cache daily, and avoids the risk of missing new metadata if the repo's timestamp interval is long
This sounds good. I think the constants are a nice compromise for those who are configuring the client and want a simple "boolean" approach of "no cache" vs "cache up to timestamp expiration".
What is your opinion on the correct behavior for airgapped environments? Would you suggest having an option to disregard an expired timestamp?
I think we should omit airgapped support for now, there's much more nuance with getting this right, given it doesn't follow the TUF spec. I agree with @kommendorkapten that for airgapped clients, a reasonable approach is to fetch the trust root out of band. The python client has been thinking about what it means to be truly offline too.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Got it, I missed @kommendorkapten's comment on that, but I agree with you both!
Signed-off-by: Fredrik Skogman <kommendorkapten@github.com>
Signed-off-by: Fredrik Skogman <kommendorkapten@github.com>
Signed-off-by: Fredrik Skogman <kommendorkapten@github.com>
I broke some tests when I updated to latest go-tuf, I will fix that. |
Signed-off-by: Fredrik Skogman <kommendorkapten@github.com>
Signed-off-by: Cody Soyland <codysoyland@github.com>
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Left one comment about how to clarify the purpose of cache validity and use consts, but it's not a blocker. Thanks for all of the work on this!
Signed-off-by: Cody Soyland <codysoyland@github.com>
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
🎉
Addresses: #547 - [x] Pending: sigstore/sigstore-go#41 Uses the new [sigstore-go@0.2.0](https://github.com/sigstore/sigstore-go/releases/tag/v0.2.0) Currently slsa-verifier has npmjs' attestation key hardcoded. But sigstore now stores the same key within their own TUF root. This PR - dynamically use the keyid specified in the sigstore bundle, rather than the hardcoded keyid. - uses an updated ([pending]( sigstore/sigstore-go#41)) sigstore-go library that allows us to fetch a signed and verified copy of the same key. --------- Signed-off-by: Ramon Petgrave <ramon.petgrave64@gmail.com>
Summary
This PR implements #38. This adds:
Credit to @kommendorkapten for some parts of the client code, which we developed in collaboration at GitHub.
To do:
Release Note
Documentation