TAP13 draft

[JustinCappos]

This includes a draft of TAP 13 "User Selection of the Top-Level Target Files Through Mapping Metadata".

[JustinCappos]

@joshuagl I'd appreciate your feedback here too.

[lukpueh]

An example of the new mapping metdata might be helpful

[JustinCappos]

Agreed. I'm a bit at a loss about how to do this since the mapping metadata isn't precisely defined in format.

Does anyone have thoughts on this?

[lukpueh]

We do have a definition in TAP 4:

{
  // For each repository, its key name is the directory where files, including
  // the root metadata file, are cached, and its value is a list of URLs where
  // files may be downloaded.
  "repositories": {
    "Django": ["https://djangoproject.com/"],
    "PyPI":   ["https://pypi.python.org/"]
  },
  // For each set of targets, specify a list of repositories where files may be
  // downloaded.
  "mapping": [
    {
      // Much like target delegation, the order of these entries indicates
      // the priority of the delegation.  The entries listed first will be
      // considered first.
      // Map the targets "/django/django-1.*.tgz" to both Django and PyPI.
      "paths":        ["/django/django-1.*.tgz"],
      "repositories": ["Django", "PyPI"],
      // At least one repository must sign for the same length and hashes
      // of the "/django/django-1.*.tgz" targets.
      "threshold": 1
      // In this case, the "terminating" attribute is set to false.
      "terminating":  false,
      // Therefore, if this mapping has not signed for "/django/django-1.*.tgz"
      // targets, the following mapping will be consulted.
    },
    {
      ...
    {
      // Map all other targets only to PyPI.
      "paths":        ["*"],
      "repositories": ["PyPI"],
      "terminating": true
      "threshold": 1
    }
  ]
}

I'll need to think about how to best integrate TAP 13 with the concepts of TAP 4.

[joshuagl]

It took me a bit to think through, but I don't think we expect the selection of top-level targets to be mutually exclusive of other mappings? For example, I could have a mapping file which both maps the targets role to my filtered view of targets file(s) and maps paths (that must exist within this filtered view) to repositories with optional threshold and terminating flags.

With that in mind I started to wonder about extending the existing "mapping" list to include a list of targets. A threshold flag seems to me to make sense for a targets mapping, such that a minimum number of repositories sign for the same targets metadata file(s). I think a terminating flag might also makes sense, such that a user could prevent the targets metadata file(s) being trusted from some of the mapped repositories?

However, I haven't been able to fully reason out how we would express the nuances above in the specification and workflows.

Therefore, I am wondering if it would it make more sense to have the targets mapping be a separate ordered list of targets files to include in the customised namespace? Making our mapping file something like:

{
  // For each repository, its key name is the directory where files, including
  // the root metadata file, are cached, and its value is a list of URLs where
  // files may be downloaded.
  "repositories": {
    "Django": ["https://djangoproject.com/"],
    "PyPI":   ["https://pypi.python.org/"]
  },
  // The names and keys of targets metadata files on the repository that
  // will be composed together into a filtered view of the repository for
  // this client.
  // TODO: must the targets metadata files exist on all repositories?
  "targets": {
      "tuf.json": {"keyids": [...]},
      "securesystemslib.json": {"keyids": [...]}
  },
  // For each set of targets, specify a list of repositories where files may be
  // downloaded.
  "mapping": [
    {
      // Much like target delegation, the order of these entries indicates
      // the priority of the delegation.  The entries listed first will be
      // considered first.
      // Map the targets "/django/django-1.*.tgz" to both Django and PyPI.
      "paths":        ["/django/django-1.*.tgz"],
      "repositories": ["Django", "PyPI"],
      // At least one repository must sign for the same length and hashes
      // of the "/django/django-1.*.tgz" targets.
      "threshold": 1
      // In this case, the "terminating" attribute is set to false.
      "terminating":  false,
      // Therefore, if this mapping has not signed for "/django/django-1.*.tgz"
      // targets, the following mapping will be consulted.
    },
    {
      ...
    {
      // Map all other targets only to PyPI.
      "paths":        ["*"],
      "repositories": ["PyPI"],
      "terminating": true
      "threshold": 1
    }
  ]
}

In the example above, would the targets need to exist on all repositories?

[mnm678]

From my understanding, in that example the top-level targets file would need to exist on all repositories. If we want it to differ between repositories, maybe we can allow the user to include one for each mirror.

On another note, how would targets mapping metadata work for a client that does not support multiple repositories? One advantage to putting the targets mapping information in a separate file from the repository mapping information is that a client can choose to use one or the other of these without additional overhead.

[lukpueh]

By developer's key, do you mean top-level targets key? It might be confusing to the reader to talk about a developer key that wasn't mentioned before here.

[lukpueh]

Also, I find it interesting to say that the client delegates to the developer (top-level targets?). If we frame it that way we could say that the mapping file and thus the client becomes the root of trust. Is that correct?

[JustinCappos]

Suggested change

	a developer’s key. TAP 8 also provides support for cases where the key need to be rotated
	what would have been the top-level targets role. TAP 8 also provides support for cases where the key need to be rotated

[mnm678]

A draft reference implementation of this TAP is available at theupdateframework/python-tuf#1103.

[jku]

The changes in the client application workflow are said to be fairly minor but ... do I understand correctly that the client must now provide the keys that will be used to verify the target metadata?

How will this work in practice:

• should the client application bootstrap now include a root.json and the mapping file?
• Are clients expected to generate this mapping file somehow, or is this something an admin would write by hand and include in the application install/bootstrap?
• What happens when target keys need to be rotated?

[mnm678]

The changes in the client application workflow are said to be fairly minor but ... do I understand correctly that the client must now provide the keys that will be used to verify the target metadata?

If the client chooses to use the targets mapping metadata, they must also provide key management for this metadata. This is probably worth more discussion in the TAP, and these are all questions an implementer should answer before using targets mapping metadata.

should the client application bootstrap now include a root.json and the mapping file?

Yes, the mapping file would be provided at setup.

Are clients expected to generate this mapping file somehow, or is this something an admin would write by hand and include in the application install/bootstrap?

Most likely, this is something that would be provided by an admin. For example a company may want to point users to some packages on a public repository, but don't want them blindly trusting all images on that repository, so the company would create mapping metadata for their users.

What happens when target keys need to be rotated?

This is a really good question. As it is currently written, rotating the target keys would require new mapping metadata. It would be good to get more feedback from potential adopters about whether this is a feasible requirement.

[mnm678]

I've been thinking about this TAP, and I wonder if it would make sense to allow for client-defined top-level targets metadata, rather than allowing the client to point to a top-level targets metadata file on the repository. This would let the client have more fine-grained control about who they trust for each target, and would make this independent of uploading to the repository.

For example, consider top-level targets on the repository that delegates to 2 companies, company A and company B, who each delegate to teams within those companies. A client may want to pin the keys for team 1 in company A and team 4 in company B.

Using this method, the client-defined top-level targets metadata file could act as a map to the specific targets on the repository. This would also be possible to achieve this with the TAP as written, but it would require the client to upload a targets file to the repository, which might present a challenge.

cc @JustinCappos @joshuagl @lukpueh

[mnm678]

There are a couple of open questions left about this TAP:

• repository vs client hosted alternate top-level targets metadata The TAP currently describes repository hosted metadata that is pointed to by the client, but this does not give the client as much control over the subset of packages on a repository that they trust. For example, a client may want to limit the use of a repository to 2 specific packages. In the current proposal they would need to upload new metadata that specifies these exact packages, which may not be possible if the repository is controlled by a third party. For this reason, I think we should consider allowing the client to provide a mapping to a local top-level targets file.

• client mapping format This depends a bit on the above, but there are a couple of mapping formats described here and in the POC. One big question is whether this should be a part of the existing repository mapping metadata.

• metadata on multiple repositories How does this TAP interact with TAP 4? Specifically, does the targets mapping apply in the same way for every repository, or should a client be able to specify a different mapping for each repository. It would give the client more flexibility if they could set different mappings for each repository, though this may require a larger change to the repository mapping metadata.

What do others think about the above? And are any of these questions blockers for merging this TAP as a draft?

cc @JustinCappos @lukpueh @joshuagl

[joshuagl]

Thank you for keeping this TAP review moving Marina!

There are a couple of open questions left about this TAP:

• repository vs client hosted alternate top-level targets metadata The TAP currently describes repository hosted metadata that is pointed to by the client, but this does not give the client as much control over the subset of packages on a repository that they trust. For example, a client may want to limit the use of a repository to 2 specific packages. In the current proposal they would need to upload new metadata that specifies these exact packages, which may not be possible if the repository is controlled by a third party. For this reason, I think we should consider allowing the client to provide a mapping to a local top-level targets file.

I think this mechanism becomes much more useful if clients can define top-level targets metadata. But I believe allowing clients to define top-level targets metadata makes it much harder to maintain the security properties of TUF (compromise resilience, and freshness).

• client mapping format This depends a bit on the above, but there are a couple of mapping formats described here and in the POC. One big question is whether this should be a part of the existing repository mapping metadata.

I'm inclined to agree with your comment above, that separate files makes more sense. When I tried to imagine a unified mapping file to describe repository mapping and targets mapping in the same file and it feels complex. Much simpler for users to have separate files.

• metadata on multiple repositories How does this TAP interact with TAP 4? Specifically, does the targets mapping apply in the same way for every repository, or should a client be able to specify a different mapping for each repository. It would give the client more flexibility if they could set different mappings for each repository, though this may require a larger change to the repository mapping metadata.

🤔 I personally need to think about this one a bit more.

What do others think about the above? And are any of these questions blockers for merging this TAP as a draft?

I'm generally in favour of merging TAPs which feel like a good idea as draft sooner. It's much easier to collaborate on a TAP once it lands in the repository.
Perhaps we could assign each draft TAP a champion/sponsor (named TAP editor to move the TAP forwards) and link a corresponding issue describing next steps to move the TAP forward?

[joshuagl]

Suggested change

	* [TAP 13: User Selection of the Top-Level Target Files Through Mapping Metadata](tap13.md)
	* [TAP 13: Client Selection of the Top-Level Target Files Through Mapping Metadata](tap13.md)

[joshuagl]

Suggested change

	* Title: User Selection of the Top-Level Target Files Through Mapping Metadata
	* Title: Client Selection of the Top-Level Target Files Through Mapping Metadata

I wonder if this title makes more sense?

[JustinCappos]

I'm generally in favour of merging TAPs which feel like a good idea as draft sooner. It's much easier to collaborate on a TAP once it lands in the repository.
Perhaps we could assign each draft TAP a champion/sponsor (named TAP editor to move the TAP forwards) and link a corresponding issue describing next steps to move the TAP forward?

I'm in favor of this in general. One concern though is that some communities have been pointing at our draft TAPs and if we have a lot of churn there, we have work they need to unwind...

[joshuagl]

I'm generally in favour of merging TAPs which feel like a good idea as draft sooner. It's much easier to collaborate on a TAP once it lands in the repository.
Perhaps we could assign each draft TAP a champion/sponsor (named TAP editor to move the TAP forwards) and link a corresponding issue describing next steps to move the TAP forward?

I'm in favor of this in general. One concern though is that some communities have been pointing at our draft TAPs and if we have a lot of churn there, we have work they need to unwind...

Oh, I hadn't realised this was the case. Is that the Uptane and/or Notary communities?

Would it make sense to introduce an additional status, say WIP, for TAPs that are being actively worked on but are deemed incomplete by the TAP editors? These TAPs could even be kept in a separate folder, so as to further emphasise their less ready state.

I envisage the non-deferred/rejected/superseded TAP states being as follows:

• WIP: a TAP which the editors deem not ready yet
• Draft: a TAP which seems logically complete, but is not yet tested by an implementation
• Accepted: a complete TAP, with a PoC, that has not yet integrated into the specification
• Final: a complete TAP, with a PoC, that is integrated into the specification

[JustinCappos]

I envisage the non-deferred/rejected/superseded TAP states being as follows:

• WIP: a TAP which the editors deem not ready yet

I think the original thinking was mostly that things in this state would be discussed via issues / email but we would not assign a number to it yet. Is this similar to what you're thinking?

I think Draft is currently somewhere in between the two steps you mention, which is logical given I think you want to better differentiate things in that status.

[joshuagl]

I envisage the non-deferred/rejected/superseded TAP states being as follows:

• WIP: a TAP which the editors deem not ready yet

I think the original thinking was mostly that things in this state would be discussed via issues / email but we would not assign a number to it yet. Is this similar to what you're thinking?

Yes. I think we wouldn't assign a number to WIP TAPs. It feels like we could have more productive, and more public, collaboration on TAPs by merging them sooner and collaborating on the TAPs using the standard GitHub workflows. We're not seeing a lot of discussion happen on the public mailing list, and I think the PR workflow is more conducive to collaborators suggesting changes to the TAPs.

Of course, there would need to be a barrier to entry. But that might only be a consensus that the TAP is a good idea, or a reasonable initial proposal, with the details to be thrashed out while the TAP is in WIP state.

[joshuagl]

I'll work on some changes to TAP 1 in the new year to see whether the introduction of a WIP status makes sense for all.

[mnm678]

Based on the changes to TAP 1, are there any objections to merging this as a draft as we resolve the remaining questions?

[joshuagl]

Let's capture the remaining questions in an issue, then we should be good to me as draft.

[mnm678]

I moved the questions to #137

[mnm678]

@joshuagl can you give an official approve/reject for merging this as a draft?

[JustinCappos]

Perhaps start with something like, "Currently if a user trusts a TUF repository, a compromise of the targets role for that repository enables an attacker to install arbitrary malicious software."

[mnm678]

I updated the first few paragraphs. Is it better?

[JustinCappos]

It doesn't really explain why the intended design would be for Alice and Bob to receive different packages. I think the motivation here from a security standpoint needs to be immediately clear.

…

On Sat, May 22, 2021 at 2:43 AM Marina Moore ***@***.***> wrote: ***@***.**** commented on this pull request. ------------------------------ In tap13.md <#118 (comment)> : > repository are compromised. # Motivation - -In TUF, it is not possible for different users to have different namespaces on -the same repository. That is, if Alice and Bob both use repository X and ask -for package foo, they will both receive the same package. -This proposal enables clients to restrict the targets they consume to +In TUF, the root and targets roles on the repository are responsible for I updated the first few paragraphs. Is it better? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#118 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAGRODYQ3BX4JU75FEEJ4L3TO2SU7ANCNFSM4OJR2SPQ> .

[jku]

Is PyPI the prime example of where this could be used?

I'm asking because I've been trying very hard to imagine how that would work and I've failed to identify a scenario where this would be useful. Is is possible to get a practical example (doesn't have to be in this draft) of the sort of mapping that would be useful in the PyPI case.

[mnm678]

In the PEP 480 model, a user could limit PyPI downloads to those that are delegated from the claimed role, and so only install end-to-end signed images.

There were also some requests for this feature with regards to container registries, so that an organization could curate a list of verified packages and keys for their customers.

[jku]

Would it be possible to write a specific example out in more detail? The core issue I have in understanding this is that people don't install stand-alone "images" from PyPI. They install a whole tree of dependencies. That tree is extremely unlikely to consist of packages signed by one or even a few roles (assuming I've understood the ideas correctly).

So installing a package that depends on a dozen other packages (that in turn depend on dozens of other packages) is the norm, not the exception... and I don't think it's reasonable to assume that these packages come from only a few delegated roles. So how does the Mapping that is useful for this use case get generated in practice? Does it contain dozens of roles? What happens when the dependency tree inevitably changes and roles need to be added/removed?

Maybe I've just misunderstood the idea -- but again, a specific example in the style of "company X has done due diligence on the Django developers and wants to trust them, but not pypi in general. The write a Mapping that..." could maybe help with that.

There were also some requests for this feature with regards to container registries, so that an organization could curate a list of verified packages and keys for their customers.

Now this seems like a much easier example to understand.

[mnm678]

Does the new text here help?

[jku]

Right I see: the user does not curate who they trust, other than trust all developers who sign their packages and do not trust any developers who don't sign -- this seems understandable and realistic.

[joshuagl]

I'd really like to see this TAP merged as Draft, to me it's a promising enhancement for open submission/self-service repositories like PyPI.

The mapping format will likely be clearer as a POC happens (and the POC should be built on a TAP 4 implementation).