Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merge all of the inbox S.S.C.* libraries into one #55690

Closed
Tracked by #64488
vcsjones opened this issue Jul 14, 2021 · 22 comments
Closed
Tracked by #64488

Merge all of the inbox S.S.C.* libraries into one #55690

vcsjones opened this issue Jul 14, 2021 · 22 comments
Assignees
Milestone

Comments

@vcsjones
Copy link
Member

vcsjones commented Jul 14, 2021

Currently, the PemEncoding and PemFields types live in System.Security.Cryptography.Encoding. This made sense at the time. However, it makes it difficult to use the PEM functionality from things in System.Security.Cryptography.Primitives, specifically AsymmetricAlgorithm, because that would be a circular dependency.

This has already impacted our ability to do something we wanted to at #34086 (comment).

With #51630, we’re going to run in to this again, with a stronger argument that the PEM-PKCS8 exports should be done on AsymmetricAlgorithm directly. This leaves us with two options:

  1. Move the types we need in to the Primitives assembly.
  2. Use reflection to call the PEM APIs. This is doable, but we will need to call the allocating APIs since working with Span<T> with reflection is not doable today.
@dotnet-issue-labeler dotnet-issue-labeler bot added area-System.Security untriaged New issue has not been triaged by the area owner labels Jul 14, 2021
@ghost
Copy link

ghost commented Jul 14, 2021

Tagging subscribers to this area: @bartonjs, @vcsjones, @krwq, @GrabYourPitchforks
See info in area-owners.md if you want to be subscribed.

Issue Details

Currently, the PemEncoding and PemFields types live in System.Security.Cryptography.Encoding. This made sense at the time. However, it makes it difficult to use the PEM functionality from things in System.Security.Cryptography.Primities, specifically SymmetricAlgorithm, because that would be a circular dependency.

This has already impacted our ability to do something we wanted to once already at #34086 (comment).

With #51630, we’re going to run in to this again, with a stronger argument that the PKCS8 exports should be done on SymmetricAlgorithm directly. This leaves us with two options:

  1. Move the types we need in to the Primitives assembly.
  2. Use reflection to call the PEM APIs. This is doable, but we will need to call the allocating APIs since working with Span<T> with reflection is not doable today.
Author: vcsjones
Assignees: -
Labels:

area-System.Security, untriaged

Milestone: -

@bartonjs bartonjs removed the untriaged New issue has not been triaged by the area owner label Jul 15, 2021
@bartonjs bartonjs added this to the 7.0.0 milestone Jul 15, 2021
@bartonjs
Copy link
Member

but we will need to call the allocating APIs since working with Span with reflection is not doable today

I think that it is, it's just more complicated 😄.

private delegate int TryWriteDelegate(ReadOnlySpan<char> label, ReadOnlySpan<byte> data, Span<char> destination, out int charsWritten);

...

Type t = ...;
MethodInfo method = t.GetMethod("TryWrite", BindingFlags.Static | BindingFlags.Public);
TryWriteDelegate twd = (TryWriteDelegate)method.CreateDelegate(typeof(TryWriteDelegate));
// squirrel that off into a field somewhere.

There've been a few things that make me feel like we might want to just merge everything except Pkcs and Xml into System.Private.Security.Cryptography.dll, but maybe it doesn't need to be quite that drastic, and this one type is all we should move.

@vcsjones
Copy link
Member Author

There've been a few things that make me feel like we might want to just merge everything except Pkcs and Xml into System.Private.Security.Cryptography.dll

If that's on the table I'd very much prefer that. I can't recall the specifics but on multiple occasions I wished it were a big assembly. I'm sure if I went digging through some of my abandoned PRs I could jog my memory. 😄

@bartonjs
Copy link
Member

with a stronger argument that the PKCS8 exports should be done on SymmetricAlgorithm directly

On Symmetric? PKCS#8 doesn't make sense there.

On Asymmetric? We define the methods there, but since each algorithm has unique ways for doing the algorithm identifier (IIRC that's where DSA shoves the domain parameters and EC shoves the curve declaration) and the payload, there's not a lot that the base class can really do without there being two protected virtuals... so we let the algorithms do it with the one.

@vcsjones
Copy link
Member Author

On Symmetric? PKCS#8 doesn't make sense there.

Yeah typed the wrong thing there. It seems some part of my brain is still working on one shots.

On Asymmetric?

For export, the only thing on AsymmetricAlgorithm were PKCS8 / SPKI PEM exports. Since those are just PUBLIC / PRIVATE key (where PRIVATE could be ENCRYPTED PRIVATE) it doesn't make sense to override them on their subclasses - we didn't even define them as virtual on the proposal.

@bartonjs
Copy link
Member

we didn't even define them as virtual on the proposal.

Ooooh, not the real PKCS8, but the PEM-PKCS8.

Yeah, since that's just "call the existing PKCS8 virtual, PEM-ify the result".

@vcsjones
Copy link
Member Author

vcsjones commented Jul 15, 2021

Ooooh, not the real PKCS8, but the PEM-PKCS8.

Yeah, I could have been more clear what I was talking about 😄

Maybe a place to start then is to make a new System.Security.Cryptography project and begin with moving .Encoding and .Primitives in there. We can move the remaining later, if it makes sense to move them. .Xml and .Pkcs should probably stay out, and probably .ProtectedData?

@bartonjs
Copy link
Member

@ericstj Anything special that should be kept in mind while considering collapsing the non-NuGet crypto libraries?

I feel like there are probably two approaches:

  • System.Private.Security.Cryptography
    • Keep the layering in the ref assemblies, but the impls are all type forwards.
    • Refs are probably harder to maintain.
  • System.Security.Cryptography
    • Existing libraries change to typeforwards
    • Refs are still autogeneratable

I don't think, off the top of my head, we need the layering for any co-dependency reasons. It was mostly a Core 1.0 artifact.

In extremis:

  • System.Security.Cryptography.Primitives
  • System.Security.Cryptography.Encoding
  • System.Security.Cryptography.Algorithms
  • System.Security.Cryptography.Csp
  • System.Security.Cryptography.Cng
  • System.Security.Cryptography.OpenSsl
  • System.Security.Cryptography.X509Certificates

I believe that all 7 are included in the shared framework on all OSes.

@vcsjones
Copy link
Member Author

There are some "reflection-y" things that happen with CryptoConfig and various formatters in S.S.C.Xml. Do type forwards work with assembly qualified type names, like System.Security.Cryptography.RSA, System.Security.Cryptography.Algorithms?

@bartonjs
Copy link
Member

Yep, it works.

@vcsjones
Copy link
Member Author

Yep, it works.

Ah, thanks.

System.Private.Security.Cryptography

Are there any resource on the benefits of the first approach? I know that's done in some places like CoreLib, Uri, and Xml, but I don't quite see why.

@vcsjones
Copy link
Member Author

Another issue we may be able to close if we merge these assemblies is #29555.

@ericstj
Copy link
Member

ericstj commented Jul 21, 2021

Keep in mind TypeForwardedFrom for any serializable types (hopefully not an issue). You will of course need to keep the old assemblies as facades for compatibility but those can be a one-time cost and effectively ignored after that.

You still may want to enforce some layering in your own codebase but that can be done through convention and code review.

You may end up with PNSEs for surface area that’s smaller than the assembly. Today you benefit from the PNSE generator but you’ll need to either check that in or we may need a better partial PNSE solution.

@bartonjs
Copy link
Member

Are there any resource on the benefits of [using a System.Private.*]?

I chatted with Eric via a different medium, and concluded that System.Private.* isn't the route we want to take.

The System.Private.* libraries started when we had the small contract assemblies in .NET Core 1.0. Whenever we felt like it was important (or maybe just expedient) to have multiple contracts backed by one implementation assembly a System.Private was born. The feeling back then, IIRC, was "if you can avoid it, please avoid it".

So the System.Private.* assemblies seem to, largely, represent "we wish we could sensibly split these apart, and want to make less work on ourselves when we do so". Since we're instead saying "we don't think it makes sense to keep these split apart", we don't need the "Private" artifact.

(And if I misunderstood, hopefully someone corrects me)

@bartonjs bartonjs changed the title Consider moving PemEncoding and PemFields to S.S.Cryptography.Primitives Merge all of the inbox S.S.C.* libraries into one Nov 1, 2021
@bartonjs bartonjs self-assigned this Nov 1, 2021
@bartonjs
Copy link
Member

bartonjs commented Nov 1, 2021

With the stage 1 PR I got to avoid platform-differentiated code. Stage 2 is being less nice.

We've evolved ourselves into having a couple of different "standards" in file pathing for how we have platform splits. For example:

  • System.Security.Cryptography.Encoding
    • All: Internal\Cryptography\OidLookup.cs
    • Windows: Internal\Cryptography\OidLookup.Windows.cs
    • Linux/FreeBSD: Internal\Cryptography\OidLookup.Unix.cs
    • Android\Apple: Internal\Cryptography\OidLookup.NoFallback.cs
  • System.Security.Cryptography.X509Certificates
    • All: Internal\Cryptography\FindPal.cs
    • All: Internal\Cryptography\IFindPal.cs
    • Android: Internal\Cryptography\Pal.Android\FindPal.cs
    • Apple: Internal\Cryptography\Pal.OSX\FindPal.cs
    • Linux/FreeBSD: Internal\Cryptography\Pal.Unix\FindPal.cs
    • Android/Apple/Linux/FreeBSD ("!Windows"): Internal\Cryptography\Pal.Unix\ManagedCertificateFinder.cs
    • Linux/FreeBSD: Internal\Cryptography\Pal.Unix\OpenSslCertificateFinder.cs
    • Windows: Internal\Cryptography\Pal.Windows\FindPal.cs

Note that sometimes Apple is split into macOS and iOS (and in one case tvOS), but not in these examples.

In addition to getting rid of "Internal\Cryptography" as part of this move, it feels like we should try to come up with a better path/name scheme.

I think my strawman is going to be

  • Namespace\As\Path: AnyOS stuff that isn't involved in platform splits. Additionally, the common portion of any type that is public and is involved in a platform split.
  • Namespace\As\Path\Pal: ("Pal" and subdirectories not contributing to the namespace): Auxiliary types included by every platform that just set up the concept of PAL-ling.
  • Namespace\As\Path\Pal\Managed: Any files, like ManagedCertificateFinder.cs, that are contributing to a "non-platform" implementation. Probably included across multiple non-Windows systems.
  • Namespace\As\Path\Pal\Windows: Windows versions of files. e.g. OidLookup.Windows.cs goes here as just OidLookup.cs.
  • Namespace\As\Path\Pal\OpenSsl: Non-"interop" files that are involved in interop logic to OpenSSL via libSystem.Security.Cryptography.Native.so
  • Namespace\As\Path\Pal\Apple: "Apple common non-interop"
  • Namespace\As\Path\Pal\macOS: When macOS and iOS disagree, the macOS version.
  • Namespace\As\Path\Pal\iOS: When macOS and iOS disagree, the iOS version.
  • Namespace\As\Path\Pal\Android: Android

iOS/macOS could also easily be subdirectories under Apple, or extension variants (e.g. System\Security\Cryptography\Pal\Apple\RSASecurityTransforms.macOS.cs)

The idea behind this strawman being that we name the directories less about where they run "Unix" and more about what crypto provider they're working with "Windows", "OpenSSL", "Apple" (Security.framework), "Android" (Android JDK/JNI), and "Managed" (not PAL'd).

So for the earlier example files:

  • System\Security\Cryptography\Pal\OidLookup.cs
  • System\Security\Cryptography\Pal\Windows\OidLookup.cs (former OidLookup.Windows.cs)
  • System\Security\Cryptography\Pal\OpenSsl\OidLookup.cs (former OidLookup.Unix.cs)
  • System\Security\Cryptography\Pal\Managed\OidLookup.cs (former OidLookup.NoFallback.cs)
  • System\Security\Cryptography\X509Certificates\Pal\FindPal.cs
  • System\Security\Cryptography\X509Certificates\Pal\IFindPal.cs
  • System\Security\Cryptography\X509Certificates\Pal\Android\FindPal.cs
  • System\Security\Cryptography\X509Certificates\Pal\Apple\FindPal.cs
  • System\Security\Cryptography\X509Certificates\Pal\OpenSsl\FindPal.cs
  • System\Security\Cryptography\X509Certificates\Pal\Managed\ManagedCertificateFinder.cs
  • System\Security\Cryptography\X509Certificates\Pal\OpenSsl\OpenSslCertificateFinder.cs
  • System\Security\Cryptography\X509Certificates\Pal\Windows\FindPal.cs

Not sure I'm happy with that, though, so definitely open to feedback.

@stephentoub I'd be happy to get your insight/opinion here.

@vcsjones
Copy link
Member Author

vcsjones commented Nov 1, 2021

"open mic" style feedback...

System\Security\Cryptography\Pal\Managed\OidLookup.cs

Having a managed "PAL" seems odd.

..\Android\FindPal.cs
..\Apple\FindPal.cs
..\OpenSsl\FindPal.cs

I really struggle with files with the same name in different directories. It makes code navigation harder (Cmd+P) and it when looking at a tab row of many open files, they all have the same name so I just end up lost. I greatly prefer file-based suffixes instead of directories that contain a handful of files with lots of overlapping names. If that is not "the way", so be it, but it's one of the things that has bugged me quite a bit.

@bartonjs
Copy link
Member

bartonjs commented Nov 1, 2021

That’s a good point. It has also frustrated me, but somehow that didn’t occur to me when I was strawmanning.

@bartonjs
Copy link
Member

bartonjs commented Nov 1, 2021

Once the files have unique names, I guess I should question whether there's value in PAL directories at all. The problem I have with them a bit right now is that "Android" can be in both "Android" and "Unix", but Linux is only in "Unix", though not necessarily all of Unix (since that could also be something shared by Apple and Android); so if they're present it feels like they should be about the provider, but I'm open to no-directory, too.

No extra directories would then end up with something like

  • System\Security\Cryptography\Pal\OidLookup.cs
  • System\Security\Cryptography\Pal\Windows\OidLookup.Windows.cs
  • System\Security\Cryptography\Pal\OpenSsl\OidLookup.OpenSsl.cs (former OidLookup.Unix.cs)
  • System\Security\Cryptography\Pal\Managed\OidLookup.NoFallback.cs
  • System\Security\Cryptography\X509Certificates\FindPal.cs
  • System\Security\Cryptography\X509Certificates\IFindPal.cs (could perhaps be combined with FindPal into an abstract class over an interface)
  • System\Security\Cryptography\X509Certificates\AndroidCertificateFinder.cs
  • System\Security\Cryptography\X509Certificates\AppleCertificateFinderl.cs
  • System\Security\Cryptography\X509Certificates\OpenSslCertificateFinder.cs (merging FindPal and OpenSslCertificateFinder together)
  • System\Security\Cryptography\X509Certificates\ManagedCertificateFinder.cs
  • System\Security\Cryptography\X509Certificates\WindowsCertificateFinder.cs

@bartonjs
Copy link
Member

bartonjs commented Nov 4, 2021

@vcsjones: Since Encodings just got folded in to the new library, AsymmetricAlgorithm and PemEncoding are in the same library now. Since that was the catalyst for the squishification, thought I'd point that out explicitly.

@stephentoub
Copy link
Member

I really struggle with files with the same name in different directories.

+1. This is a key reason we went with *.Suffix.cs.

Can we get rid of the "\Pal" directory as well? Doesn't seem to be adding much benefit. Ideally we end up with all the partials right next to each other, e.g.
OidLookup.cs
OidLookup.Windows.cs
OidLookup.OpenSsl.cs
etc.

@bartonjs
Copy link
Member

bartonjs commented Nov 9, 2021

Can we get rid of the "\Pal" directory as well? Doesn't seem to be adding much benefit. Ideally we end up with all the partials right next to each other, e.g.

Oh, good, that's the approach I'm currently pursuing with squishing Algorithms down into the unified library 😄.

@bartonjs
Copy link
Member

Closed via #64307.

@ghost ghost locked as resolved and limited conversation to collaborators Mar 2, 2022
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Projects
None yet
Development

No branches or pull requests

4 participants