Add PGO for F# compiler

[psfinaki]

This should eventually make the FSC about 30% faster.

The latest FCS package contains these MIBC files:

In the SDK part of it, we need to unpack them, apply to the compiler, and repack.

[vzarytovskii]

This will also require adding mibc dependencies via darc, so they are auto-updated.

[psfinaki]

This will also require adding mibc dependencies via darc, so they are auto-updated.

Right. I added them to Version.Details.xml - is the right way to go?

[vzarytovskii]

This will also require adding mibc dependencies via darc, so they are auto-updated.

Right. I added them to Version.Details.xml - is the right way to go?

Yes, but they also should be enabled in darc for the msbuild, so they are updated automatically. But this should explicitly be agreed upon with sdk folks. @baronfel is it the way to go? Or should we pack profiles with our nuget (or separate nuget) and when repacking, use them from there?

I would personally say the latter is a better way to go, since we probably also want to use those profiles when building proto.

[baronfel]

I'm not sure what's best from an engineering PoV - @joeloff do you have an opinion here?

[vzarytovskii]

Thinking again about this, I think us packing profiles should be better, since we know how to manage them and which channel is correct for which branch (we had to use 8 before and now switched to 9).

[psfinaki]

We were looking at this with @KevinRansom and most likely F# is the first one to try to do this via SDK (there is no notion of embed-pgo-data and not even a way to give extra args to xgen2). Hence the precise way to do this should indeed be agreed on in case some other .NET component decides to copypaste get inspired by this.

For reference, here is the alternative approach in the F# repo, in case we decide to do it from from within the compiler itself.

[vzarytovskii]

I vote we distribute profiles with our packages (maybe new package), since

1. We know which channel to consume them from for any given branch (it's going to be 9 for a while for all branches, then eventually 10 for net10).
2. Ae want to use them when we build proto compiler, and to not duplicate effort, targets, etc, we download them once in our CI, use for proto and pack them for product, and when repacking here, we just assume they're part of our package(s).

[psfinaki]

Any other opinions here? @baronfel @joeloff @KevinRansom?
We'd need to decide soon-ish if we want to get this into .NET 9 which would be definitely nice to have.

I am biased since we kind of have a working thing in F# whereas this PR will take some time to finalize so ofc laziness is involved but it's important to do things "right" here.

Certainly we can also merge the F# repo implementation and then revert later if needed.

[vzarytovskii]

I thought about it over the weekend, and probably going to insist that we do it in our repo:

1. Download profiles there, as we already do, use it on Proto compiler
2. Pack them when publishing product compiler, from signed CI (together or separate package, doesn't really matter).
3. Unpack them in this repo, and reapply while repacking.

This way we are flexible to do whatever we want without making changes to this repo. We know exactly which channel to pull packages from for every given branch. We can transparently change implementation if we want it (either use runtime provided ones, or generate our own), without changes to this repo. It is unlikely that we will be forgetting to update darc for sdk every time we update branches, etc.

[psfinaki]

/azp run

[azure-pipelines]

Azure Pipelines successfully started running 4 pipeline(s).

[psfinaki]

/azp run

[azure-pipelines]

Azure Pipelines successfully started running 4 pipeline(s).

[psfinaki]

CI seems to be flaky but overall this looks like a viable approach. The MIBC files are present in the packages already:

@KevinRansom @vzarytovskii please rereview.

[psfinaki]

Okay, this is green 😎

[KevinRansom]

When the sdk builds it builds cross target --- ie the tooling used to build the product doesnot necessarily match the build target.

When you do mibc you need to match the build target not the tooling platform.

[psfinaki]

Hmmm alright, I am trying to leverage NETCoreSdkRuntimeIdentifier now - is this the right thing?

[psfinaki]

Probably not :)

[KevinRansom]

When the sdk builds it builds cross target --- ie the tooling used to build the product doesnot necessarily match the build target.

When you do mibc you need to match the build target not the tooling platform.

[psfinaki]

Okay, that's green again... @mmitche @vzarytovskii @baronfel @KevinRansom how does this look like to you?

[baronfel]

Is this valid @mmitche? How portable are PGO profiles, can they float across different libc ABIs?

[mmitche]

Not sure. I'm going to draw a random name out of the hat on that one and tag @agocke for general knowledge.

[psfinaki]

Maybe @EgorBo?

[psfinaki]

@KevinRansom do you have some thoughts here?