Skip to content
This repository has been archived by the owner on Jan 23, 2023. It is now read-only.
/ corefx Public archive

Auto-generate the reference assemblies to normalize them #39681

Merged
30 commits merged into from
Jul 25, 2019

Conversation

ahsonkhan
Copy link
Member

Auto-generated the reference assemblies using /t:GenerateReferenceSource on all projects that already contained a /ref/ directory.
https://github.com/dotnet/corefx/blob/master/Documentation/coding-guidelines/updating-ref-source.md

  • Sorting alphabetically to be consistent, and fix formatting.
  • Fully qualify the types.
  • Enums are ordered numerically.
  • Follow consistent file name guidelines (fixing the Utf8String ref specifically).
  • Left System.Runtime as is for now other than a cosmetic change.
  • Removed nullability annotation from explicit interface implementations (based on discussion with @safern).

Partially addresses https://github.com/dotnet/corefx/issues/29737

There is no additional/removal of APIs in this change. The diff between this PR and master is empty.

@@ -113,7 +113,7 @@ public partial class ConcurrentDictionary<TKey, TValue> : System.Collections.Gen
void System.Collections.Generic.IDictionary<TKey, TValue>.Add(TKey key, TValue value) { }
bool System.Collections.Generic.IDictionary<TKey, TValue>.Remove(TKey key) { throw null; }
void System.Collections.ICollection.CopyTo(System.Array array, int index) { }
void System.Collections.IDictionary.Add(object key, object? value) { }
void System.Collections.IDictionary.Add(object key, object value) { }
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@safern, is this expected?

It don't think it actually matters in such an explicit interface case, because no one will call via this signature, but... it's a little strange to be undoing it.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I mentioned this in the original PR comment. I talked to @safern about this, and his change to GenAPI (to make it nullable aware, which I used here - dotnet/arcade#2679) removes the nullablility annotations on explicit interface implementations.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The reason why it removes it is because we’re no longer preserving nullable metadata on non-public members, since explicit interfaces are non-public the compiler drops them, so we don’t have a way of knowing it’s nullability in GenAPI. I don’t love the idea but i can’t think of a way to preserve its nullability but by directly using the information in the interface declaration itself.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just to highlight explicitly, this change does cause a compiler warning which we are disabling: CS8617

dotnet/roslyn#30845

cc @AlekseyTs, @gafter

@stephentoub
Copy link
Member

I'm curious, what process do you use to generate this... do you have a script that iterates through all of the project directories and builds with /t:GenerateReferenceSource, or is there a top-level build command for doing that?

public const float PositiveInfinity = (float)1.0 / (float)0.0;
public const float NegativeInfinity = (float)-1.0 / (float)0.0;
public const float NaN = (float)0.0 / (float)0.0;
public const float Epsilon = 1.4E-45f;
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@tannergooding - for some reason the auto-gen changes the value to 1.401298E-45f.

Similarly, the other values (MaxValue/MinValue) show up incorrectly as well. I kept the values that already exist but not sure what the intention is here:

public const float MaxValue = 3.40282347E+38f;
public const float MinValue = -3.40282347E+38f;

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm guessing because the improved algorithm identifies the shorter forms as bit-equivalent in the floating point representation, and more desirable as they're shorter.

Copy link
Member Author

@ahsonkhan ahsonkhan Jul 23, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The current ref isn't shorter in all cases, and what GenAPI is proposing isn't either.

Current (and what's in master/this PR):

public const float Epsilon = 1.4E-45f;
public const float MaxValue = 3.40282346638528859E+38f;

What GenAPI was suggesting changing this to:

public const float Epsilon = 1.401298E-45f; // this is a longer form
public const float MaxValue = 3.40282347E+38f; // this is a shorter form

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ah, @tannergooding would have the explain that.

@ahsonkhan
Copy link
Member Author

I'm curious, what process do you use to generate this... do you have a script that iterates through all of the project directories and builds with /t:GenerateReferenceSource, or is there a top-level build command for doing that?

Basically yes, but its not a script (mainly because I am not super familiar with the how msbuild targets work, particularly in corefx). I wrote a C# tool that does this (enumerate through all ref directories and run dotnet msbuild /t:GenerateReferenceSource).

@ahsonkhan
Copy link
Member Author

ahsonkhan commented Jul 23, 2019

Failing CI leg seems unrelated - corefx-ci (Windows x64_Debug):
https://mc.dot.net/#/user/dotnet-bot/pr~2Fdotnet~2Fcorefx~2Frefs~2Fpull~2F39681~2Fmerge/test~2Ffunctional~2Fcli~2F~2F/20190722.37/workItem/System.Security.Cryptography.X509Certificates.Tests/wilogs
System.Security.Cryptography.X509Certificates.Tests
Windows.10.Amd64.ClientRS4.ES.Open-x64:Debug

C:\dotnetbuild\work\aefda20e-4b85-49d2-bdec-4b4caec98676\Work\8c6601c9-f8d1-42cf-b958-2ea0fd0376fd\Exec>"C:\dotnetbuild\work\aefda20e-4b85-49d2-bdec-4b4caec98676\Payload\dotnet.exe" exec --runtimeconfig System.Security.Cryptography.X509Certificates.Tests.runtimeconfig.json xunit.console.dll System.Security.Cryptography.X509Certificates.Tests.dll -xml testResults.xml -nologo -nocolor -notrait category=nonnetcoreapptests -notrait category=nonwindowstests -notrait category=IgnoreForCI -notrait category=OuterLoop -notrait category=failing  
  Discovering: System.Security.Cryptography.X509Certificates.Tests (method display = ClassAndMethod, method display options = None)
  Discovered:  System.Security.Cryptography.X509Certificates.Tests (found 451 of 469 test cases)
  Starting:    System.Security.Cryptography.X509Certificates.Tests (parallel test collections = on, max threads = 4)
----- end 23/07/2019  1:30:25,83 ----- exit code -1073740791 ----------------------------------------------------------
 23/07/2019- 1:30:25,83
 23/07/2019- 1:30:26,33
Got args https://dev.azure.com/dnceng/ public 7601934 eyJ0eXAiOiJKV1QiLCJhbGciOiJSUzI1NiIsIng1dCI6Im9PdmN6NU1fN3AtSGpJS2xGWHo5M3VfVjBabyJ9.eyJuYW1laWQiOiJlNDI4NWM4Yy0zZmQ1LTQyNjctOWIxYy00MjE5NWM0N2E1NTMiLCJzY3AiOiJMb2NhdGlvblNlcnZpY2UuQ29ubmVjdCBSZWFkQW5kUHVibGlzaFRlc3Q6OWVlNmQ0NzgtZDI4OC00N2Y3LWFhY2MtZjZlNmQwODJhZTZkIFJlYWRBbmRVcGRhdGVCdWlsZEJ5VXJpOjllZTZkNDc4LWQyODgtNDdmNy1hYWNjLWY2ZTZkMDgyYWU2ZC9kb3RuZXQvY29yZWZ4LzE5NjpCdWlsZC9CdWlsZC8yNzUyNTMiLCJhdWkiOiIyYmI2MzQxYi0zYzBlLTRhNzItOWI3ZC1lMDc3MmQ2NTljNzIiLCJzaWQiOiJmNjU5OTdmZC00MDBlLTRkNTctOWE4ZS1lODUyMzRkMmIzNDkiLCJpc3MiOiJhcHAudnN0b2tlbi52aXN1YWxzdHVkaW8uY29tIiwiYXVkIjoiYXBwLnZzdG9rZW4udmlzdWFsc3R1ZGlvLmNvbXx2c286YjU1ZGU0ZWQtNGI1YS00MjE1LWE4ZTQtMGEwYTVmNzFlN2Q4IiwibmJmIjoxNTYzODQxMDI5LCJleHAiOjE1NjM4NTEyMjl9.CPXfNF5h_AjXcjlVFgvKXHrN24Zoju08Jivwe3v8L-18L2u3AeG_3uXBXr8OjfxIwzphifPcMFLlO3B4hbPLiYYMc1RAS2lwtkTTkdVgrVfRog00K6JB_Od5klvGV0cwZ8CR022AU1u3gwN4ZvFoZ2gEZ4JmI_yOEj5sEegavnyN1FTH3HLr3FwYayr-BFNKS_YgW5xcy5Fn1aBsjjEni4hyabcPVfouizzsldSshT1l_OtJmwgtaDaQ7WDtAS9pS2I0RlGsGNvR2UDdgJ1p2HmZRBo7sxm7wr1YJ-gPO2PgqYg4ptOr2Am_KYuqzeRU4CpzJIpm0b4U0jlzsMCisg
Main thread starting workers
Worker 0: starting...
Worker 1: starting...
Worker 2: starting...
Worker 3: starting...
Worker 4: starting...
Worker 5: starting...
Worker 6: starting...
Worker 7: starting...
Worker 8: starting...
Worker 9: starting...
Beginning reading of test results.
Uploading results in batches of size 1000
Searching 'C:\dotnetbuild\work\aefda20e-4b85-49d2-bdec-4b4caec98676\Work\8c6601c9-f8d1-42cf-b958-2ea0fd0376fd\Exec\..' for log files
Found log 'C:\dotnetbuild\work\aefda20e-4b85-49d2-bdec-4b4caec98676\Work\8c6601c9-f8d1-42cf-b958-2ea0fd0376fd\Exec\..\console.235352a3.log'
Uri 'https://helixexternalresults.blob.core.windows.net/dotnet-corefx-refs-pull-39681-merge-aefda20e4b8549d2bd/System.Security.Cryptography.X509Certificates.Tests/console.235352a3.log?sv=2017-07-29&sr=c&sig=qGEoeAvZ2vowco6WTmaPaOG4BhaUUQp83bgUFaGm1Cw%3D&se=2019-08-02T00%3A49%3A35Z&sp=rl'
Generated log list: <ul><li><a href='https://helixexternalresults.blob.core.windows.net/dotnet-corefx-refs-pull-39681-merge-aefda20e4b8549d2bd/System.Security.Cryptography.X509Certificates.Tests/console.235352a3.log?sv=2017-07-29&sr=c&sig=qGEoeAvZ2vowco6WTmaPaOG4BhaUUQp83bgUFaGm1Cw%3D&se=2019-08-02T00%3A49%3A35Z&sp=rl' target='_blank'>console.235352a3.log</a></li></ul>
Searching 'C:\dotnetbuild\work\aefda20e-4b85-49d2-bdec-4b4caec98676\Work\8c6601c9-f8d1-42cf-b958-2ea0fd0376fd\Exec' for test results files
No results file found in any of the following formats: xunit, junit, trx
Main thread finished queueing batches
Worker 0: uploaded 1 results
Main thread exiting
 23/07/2019- 1:30:27,53
2019-07-23 01:30:28,067: ERROR: xunit-reporter(69): main: Unable to report xunit results: no test results xml file found.

@stephentoub
Copy link
Member

That's https://github.com/dotnet/corefx/issues/38824.
cc: @bartonjs

@ahsonkhan
Copy link
Member Author

@stephentoub, do you think its worth porting this change to the release/3.0 branch to minimize diffs in the future (if someone was to modify the ref)? I don't see a great reason other than general clean up.

@ahsonkhan ahsonkhan self-assigned this Jul 23, 2019
@stephentoub
Copy link
Member

@stephentoub, do you think its worth porting this change to the release/3.0 branch to minimize diffs in the future (if someone was to modify the ref)?

I don't think so. We generally see very few, if not zero, changes to refs as part of servicing, and there's enough churn here that it's not clear how low the risk is of an unintentional break.

But, I'll let @danmosemsft comment if he disagrees. If he wants to take it, that's fine.

@danmoseley
Copy link
Member

Is there any way this would significantly impact customers using 3.0 if we didn't?

@bartonjs
Copy link
Member

Is there any way this would significantly impact customers using 3.0 if we didn't?

If the change has the effects it hopes to: There's no impact to customers either way. (Hence, I'd say we don't want it, we never touch the ref.cs files in servicing)

@danmoseley
Copy link
Member

If the change has the effects it hopes to: There's no impact to customers either way.

OK. I had a quick eyeball for renamed parameter or added API and I didn't see any.

@ahsonkhan
Copy link
Member Author

Updating System.Runtime ref as well (filed an issue for the one outlier: dotnet/arcade#3368) related to ValueTuple that had to be manually reverted back.

@@ -33,6 +33,7 @@ public partial struct Single
// We need to add this into the manual ref assembly to preserve it because the
// implementation doesn't have any reference field, hence GenApi will not emit it.
private object _dummy;
// Placing the value type field in the manual ref as well to avoid the error CS0282: There is no defined ordering between fields in multiple declarations of partial struct 'TypedReference'.
private int _dummyPrimitive;
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Since the value type field does get generated by GenAPI (for the IntPtr fields), wanted to add an explicit comment to highlight why we want to keep that in the manual file instead.

https://github.com/dotnet/coreclr/blob/10bf24055157e962b50c6bb25865f69fb055cd98/src/System.Private.CoreLib/src/System/TypedReference.cs#L19-L22

cc @safern since you added this in your change here: #37737

@ahsonkhan ahsonkhan added the auto-merge Automatically merge PR once CI passes. label Jul 24, 2019
@ghost
Copy link

ghost commented Jul 24, 2019

Hello @ahsonkhan!

Because this pull request has the auto-merge label, I will be glad to assist with helping to merge this pull request once all check-in policies pass.

p.s. you can customize the way I help with merging this pull request, such as holding this pull request until a specific person approves. Simply @mention me (@msftbot) and give me an instruction to get started! Learn more here.

public const FileAttribute vbSystem = FileAttribute.System;
public const MsgBoxStyle vbSystemModal = MsgBoxStyle.SystemModal;
public const Microsoft.VisualBasic.VariantType vbObject = Microsoft.VisualBasic.VariantType.Object;
public const int vbObjectError = -2147221504;
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

cc @cston, @333fred to spot check some of these changes. From my perspective, most of these normalizations are correct and cosmetic (there should be no observable change from the callers view).

[System.ObsoleteAttribute("This member has been deprecated. Please use FilePutObject to write Object types, or coerce FileNumber and RecordNumber to Integer for writing non-Object types. http://go.microsoft.com/fwlink/?linkid=14202")]
public static void FilePut(object FileNumber, object Value, object RecordNumber/* = -1*/) { }
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Note, we are losing this comment. Maybe it would be worth moving this to a .Manual file (possibly outside the PR).

@ghost ghost merged commit bdd0814 into dotnet:master Jul 25, 2019
@ahsonkhan ahsonkhan deleted the NormalizeRefAssembly branch July 25, 2019 21:52
@karelz karelz added this to the 5.0 milestone Aug 3, 2019
macrogreg pushed a commit to open-telemetry/opentelemetry-dotnet-instrumentation that referenced this pull request Sep 24, 2020
…fx#39681)

* Update Microsoft.VisualBasic.Core ref

* Update System.Collections.Concurrent ref

* Update System.Collections.Immutable ref

* Update System.Collections ref

* Update System.ComponentModel.Composition ref

* Update System.Data.Common ref

* Update System.Data.OleDb ref

* Update System.Data.SqlClient ref for netcoreapp

* Update System.Diagnostics.DiagnosticSource ref and DiagnosticSourceActivity.cs

* Update System.IO.Packaging ref for netcoreapp

* Update System.IO.Pipelines ref

* Update System.Linq ref

* Update System.Net.Http ref

* Update System.Net.Security ref

* Update System.Numerics.Tensors ref

* Update System.Reflection.Emit for netcore

* Update System.Security.AccessControl ref

* Update System.Security.Permissions ref

* Update System.ServiceModel.Syndication ref

* Update System.Text.Encodings.Web ref

* Update System.Threading.Channels ref for netcoreapp

* Update System.Utf8String.Experimental ref

* Update System.Utf8String.Experimental csproj to reference new file.

* Upat eSystem.Runtime ref and Manual ref

* Update Microsoft.Bcl.AsyncInterfaces ref

* Disable warning CS8617 for missing nullability annotations within
explicit interface methods.

* Revert System.Runtime.cs back to what's in master.

* Update System.Runtime ref (single change - System.Char -> char).

* Revert System.Collections.Immutable back to what's in master (keep
readonly on private fields).

* Update System.Runtime ref and add comment in Manual ref file for
TypedReference.


Commit migrated from dotnet/corefx@bdd0814
picenka21 pushed a commit to picenka21/runtime that referenced this pull request Feb 18, 2022
…fx#39681)

* Update Microsoft.VisualBasic.Core ref

* Update System.Collections.Concurrent ref

* Update System.Collections.Immutable ref

* Update System.Collections ref

* Update System.ComponentModel.Composition ref

* Update System.Data.Common ref

* Update System.Data.OleDb ref

* Update System.Data.SqlClient ref for netcoreapp

* Update System.Diagnostics.DiagnosticSource ref and DiagnosticSourceActivity.cs

* Update System.IO.Packaging ref for netcoreapp

* Update System.IO.Pipelines ref

* Update System.Linq ref

* Update System.Net.Http ref

* Update System.Net.Security ref

* Update System.Numerics.Tensors ref

* Update System.Reflection.Emit for netcore

* Update System.Security.AccessControl ref

* Update System.Security.Permissions ref

* Update System.ServiceModel.Syndication ref

* Update System.Text.Encodings.Web ref

* Update System.Threading.Channels ref for netcoreapp

* Update System.Utf8String.Experimental ref

* Update System.Utf8String.Experimental csproj to reference new file.

* Upat eSystem.Runtime ref and Manual ref

* Update Microsoft.Bcl.AsyncInterfaces ref

* Disable warning CS8617 for missing nullability annotations within
explicit interface methods.

* Revert System.Runtime.cs back to what's in master.

* Update System.Runtime ref (single change - System.Char -> char).

* Revert System.Collections.Immutable back to what's in master (keep
readonly on private fields).

* Update System.Runtime ref and add comment in Manual ref file for
TypedReference.


Commit migrated from dotnet/corefx@bdd0814
This pull request was closed.
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
area-Meta auto-merge Automatically merge PR once CI passes.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

7 participants