Complex.Sqrt comply with IEEE 754 and use double.Hypot

[lilinus]

Contributes to #104233

[dotnet-policy-service]

Tagging subscribers to this area: @dotnet/area-system-numerics
See info in area-owners.md if you want to be subscribed.

[tannergooding]

Looks like there's some test failures caused by using Hypot.

In particular Complex.Abs for Complex(NaN, -inf) is saying it expects NaN, but double.Hypot produces +inf instead. The behavior of double.Hypot strictly matches the IEEE 754 rules which dictate:

For the hypot operation, hypot(±0, ±0) is +0, hypot(±∞, qNaN) is +∞, and hypot(qNaN, ±∞) is +∞.

The C language specification (which is typically considered the source of truth for IEEE 754 compliance of Complex arithmetic) in G.6 Complex artihmetic dictates

Each of the functions cabs and carg is specified by a formula in terms of a real function (whose special cases are covered in Annex F):

cabs(x + iy) = hypot(x, y)
carg(x + iy) = atan2(y, x)

Where Annex F then matches the IEEE 754 rules, thus the new behavior is strictly correct and is fixing a latent bug in this edge case.

---

There is a similar failure for Complex.Sqrt given Complex(-inf, NaN) where it is expecting the imaginary result to be NaN but the actual result is +inf

G.6.4.2 The csqrt functions in the C language specification dictates:

— csqrt(conj(z)) = conj(csqrt(z)).
— csqrt(±0 + i0) returns +0 + i0.
— csqrt(x + i∞) returns +∞+ i∞, for all x (including NaN).
— csqrt(x + iNaN) returns NaN + iNaN and optionally raises the "invalid" floating-point exception, for finite x.
— csqrt(−∞ + iy) returns +0 + i∞, for finite positive-signed y.
— csqrt(+∞+ iy) returns +∞+ i0, for finite positive-signed y.
— csqrt(−∞ + iNaN) returns NaN±i∞ (where the sign of the imaginary part of the result is unspecified).
— csqrt(+∞+ iNaN) returns +∞+ iNaN.
— csqrt(NaN + iy) returns NaN + iNaN and optionally raises the "invalid" floating-point exception, for finite y.
— csqrt(NaN + iNaN) returns NaN + iNaN.

And thus this is another case where the new behavior is strictly more correct. There's one other edge here that appears to not be handled correctly as well, but which aren't failing the tests. In particular Complex(NaN, +inf) should be producing Complex(+inf, +inf) but is actually producing Complex(NaN, NaN)