Proposal: allow generic methods to specify T? without constraining to class or struct

[roji]

Currently, specifying T? in a generic method signature requires it to be constrained to either a struct or class. The proposal is to allow the following without any constraints:

T? Foo<T>() {...}

For value types, the method would return Nullable<T> as usual, and for reference types it would return a nullable reference for the purposes of compiler analysis.

As an example, when looking at a better (and more null-friendly) API for fetching database column values on DbDataReader, I am currently blocked by the inability to define a generic method that returns T?, and that can work on both value and reference types. The only alternative would be to define two methods - one constrained on class and the other on struct - but since overloading on generic constraints isn't supported they must have different names. This would force the user to deal with the value/reference type distinction - figuring out which method to call - and seems like a non-starter for the kind of API we're looking for.

More generally, it seems that C# 8 nullability introduces more difficulties when intersecting with the value/reference type divide - developers are unfortunately forced to think about this distinction more than before nullability existed. It seems like it's important to smooth that out (see #1865 in general for diminishing the feature gap). As an example, C# already allows one to use (nullable) references and Nullable<T> interchangeably in several contexts: the null coalescing and null-conditional operators accept both. It would be good to go further in this direction.

This issue may be a dup of #2146, although that issue seems a bit unclear. See also dotnet/roslyn#30953 which is somewhat related, although that issue is about returning default, whereas here the idea is to allow returning null for both reference and value types.

/cc @divega @ajcvickers

[john-h-k]

This isn't currently possible in C# 8?!

[roji]

This isn't currently possible in C# 8?!

Not yet :)

[HaloFour]

For value types, the method would return Nullable<T> as usual, and for reference types it would return a nullable reference for the purposes of compiler analysis.

How do you propose that actually work? A method can't selectively change its return value. A generic method has the same signature and body regardless of the generic type arguments, it can't return T sometimes and Nullable<T> other times.

[DanFTRX]

@HaloFour Could it possibly generate 2 methods at compile time, one with a Nullable constrained to struct and one with a nullable reference? I know it would not be valid language code to define two different return types with same function name due to ambiguity, but is that a runtime restriction or a compile restriction? (I know its just a compile restriction for Java as I've done it in the past, not sure about .NET). If its just a compile time ambiguity, then it is at least be possible to do this.

[roji]

You're right of course - this isn't just a compiler-level feature, it would require deeper changes in other places. For example, the combination of a [Nullable] attribute along with the absence of a struct/class constraint could be interpreted (by the JIT, by the compiler of consuming code...) as meaning Nullable<T> when T is a struct, or T? when it is a class. Unfortunately that would be a breaking change as older compilers will only see a T return value. I'm hoping someone more knowledgeable can take this further.

I do realize that on the implementation level this is a pretty big/complicated ask, I'm only expressing the need at the language level. I'm mostly concerned that nullability is making things significantly more complicated than before.

PS If overloading on generic constraints were allowed (#2013) this would be much less of a problem, but that doesn't seem to be adopted in any way either.

[roji]

@DanFTRX the problem with generating two methods is that they would collide - same name with the same parameters...

[DanFTRX]

@roji I'm aware that it creates ambiguity at compile time, but that can be solved with compiler analytics since it would be constrained this specific situation. I do know from when I worked with Java in the past, it is possible to generate valid bytecode directly for 2 methods same names same parameters with the Java runtime, I assumed since C# is similar in regards to being a compiled runtime language, it might be possible to do it here as well.

[HaloFour]

I think your only option would be for the compiler to emit two different versions of the method.

The CLR does allow overloads based on different returning types, or even just on the application of modopt/modreq modifiers on parameters. My concern would be how they would work with downlevel compilers or other languages.

[roji]

@HaloFour you're right, that sounds like a more viable approach than anything impacting the runtime. So perhaps generating two methods - one for value types and one for reference types - is the right way here.

[Joe4evr]

I think that this requires a more configurable approach than what is proposed. Consider that there are also various open generic methods that don't return a Nullable<T> for value types, e.g. the XxxOrDefault() methods in LINQ. In order to properly integrate with the Nullability Analysis, a new recognized attribute should be able to be applied to open generic methods to specify how the return value has to be interpreted. I'm thinking something like this:

[return: OpenNullable(ValueTypeHandling.DefaultValue)] //compiler emits one method and in consuming code ignores the nullability flow if invoked with 'T' as a value type
public T? FirstOrDefault<T>(this IEnumerable<T> sequence);

[return: OpenNullable(ValueTypeHandling.Nullable)] //compiler emits two methods as suggested by HaloFour
public T? ReadValue<T>();

[roji]

@Joe4evr what you're thinking about seems to be covered by dotnet/roslyn#30953?

Methods like xxxOrDefault(), which don't return a Nullable<T> for value types, shouldn't have a return value of T? - this has been a synonym for Nullable<T> for a long time. The current plan seems to be to handle your case with a [MaybeNull] attribute, which would mean that T is nullable only if it's a reference type.

[YairHalberstadt]

I have no idea how practical this actually is to implement, but I think it is extremely necessary if we want to be able to write truly generic code whilst dealing with nullability.

[omariom]

If it is not a virtual method, this approach could help
https://twitter.com/xoofx/status/1087770199854665729?s=20

[JeroenBos]

@roji: the name and parameters would collide yes, but what if type parameter constraint also distinguished? That would neatly separate these two cases. The method with class constraint would return T, and with struct would return T?

[Logerfo]

@JeroenBos that would be a breaking change, since the constraints are not considered as part of the method signature today. Changing this would make the new language spec only available for a new runtime version.

[YohDeadfall]

But any proposal that redefines existing syntax to mean something else just in the unconstrained generic context - just to make the attribute shorter - seems like a problematic direction to me.

The current trend is to make construction simpler. Remember patter matching, for example, or the null coalescing assignment operator.

I think that having ? as a may-be-nullable constraint is a good idea. It already exists and it's the class? constraint, but it's for reference types only and means nullable references are valid too. The following example demonstrates its behavior:

public static void DoWithNullable<T>(T value) where T : class?
{
}
    
public static void DoWintNonNullable<T>(T value) where T : class
{
}
    
public static void Main(string[] args)
{
    DoWithNullable<string?>(null); // OK
    DoWithNullable<string>("value"); // OK
       
    DoWintNonNullable<string?>(null); // Warning CS8634: Nullability of type argument 'string?' doesn't match 'class' constraint.
    DoWintNonNullable<string>("value"); // OK
}

The problem here is that there is no way to express the same thing for a struct and for any type (either reference or value). So having struct? and any? constraints could solve the issue, but at some cost. This is what @sharwell proposed.

[YohDeadfall]

Just for you information, class? is a syntax sugar which adds an attribute:

public static void DoWithNullable<[Nullable(2)] T>([Nullable(1)] T value) where T : class
{
}

public static void DoWintNonNullable<[Nullable(1)] T>([Nullable(1)] T value) where T : class
{
}

@jcouv Could you explain the meaning of magic values which NullableAttribute accepts?

[yaakov-h]

public static void DoWintNonNullable<[Nullable(1)] T>([Nullable(1)] T value) where T : class
{
}

This is incorrect. There is no [Nullable] attribute emitted in this case.

[YohDeadfall]

To see it you should do a little trick, add #nullable enable (:

[yaakov-h]

ah right, without that it gets emitted as oblivious...

and therein lies the answer!

Each type reference is accompanied by a NullableAttribute with an array of bytes, where 0 is Oblivious, 1 is NotAnnotated and 2 is Annotated. All value types are marked with flag 0 (oblivious).

https://github.com/dotnet/roslyn/blob/910a3331f730a435c3ced92888568924a789771c/docs/features/nullable-reference-types.md

[YohDeadfall]

@yaakov-h Thanks for answering the question to Julien! Out of curiosity, why is byte used instead of an enum?

[yaakov-h]

No idea. At a guess, to reduce the amount of code that needs to be embedded in the target assembly?

[roji]

See https://github.com/dotnet/csharplang/blob/master/meetings/2019/LDM-2019-05-15.md#unconstrained-t

[roji]

@jcouv in this section or the recent nullability LDM, it's stated that "There is therefore no real need for the ability to say T? on an unconstrained T, since [AllowNull]and[MaybeNull]` can be specified together.

However, the point of unconstrained T? (at least in this issue) is to have Nullable<T> when T is a value type, without having to constrain the generic method to value types only. Is this concern also being addressed in some way?

[jcouv]

@roji There are two views of what T? might mean. One view is that it means Nullable<T> when T is a struct, and the other is that it doesn't change the type (it's still a runtime T) but provides an annotation for reference types.

So far in the nullable feature, annotations do not change emitted/runtime types, so we mostly focused on the latter interpretation.
In the former interpretation, it is not even clear what would be emitted in IL: is it a T or a Nullable<T>?
It might be possible to do this with some runtime support, but it's not clear yet that the ability to express T or Nullable<T> as a type is that useful or desirable given the complexity.

[roji]

@jcouv thanks for the clarification.

Yeah, this issue is about T? meaning Nullable<T> for value types (the former interpretation), and above are some ideas on how that could happen (e.g. compiler emitting two methods - one for reference types and one for value types - overloaded on the return type). It's indeed complex, but it does seem highly desirable to have a way for a generic method to specify a nullable "something", without the user having to care whether it's a value or reference type.

The "latter interpretation", where T? is T in runtime seems problematic in that T? means Nullable<T> in every other context when T is a value type. It would be odd if T? meant something different in an unconstrained generic context and in a struct-constrained context. In addition, with the new [MaybeNull] and [AllowNull] it's not clear why this would even be necessary as the attributes suffice.

[KillyMXI]

I want to bring my perspective on nullable return types, especially out parameters. I believe this issue is the best fitting one to add to, but let me know if it is better to open a new issue.

When writing generic functional code, some language legacy forces us to write duplicate code. Most notorious example is when we have to implement the same code for IEnumerable<T> and T[]. Iirc, authors of extension libraries such as LanguageExt can bring up more examples, but that's beyond the point of this comment so I won't look it up now.

From what I see, the different handling of Nullable<T> and nullable reference types might surpass the pain of duplicating code existed before.

And what I find rather ironic, the new language feature doesn't work well with the established Try pattern.

---

Example 1:

(Taken from this StackOverflow question.)

Let's say we write an implementation of the Option/Maybe monad and want to provide a convenience method to make it integrate better into existing code by implementing a common .Net pattern...

public abstract Option<T> {} // exact implementation skipped as irrelevant
public class None<T> : Option<T>
public class Some<T> : Option<T> 

public static class OptionExtensions
{
   public static bool TryGetValue<T>(this Option<T> option, out T value)
   {
       if (option is Some<T> some)
       {
           value = some.Value;
           return true;
       }

       value = default;
       return false;
   }
}

Once we add NotNullWhenAttribute to out T value, we have to make the choice - do we need T to be a reference or value type.

But we need both of them. We don't want to narrow it down!

Thanks to @omariom for bringing up the workaround. Since the issue #1628 was fixed, we now can have multiple extension methods that differ only in where constraints within the same class. But we still have to have duplicate them for classes and structs.

For this particular example it looks acceptable. Just two copies...

First, we introduced non-trivial code duplication. I suspect there will often be nontrivial code, not shareable between these extension classes, or otherwise there must be a better workaround...

Second, what if we have more than one nullable type in the return values? More examples for that:

Example 2:

Not going too far from our monads. My own issue after which I started to look around and found the above-mentioned SO question - I wanted to do the same thing with the Either monad.

public abstract class Either<L, R>
{
    // most of the implementation is skipped as irrelevant
    public abstract T Match<T>(Func<L, T> f, Func<R, T> g);

    private class Left : Either<L, R>{ /* skipped */ }
    private class Right : Either<L, R>{ /* skipped */ }
}

public static class EitherExtensions
{
    public static bool TryGetValue<L, R>(
        this Either<L, R> either,
        [NotNullWhen(false)] out L? left,
        [NotNullWhen(true)] out R? right
        )
        where L : class
        where R : class
    {
        L? tempLeft = null;
        R? tempRight = null;
        bool result = either.Match(
            l => { tempLeft = l; return false; },
            r => { tempRight = r; return true; }
            );
        left = tempLeft;
        right = tempRight;
        return result;
    }
    
    /* Three (3) more copies of TryGetValue<L, R> are skipped for brevity.
       Thankfully I can do this in an example code.
       Although it takes a lot from the impression I hoped to make... */
}

I have two independent types - L and R. Even if I expect L to be almost always a string, I don't want to have a pressure to choose between code amount and universality. So I had to make four copies of the extension method - 68 lines of code instead of 17.

Example 3:

I have a bunch of extension methods to work with nullables as poor man's Option...

No out parameters here, but the issue is the same - compiler has to know which kind of nullables there are, without any code differences.

public static R? MapNullable<T, R>(this T? item, Func<T, R?> fn)
    where T : struct
    where R : struct
    /* !item.HasValue ? null : fn(item.Value) */
    => item == null ? null : fn(item.Value);

public static R? MapNullable<T, R>(this T? item, Func<T, R> fn)
    where T : struct
    where R : struct
    => item == null ? (R?)null : fn(item.Value);

public static R MapNullable<T, R>(this T? item, Func<T?, R> fn)
    where T : struct
    => fn(item);

// ... few more MapNullable<> variations skipped

This time, contents of the functions is trivial. But the amount of them makes it hard to maintain them if I want to have them universal across all class and struct combinations. Some of them have to be copied twice, some - four times... Some - precisely, some - not quite. And it all adds up.

Granted, I'm not heavily dependent on these methods and can get rid of them, but the same issue will surface in one shape or another for different people.

---

I'm waiting to run into a problem when there will be a need for a function with a combination of multiple different input types, return/out types, collection kinds to see how big of a combinatorial explosion we can realistically get.

It seems like commenters in the issue #2309 also had a short discussion on what can be done about this, but also without any conclusion.

This is too much of PITA for programmers to handle this manually. If there is a single point to remove the frustration for a lot of people - it must be done.

This issue need to have higher priority and worked on more actively. I hope to see more in-depth discussion about implementation possibilities and roadblocks.

---

For input parameters the challenge seems to be to reconcile some existing API's. We are expected to operate with Nullable<T> using it's HasValue and Value properties, while there is no such thing for bare reference types.

I'm not a fan of an idea to introduce a ReferenceNullable<T> wrapper for classes - it will just complicate things further. I think something has to be done about Nullable<T> to bring it closer to the new nullable behavior and let that work when it is known only as T?. We don't have to rely on HasValue already. Teaching the compiler to replace Nullable<T> with the underlying value when appropriate must be more challenging task though...

The limitation for output types seems to be just a collateral damage from that. Correct me if I'm missing something.

If the limitation can be removed just for return types at least - it will be really helpful already. I suspect there are also challenges in the inner workings to make that possible. I would like to hear a competent feedback on that. Is it common for C# compiler to emit multiple copies of a function for different underlying types?

[roji]

This seems like it's covered by dotnet/roslyn#29146, closing for now.

[roji]

Read that too quickly, dotnet/roslyn#29146 is about expressing the default of T for unconstrained generic types. This proposal is about being able to express a single generic method where T? (or some other syntax) express Nullable for value types, and NRTs for reference types. In other words, T? would refer to two different runtime types depending on whether T is a value or reference type (int? for int, string for string).

Reopening even though there seems to be little chance this will be implemented (#2194 (comment)).

[Dreamescaper]

So I assume that it's still not possible to represent nullable unconstraint return type in C# (i.e. string? when T is string and int? when T is int). And current approach will probably never be implemented (unfortunately), as it will be a breaking change.

Are there any other proposals to allow that? I wasn't able to find anything.

[FaustVX]

You could define your method twice:

void M<T>(T? input) where T : class { ... }
void M<T>(T? input) where T : struct { ... }

But the parameters list must contain a parameter of type T? (to have 2 different signatures)

[jnm2]

@Dreamescaper It's too late to allow that since T? on an unconstrained generic T has an established meaning now. If I remember right, implementing the way everyone wanted would have not been expressible in today's IL metadata format. It would need runtime support and assemblies using it would not be understandable to current versions of all tools that read IL today.