[Work items] Nullable reference types

[cston]

Preview 2:

• Intellisense (@333fred)
  • Public API
  • Display inferred nullability of variables, fields, properties, type arguments
  • Should we display some hint about oblivious vs. non-null?
• Nullable Specification (Mads)
• #nullable
  • binding (no more scoped context) Remove remaining artifacts of symbol-centric NonNullTypes context. #31607
  • #pragma warning nullable (enable|disable|restore) PR Add support for restore and safeonly for #nullable directive. #31806
  • Code fixer
• Tuples Track nullable state for tuple items #29970
• More attributes e.g.: [MaybeNull]
  • Finalize BCL types
• ! suppress conversion warnings ! should suppress conversion warnings #30376
• e switch { … }, switch (e) { … } Nullable reference types: no nullability flow check in switch statements. #23944
• Avoid inferring unspeakable type arguments (PR Use speakable annotations in method type re-inference #31813)
• Dogfood
  • WPFRunner
  • CodeAnalysis
• Nullable value types Track Nullable Value Type Null State #29817, PR Track nullability of nullable value types #31499
• [Nullable] metadata encoding (PR Adjust the way nullable annotations are represented in metadata #31212)
  • Sync with F# on format
• Analyze yield return Null warnings not reported for iterator methods #23701

Non-blocking:

• Annotations Need mechanism for injecting annotations #29821
  • File format, MSBuild and loading logic
  • Some IDE support
• Intellisense
  • Display member annotations QuickInfo doesn't display nullability on properties #26830
  • Display member annotations from metadata even when feature disabled
• Codegen for null-check on parameter!
• Lambdas: track nullability of captured variables Analyze captured variables at the location lambda is converted to a delegate #29617
• Local functions: track nullability of captured variables Analyze local functions in NullableWalker #27233
• Other annotations
  • Other attributes (NullEquals, NullInNullOut, NotNulled) Recognize annotations on methods that affect nullability analysis #26761
• [Nullable]
  • Include NullableAttribute in framework? See Use synthesized attribute infrastructure for System.NullableAttribute #22820 (comment)
• Add attribute types to Core 3
• Port attribute types to mono
• Coordinate integration with Razor
• NullableAttribute: Store flags into a field NullableAttribute constructor parameter should be stored in a field #30143
• Codefixer to add #nullable enable and NullableReferenceTypes Nullable: Fixer to add #nullable and project-level setting #30099
  • project system property or API Add support for Nullable build setting project-system#4058
• Other annotations
  • Nullable property with non-nullable setter Need annotation for nullable property with non-nullable setter #26621
• TypeSymbolWithAnnotations
  • Revert properties (MethodSymbol.ReturnType, etc.) to TypeSymbol and add TypeSymbolWithAnnotations properties
  • Review use of TypeMap.AsTypeSymbol and TypeMap.AsTypeSymbolWithAnnotations for cases where nullability is dropped (@jcouv)
• SymbolDisplay
  • Include ?
  • Include ! and ~ Add SymbolDisplay option to append suffix for unannotated reference types #28242
• Conversions
  • Allow variance of out parameter: void M(out string x) invoked with M(out string? x);
  • Infer nullability in out var out var treated as non-nullable #25347
  • Infer nullability in deconstruction Nullability should be inferred from deconstruction #29618
  • Which conversions remain?
• Flow analysis
  • De-reference should update null state Should not report warning for second dereference of nullable value #26749
  • case patterns (case null:, case string:, case string s:) Nullable reference types: no nullability flow check in switch statements. #23944
  • x in var x = obliviousListOfString; should be List<string!>!
  • Anonymous types Inferred nullability should flow into anonymous type fields #24018
  • Query clauses and range variables Implement nullability analysis in LINQ queries #29863
  • foreach (var x in maybeNull) { } Report warning using foreach with a nullable value #23493
  • No warnings in unreachable code Handle unreachable code in the nullable walker #28798
• Fields and properties
  • Report uninitialized warnings for static fields and properties Nullable Reference Types: CS8618 and/or CS8625 should be consistently reported for static fields/properties. #27014
  • Track nullability of fields and properties
  • Report warning for null initializer null initializer for non-nullable property (or field) should warn #26628
  • Report warning before assignment Nullable Reference Types: A warning should be reported when dereferencing a field/property before it is assigned #27016
  • No warnings for constructor that throws exception Reporting "field is uninitialized" for constructor that throws an exception #25529
• Use VisitType to implement ContainsNullableReferenceTypes. See TestDeepTypeAccessibilityBug18018 test.
• ! operator
  • Warning for unnecessary !? Report hidden diagnostic for unnecessary ! #25372
  • Error for !!. See StaticNullChecking.SuppressNullableWarning_Multiple !! should be disallowed #29902
  • Error for a! = b;. See StaticNullChecking.SuppressNullableWarning_Assignment and InvalidOperationException in StackOptimizerPass1.IsIndirectAssignment with x! = null; #28377
  • ERR_NotNullableOperatorNotReferenceType should be a warning not an error
• Method type inference
  • MergeDynamic and MergeTupleNames ignore nullability
  • InferExtensionMethodTypeArguments ignores nullability
  • MethodGroupReturnTypeInference ignores nullability
• Flow analysis
  • Intersection with other C# 8.0 features (switch expression, null-coalescing assignment, ...)
    • ??=
    • recursive patterns
• Dogfood
  • Update public API of Microsoft.CodeAnalysis.dll and Microsoft.CSharp.CodeAnalysis.dll
  • Update tool for tracking public APIs
• EE
  • Include ? annotations in Type for Locals and Watch

Productivity Migrated to project https://github.com/dotnet/roslyn/projects/43

• QuickInfo

• Authoring experience for external annotations

• Can we make it easier to see what context we're in?

• null assignment suggests declaring as nullable Add a DeclareAsNullable fixer #26630

• null test suggests declaring as nullable Add DeclareAsNullable refactoring #26661

• Fix interface implementation or override Nullability fixer for interface implementations #26833

• Grey out unnecessary ! Gray out unnecessary non-null suppression operator #26736

• Grey out unnecessary ?

• Using ? or ! suggests turning feature on (issue Nullable: Fixer to add #nullable and project-level setting #30099)

  • locally Create AddNonNullTypes code fixer #30098
  • globally

• help move #nullable up/down the scopes

• classify #nullable (PR Add syntax classification for nullable directives #31579)

• Indent #nullable to the margin

• Marshalling options in OOP process (PR add new compilation options to data sync to OOP. #32726)

• Performance

  • Measure performance compiling Roslyn

• IDE: Metadata view

Work related to project-system:

• project-system API for CPS Add NullableReferenceTypes property project-system#4277
• project-system API for csproj/legacy
• add drop-down in UI
• SDK work to set default value
• Code fixer to set NullableReferenceTypes project property (setting Project.Configuration.Properties["NullableReferenceTypes"] and declaring that "NullableReferenceTypes" BooleanProperty)

Preview 1:

• Fix installer Installing NonNullable Reference Types preview fails #29987
• Merge back to a preview branch
  • Remove/replace PROTOTYPE markers
  • IDE test pass
• [NonNullTypes]
  • Disable warnings with [NonNullTypes(Warnings=false)] Add NonNullTypesAttribute.Warnings property #29616 (no longer applicable)
  • Fix/verify [NonNullTypes] in PE symbols (no longer applicable)
  • Injecting NonNullTypesAttribute Inject NonNullTypesAttribute #29180
  • Use explicit NonNullTypes context in every TSWA Every TypeSymbolWithAnnotations should have an explicit NonNullTypes context #28766
  • Warning for ! if no attribute (needs a new solution)
  • Enable warnings with [NonNullTypes]
  • Warning for ? if no attribute or if [NonNullTypes(false)]
  • Remove syntax-based cycle breaking (modules, types, members)
  • Enable feature with [NonNullTypes] or [NonNullTypes(value)] (
    Bind NonNullTypes for types #28491)
• Method type inference
  • Incorrect nullability arguments include ~ Incorrect nullability from method type inference #27961
• Flow analysis
  • (object)x != null Warnings reported after comparing with object == or != #27928
  • L-values Suppression operator on L-values #27522
  • t.ToString(); for unconstrained T
  • tuples Missing warning on tuple nullability #28951
  • ?. Infer nullability of receiver and member from ?. expression #25870
  • s is string Pattern test should inform nullability #26745
  • is patterns
    • x is C y
    • x is K where K is a constant other than null
• Type parameters
  • Substitution with ~ and ? or ! (oblivious doesn't survive, LDM 7-11)
  • Support object constraint; use object? for unconstrained Support Object type as a generic type constraint. #29809
  • Check constraints No warning for mismatched nullability in constraint #27742
  • Error for T? unless T is constrained to a reference type or a value type Report error for T? when T is unconstrained #28804
  • Support class? constraint
• Switch old tests to C# 8 (update default test options)
• Track nullability of unconstrained type parameters Track nullability of unconstrained type parameters #28956
• Performance
  • Close allocation gap with null unaware C#
• Merge 15.8 branch
• Compiler command-line switch Add "Nullable" compilation option. #30479 and MSBuild setting

[jcouv]

Championed issue: dotnet/csharplang#36

Notes on project property:

• Instructions to test project-system and roslyn together: https://github.com/dotnet/project-system/blob/master/docs/repo/getting-started.md#deploying-to-a-different-hive

• project.ConfigurationManager.ConfigurationRow(configurationName).Item(1).Properties["NullableReferenceTypes"]

• Revisit rules for un-initialized fields? (see thread with Oren)

• should we check implementations of methods that use [NotNullWhenTrue]?

• Test #nullable within #ifdef directive

• Test with Pair or tuple: F((maybeNull, oblivious), (maybeNull, nonNull), (maybeNull, maybeNull))

• Oblivious means "assume the best"

• What are the scoping rules of #nullable in using aliases?

• How does the public API work for nullable value types?

• scripting

• Document breaking change: if you were defining a NonNullTypes attribute, then the compiler will now reject it.

• External annotations and determinism

• LDM: should we warn when assigning to oblivious? MOblivious(null);

• LDM: var x = oblivious; we probably need to keep x as oblivious here. Or do we assume ??

• LDM: what do we emit for string? in C# 7 (new compiler)? I assume the compiler should produce an error.

• Verify older compilers consuming newer library that uses the new constraints

• LDM: is T? where T : class? allowed or meaningful? (error CS8627)

• block T? if T is unconstrained

• verify behavior with LangVersion 7.3 or no [module: NonNullTypes].

Dogfood experience:

• I want to review all the public APIs. How do I do that?
  • PublicAPI analyzer/fixer should be updated with nullability.
• Problem with linked source files (CS0453)
• Need a way to dump all the APIs which an assembly depends on (so you can think about annotating them)
• FixOnce for less trivial cases (PR Add DeclareAsNullable refactoring #26661)
  • local == null and local != null should offer me to fix the declaration of local
• Highlight oblivious methods and offer dialog to annotate, or write a tool to annotate BCL.
• Need to clarify guidelines: my current assumption is that if a null parameter guarantees to throw, then the parameter should be marked as non-nullable.
• Confirm behavior for [EnsuresTrues] on a conditional method like Debug.Assert
• I think we need URTANN
• CodeAnalysis warnings did not contain that many W warnings (gist)
• FixAll for trivial cases (PR Add a DeclareAsNullable fixer #26630)
  • return null
  • parameter with default value null
  • local declared with null
• It's useful to have different annotation for getter and for setter. Maybe always set non-null. But before the first set, get could return null. (Need annotation for nullable property with non-nullable setter #26621)
• External attribute annotation for IsNullOrEmpty and Debug.Assert (PR Use [NotNullWhenFalse] and [EnsuresNotNull] to drive nullability analysis #26656)

Some various notes from LDM discussions and others:

• Compiler or analyzer warning on using ? in a scope where [NonNullTypes(false)] (warning CS8632)

• [module: NonNullTypes] is confusing (most developers aren't aware of module: target.

• Is !! ok? (answer: yes)

• Confirm whether we want to allow out x! (currently allowed) or x! = y (currently disallowed).

• How do you version APIs that have nullability or flow annotations?

• Analyzer for bad usage of attributes, or to recommend using them?

• Nullability in Razor pages

• Once the feature can be turned on/off independently of LangVersion (issue Add flag to control nullable feature #26692), we should ensure UpgradeProject experience is still good.

• Can we bootstrap the BCL annotations using existing CodeContracts?

• Should we warn if you use the annotations, but the flag is off? Maybe grey them out? (Champion "Nullable reference types" (16.3, Core 3) csharplang#36 (comment))

• How do we track nullable structs? S? s = .... Does checking .HasValue affect flow analysis?

• Does ! produce a result that is oblivious or non-null? (! stickiness) (answer: non-null)

• Should referencing an un-annotated library produce a warning? (answer: no longer applicable)

• Should MakeArray(new C()) produce a result that is Array<C> or Array<C?>? How do we know that that method doesn't add nulls? On the other hand, how does a method indicate that it returns a non-nullable when the type argument is non-nullable? (maybe we need a constraint: Array<T2> MakeArray(T element) where ...)

• Update templates (so new projects have the feature turned on by default)

• What should IDE show at various points in the method? Does having the feature on or off affect what is shown?

• Could users annotate some methods to reset flow analysis? (for example, p.ChangeMiddleNameToNull())

• Are we going to apply similar flow analysis and warnings to nullable value types?

• How are we going to add annotations to BCL, and older frameworks in particular?

• Lambda with multiple returns, one is a non-null object and the other is nullable local which flow analysis knows isn't null. What is the return type? Same question for other "best type" scenarios (ternary, method type inference).

• Should nullability be exposed in APIs (IOperation) or UI?

• Should there be a warning on useless !? (maybe grey it out too?)

• Should test that nullable annotations from a metadata reference are always displayed in QuickInfo (even in a C# 7 project).

• Probably this already works, but we should check the intersection between nullability and patterns/switch features: case int? x: // error already, case string? y: // should also be error, case string z: // should be non-null, incomplete switch expression that doesn't handle null should warn (even if input is considered non-null?).

• string.IsNullOrEmpty should inform flow analysis

  • We'll probably want to annotate some key APIs (like TryGetValue)
  • I assume we'll need some attribute to annotate such APIs. Could the compiler generate the attribute, or an analyzer suggest it, by inspecting the body of the method?

• TryGetValue should inform flow analysis

• string? x = null; x = nonNull; var y = new [] { x }; What is the type of y?

• string? x = null; x = nonNull; var y = MakeArray(x, 10); What is the type of y?

• Should we warn if conditionally accessing a value that is known to be non-null? I suspect we don't. (Warn if a value is never or always null #22743)

• x is null should inform flow analysis (Pattern test should inform nullability #26745)

• From a customer:

class MyClass
{
    private string m_IdAsString; // warning CS8618: Non-nullable field 'm_IdAsString' is uninitialized.
    public Guid Id
    {
        get { return new Guid( m_IdAsString ); }
        set { m_IdAsString = value.ToString(); }
    }
    public MyClass()
    {
        Id = Guid.Empty;
    }
}

• Variance with out variables

    static void F(out string x) => throw new Exception();
    static void G(out IEnumerable<string> y) => throw new Exception();

    F(out string? x);              // warning?
    G(out IEnumerable<string?> y); // warning?

• Inferred null state from !

    static void F<T>(T x, out T y) => throw new Exception();

    string? x = null;
    string y;
    F(x, out y);  // warning
    F(x!, out y); // warning?

    static void F(object? x)
    {
        object y;
        y = x;        // warning: `x` may be null
        y.ToString(); // warning?
        y = x!;       // no warning
        y.ToString(); // warning?
    }

• Warnings for unnecessary !

    static void F(object? x)
    {
        if (x == null) return;
        x!.ToString(); // warning: unnecessary `!`?
        // maybe the IDE offers to remove uncessary ! operators?
    }

• How to tell the nullable walker about flow of various methods (in most general sense, could be represented by a code snippet):

  • ThrowIfNull
  • IsNullOrEmpty
  • Debug.Assert(x 1= null)
  • FirstOrDefault

• Tracking null state of non-nullable locals

• Constraining T to object or class

• VSIX feed: https://dotnet.myget.org/F/roslyn-nonnull/vsix/

[Joe4evr]

string.IsNullOrEmpty should inform flow analysis

Nit: Should probably add .IsNullOrWhitespace while you're at it.

[jcouv]

I looked at the remaining issues. Extracted one or two as discrete issues.
I think we can now close this umbrella issue because we now use the Nullable Board.