Resolve issue when specific types shared data after creation off gene…

…ric template (microsoft#600)

* Buildable

* Part 14

* First passing test

* Simplify configuration

* Style

* Fix test and move code to folders

* Builtins import

* Fix microsoft#470

* Fluents

* Add search path

* Import analysis, part I

* Simplify builtins handling

* Remove IMember

* Handle import specific

* More tests

* Add typeshed

* Renames

* Make sure lazy modules are loaded

* Renames

* Move/rename

* Rework importing

* Derivation rework

* Part 2

* Part 3

* Buildable

* Module members

* Async walk

* Imports test pass

* Remove lazy types

* Fix from import

* Stubs

* Double overloads

* Fix datetime test

* Couple more tests + fluents

* Few more tests

* Additionl test + union type

* Built-in scrape tests

* Full stdlib scrape test

* Complete async AST walker

* Conditional defines test + variable loc cleanup

* More stub tests
Fix stub loading for packages (port from DDG)
Split walker into multiple files

* Add some (broken mostly) tests from DDG

* Move document tests

* Function arg eval, part I

* Instance/factory

* Builds

* Test fixes

* Fix static and instance call eval

* More tests

* More ported tests

* Specialize builtin functions

* Make walkers common and handle nested functions

* Moar tests

* Parser fixes + more tests

* Handle negative numbers

* Fix null ref

* Basic list support

* Few more list tests

* Basic iterators

* Support __iter__

* Iterators

* Fix couple of tests

* Add decorator test

* Generics, part I

* Generics, part 2

* Generics, part 3

* Basic TypeVar test

* Typings, part 4

* Fix test

* Generics, part 6

* Generics, part 7

* More tests (failing)

* Forward ref fixes

* Reorg

* Improve symbol resolution + test fixes

* Test fixes

* Dictionary, part I

* Part 11

* Fix test

* Tests

* Tests

* More dict work

* List ctor

* Skip some tests for now

* Fix iterators

* Tuple slicing

* Polish type comparo in return types

* Add Mapping and tests

* Add Iterable

* Fix typo

* Add Iterator[T] + test

* Simplify typing types

* Class reduction

* Fix tests

* Test fix

* Handle 'with' statement

* Handle try-except

* Class method inheritance + NewType

* Container types

* Containers test

* Tests

* Handle generic type alias

* Named tuple

* Global/non-local

* Handle tuples in for
Handle custom iterators

* Basic generator

* Any/AnyStr

* Test fixes

* Type/Optional/etc handling

* Proper doc population

* Tests + range

* Argument match

* Basic argset and diagnostics

* Argset tests

* Exclude WIP

* Exclude WIP

* Arg eval

* Arg match, part 2

* Tests and generic arg comparisons

* Function eval with arguments

* Baselines

* Fix test

* Undo AST formatting change and update baseline

* LS cleanup 1

* Fix list ctor argument unpacking

* Cleanup 2

* Builds

* Partial completions

* Partial

* Partial

* Simple test

* Tests

* Basic startup

* Clean up a bit

* Remove debug code

* Port formatter tests

* Fix tokenizer crash

* Async fixes

* Hover

* Basic hover

* Adjust expression options

* Basic signature help

* Fix class/instance

* Update test

* Fix builtin creation exception

* Fix tests

* Actually provide declared module

* Completion test (partial)

* Undo

* Fix null await
Fix override completions + test

* Exports filtering
Prevent augmenting imported types

* Filter variables & exports

* Ported tests

* Test fixes

* More ported tests

* Fix exception completions

* Import completions

* Scope completions

* With completions

* Test fixes

* WIP

* Test fix

* Better arg match

* Temp disable WIP

* First cut

* Fix type leak

* WIP

* Remove ConfigureAwait and handle canceled and failed in the analysis notifications

* WIP

* Generic class base

* Generic forward reference resolution

* Suppress completion in strings + test

* Prevent recursion on generic resolution
Better match arguments

* Handle call expression in generics

* Relax condition as it happens in tensorflow

* Fix typeshed version search
Make writing cached modules async
Fix module doc fetching

* Hover tests

* Fix prom import hover

* Hover tests

* Synchronize test cache writing

* First cut

* Test

* Fixes

* Add tests for os.path
Null ref fix

* Fix cache check

* Improve resolution of builtins and typing in stubs

* Merge tests

* Add ntst for requests

* Handle typeshed better

* Fix custom stub handling

* Better sync

* Move files

* Fix parameter locations

* Hover improvement

* PEP hints

* One more test for PEP hints

* Better handle hover over import as

* Text based generic constraints

* Handle with better with generic stubs

* Undo debug

* Handle non-binary open()
Temporary fix 'with' handler since we haven't specialized IO/TextIO/BinaryIO yet.

* Output syntax errors

* Properly clear

* - Fix async issue with analysis completion
- Clean up diagnostics service interface
- Use real DS in tests

* Use proper scope when analyzing module

* Severity mapping and reporting

* Add publishing test
Add NSubstitute
Move all services to the same namespace.

* Unused var

* Test forced publish on close

* Fix typo

* Update test framework

* Import location

* Remove incorrect reference

* Diagnostic severity mapping test
Fix post-mortem earlier PR comment

* Minor fixes

* Better handle return types in classes created from templates.

* Move interface to the main class part

* Dynamic return type

* Add hover and signature tests

* Baseline update

* Flicker reduction

* - Correct reported unresolved import name
- Add tests

* PR feedback

* Resolve merge issues

* Fix completion doc + test

* Restore formatting

* Bunch of null checks

* Fix generic base classes
Add tests and changes to couple more cases

* PR feedback

src/Analysis/Ast/Impl/Analyzer/Evaluation/ExpressionEval.Callables.cs

@@ -157,7 +157,7 @@ public async Task<IMember> GetValueFromFunctionTypeAsync(IPythonFunctionType fn,
             if (instanceType == null || fn.DeclaringType == null || fn.IsSpecialized ||
                 instanceType.IsSpecialized || fn.DeclaringType.IsSpecialized ||
                 instanceType.Equals(fn.DeclaringType) ||
-                fn.IsStub || !string.IsNullOrEmpty(fn.Overloads[args.OverloadIndex].ReturnDocumentation)) {
+                fn.IsStub || !string.IsNullOrEmpty(fn.Overloads[args.OverloadIndex].GetReturnDocumentation(null))) {
 
                 if (fn.IsSpecialized && fn is PythonFunctionType ft) {
                     foreach (var moduleName in ft.Dependencies) {
@@ -218,7 +218,7 @@ private async Task<ArgumentSet> FindOverloadAsync(IPythonFunctionType fn, IPytho
             var result = noErrorsMatches.Any()
                 ? noErrorsMatches.FirstOrDefault(args => IsMatch(args, fn.Overloads[args.OverloadIndex].Parameters))
                 : null;
-            
+
             // Optimistically pick the best available.
             return result ?? orderedSets.FirstOrDefault();
         }

src/Analysis/Ast/Impl/Analyzer/Evaluation/ExpressionEval.Collections.cs

@@ -31,7 +31,7 @@ public async Task<IMember> GetValueFromIndexAsync(IndexExpression expr, Cancella
             }
 
             var target = await GetValueFromExpressionAsync(expr.Target, cancellationToken);
-            // Try generics
+            // Try generics first since this may be an expression like Dict[int, str]
             var result = await GetValueFromGenericAsync(target, expr, cancellationToken);
             if (result != null) {
                 return result;
@@ -48,7 +48,9 @@ public async Task<IMember> GetValueFromIndexAsync(IndexExpression expr, Cancella
                     instance = new PythonInstance(type);
                 }
                 var index = await GetValueFromExpressionAsync(expr.Index, cancellationToken);
-                return type.Index(instance, index);
+                if (index != null) {
+                    return type.Index(instance, index);
+                }
             }
 
             return UnknownType;
@@ -93,7 +95,7 @@ public async Task<IMember> GetValueFromSetAsync(SetExpression expression, Cancel
 
         public async Task<IMember> GetValueFromGeneratorAsync(GeneratorExpression expression, CancellationToken cancellationToken = default) {
             var iter = expression.Iterators.OfType<ComprehensionFor>().FirstOrDefault();
-            if(iter != null) { 
+            if (iter != null) {
                 return await GetValueFromExpressionAsync(iter.List, cancellationToken) ?? UnknownType;
             }
             return UnknownType;

src/Analysis/Ast/Impl/Analyzer/Evaluation/ExpressionEval.Generics.cs

@@ -13,14 +13,14 @@
 // See the Apache Version 2.0 License for specific language governing
 // permissions and limitations under the License.
 
-using System;
 using System.Collections.Generic;
 using System.Linq;
 using System.Threading;
 using System.Threading.Tasks;
 using Microsoft.Python.Analysis.Specializations.Typing;
 using Microsoft.Python.Analysis.Specializations.Typing.Types;
 using Microsoft.Python.Analysis.Types;
+using Microsoft.Python.Analysis.Values;
 using Microsoft.Python.Core;
 using Microsoft.Python.Parsing.Ast;
 
@@ -47,8 +47,10 @@ private async Task<IMember> GetValueFromGenericAsync(IMember target, Expression
             // rather than specific type instantiation as in List[str].
 
             IPythonType[] specificTypes;
+            var returnInstance = false;
             switch (expr) {
-                case IndexExpression indexExpr: {
+                // Indexing returns type as from A[int]
+                case IndexExpression indexExpr:
                     // Generic[T1, T2, ...] or A[type]()
                     var indices = await EvaluateIndexAsync(indexExpr, cancellationToken);
                     // See which ones are generic parameters as defined by TypeVar() 
@@ -66,6 +68,7 @@ private async Task<IMember> GetValueFromGenericAsync(IMember target, Expression
                     if (genericTypeArgs.Length > 0 && genericTypeArgs.Length != indices.Count) {
                         // TODO: report that some type arguments are not declared with TypeVar.
                     }
+
                     if (specificTypes.Length > 0 && specificTypes.Length != indices.Count) {
                         // TODO: report that arguments are not specific types or are not declared.
                     }
@@ -84,20 +87,21 @@ private async Task<IMember> GetValueFromGenericAsync(IMember target, Expression
 
                         if (specificTypes.Length > 0) {
                             // If target is a generic type and indexes are specific types, create specific class
-                            return await gt.CreateSpecificTypeAsync(new ArgumentSet(specificTypes), Module, GetLoc(expr), cancellationToken);
+                            return gt.CreateSpecificType(new ArgumentSet(specificTypes), Module, GetLoc(expr));
                         } else {
                             // TODO: report too few type arguments for the Generic[].
                             return UnknownType;
                         }
                     }
-
                     break;
-                }
+
                 case CallExpression callExpr:
                     // Alternative instantiation:
                     //  class A(Generic[T]): ...
                     //  x = A(1234)
-                    specificTypes = (await EvaluateICallArgsAsync(callExpr, cancellationToken)).Select(x => x.GetPythonType()).ToArray();
+                    specificTypes = (await EvaluateCallArgsAsync(callExpr, cancellationToken)).Select(x => x.GetPythonType()).ToArray();
+                    // Callable returns instance (as opposed to a type with index expression)
+                    returnInstance = true;
                     break;
 
                 default:
@@ -109,9 +113,12 @@ private async Task<IMember> GetValueFromGenericAsync(IMember target, Expression
             // as we resolve classes on demand. Therefore we don't know if class is generic
             // or not at the time of the PythonClassType creation.
             // TODO: figure out if we could make GenericClassType: PythonClassType, IGenericType instead.
-            return target is PythonClassType cls 
-                ? await cls.CreateSpecificTypeAsync(new ArgumentSet(specificTypes), Module, GetLoc(expr), cancellationToken) 
-                : null;
+            if (target is PythonClassType cls) {
+                var location = GetLoc(expr);
+                var type = cls.CreateSpecificType(new ArgumentSet(specificTypes), Module, location);
+                return returnInstance ? new PythonInstance(type, GetLoc(expr)) : (IMember)type;
+            }
+            return null;
         }
 
         private async Task<IReadOnlyList<IMember>> EvaluateIndexAsync(IndexExpression expr, CancellationToken cancellationToken = default) {
@@ -124,16 +131,16 @@ private async Task<IReadOnlyList<IMember>> EvaluateIndexAsync(IndexExpression ex
                 }
             } else {
                 var index = await GetValueFromExpressionAsync(expr.Index, cancellationToken);
-                indices.Add(index);
+                indices.Add(index ?? UnknownType);
             }
             return indices;
         }
 
-        private async Task<IReadOnlyList<IMember>> EvaluateICallArgsAsync(CallExpression expr, CancellationToken cancellationToken = default) {
+        private async Task<IReadOnlyList<IMember>> EvaluateCallArgsAsync(CallExpression expr, CancellationToken cancellationToken = default) {
             var indices = new List<IMember>();
             cancellationToken.ThrowIfCancellationRequested();
-            foreach (var e in expr.Args.Select(a => a.Expression).ExcludeDefault()) {
-                var value = await GetValueFromExpressionAsync(e, cancellationToken);
+            foreach (var e in expr.Args.Select(a => a.Expression)) {
+                var value = await GetValueFromExpressionAsync(e, cancellationToken) ?? UnknownType;
                 indices.Add(value);
             }
             return indices;

src/Analysis/Ast/Impl/Analyzer/Evaluation/ExpressionEval.Operators.cs

@@ -39,7 +39,7 @@ private async Task<IMember> GetValueFromUnaryOpAsync(UnaryExpression expr, Cance
                 case PythonOperator.Pos:
                     return await GetValueFromUnaryOpAsync(expr, "__pos__", cancellationToken);
             }
-            return null;
+            return UnknownType;
         }
 
         private async Task<IMember> GetValueFromUnaryOpAsync(UnaryExpression expr, string op, CancellationToken cancellationToken = default) {
@@ -57,7 +57,7 @@ private async Task<IMember> GetValueFromUnaryOpAsync(UnaryExpression expr, strin
                     ? new PythonConstant(-value, c.Type, GetLoc(expr))
                     : instance;
             }
-            return null;
+            return UnknownType;
         }
 
         private async Task<IMember> GetValueFromBinaryOpAsync(Expression expr, CancellationToken cancellationToken = default) {
@@ -87,8 +87,8 @@ private async Task<IMember> GetValueFromBinaryOpAsync(Expression expr, Cancellat
                     return Interpreter.GetBuiltinType(BuiltinTypeId.Bool);
             }
 
-            var left = await GetValueFromExpressionAsync(binop.Left, cancellationToken);
-            var right = await GetValueFromExpressionAsync(binop.Right, cancellationToken);
+            var left = await GetValueFromExpressionAsync(binop.Left, cancellationToken) ?? UnknownType;
+            var right = await GetValueFromExpressionAsync(binop.Right, cancellationToken) ?? UnknownType;
 
             switch (binop.Operator) {
                 case PythonOperator.Divide:

src/Analysis/Ast/Impl/Analyzer/Evaluation/ExpressionEval.cs

@@ -192,7 +192,7 @@ private async Task<IMember> GetValueFromMemberAsync(MemberExpression expr, Cance
             }
 
             instance = instance ?? m as IPythonInstance;
-            var type = m.GetPythonType(); // Try inner type
+            var type = m?.GetPythonType(); // Try inner type
             var value = type?.GetMember(expr.Name);
 
             // Class type GetMember returns a type. However, class members are
@@ -226,7 +226,7 @@ private async Task<IMember> GetValueFromConditionalAsync(ConditionalExpression e
             var trueValue = await GetValueFromExpressionAsync(expr.TrueExpression, cancellationToken);
             var falseValue = await GetValueFromExpressionAsync(expr.FalseExpression, cancellationToken);
 
-            return trueValue ?? falseValue;
+            return trueValue ?? falseValue ?? UnknownType;
         }
 
         public void ReportDiagnostics(Uri documentUri, DiagnosticsEntry entry) {

src/Analysis/Ast/Impl/Analyzer/Handlers/AssignmentHandler.cs

@@ -32,7 +32,7 @@ public async Task HandleAssignmentAsync(AssignmentStatement node, CancellationTo
                 return;
             }
 
-            var value = await Eval.GetValueFromExpressionAsync(node.Right, cancellationToken);
+            var value = await Eval.GetValueFromExpressionAsync(node.Right, cancellationToken) ?? Eval.UnknownType;
             // Check PEP hint first
             var valueType = Eval.GetTypeFromPepHint(node.Right);
             if (valueType != null) {

src/Analysis/Ast/Impl/Analyzer/Handlers/TryExceptHandler.cs

@@ -28,7 +28,7 @@ public async Task<bool> HandleTryExceptAsync(TryStatement node, CancellationToke
             foreach (var handler in node.Handlers.MaybeEnumerate()) {
                 if (handler.Test != null && handler.Target is NameExpression nex) {
                     var value = await Eval.GetValueFromExpressionAsync(handler.Test, cancellationToken);
-                    Eval.DeclareVariable(nex.Name, value, VariableSource.Declaration, nex);
+                    Eval.DeclareVariable(nex.Name, value ?? Eval.UnknownType, VariableSource.Declaration, nex);
                 }
                 await handler.Body.WalkAsync(Walker, cancellationToken);
             }

src/Analysis/Ast/Impl/Analyzer/Handlers/TupleExpressionHandler.cs

@@ -22,8 +22,8 @@
 using Microsoft.Python.Parsing.Ast;
 
 namespace Microsoft.Python.Analysis.Analyzer.Handlers {
-    internal sealed class TupleExpressionHandler: StatementHandler {
-        public TupleExpressionHandler(AnalysisWalker walker): base(walker) { }
+    internal sealed class TupleExpressionHandler : StatementHandler {
+        public TupleExpressionHandler(AnalysisWalker walker) : base(walker) { }
 
         public async Task HandleTupleAssignmentAsync(TupleExpression lhs, Expression rhs, IMember value, CancellationToken cancellationToken = default) {
             cancellationToken.ThrowIfCancellationRequested();
@@ -34,9 +34,7 @@ public async Task HandleTupleAssignmentAsync(TupleExpression lhs, Expression rhs
                 for (var i = 0; i < Math.Min(names.Length, returnedExpressions.Length); i++) {
                     if (returnedExpressions[i] != null && !string.IsNullOrEmpty(names[i])) {
                         var v = await Eval.GetValueFromExpressionAsync(returnedExpressions[i], cancellationToken);
-                        if (v != null) {
-                            Eval.DeclareVariable(names[i], v, VariableSource.Declaration, returnedExpressions[i]);
-                        }
+                        Eval.DeclareVariable(names[i], v ?? Eval.UnknownType, VariableSource.Declaration, returnedExpressions[i]);
                     }
                 }
                 return;

src/Analysis/Ast/Impl/Analyzer/Handlers/WithHandler.cs

@@ -28,7 +28,7 @@ public async Task HandleWithAsync(WithStatement node, CancellationToken cancella
             cancellationToken.ThrowIfCancellationRequested();
             foreach (var item in node.Items.Where(x => x.Variable != null)) {
                 var contextManager = await Eval.GetValueFromExpressionAsync(item.ContextManager, cancellationToken);
-                var cmType = contextManager.GetPythonType();
+                var cmType = contextManager?.GetPythonType();
 
                 var enter = cmType?.GetMember(node.IsAsync ? @"__aenter__" : @"__enter__")?.GetPythonType<IPythonFunctionType>();
                 if (enter != null) {

src/Analysis/Ast/Impl/Analyzer/Symbols/FunctionEvaluator.cs

@@ -87,7 +87,7 @@ public override async Task EvaluateAsync(CancellationToken cancellationToken = d
         }
 
         public override async Task<bool> WalkAsync(AssignmentStatement node, CancellationToken cancellationToken = default) {
-            var value = await Eval.GetValueFromExpressionAsync(node.Right, cancellationToken);
+            var value = await Eval.GetValueFromExpressionAsync(node.Right, cancellationToken) ?? Eval.UnknownType;
 
             foreach (var lhs in node.Left) {
                 switch (lhs) {
@@ -180,7 +180,7 @@ private async Task DeclareParameterAsync(Parameter p, int index, ParameterInfo p
             } else {
                 var defaultValue = await Eval.GetValueFromExpressionAsync(p.DefaultValue, cancellationToken) ?? Eval.UnknownType;
 
-                paramType = defaultValue.GetPythonType();
+                paramType = defaultValue?.GetPythonType();
                 if (!paramType.IsUnknown()) {
                     pi?.SetDefaultValueType(paramType);
                 }

src/Analysis/Ast/Impl/Extensions/AnalysisExtensions.cs

@@ -54,7 +54,7 @@ private static PythonFunctionType GetOrCreateFunction(this IDocumentAnalysis ana
             // 'type()' in code is a function call, not a type class instantiation.
             if (!(analysis.GlobalScope.Variables[name]?.Value is PythonFunctionType f)) {
                 f = PythonFunctionType.ForSpecialization(name, analysis.Document);
-                f.AddOverload(new PythonFunctionOverload(name, analysis.Document, LocationInfo.Empty));
+                f.AddOverload(new PythonFunctionOverload(name, analysis.Document, _ => LocationInfo.Empty));
                 analysis.GlobalScope.DeclareVariable(name, f, VariableSource.Declaration, LocationInfo.Empty);
             }
             return f;

src/Analysis/Ast/Impl/Extensions/MemberExtensions.cs

@@ -16,7 +16,6 @@
 using Microsoft.Python.Analysis.Specializations.Typing;
 using Microsoft.Python.Analysis.Types;
 using Microsoft.Python.Analysis.Values;
-using Microsoft.Python.Parsing.Ast;
 
 namespace Microsoft.Python.Analysis {
     public static class MemberExtensions {

src/Analysis/Ast/Impl/Specializations/Specialized.cs

@@ -19,15 +19,15 @@ namespace Microsoft.Python.Analysis.Specializations {
     internal static class Specialized {
         public static IPythonPropertyType Property(string name, IPythonModule declaringModule, IPythonType declaringType, IMember returnValue) {
             var prop = new PythonPropertyType(name, declaringModule, declaringType, false, LocationInfo.Empty);
-            var o = new PythonFunctionOverload(prop.Name, declaringModule, LocationInfo.Empty);
+            var o = new PythonFunctionOverload(prop.Name, declaringModule, _ => LocationInfo.Empty);
             o.AddReturnValue(returnValue);
             prop.AddOverload(o);
             return prop;
         }
 
         public static IPythonFunctionType Function(string name, IPythonModule declaringModule, IPythonType declaringType, string documentation, IMember returnValue) {
             var prop = new PythonFunctionType(name, declaringModule, declaringType, documentation, LocationInfo.Empty);
-            var o = new PythonFunctionOverload(prop.Name, declaringModule, LocationInfo.Empty);
+            var o = new PythonFunctionOverload(prop.Name, declaringModule, _ => LocationInfo.Empty);
             o.AddReturnValue(returnValue);
             prop.AddOverload(o);
             return prop;

src/Analysis/Ast/Impl/Specializations/Typing/Definitions/IGenericClassBaseType.cs

@@ -14,12 +14,13 @@
 // permissions and limitations under the License.
 
 using System.Collections.Generic;
+using Microsoft.Python.Analysis.Types;
 
 namespace Microsoft.Python.Analysis.Specializations.Typing {
     /// <summary>
     /// Represents Generic[T1, T2, ...]. Used as a base class to generic classes.
     /// </summary>
-    public interface IGenericClassBaseType {
+    public interface IGenericClassBaseType: IPythonType {
         IReadOnlyList<IGenericTypeParameter> TypeArgs { get; }
     }
 }

src/Analysis/Ast/Impl/Specializations/Typing/Types/GenericClassBaseType.cs

@@ -17,6 +17,9 @@
 using Microsoft.Python.Analysis.Types;
 
 namespace Microsoft.Python.Analysis.Specializations.Typing.Types {
+    /// <summary>
+    /// Represents Generic[T1, T2, ...]
+    /// </summary>
     internal sealed class GenericClassBaseType: PythonClassType, IGenericClassBaseType {
         internal GenericClassBaseType(IReadOnlyList<IGenericTypeParameter> typeArgs, IPythonModule declaringModule, LocationInfo location) 
             : base("Generic", declaringModule, location) {