Report unresolved imports (microsoft#593)

* Part 7 * Buildable * PR feedback * Merge conflict * Fix microsoft#446 * Fix microsoft#446 * Part 8 * Part 9 * Buildable * Part 10 * Part 11 * Part 12 * Buildable * Part 14 * First passing test * Simplify configuration * Style * Fix test and move code to folders * Builtins import * Fix microsoft#470 * Fluents * Add search path * Import analysis, part I * Simplify builtins handling * Remove IMember * Handle import specific * More tests * Add typeshed * Renames * Make sure lazy modules are loaded * Renames * Move/rename * Rework importing * Derivation rework * Part 2 * Part 3 * Buildable * Module members * Async walk * Imports test pass * Remove lazy types * Fix from import * Stubs * Double overloads * Fix datetime test * Couple more tests + fluents * Few more tests * Additionl test + union type * Built-in scrape tests * Full stdlib scrape test * Complete async AST walker * Conditional defines test + variable loc cleanup * More stub tests Fix stub loading for packages (port from DDG) Split walker into multiple files * Add some (broken mostly) tests from DDG * Move document tests * Function arg eval, part I * Instance/factory * Builds * Test fixes * Fix static and instance call eval * More tests * More ported tests * Specialize builtin functions * Make walkers common and handle nested functions * Moar tests * Parser fixes + more tests * Handle negative numbers * Fix null ref * Basic list support * Few more list tests * Basic iterators * Support __iter__ * Iterators * Fix couple of tests * Add decorator test * Generics, part I * Generics, part 2 * Generics, part 3 * Basic TypeVar test * Typings, part 4 * Fix test * Generics, part 6 * Generics, part 7 * More tests (failing) * Forward ref fixes * Reorg * Improve symbol resolution + test fixes * Test fixes * Dictionary, part I * Part 11 * Fix test * Tests * Tests * More dict work * List ctor * Skip some tests for now * Fix iterators * Tuple slicing * Polish type comparo in return types * Add Mapping and tests * Add Iterable * Fix typo * Add Iterator[T] + test * Simplify typing types * Class reduction * Fix tests * Test fix * Handle 'with' statement * Handle try-except * Class method inheritance + NewType * Container types * Containers test * Tests * Handle generic type alias * Named tuple * Global/non-local * Handle tuples in for Handle custom iterators * Basic generator * Any/AnyStr * Test fixes * Type/Optional/etc handling * Proper doc population * Tests + range * Argument match * Basic argset and diagnostics * Argset tests * Exclude WIP * Exclude WIP * Arg eval * Arg match, part 2 * Tests and generic arg comparisons * Function eval with arguments * Baselines * Fix test * Undo AST formatting change and update baseline * LS cleanup 1 * Fix list ctor argument unpacking * Cleanup 2 * Builds * Partial completions * Partial * Partial * Simple test * Tests * Basic startup * Clean up a bit * Remove debug code * Port formatter tests * Fix tokenizer crash * Async fixes * Hover * Basic hover * Adjust expression options * Basic signature help * Fix class/instance * Update test * Fix builtin creation exception * Fix tests * Actually provide declared module * Completion test (partial) * Undo * Fix null await Fix override completions + test * Exports filtering Prevent augmenting imported types * Filter variables & exports * Ported tests * Test fixes * More ported tests * Fix exception completions * Import completions * Scope completions * With completions * Test fixes * WIP * Test fix * Better arg match * Temp disable WIP * First cut * Fix type leak * WIP * Remove ConfigureAwait and handle canceled and failed in the analysis notifications * WIP * Generic class base * Generic forward reference resolution * Suppress completion in strings + test * Prevent recursion on generic resolution Better match arguments * Handle call expression in generics * Relax condition as it happens in tensorflow * Fix typeshed version search Make writing cached modules async Fix module doc fetching * Hover tests * Fix prom import hover * Hover tests * Synchronize test cache writing * First cut * Test * Fixes * Add tests for os.path Null ref fix * Fix cache check * Improve resolution of builtins and typing in stubs * Merge tests * Add ntst for requests * Handle typeshed better * Fix custom stub handling * Better sync * Move files * Fix parameter locations * Hover improvement * PEP hints * One more test for PEP hints * Better handle hover over import as * Text based generic constraints * Handle with better with generic stubs * Undo debug * Handle non-binary open() Temporary fix 'with' handler since we haven't specialized IO/TextIO/BinaryIO yet. * Output syntax errors * Properly clear * - Fix async issue with analysis completion - Clean up diagnostics service interface - Use real DS in tests * Use proper scope when analyzing module * Severity mapping and reporting * Add publishing test Add NSubstitute Move all services to the same namespace. * Unused var * Test forced publish on close * Fix typo * Update test framework * Import location * Remove incorrect reference * Diagnostic severity mapping test Fix post-mortem earlier PR comment * Minor fixes * Move interface to the main class part * Flicker reduction * - Correct reported unresolved import name - Add tests * PR feedback
jakebailey · Feb 13, 2019 · f3b34e3 · f3b34e3
1 parent 4ea596d
commit f3b34e3
Show file tree

Hide file tree

Showing 22 changed files with 311 additions and 91 deletions.
diff --git a/src/Analysis/Ast/Impl/Analyzer/Definitions/IExpressionEvaluator.cs b/src/Analysis/Ast/Impl/Analyzer/Definitions/IExpressionEvaluator.cs
@@ -67,6 +67,8 @@ public interface IExpressionEvaluator {
         IPythonModule Module { get; }
         IPythonInterpreter Interpreter { get; }
         IServiceContainer Services { get; }
+
+        void ReportDiagnostics(Uri documentUri, DiagnosticsEntry entry);
         IEnumerable<DiagnosticsEntry> Diagnostics { get; }
     }
 }
diff --git a/src/Analysis/Ast/Impl/Analyzer/DocumentAnalysis.cs b/src/Analysis/Ast/Impl/Analyzer/DocumentAnalysis.cs
@@ -18,14 +18,11 @@
 using System.IO;
 using System.Linq;
 using Microsoft.Python.Analysis.Analyzer.Evaluation;
-using Microsoft.Python.Analysis.Analyzer.Expressions;
 using Microsoft.Python.Analysis.Diagnostics;
-using Microsoft.Python.Analysis.Types;
 using Microsoft.Python.Analysis.Documents;
 using Microsoft.Python.Analysis.Values;
 using Microsoft.Python.Core;
 using Microsoft.Python.Core.Diagnostics;
-using Microsoft.Python.Core.Text;
 using Microsoft.Python.Parsing;
 using Microsoft.Python.Parsing.Ast;
 
@@ -67,6 +64,11 @@ public DocumentAnalysis(IDocument document, int version, IGlobalScope globalScop
         /// Expression evaluator used in the analysis.
         /// </summary>
         public IExpressionEvaluator ExpressionEvaluator { get; }
+
+        /// <summary>
+        /// Analysis diagnostics.
+        /// </summary>
+        public IEnumerable<DiagnosticsEntry> Diagnostics => ExpressionEvaluator.Diagnostics;
         #endregion
     }
 
@@ -88,5 +90,4 @@ public EmptyAnalysis(IServiceContainer services, IDocument document) {
         public IExpressionEvaluator ExpressionEvaluator { get; }
         public IEnumerable<DiagnosticsEntry> Diagnostics => Enumerable.Empty<DiagnosticsEntry>();
     }
-
 }
diff --git a/src/Analysis/Ast/Impl/Analyzer/Evaluation/ExpressionEval.cs b/src/Analysis/Ast/Impl/Analyzer/Evaluation/ExpressionEval.cs
@@ -33,7 +33,7 @@ namespace Microsoft.Python.Analysis.Analyzer.Evaluation {
     /// Helper class that provides methods for looking up variables
     /// and types in a chain of scopes during analysis.
     /// </summary>
-    internal sealed partial class ExpressionEval : IExpressionEvaluator {
+    internal sealed partial class ExpressionEval: IExpressionEvaluator {
         private readonly Stack<Scope> _openScopes = new Stack<Scope>();
         private readonly List<DiagnosticsEntry> _diagnostics = new List<DiagnosticsEntry>();
         private readonly object _lock = new object();
@@ -229,11 +229,11 @@ private async Task<IMember> GetValueFromConditionalAsync(ConditionalExpression e
             return trueValue ?? falseValue;
         }
 
-        private void ReportDiagnostics(Uri documentUri, IEnumerable<DiagnosticsEntry> entries) {
+        public void ReportDiagnostics(Uri documentUri, DiagnosticsEntry entry) {
             // Do not add if module is library, etc. Only handle user code.
             if (Module.ModuleType == ModuleType.User) {
                 lock (_lock) {
-                    _diagnostics.AddRange(entries);
+                    _diagnostics.Add(entry);
                 }
             }
         }

diff --git a/src/Analysis/Ast/Impl/Analyzer/Handlers/FromImportHandler.cs b/src/Analysis/Ast/Impl/Analyzer/Handlers/FromImportHandler.cs
@@ -33,37 +33,36 @@ public async Task<bool> HandleFromImportAsync(FromImportStatement node, Cancella
             }
 
             var rootNames = node.Root.Names;
-            IImportSearchResult imports = null;
             if (rootNames.Count == 1) {
                 var rootName = rootNames[0].Name;
                 if (rootName.EqualsOrdinal("__future__")) {
                     return false;
                 }
             }
 
-            imports = ModuleResolution.CurrentPathResolver.FindImports(Module.FilePath, node);
-            // If we are processing stub, ignore imports of the original module.
-            // For example, typeshed stub for sys imports sys.
-            if (Module.ModuleType == ModuleType.Stub && imports is ModuleImport mi && mi.Name == Module.Name) {
-                return false;
-            }
-
+            var imports = ModuleResolution.CurrentPathResolver.FindImports(Module.FilePath, node);
             switch (imports) {
+                case ModuleImport moduleImport when moduleImport.FullName == Module.Name && Module.ModuleType == ModuleType.Stub:
+                    // If we are processing stub, ignore imports of the original module.
+                    // For example, typeshed stub for 'sys' imports sys.
+                    break;
                 case ModuleImport moduleImport when moduleImport.FullName == Module.Name:
                     ImportMembersFromSelf(node);
-                    return false;
+                    break;
                 case ModuleImport moduleImport:
                     await ImportMembersFromModuleAsync(node, moduleImport.FullName, cancellationToken);
-                    return false;
+                    break;
                 case PossibleModuleImport possibleModuleImport:
-                    await HandlePossibleImportAsync(node, possibleModuleImport, cancellationToken);
-                    return false;
+                    await HandlePossibleImportAsync(possibleModuleImport, possibleModuleImport.PossibleModuleFullName, Eval.GetLoc(node.Root), cancellationToken);
+                    break;
                 case PackageImport packageImports:
                     await ImportMembersFromPackageAsync(node, packageImports, cancellationToken);
-                    return false;
-                default:
-                    return false;
+                    break;
+                case ImportNotFound notFound:
+                    MakeUnresolvedImport(null, notFound.FullName, Eval.GetLoc(node.Root));
+                    break;
             }
+            return false;
         }
 
         private void ImportMembersFromSelf(FromImportStatement node) {

diff --git a/src/Analysis/Ast/Impl/Analyzer/Handlers/ImportHandler.cs b/src/Analysis/Ast/Impl/Analyzer/Handlers/ImportHandler.cs
@@ -18,13 +18,17 @@
 using System.Threading;
 using System.Threading.Tasks;
 using Microsoft.Python.Analysis.Core.DependencyResolution;
+using Microsoft.Python.Analysis.Diagnostics;
 using Microsoft.Python.Analysis.Modules;
 using Microsoft.Python.Analysis.Types;
 using Microsoft.Python.Analysis.Values;
+using Microsoft.Python.Core;
+using Microsoft.Python.Parsing;
 using Microsoft.Python.Parsing.Ast;
+using ErrorCodes = Microsoft.Python.Analysis.Diagnostics.ErrorCodes;
 
 namespace Microsoft.Python.Analysis.Analyzer.Handlers {
-    internal sealed partial class ImportHandler: StatementHandler {
+    internal sealed partial class ImportHandler : StatementHandler {
         public ImportHandler(AnalysisWalker walker) : base(walker) { }
 
         public async Task<bool> HandleImportAsync(ImportStatement node, CancellationToken cancellationToken = default) {
@@ -54,14 +58,14 @@ public async Task<bool> HandleImportAsync(ImportStatement node, CancellationToke
                         Eval.DeclareVariable(memberName, Module, VariableSource.Declaration, location);
                         break;
                     case ModuleImport moduleImport:
-                        module = await HandleImportAsync(node, moduleImport, cancellationToken);
+                        module = await HandleImportAsync(moduleImport, location, cancellationToken);
                         break;
                     case PossibleModuleImport possibleModuleImport:
-                        module = await HandlePossibleImportAsync(node, possibleModuleImport, cancellationToken);
+                        module = await HandlePossibleImportAsync(possibleModuleImport, possibleModuleImport.PossibleModuleFullName, location, cancellationToken);
                         break;
                     default:
                         // TODO: Package import?
-                        MakeUnresolvedImport(memberName, moduleImportExpression);
+                        MakeUnresolvedImport(memberName, moduleImportExpression.MakeString(), Eval.GetLoc(moduleImportExpression));
                         break;
                 }
 
@@ -72,20 +76,21 @@ public async Task<bool> HandleImportAsync(ImportStatement node, CancellationToke
             return false;
         }
 
-        private async Task<IPythonModule> HandleImportAsync(ImportStatement node, ModuleImport moduleImport, CancellationToken cancellationToken) {
+        private async Task<IPythonModule> HandleImportAsync(ModuleImport moduleImport, LocationInfo location, CancellationToken cancellationToken) {
             var module = await ModuleResolution.ImportModuleAsync(moduleImport.FullName, cancellationToken);
             if (module == null) {
-                MakeUnresolvedImport(moduleImport.FullName, node);
+                MakeUnresolvedImport(moduleImport.FullName, moduleImport.FullName, location);
                 return null;
             }
             return module;
         }
 
-        private async Task<IPythonModule> HandlePossibleImportAsync(Node node, PossibleModuleImport possibleModuleImport, CancellationToken cancellationToken) {
+        private async Task<IPythonModule> HandlePossibleImportAsync(
+            PossibleModuleImport possibleModuleImport, string moduleName, LocationInfo location, CancellationToken cancellationToken) {
             var fullName = possibleModuleImport.PrecedingModuleFullName;
             var module = await ModuleResolution.ImportModuleAsync(possibleModuleImport.PrecedingModuleFullName, cancellationToken);
             if (module == null) {
-                MakeUnresolvedImport(possibleModuleImport.PrecedingModuleFullName, node);
+                MakeUnresolvedImport(possibleModuleImport.PrecedingModuleFullName, moduleName, location);
                 return null;
             }
 
@@ -95,7 +100,7 @@ private async Task<IPythonModule> HandlePossibleImportAsync(Node node, PossibleM
                 var childModule = module.GetMember<IPythonModule>(namePart);
                 if (childModule == null) {
                     var unresolvedModuleName = string.Join(".", nameParts.Take(i + 1).Prepend(fullName));
-                    MakeUnresolvedImport(unresolvedModuleName, node);
+                    MakeUnresolvedImport(unresolvedModuleName, moduleName, location);
                     return null;
                 }
                 module = childModule;
@@ -143,7 +148,12 @@ private void AssignImportedVariables(IPythonModule module, DottedName moduleImpo
             }
         }
 
-        private void MakeUnresolvedImport(string name, Node node)
-            => Eval.DeclareVariable(name, new SentinelModule(name, Eval.Services), VariableSource.Import, Eval.GetLoc(node));
+        private void MakeUnresolvedImport(string variableName, string moduleName, LocationInfo location) {
+            if (!string.IsNullOrEmpty(variableName)) {
+                Eval.DeclareVariable(variableName, new SentinelModule(moduleName, Eval.Services), VariableSource.Import, location);
+            }
+            Eval.ReportDiagnostics(Eval.Module.Uri, new DiagnosticsEntry(
+                Resources.ErrorUnresolvedImport.FormatInvariant(moduleName), location.Span, ErrorCodes.UnresolvedImport, Severity.Warning));
+        }
     }
 }
diff --git a/src/Analysis/Ast/Impl/Definitions/IDocumentAnalysis.cs b/src/Analysis/Ast/Impl/Definitions/IDocumentAnalysis.cs
@@ -13,7 +13,9 @@
 // See the Apache Version 2.0 License for specific language governing
 // permissions and limitations under the License.
 
+using System.Collections.Generic;
 using Microsoft.Python.Analysis.Analyzer;
+using Microsoft.Python.Analysis.Diagnostics;
 using Microsoft.Python.Analysis.Documents;
 using Microsoft.Python.Analysis.Values;
 using Microsoft.Python.Parsing.Ast;
@@ -49,5 +51,10 @@ public interface IDocumentAnalysis {
         /// Expression evaluator used in the analysis.
         /// </summary>
         IExpressionEvaluator ExpressionEvaluator { get; }
+
+        /// <summary>
+        /// Analysis diagnostics.
+        /// </summary>
+        IEnumerable<DiagnosticsEntry> Diagnostics { get; }
     }
 }
diff --git a/src/Analysis/Ast/Impl/Diagnostics/DiagnosticsSeverityMap.cs b/src/Analysis/Ast/Impl/Diagnostics/DiagnosticsSeverityMap.cs
@@ -0,0 +1,45 @@
+// Copyright(c) Microsoft Corporation
+// All rights reserved.
+//
+// Licensed under the Apache License, Version 2.0 (the License); you may not use
+// this file except in compliance with the License. You may obtain a copy of the
+// License at http://www.apache.org/licenses/LICENSE-2.0
+//
+// THIS CODE IS PROVIDED ON AN  *AS IS* BASIS, WITHOUT WARRANTIES OR CONDITIONS
+// OF ANY KIND, EITHER EXPRESS OR IMPLIED, INCLUDING WITHOUT LIMITATION ANY
+// IMPLIED WARRANTIES OR CONDITIONS OF TITLE, FITNESS FOR A PARTICULAR PURPOSE,
+// MERCHANTABILITY OR NON-INFRINGEMENT.
+//
+// See the Apache Version 2.0 License for specific language governing
+// permissions and limitations under the License.
+
+using System.Collections.Generic;
+using Microsoft.Python.Core;
+using Microsoft.Python.Parsing;
+
+namespace Microsoft.Python.Analysis.Diagnostics {
+    public sealed class DiagnosticsSeverityMap {
+        private readonly Dictionary<string, Severity> _map = new Dictionary<string, Severity>();
+
+        public DiagnosticsSeverityMap() { }
+
+        public DiagnosticsSeverityMap(string[] errors, string[] warnings, string[] information, string[] disabled) {
+            _map.Clear();
+            // disabled > error > warning > information
+            foreach (var x in information.MaybeEnumerate()) {
+                _map[x] = Severity.Information;
+            }
+            foreach (var x in warnings.MaybeEnumerate()) {
+                _map[x] = Severity.Warning;
+            }
+            foreach (var x in errors.MaybeEnumerate()) {
+                _map[x] = Severity.Error;
+            }
+            foreach (var x in disabled.MaybeEnumerate()) {
+                _map[x] = Severity.Suppressed;
+            }
+        }
+        public Severity GetEffectiveSeverity(string code, Severity defaultSeverity)
+            => _map.TryGetValue(code, out var severity) ? severity : defaultSeverity;
+    }
+}
diff --git a/src/Analysis/Ast/Impl/Diagnostics/ErrorCodes.cs b/src/Analysis/Ast/Impl/Diagnostics/ErrorCodes.cs
@@ -21,5 +21,6 @@ public static class ErrorCodes {
         public const string UnknownParameterName = "unknown-parameter-name";
         public const string ParameterAlreadySpecified = "parameter-already-specified";
         public const string ParameterMissing = "parameter-missing";
+        public const string UnresolvedImport = "unresolved-import";
     }
 }
diff --git a/src/Analysis/Ast/Impl/Diagnostics/IDiagnosticsService.cs b/src/Analysis/Ast/Impl/Diagnostics/IDiagnosticsService.cs
@@ -38,5 +38,12 @@ public interface IDiagnosticsService {
         /// the diagnostic publishing to the client.
         /// </summary>
         int PublishingDelay { get; set; }
+
+        /// <summary>
+        /// Provides map of error codes to severity when user wants
+        /// to override default severity settings or suppress particular
+        /// diagnostics completely.
+        /// </summary>
+        DiagnosticsSeverityMap DiagnosticsSeverityMap { get; set; }
     }
 }
diff --git a/src/Analysis/Ast/Impl/Modules/PythonModule.cs b/src/Analysis/Ast/Impl/Modules/PythonModule.cs
@@ -385,7 +385,7 @@ private void Parse(CancellationToken cancellationToken) {
 
                 // Do not report issues with libraries or stubs
                 if (sink != null) {
-                    _diagnosticsService?.Replace(Uri, _parseErrors);
+                    _diagnosticsService?.Replace(Uri, _parseErrors.Concat(Analysis.Diagnostics));
                 }
 
                 _parsingTask = null;
@@ -460,6 +460,11 @@ public virtual bool NotifyAnalysisComplete(IDocumentAnalysis analysis) {
                     OnAnalysisComplete();
                     ContentState = State.Analyzed;
 
+                    // Do not report issues with libraries or stubs
+                    if (ModuleType == ModuleType.User) {
+                        _diagnosticsService?.Replace(Uri, _parseErrors.Concat(Analysis.Diagnostics));
+                    }
+
                     var tcs = _analysisTcs;
                     _analysisTcs = null;
                     tcs.TrySetResult(analysis);

diff --git a/src/Analysis/Ast/Impl/Modules/Resolution/MainModuleResolution.cs b/src/Analysis/Ast/Impl/Modules/Resolution/MainModuleResolution.cs
@@ -42,7 +42,7 @@ internal async Task InitializeAsync(CancellationToken cancellationToken = defaul
 
             // Initialize built-in
             var moduleName = BuiltinTypeId.Unknown.GetModuleName(_interpreter.LanguageVersion);
-            var modulePath = ModuleCache.GetCacheFilePath(_interpreter.Configuration.InterpreterPath ?? "python.exe");
+            var modulePath = ModuleCache.GetCacheFilePath(_interpreter.Configuration.InterpreterPath);
 
             var b = new BuiltinsPythonModule(moduleName, modulePath, _services);
             _modules[BuiltinModuleName] = BuiltinsModule = b;

diff --git a/src/Analysis/Ast/Test/FluentAssertions/AssertionsFactory.cs b/src/Analysis/Ast/Test/FluentAssertions/AssertionsFactory.cs
@@ -32,5 +32,8 @@ internal static class AssertionsFactory {
         public static VariableAssertions Should(this IVariable v) => new VariableAssertions(v);
 
         public static RangeAssertions Should(this Range? range) => new RangeAssertions(range);
+
+        public static SourceSpanAssertions Should(this SourceSpan span) => new SourceSpanAssertions(span);
+        public static SourceSpanAssertions Should(this SourceSpan? span) => new SourceSpanAssertions(span.Value);
     }
 }
diff --git a/.../FluentAssertions/SourceSpanAssertions.cs → .../FluentAssertions/SourceSpanAssertions.cs b/.../FluentAssertions/SourceSpanAssertions.cs → .../FluentAssertions/SourceSpanAssertions.cs
@@ -18,8 +18,8 @@
 using Microsoft.Python.Core.Text;
 using static Microsoft.Python.Analysis.Tests.FluentAssertions.AssertionsUtilities;
 
-namespace Microsoft.Python.LanguageServer.Tests.FluentAssertions {
-    internal sealed class SourceSpanAssertions {
+namespace Microsoft.Python.Analysis.Tests.FluentAssertions {
+    public sealed class SourceSpanAssertions {
         public SourceSpan? Subject { get; }
 
         public SourceSpanAssertions(SourceSpan? span) {