Release v3.1.2 (#853)

See Coconut's [documentation](http://coconut.readthedocs.io/en/develop/DOCS.html) for more information on all of the features listed below. Bugfixes: * #851, #852: Fixed comments inside of parentheses in the Jupyter kernel. Language features: * #846: `reduce`, `takewhile`, and `dropwhile` now support keyword arguments. * #848: Class and data patterns now support keyword argument name elision. * #847: New pattern-matching syntax for matching anonymous named tuples. Compiler features: * #843: Added compiler warnings for (some cases of) undefined variables.
evhub · Sep 1, 2024 · 4659175 · 4659175
2 parents ac70b14 + 120c273
commit 4659175
Show file tree

Hide file tree

Showing 24 changed files with 304 additions and 127 deletions.
diff --git a/.pre-commit-config.yaml b/.pre-commit-config.yaml
@@ -34,7 +34,7 @@ repos:
         args:
         - --autofix
 -   repo: https://github.com/pycqa/flake8
-    rev: 7.0.0
+    rev: 7.1.1
     hooks:
     -   id: flake8
         args:

diff --git a/DOCS.md b/DOCS.md
@@ -1006,7 +1006,7 @@ Coconut also allows a single `?` before attribute access, function calling, part
 
 When using a `None`-aware operator for member access, either for a method or an attribute, the syntax is `obj?.method()` or `obj?.attr` respectively. `obj?.attr` is equivalent to `obj.attr if obj is not None else obj`. This does not prevent an `AttributeError` if `attr` is not an attribute or method of `obj`.
 
-The `None`-aware indexing operator is used identically to normal indexing, using `?[]` instead of `[]`. `seq?[index]` is equivalent to the expression `seq[index] is seq is not None else seq`. Using this operator will not prevent an `IndexError` if `index` is outside the bounds of `seq`.
+The `None`-aware indexing operator is used identically to normal indexing, using `?[]` instead of `[]`. `seq?[index]` is equivalent to the expression `seq[index] if seq is not None else seq`. Using this operator will not prevent an `IndexError` if `index` is outside the bounds of `seq`.
 
 Coconut also supports None-aware [pipe operators](#pipes) and [function composition pipes](#function-composition).
 
@@ -1204,6 +1204,7 @@ base_pattern ::= (
     | NAME "(" patterns ")"          # classes or data types
     | "data" NAME "(" patterns ")"   # data types
     | "class" NAME "(" patterns ")"  # classes
+    | "(" name "=" pattern ... ")"   # anonymous named tuples
     | "{" pattern_pairs              # dictionaries
         ["," "**" (NAME | "{}")] "}" #  (keys must be constants or equality checks)
     | ["s" | "f" | "m"] "{"
@@ -1269,7 +1270,8 @@ base_pattern ::= (
   - Classes or Data Types (`<name>(<args>)`): will match as a data type if given [a Coconut `data` type](#data) (or a tuple of Coconut data types) and a class otherwise.
   - Data Types (`data <name>(<args>)`): will check that whatever is in that position is of data type `<name>` and will match the attributes to `<args>`. Generally, `data <name>(<args>)` will match any data type that could have been constructed with `makedata(<name>, <args>)`. Includes support for positional arguments, named arguments, default arguments, and starred arguments. Also supports strict attributes by prepending a dot to the attribute name that raises `AttributError` if the attribute is not present rather than failing the match (e.g. `data MyData(.my_attr=<some_pattern>)`).
   - Classes (`class <name>(<args>)`): does [PEP-634-style class matching](https://www.python.org/dev/peps/pep-0634/#class-patterns). Also supports strict attribute matching as above.
-- Mapping Destructuring:
+  - Anonymous Named Tuples (`(<name>=<pattern>, ...)`): checks that the object is a `tuple` of the given length with the given attributes. For matching [anonymous `namedtuple`s](#anonymous-namedtuples).
+- Dict Destructuring:
   - Dicts (`{<key>: <value>, ...}`): will match any mapping (`collections.abc.Mapping`) with the given keys and values that match the value patterns. Keys must be constants or equality checks.
   - Dicts With Rest (`{<pairs>, **<rest>}`): will match a mapping (`collections.abc.Mapping`) containing all the `<pairs>`, and will put a `dict` of everything else into `<rest>`. If `<rest>` is `{}`, will enforce that the mapping is exactly the same length as `<pairs>`.
 - Set Destructuring:
@@ -1735,7 +1737,7 @@ The syntax for a statement lambda is
 ```
 [async|match|copyclosure] def (arguments) => statement; statement; ...
 ```
-where `arguments` can be standard function arguments or [pattern-matching function definition](#pattern-matching-functions) arguments and `statement` can be an assignment statement or a keyword statement. Note that the `async`, `match`, and [`copyclosure`](#copyclosure-functions) keywords can be combined and can be in any order.
+where `arguments` can be standard function arguments or [pattern-matching function definition](#pattern-matching-functions) arguments and `statement` can be any non-compound statement—that is, any statement that doesn't open a code block below it (so `def x => assert x` is fine but `def x => if x: True` is not). Note that the `async`, `match`, and [`copyclosure`](#copyclosure-functions) keywords can be combined and can be in any order.
 
 If the last `statement` (not followed by a semicolon) in a statement lambda is an `expression`, it will automatically be returned.
 
@@ -2233,7 +2235,7 @@ as a shorthand for
 f(long_variable_name=long_variable_name)
 ```
 
-Such syntax is also supported in [partial application](#partial-application) and [anonymous `namedtuple`s](#anonymous-namedtuples).
+Such syntax is also supported in [partial application](#partial-application), [anonymous `namedtuple`s](#anonymous-namedtuples), and [`class`/`data`/anonymous `namedtuple` patterns](#match).
 
 _Deprecated: Coconut also supports `f(...=long_variable_name)` as an alternative shorthand syntax._
 
@@ -2262,7 +2264,7 @@ main_func(
 
 ### Anonymous Namedtuples
 
-Coconut supports anonymous [`namedtuple`](https://docs.python.org/3/library/collections.html#collections.namedtuple) literals, such that `(a=1, b=2)` can be used just as `(1, 2)`, but with added names. Anonymous `namedtuple`s are always pickleable.
+Coconut supports anonymous [`namedtuple`](https://docs.python.org/3/library/collections.html#collections.namedtuple) literals, such that `(a=1, b=2)` can be used just as `(1, 2)`, but with added names. Anonymous `namedtuple`s are always pickleable and support [`__match_args__`](https://peps.python.org/pep-0622/) on all Python versions.
 
 The syntax for anonymous namedtuple literals is:
 ```coconut
@@ -3803,9 +3805,9 @@ _Can’t be done quickly without Coconut’s iterable indexing, which requires m
 
 #### `reduce`
 
-**reduce**(_function_, _iterable_[, _initial_], /)
+**reduce**(_function_, _iterable_[, _initial_])
 
-Coconut re-introduces Python 2's `reduce` built-in, using the `functools.reduce` version.
+Coconut re-introduces Python 2's `reduce` built-in, using the `functools.reduce` version. Additionally, unlike `functools.reduce`, Coconut's `reduce` always supports keyword arguments.
 
 ##### Python Docs
 
@@ -3935,9 +3937,9 @@ result = itertools.zip_longest(range(5), range(10))
 
 #### `takewhile`
 
-**takewhile**(_predicate_, _iterable_, /)
+**takewhile**(_predicate_, _iterable_)
 
-Coconut provides `itertools.takewhile` as a built-in under the name `takewhile`.
+Coconut provides `itertools.takewhile` as a built-in under the name `takewhile`. Additionally, unlike `itertools.takewhile`, Coconut's `takewhile` always supports keyword arguments.
 
 ##### Python Docs
 
@@ -3969,9 +3971,9 @@ negatives = itertools.takewhile(lambda x: x < 0, numiter)
 
 #### `dropwhile`
 
-**dropwhile**(_predicate_, _iterable_, /)
+**dropwhile**(_predicate_, _iterable_)
 
-Coconut provides `itertools.dropwhile` as a built-in under the name `dropwhile`.
+Coconut provides `itertools.dropwhile` as a built-in under the name `dropwhile`. Additionally, unlike `itertools.dropwhile`, Coconut's `dropwhile` always supports keyword arguments.
 
 ##### Python Docs
 

diff --git a/Makefile b/Makefile
@@ -26,27 +26,27 @@ dev-py3: clean setup-py3
 .PHONY: setup
 setup:
 	-python -m ensurepip
-	python -m pip install --upgrade setuptools wheel pip pytest_remotedata cython
+	python -m pip install --upgrade setuptools wheel pip cython
 
 .PHONY: setup-py2
 setup-py2:
 	-python2 -m ensurepip
-	python2 -m pip install --upgrade "setuptools<58" wheel pip pytest_remotedata cython
+	python2 -m pip install --upgrade "setuptools<58" wheel pip cython
 
 .PHONY: setup-py3
 setup-py3:
 	-python3 -m ensurepip
-	python3 -m pip install --upgrade setuptools wheel pip pytest_remotedata cython
+	python3 -m pip install --upgrade setuptools wheel pip cython
 
 .PHONY: setup-pypy
 setup-pypy:
 	-pypy -m ensurepip
-	pypy -m pip install --upgrade "setuptools<58" wheel pip pytest_remotedata
+	pypy -m pip install --upgrade "setuptools<58" wheel pip
 
 .PHONY: setup-pypy3
 setup-pypy3:
 	-pypy3 -m ensurepip
-	pypy3 -m pip install --upgrade setuptools wheel pip pytest_remotedata
+	pypy3 -m pip install --upgrade setuptools wheel pip
 
 .PHONY: install
 install: setup

diff --git a/__coconut__/__init__.pyi b/__coconut__/__init__.pyi
@@ -1865,6 +1865,18 @@ def _coconut_mk_anon_namedtuple(
     fields: _t.Tuple[_t.Text, ...],
     types: _t.Optional[_t.Tuple[_t.Any, ...]] = None,
 ) -> _t.Callable[..., _t.Tuple[_t.Any, ...]]: ...
+@_t.overload
+def _coconut_mk_anon_namedtuple(
+    fields: _t.Tuple[_t.Text, ...],
+    types: _t.Optional[_t.Tuple[_t.Any, ...]],
+    of_args: _T,
+) -> _T: ...
+@_t.overload
+def _coconut_mk_anon_namedtuple(
+    fields: _t.Tuple[_t.Text, ...],
+    *,
+    of_args: _T,
+) -> _T: ...
 
 
 # @_t.overload

diff --git a/coconut/_pyparsing.py b/coconut/_pyparsing.py
@@ -46,7 +46,7 @@
     never_clear_incremental_cache,
     warn_on_multiline_regex,
     num_displayed_timing_items,
-    use_cache_file,
+    use_pyparsing_cache_file,
     use_line_by_line_parser,
     incremental_use_hybrid,
 )
@@ -254,7 +254,7 @@ def enableIncremental(*args, **kwargs):
     and hasattr(MatchFirst, "setAdaptiveMode")
 )
 
-USE_CACHE = SUPPORTS_INCREMENTAL and use_cache_file
+USE_CACHE = SUPPORTS_INCREMENTAL and use_pyparsing_cache_file
 USE_LINE_BY_LINE = USE_COMPUTATION_GRAPH and use_line_by_line_parser
 
 if MODERN_PYPARSING:

diff --git a/coconut/command/util.py b/coconut/command/util.py
@@ -671,7 +671,7 @@ class Prompt(object):
     style = None
     runner = None
     lexer = None
-    suggester = None if prompt_use_suggester else False
+    suggester = True if prompt_use_suggester else None
 
     def __init__(self, setup_now=False):
         """Set up the prompt."""
@@ -686,7 +686,7 @@ def setup(self):
         We do this lazily since it's expensive."""
         if self.lexer is None:
             self.lexer = PygmentsLexer(CoconutLexer)
-        if self.suggester is None:
+        if self.suggester is True:
             self.suggester = AutoSuggestFromHistory()
 
     def set_style(self, style):

diff --git a/coconut/compiler/compiler.py b/coconut/compiler/compiler.py
@@ -246,12 +246,17 @@ def import_stmt(imp_from, imp, imp_as, raw=False):
     )
 
 
-def imported_names(imports):
-    """Yields all the names imported by imports = [[imp1], [imp2, as], ...]."""
+def get_imported_names(imports):
+    """Returns all the names imported by imports = [[imp1], [imp2, as], ...] and whether there is a star import."""
+    saw_names = []
+    saw_star = False
     for imp in imports:
         imp_name = imp[-1].split(".", 1)[0]
-        if imp_name != "*":
-            yield imp_name
+        if imp_name == "*":
+            saw_star = True
+        else:
+            saw_names.append(imp_name)
+    return saw_names, saw_star
 
 
 def special_starred_import_handle(imp_all=False):
@@ -529,7 +534,8 @@ def reset(self, keep_state=False, filename=None):
         # but always overwrite temp_vars_by_key since they store locs that will be invalidated
         self.temp_vars_by_key = {}
         self.parsing_context = defaultdict(list)
-        self.unused_imports = defaultdict(list)
+        self.name_info = defaultdict(lambda: {"imported": [], "referenced": [], "assigned": []})
+        self.star_import = False
         self.kept_lines = []
         self.num_lines = 0
         self.disable_name_check = False
@@ -942,6 +948,11 @@ def strict_err(self, *args, **kwargs):
         if self.strict:
             raise self.make_err(CoconutStyleError, *args, **kwargs)
 
+    def strict_warn(self, *args, **kwargs):
+        internal_assert("extra" not in kwargs, "cannot pass extra=... to strict_warn")
+        if self.strict:
+            self.syntax_warning(*args, extra="remove --strict to dismiss", **kwargs)
+
     def syntax_warning(self, message, original, loc, **kwargs):
         """Show a CoconutSyntaxWarning. Usage:
             self.syntax_warning(message, original, loc)
@@ -1319,21 +1330,30 @@ def streamline(self, grammars, inputstring=None, force=False, inner=False):
             elif inputstring is not None and not inner:
                 logger.log("No streamlining done for input of length {length}.".format(length=input_len))
 
+    def qa_error(self, msg, original, loc):
+        """Strict error or warn an error that should be disabled by a NOQA comment."""
+        ln = self.adjust(lineno(loc, original))
+        comment = self.reformat(" ".join(self.comments[ln]), ignore_errors=True)
+        if not self.noqa_regex.search(comment):
+            self.strict_err_or_warn(
+                msg + " (add '# NOQA' to suppress)",
+                original,
+                loc,
+                endpoint=False,
+            )
+
     def run_final_checks(self, original, keep_state=False):
         """Run post-parsing checks to raise any necessary errors/warnings."""
-        # only check for unused imports if we're not keeping state accross parses
+        # only check for unused imports/etc. if we're not keeping state accross parses
         if not keep_state:
-            for name, locs in self.unused_imports.items():
-                for loc in locs:
-                    ln = self.adjust(lineno(loc, original))
-                    comment = self.reformat(" ".join(self.comments[ln]), ignore_errors=True)
-                    if not self.noqa_regex.search(comment):
-                        self.strict_err_or_warn(
-                            "found unused import " + repr(self.reformat(name, ignore_errors=True)) + " (add '# NOQA' to suppress)",
-                            original,
-                            loc,
-                            endpoint=False,
-                        )
+            for name, info in self.name_info.items():
+                if info["imported"] and not info["referenced"]:
+                    for loc in info["imported"]:
+                        self.qa_error("found unused import " + repr(self.reformat(name, ignore_errors=True)), original, loc)
+                if not self.star_import:  # only check for undefined names when there are no * imports
+                    if name not in all_builtins and info["referenced"] and not (info["assigned"] or info["imported"]):
+                        for loc in info["referenced"]:
+                            self.qa_error("found undefined name " + repr(self.reformat(name, ignore_errors=True)), original, loc)
 
     def parse_line_by_line(self, init_parser, line_parser, original):
         """Apply init_parser then line_parser repeatedly."""
@@ -3473,25 +3493,30 @@ def __new__(_coconut_cls, {all_args}):
 
         return self.assemble_data(decorators, name, namedtuple_call, inherit, extra_stmts, stmts, base_args, paramdefs)
 
-    def make_namedtuple_call(self, name, namedtuple_args, types=None):
+    def make_namedtuple_call(self, name, namedtuple_args, types=None, of_args=None):
         """Construct a namedtuple call."""
         if types:
             wrapped_types = [
                 self.wrap_typedef(types.get(i, "_coconut.typing.Any"), for_py_typedef=False)
                 for i in range(len(namedtuple_args))
             ]
-            if name is None:
-                return "_coconut_mk_anon_namedtuple(" + tuple_str_of(namedtuple_args, add_quotes=True) + ", " + tuple_str_of(wrapped_types) + ")"
-            else:
-                return '_coconut.typing.NamedTuple("' + name + '", [' + ", ".join(
-                    '("' + argname + '", ' + wrapped_type + ")"
-                    for argname, wrapped_type in zip(namedtuple_args, wrapped_types)
-                ) + "])"
         else:
-            if name is None:
-                return "_coconut_mk_anon_namedtuple(" + tuple_str_of(namedtuple_args, add_quotes=True) + ")"
-            else:
-                return '_coconut.collections.namedtuple("' + name + '", ' + tuple_str_of(namedtuple_args, add_quotes=True) + ')'
+            wrapped_types = None
+        if name is None:
+            return (
+                "_coconut_mk_anon_namedtuple("
+                + tuple_str_of(namedtuple_args, add_quotes=True)
+                + ("" if wrapped_types is None else ", " + tuple_str_of(wrapped_types))
+                + ("" if of_args is None else ", of_args=" + tuple_str_of(of_args) + "")
+                + ")"
+            )
+        elif wrapped_types is None:
+            return '_coconut.collections.namedtuple("' + name + '", ' + tuple_str_of(namedtuple_args, add_quotes=True) + ')' + ("" if of_args is None else tuple_str_of(of_args))
+        else:
+            return '_coconut.typing.NamedTuple("' + name + '", [' + ", ".join(
+                '("' + argname + '", ' + wrapped_type + ")"
+                for argname, wrapped_type in zip(namedtuple_args, wrapped_types)
+            ) + "])" + ("" if of_args is None else tuple_str_of(of_args))
 
     def assemble_data(self, decorators, name, namedtuple_call, inherit, extra_stmts, stmts, match_args, paramdefs=()):
         """Create a data class definition from the given components.
@@ -3597,8 +3622,7 @@ def anon_namedtuple_handle(self, original, loc, tokens):
             names.append(name)
             items.append(item)
 
-        namedtuple_call = self.make_namedtuple_call(None, names, types)
-        return namedtuple_call + "(" + ", ".join(items) + ")"
+        return self.make_namedtuple_call(None, names, types, of_args=items)
 
     def single_import(self, loc, path, imp_as, type_ignore=False):
         """Generate import statements from a fully qualified import and the name to bind it to."""
@@ -3731,13 +3755,17 @@ def import_handle(self, original, loc, tokens):
         else:
             raise CoconutInternalException("invalid import tokens", tokens)
         imports = list(imports)
-        if imp_from == "*" or imp_from is None and "*" in imports:
+        imported_names, star_import = get_imported_names(imports)
+        self.star_import = self.star_import or star_import
+        if star_import:
+            self.strict_warn("found * import; these disable Coconut's undefined name detection", original, loc)
+        if imp_from == "*" or (imp_from is None and star_import):
             if not (len(imports) == 1 and imports[0] == "*"):
                 raise self.make_err(CoconutSyntaxError, "only [from *] import * allowed, not from * import name", original, loc)
             self.syntax_warning("[from *] import * is a Coconut Easter egg and should not be used in production code", original, loc)
             return special_starred_import_handle(imp_all=bool(imp_from))
-        for imp_name in imported_names(imports):
-            self.unused_imports[imp_name].append(loc)
+        for imp_name in imported_names:
+            self.name_info[imp_name]["imported"].append(loc)
         return self.universal_import(loc, imports, imp_from=imp_from)
 
     def complex_raise_stmt_handle(self, loc, tokens):
@@ -4575,7 +4603,7 @@ def string_atom_handle(self, original, loc, tokens, allow_silent_concat=False):
             return tokens[0]
         else:
             if not allow_silent_concat:
-                self.strict_err_or_warn("found Python-style implicit string concatenation (use explicit '+' instead)", original, loc)
+                self.strict_err_or_warn("found implicit string concatenation (use explicit '+' instead)", original, loc)
             if any(s.endswith(")") for s in tokens):  # has .format() calls
                 # parens are necessary for string_atom_handle
                 return "(" + " + ".join(tokens) + ")"
@@ -4989,8 +5017,10 @@ def name_handle(self, original, loc, tokens, assign=False, classname=False, expr
                             )
                         return typevars[name]
 
-        if not assign:
-            self.unused_imports.pop(name, None)
+        if assign:
+            self.name_info[name]["assigned"].append(loc)
+        else:
+            self.name_info[name]["referenced"].append(loc)
 
         if (
             assign