bpo-37206: Unrepresentable default values no longer represented as None. #13933

serhiy-storchaka · 2019-06-09T19:26:37Z

https://bugs.python.org/issue37206

methane · 2019-06-10T15:14:34Z

Lib/test/clinic.test

@@ -92,7 +92,7 @@ test_objects_converter
 [clinic start generated code]*/

 PyDoc_STRVAR(test_objects_converter__doc__,
-"test_objects_converter($module, a, b=None, /)\n"
+"test_objects_converter($module, a, b=<unspecified>, /)\n"


It would be considerable that <unspecified> is obvious.
PEP 570 enabled us to make text signature and document consistent.
In document, we use [, b] for optional argument without public default value.

I would prefer [, b] if it be what the user see. But the average user will not see this, because the inspect module fails to parse both forms and the user will see just (...) in the help output. The purpose of this change to prevent displaying a wrong result (b=None when None is not acceptable). I do not see a sense of spending additional effort if the result is the same. I will make the representation using brances once I have found how to make it visible to the user.

ned-deily · 2019-06-10T19:14:15Z

I'm not going to get into whether this PR is appropriate for 3.8 but, assuming it were, I would be very concerned about making a change of this size and complexity in 3.7 at this stage in its lifecycle.

gpshead · 2019-06-11T05:08:08Z

Tools/clinic/clinic.py

@@ -4400,7 +4399,7 @@ def bad_node(self, node):
                # mild hack: explicitly support NULL as a default value
                if isinstance(expr, ast.Name) and expr.id == 'NULL':
                    value = NULL
-                    py_default = 'None'
+                    py_default = '<unspecified>'


I'd also caution against the use of words like "unspecified" or "undefined" because that can imply to a reader that the behavior is arbitrary and up to the Python implementation when not supplied. We do define these behaviors, they just require reading further documentation.

<see docs> might be better? the brackets make it clear that it isn't an actual Python value no matter what text goes in there.

It does not matter what invalid syntax is used here. The end user will never see it. This is the simplest way to make the insect module not reporting the wrong result. The effect is the same as there is not any signature (as was before using Argument Clinic).

What do you think about <unrepresentable> or <nonrepresentable>?

(This will not be seen by an average Python user, it is only for core developers)

gpshead · 2019-06-11T05:13:19Z

Tools/clinic/clinic.py

@@ -3204,6 +3201,8 @@ def converter_init(self, *, accept={str}, encoding=None, zeroes=False):
            # sorry, clinic can't support preallocated buffers
            # for es# and et#
            self.c_default = "NULL"
+        if NoneType in accept and self.c_default == "Py_None":


is this really okay? we hold the GIL and the code you've modified at a quick first glance isn't using the value when it is == Py_None, but we never Py_INCREF'd the Py_None and there is nothing preventing us from treating it as if we had.

It also potentially adds situations where None is accepted as a parameter where it wasn't before? I'd need to go audit all of the APIs this change is trying to update to see if each of them intends to allow None as a value for a review of that. In that case if someone did pass in None from Python code, it should've been Py_INCREF'd - who's doing the DECREF before clobbering it?

This is the str converter. The target's type is const char *. Py_None is invalid value here. None is converted to NULL. For example see _io.open and various encode/decode methods.

bedevere-bot · 2019-06-11T05:13:27Z

When you're done making the requested changes, leave the comment: I have made the requested changes; please review again.

And if you don't make the requested changes, you will be poked with soft cushions!

serhiy-storchaka

Currenty it is just the simlest way to make the insect module failing to parse the signature instead of returning the false result. The end user will see the same resilt as if there is no any signature.

serhiy-storchaka · 2019-06-11T06:25:18Z

Lib/test/clinic.test

@@ -92,7 +92,7 @@ test_objects_converter
 [clinic start generated code]*/

 PyDoc_STRVAR(test_objects_converter__doc__,
-"test_objects_converter($module, a, b=None, /)\n"
+"test_objects_converter($module, a, b=<unspecified>, /)\n"


I would prefer [, b] if it be what the user see. But the average user will not see this, because the inspect module fails to parse both forms and the user will see just (...) in the help output. The purpose of this change to prevent displaying a wrong result (b=None when None is not acceptable). I do not see a sense of spending additional effort if the result is the same. I will make the representation using brances once I have found how to make it visible to the user.

serhiy-storchaka · 2019-06-11T06:32:14Z

Tools/clinic/clinic.py

@@ -3204,6 +3201,8 @@ def converter_init(self, *, accept={str}, encoding=None, zeroes=False):
            # sorry, clinic can't support preallocated buffers
            # for es# and et#
            self.c_default = "NULL"
+        if NoneType in accept and self.c_default == "Py_None":


This is the str converter. The target's type is const char *. Py_None is invalid value here. None is converted to NULL. For example see _io.open and various encode/decode methods.

serhiy-storchaka

Sorry for being terse. Posted from a phone.

serhiy-storchaka · 2019-06-11T06:43:12Z

Tools/clinic/clinic.py

@@ -4400,7 +4399,7 @@ def bad_node(self, node):
                # mild hack: explicitly support NULL as a default value
                if isinstance(expr, ast.Name) and expr.id == 'NULL':
                    value = NULL
-                    py_default = 'None'
+                    py_default = '<unspecified>'


It does not matter what invalid syntax is used here. The end user will never see it. This is the simplest way to make the insect module not reporting the wrong result. The effect is the same as there is not any signature (as was before using Argument Clinic).

miss-islington · 2019-09-14T09:24:08Z

Thanks @serhiy-storchaka for the PR 🌮🎉.. I'm working now to backport this PR to: 3.8.
🐍🍒⛏🤖

miss-islington · 2019-09-14T09:24:21Z

Sorry, @serhiy-storchaka, I could not cleanly backport this to 3.8 due to a conflict.
Please backport using cherry_picker on command line.
cherry_picker 279f44678c8b84a183f9eeb85e0b086228154497 3.8

… as None. (pythonGH-13933) In ArgumentClinic, value "NULL" should now be used only for unrepresentable default values (like in the optional third parameter of getattr). "None" should be used if None is accepted as argument and passing None has the same effect as not passing the argument at all.. (cherry picked from commit 279f446) Co-authored-by: Serhiy Storchaka <storchaka@gmail.com>

bedevere-bot · 2019-09-14T09:32:09Z

GH-16141 is a backport of this pull request to the 3.8 branch.

bedevere-bot · 2019-09-14T10:01:46Z

⚠️⚠️⚠️ Buildbot failure ⚠️⚠️⚠️

Hi! The buildbot x86 Gentoo Non-Debug with X 3.x has failed when building commit 279f446.

What do you need to do:

Don't panic.
Check the buildbot page in the devguide if you don't know what the buildbots are or how they work.
Go to the page of the buildbot that failed (https://buildbot.python.org/all/#builders/99/builds/3302) and take a look at the build logs.
Check if the failure is related to this commit (279f446) or if it is a false positive.
If the failure is related to this commit, please, reflect that on the issue and make a new Pull Request with a fix.

You can take a look at the buildbot page here:

https://buildbot.python.org/all/#builders/99/builds/3302

Failed tests:

test_smtpnet

Failed subtests:

test_connect_default_port - test.test_smtpnet.SmtpSSLTest

Summary of the results of the build (if available):

== Tests result: FAILURE then FAILURE ==

410 tests OK.

10 slowest tests:

test_concurrent_futures: 3 min 3 sec
test_multiprocessing_spawn: 2 min 54 sec
test_tools: 2 min 30 sec
test_tokenize: 2 min 23 sec
test_multiprocessing_forkserver: 1 min 43 sec
test_lib2to3: 1 min 28 sec
test_asyncio: 1 min 25 sec
test_multiprocessing_fork: 1 min 16 sec
test_capi: 1 min 4 sec
test_io: 55 sec 563 ms

1 test failed:
test_smtpnet

8 tests skipped:
test_devpoll test_kqueue test_msilib test_startfile
test_winconsoleio test_winreg test_winsound test_zipfile64

1 re-run test:
test_smtpnet

Total duration: 21 min 36 sec

Click to see traceback logs

Traceback (most recent call last):
  File "/buildbot/buildarea/cpython/3.x.ware-gentoo-x86.nondebug/build/Lib/test/test_smtpnet.py", line 58, in test_connect_default_port
    server = smtplib.SMTP_SSL(self.testServer)
  File "/buildbot/buildarea/cpython/3.x.ware-gentoo-x86.nondebug/build/Lib/smtplib.py", line 1032, in __init__
    SMTP.__init__(self, host, port, local_hostname, timeout,
  File "/buildbot/buildarea/cpython/3.x.ware-gentoo-x86.nondebug/build/Lib/smtplib.py", line 256, in __init__
    raise SMTPConnectError(code, msg)
smtplib.SMTPConnectError: (421, b'4.4.5 Server busy, try again later.')

… as None. (GH-13933) (GH-16141) In ArgumentClinic, value "NULL" should now be used only for unrepresentable default values (like in the optional third parameter of getattr). "None" should be used if None is accepted as argument and passing None has the same effect as not passing the argument at all. (cherry picked from commit 279f446) Co-authored-by: Serhiy Storchaka <storchaka@gmail.com>

serhiy-storchaka added type-bug An unexpected behavior, bug, or error needs backport to 3.7 labels Jun 9, 2019

serhiy-storchaka requested review from 1st1, asvetlov, methane and tiran as code owners June 9, 2019 19:26

the-knights-who-say-ni added the CLA signed label Jun 9, 2019

bedevere-bot added the awaiting core review label Jun 9, 2019

bpo-37206: Unrepresentable default values no longer represented as None.

4d65caa

serhiy-storchaka force-pushed the clinic-unrepresentable-default branch from 610914c to 4d65caa Compare June 9, 2019 19:32

methane reviewed Jun 10, 2019

View reviewed changes

gpshead removed the needs backport to 3.7 label Jun 11, 2019

gpshead requested changes Jun 11, 2019

View reviewed changes

bedevere-bot added awaiting changes and removed awaiting core review labels Jun 11, 2019

serhiy-storchaka commented Jun 11, 2019

View reviewed changes

serhiy-storchaka added 5 commits September 13, 2019 22:57

Merge branch 'master' into clinic-unrepresentable-default

6dd2ccf

<unspecified> -> <unrepresentable>

88b1167

str.lstrip() and str.rstrip() accept None

898bf2a

sys.exit() should have the same effect as sys.exit(None).

2a2e5c6

Simplify.

0f1e93b

serhiy-storchaka merged commit 279f446 into python:master Sep 14, 2019

serhiy-storchaka deleted the clinic-unrepresentable-default branch September 14, 2019 09:24

bedevere-bot removed the awaiting changes label Sep 14, 2019

miss-islington assigned serhiy-storchaka Sep 14, 2019

bedevere-bot removed the needs backport to 3.8 label Sep 14, 2019

serhiy-storchaka removed their assignment Dec 29, 2020

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

bpo-37206: Unrepresentable default values no longer represented as None. #13933

bpo-37206: Unrepresentable default values no longer represented as None. #13933

serhiy-storchaka commented Jun 9, 2019 •

edited by bedevere-bot

Loading

methane Jun 10, 2019

serhiy-storchaka Jun 11, 2019

ned-deily commented Jun 10, 2019

gpshead Jun 11, 2019

serhiy-storchaka Jun 11, 2019

serhiy-storchaka Aug 30, 2019

gpshead Jun 11, 2019

serhiy-storchaka Jun 11, 2019

bedevere-bot commented Jun 11, 2019

serhiy-storchaka left a comment

serhiy-storchaka Jun 11, 2019

serhiy-storchaka Jun 11, 2019

serhiy-storchaka left a comment

serhiy-storchaka Jun 11, 2019

miss-islington commented Sep 14, 2019

miss-islington commented Sep 14, 2019

bedevere-bot commented Sep 14, 2019

bedevere-bot commented Sep 14, 2019

bpo-37206: Unrepresentable default values no longer represented as None. #13933

bpo-37206: Unrepresentable default values no longer represented as None. #13933

Conversation

serhiy-storchaka commented Jun 9, 2019 • edited by bedevere-bot Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ned-deily commented Jun 10, 2019

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

bedevere-bot commented Jun 11, 2019

serhiy-storchaka left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

serhiy-storchaka left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

miss-islington commented Sep 14, 2019

miss-islington commented Sep 14, 2019

bedevere-bot commented Sep 14, 2019

bedevere-bot commented Sep 14, 2019

⚠️⚠️⚠️ Buildbot failure ⚠️⚠️⚠️

serhiy-storchaka commented Jun 9, 2019 •

edited by bedevere-bot

Loading