Add = to f-strings for easier debugging.

[ericvsmith]

BPO	36817
Nosy	@warsaw, @larryhastings, @ericvsmith, @serhiy-storchaka, @ilevkivskyi, @Carreau, @pablogsal
PRs	bpo-36817: Add f-string debugging using '='. #13123 bpo-36774: Add !d to f-strings to make debugging easier. #13059 bpo-36817: Fix reference leak for expr_text in f-string = parsing #13249 bpo-36817: Do not decrement reference for expr_text on fstring = parsing failure #13256

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = 'https://github.com/ericvsmith'
closed_at = <Date 2019-05-11.19:54:49.046>
created_at = <Date 2019-05-06.18:10:56.540>
labels = ['interpreter-core', 'type-feature', '3.8']
title = 'Add = to f-strings for easier debugging.'
updated_at = <Date 2019-05-22.16:20:16.071>
user = 'https://github.com/ericvsmith'

bugs.python.org fields:

activity = <Date 2019-05-22.16:20:16.071>
actor = 'pablogsal'
assignee = 'eric.smith'
closed = True
closed_date = <Date 2019-05-11.19:54:49.046>
closer = 'pablogsal'
components = ['Interpreter Core']
creation = <Date 2019-05-06.18:10:56.540>
creator = 'eric.smith'
dependencies = []
files = []
hgrepos = []
issue_num = 36817
keywords = ['patch']
message_count = 18.0
messages = ['341582', '341583', '341668', '341682', '341691', '341694', '341696', '341722', '341732', '341774', '341940', '342208', '342211', '342220', '342231', '343106', '343210', '343211']
nosy_count = 7.0
nosy_names = ['barry', 'larry', 'eric.smith', 'serhiy.storchaka', 'levkivskyi', 'mbussonn', 'pablogsal']
pr_nums = ['13123', '13059', '13249', '13256']
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'enhancement'
url = 'https://bugs.python.org/issue36817'
versions = ['Python 3.8']

[ericvsmith]

This is an alternative proposal to bpo-36774.

We (Eric V. Smith and Larry Hastings) propose a minor language
change. This level of change doesn't require a PEP, so in this
post-BDFL world what we need is "a consensus among core developers".
So please vote! Note that "+" is typed using "shift-=", and the "1"
key can be found very nearby.

Python programmers often use "printf-style" debugging. In the
(really) bad old days this was pretty wordy:
print "foo=", foo, "bar=", bar

f-strings make this slightly nicer to type:
print(f"foo={foo} bar={bar}")

But you still have to repeat yourself: you have to write
out the *string* "foo", and then the *expession* "foo".
Wouldn't it be nice if you didn't have to?

f-strings are uniquely able to help with this. Their implementation
requires them to know the original text of the expression which they
then compile. It's not difficult for f-strings to retain the text
and prepend it; the tricky part is figuring out how to spell it.

The initial idea was to use an f-string "conversion", which we
originally spelled "!=":
f'{foo!=}'
This spelling won't work, because f-strings permit arbitrary Python
expressions, and != of course tests for inequality.

We considered other spellings:
!d (debug)
!e (equals)
!x (?)
!! (easy to type)
We'd planned to go with !d. In fact Eric gave a lightning talk
about this on Friday night and used this spelling.

And then! On Saturday, the best spelling revealed itself! Behold
the majesty of:
{foo=}
This code:
foo=5
print(f"{foo=}")
would print
foo=5

With this spelling change, we've also refined the semantics.

By default, f-strings use format() (technically they call
__format__ on the value). But the point of this is for debugging.
But you want repr() for debugging. When you use this on a string,
you want to see the quoted string; when you use this on a datetime
object, you want to see the datetime repr, not the default
formatted string.

Second, this is now composable with conversions. So you can use
{foo=!s}
to use str() instead of repr() on the value.

Relatedly, we've added a new conversion: "!f" means "use format()",
which you could never explicitly specify before. For example, to only
format pi to two decimal places:
f"{math.pi=!f:.2f}" => "3.14"

Finally, and this is the best part: what if you want whitespace around
the equals sign? Well, to the left is no problem; whitespace is preserved
from the original text inside the curly braces:
f"{ chr(65) =}" => " chr(65) ='A'"
But we also explicitly permit, and preserve, whitespace *after* the
equals sign:
f"{chr(65) = }" => "chr(65) = 'A'"

What's particularly elegant is that we simply preserve all the
characters up to the final delimiter. The equals sign sets a flag
but doesn't stop flushing through text. So this:
vvvvvvvvvv
f"{chr(65) = }"
is *exactly* the same as this:
vvvvvvvvvv
"chr(65) = 'A'"

Please vote!

Eric and /arry

[warsaw]

I'll assume you can resolve any weird corner cases, in which case +1

[ilevkivskyi]

+1 from me (as a big fan of print-debugging).

[serhiy-storchaka]

I like this!

Except that I think that !f is not needed. You can use repr by default only when no format spec is specified, and add explicit !r if you want to use repr with the format spec. If you want to format the value without repr and the format spec -- specify the empty format spec: f"{foo=:}".

[ericvsmith]

Except that I think that !f is not needed. You can use repr by default only when no format spec is specified, and add explicit !r if you want to use repr with the format spec. If you want to format the value without repr and the format spec -- specify the empty format spec: f"{foo=:}".

I had this working in bpo-36774, but it seems like a little too much magic. It also prevents you from formatting the result of the repr, which works in f-strings without the =.

Say you wanted a fixed width output. You need to apply a format to the value of the repr:

>>> nums = [1/3, 1.0, 10.0, math.pi]
>>> for n in nums:
...   print(f'*{n=}*')
... 
*n=0.3333333333333333*
*n=1.0*
*n=10.0*
*n=3.141592653589793*

>>> for n in nums:
...   print(f'*{n=:30}*')
... 
*n=0.3333333333333333            *
*n=1.0                           *
*n=10.0                          *
*n=3.141592653589793             *

If the presence of a format spec meant automatically apply the format to the value being printed, this wouldn't be possible.

[serhiy-storchaka]

You can use f'*{n=!r:30}*' if you want to format the result of the repr.

In you example the format spec is applied to both the value and the literal representation of the expression. Is it an error? I do not think this is an expected behavior. If you want to apply it to both the literal expression and its value you can use the nested f-string: f"*{f'{n=}':30}*".

There is not too much more magic here: if both converter and format specifier are omitted use !r because it is a common special case. I think it is better than the other difference in the default converter used for debugging and normal formatting.

[ericvsmith]

In you example the format spec is applied to both the value and the literal representation of the expression. Is it an error? I do not think this is an expected behavior.

No, you're misreading it. I admit that my example wasn't great. Try this one:

>>> for n in nums:
...   print(f'*{n=:+<30}*')
... 
*n=0.3333333333333333++++++++++++*
*n=1.0+++++++++++++++++++++++++++*
*n=10.0++++++++++++++++++++++++++*
*n=3.141592653589793+++++++++++++*

If you want to apply it to both the literal expression and its value you can use the nested f-string: f"*{f'{n=}':30}*".

Correct. There's a similar discussion in bpo-36774.

[larryhastings]

I think that !f is not needed. You can use repr by default only when
no format spec is specified, and add explicit !r if you want to use
repr with the format spec.

Actually that's how !d worked. We changed the behavior because it was too "magical". We need to keep the f-strings format spec simple so it was easier to remember. I for one already have difficulty remembering how f-string formatting works, I don't want to make add even more complications.

In the current proposal, the special syntax must be specified in a particular order, and the order is easy to remember because information always flows from left-to-right. The "=" must come before the "!" and/or the ":", and the "!" must come before the ":". Like so:

f'{foo
=
!s
:20}'

Modification information strictly flows from left to right:

• The = changes the "conversion function" to repr, but then you can override the conversion function with !.

• The : format spec runs __format__ on the stuff to its left; if you're using the "format" conversion function, it applies the spec directly, otherwise it calls format with that spec to the output (the string) you got from the conversion function.

If we made the default conversion function when using = dependent on the presence or absence of the format spec, now we have information flowing to the left, all the way from the end to the beginning. Eric and I agree: this is too magical and too hard to remember. We want to keep it simple.

(True story: Eric had the !d implementation already done and ready for checkin. When he changed it to this = syntax he actually mostly *threw out* code, because this way is simpler and more regular. Hopefully you're thinking "well THAT sounds nice!"--we agree.)

[ericvsmith]

After discussing this with Guido and Larry, we're going to go with the "implicit format mode", as outlined by Serhiy, and drop the !f feature.

So the rules are:
{x=} -> "x="+repr(x)
{x=:.2f} -> "x="+format(x, ".2f")
{x=:} -> "x="+format(x, "")
{x=:!s:20} -> "x="+format(str(x), "20")
{x=:!r:20} -> "x="+format(repr(x), "20")

I think the 95% case will be {x=}, the 99%+ case will be {x=:2f} case. So I'm happy with this outcome. All functionality you had available with !f is still available, but with slightly different spellings. The most common cases now have the shortest spellings.

[ericvsmith]

The most recent version of the PR 0ec4dae has the behavior without !f and auto-selecting what used to be !f. It's ready for a final review before I commit it.

[ericvsmith]

New changeset 9a4135e by Eric V. Smith in branch 'master':
bpo-36817: Add f-string debugging using '='. (GH-13123)
9a4135e

[pablogsal]

Commit 9a4135e introduced a reference leak:

https://buildbot.python.org/all/#/builders/80/builds/587/steps/3/logs/stdio

Bisect results for test_future:

0:00:00 load avg: 10.04 [1/1] test_future
beginning 9 repetitions
123456789
.........
test_future leaked [24, 24, 24, 24] references, sum=96
test_future leaked [24, 24, 24, 24] memory blocks, sum=96
test_future failed

== Tests result: FAILURE ==

1 test failed:
test_future

Total duration: 300 ms
Tests result: FAILURE
9a4135e is the first bad commit
commit 9a4135e
Author: Eric V. Smith <ericvsmith@users.noreply.github.com>
Date: Wed May 8 16:28:48 2019 -0400

bpo-36817: Add f-string debugging using '='. (GH-13123)

If a "=" is specified a the end of an f-string expression, the f-string will evaluate to the text of the expression, followed by '=', followed by the repr of the value of the expression.

:040000 040000 303c86dc65bb09cade6b8f2a0fa3f97715f1793b 7ddfc1c6c5f86bf6d0a38a64ff1415c7ca55a5fe M Doc
:040000 040000 88a8dc9ffb46d652d086168e18e6a1265e0bfb92 847032e70ebd74ad7e828f8013acc7bdc4570779 M Include
:040000 040000 119e11a4f2fa3966ca300d06dd44c46dcbae11a8 ed8c093aecd82a6b886d83218681df886f653206 M Lib
:040000 040000 9c8d303bc3b468a06c39389520795bd34cfc7534 36b368a8e4ddd3f470c99b68b00a4a3a30c8c2b1 M Misc
:040000 040000 ac4c322bf5eb2685c31b375bb8a7d49a7b6a9ed9 8e311dcfcafb1260eb5c617783318088a029b067 M Parser
:040000 040000 e9db8405b51b7f72015f70556ce611c371450731 9c3da4ade5f48b3a2e842d845cafa58ba21089a3 M Python
bisect run success

[pablogsal]

Opened PR13249 to fix the leak

[pablogsal]

New changeset 5833e94 by Pablo Galindo in branch 'master':
bpo-36817: Fix reference leak for expr_text in f-string = parsing (GH-13249)
5833e94

[pablogsal]

New changeset 26f55c2 by Pablo Galindo in branch 'master':
bpo-36817: Do not decrement reference for expr_text on fstring = parsing failure (GH-13256)
26f55c2

[Carreau]

I'm not quite sure I completely understand how this is implemented and all the possibilities; – so I would appreciate reviews on the issue (and patch) to handle this in ast-unparse.

See https://bugs.python.org/issue37003

Thanks,

[pablogsal]

The reason the main CI did not catch this is that test_tools is only executed on a (random) subset of all the files if I remember correctly because when executed on all files it massively increases the time of the CI.

[pablogsal]

Anecdotally, this happened as well when in the implementation of PEP-572

[Teut2711]

Don't know how good this implemented feature is but , one of the mess is the getting item from dict in python since you have to put a [] and then "" in f strings.

d = {"a":90, "b":67}
print(f"a is {d['a']} and b is {d['b']}")

Any intention to improve this in future?

[ericvsmith]

@Teut2711 : I'm not sure what you're proposing. If you have a concrete suggestion, please discuss it at https://discuss.python.org/c/ideas/6