IDLE: Shell calltips bugged by printed unclosed "(''' "

[SectorCorruptor]

When executing

help()

and giving an input of, say, ghi to the prompt, help terminates with an error.

However, the IDLE calltip for help changes from

(*args, **kwds)
Define the builtin 'help'

to

Import can't find module, or can't find name in module.

(This is after ending the help session and typing help()

What is weird is that if I do the same thing again, except that I provide an input of abc, as an example of a valid name, the calltip restores back to what it originally was.

Besides, the calltip also changes for other names. If I reproduce this issue, and then I type print(, the same calltip

Import can't find module, or can't find name in module.

comes.

Is this intended behaviour?

NOTE: This behaviour is not specific to help. See #105689 (comment).

My environment

• CPython versions tested on: 3.11.3
• Operating system and architecture: Windows 11

(Note: I wanted to attach a picture, but calltips go away as soon as I take a screenshot)

[terryjreedy]

The first bug, which is not an IDLE issue, is the regression of help() crashing upon invalid names. It should catch the exception and print stuff, including a new help prompt, as it used to do. #105702. The issue here is whether IDLE could be made immune to this or whether something like inspect, which IDLE uses, is messed up. So I will try to find out where the wrong tip is coming from.

[SectorCorruptor]

It seems to me that the first issue only appeared in 3.11 or so. Before that, it would say:

No Python documentation found for 'thing'.
Use help() to get the interactive help utility.
Use help(str) for help on the str class.

and continue.
Now, it's an error, and the prompt ends there.

[terryjreedy]

The discussion of the help bug is on the issue I opened and the issue it duplicated, #102541. There was a related issue just before 3.11.0 was released.

The wrong message line Import can't find module, or can't find name in module. comes from exceptions.c, 1674-9.

ComplexExtendsException(PyExc_Exception, ImportError,
                        ImportError, 0 /* new */,
                        ImportError_methods, ImportError_members,
                        0 /* getset */, ImportError_str,
                        "Import can't find module, or can't find name in "
                        "module.");

Among whatever else this call does, it makes the line the doc string of ImportError. Since ImportError has no accessible signature, the line is the calltip for ImportError. For some reason I will investigate another day, calltip.getargspec(ob) is being called with ob == ImportError, not with the function the user types.

[SectorCorruptor]

But why does that also affect every other object's calltip (like print's?)

[SectorCorruptor]

Also, I've found another bug.

What is weird is that if I do the same thing again, except that I provide an input of abc, as an example of a valid name, the calltip restores back to what it originally was.

Apparently, only some inputs will restore it back to its original value. Trying print does not change it back.

[terryjreedy]

But why does that also affect every other object's calltip (like print's?)

Because IDLE is displaying the calltip for ImportError regardless of what is typed. I am looking into this now. Directly causing ImportError does not do this. help is doing something strange.

[terryjreedy]

Progress: let m = help error line number with raise ImportError('''\ and n = user input line number with, for instance, print(. (Initially, n = m+5 because of the additional help error lines.) The latter invokes calltip.Calltip.open_calltip, which starts with

        hp = HyperParser(self.editwin, "insert")
        sur_paren = hp.get_surrounding_brackets('(')

sur_paren should be and normally is ('n.0', 'n.end') (with n replaced with the actual line number). But after the help error, it is instead ('m.21', 'n.end') (21 is the column with '('.) Hyperparser subsequently looks backward from the error '(' to find 'ImportError'. Even after more input and output, sur_paren continues to find 'm.21'. Print the error message from help() has somehow left Shell in an incorrect state.

The HyperParser initialization correctly sees 'insert' as referring to line n. The problem is with get_surrounding_brackets. This is exposed to users as Edit => Show Surrounding Parens (default hotkey ^0 (zero) as least on Windows). Normally, if one types help( and ^0, only the last '(' is highlighted. After the help error message that prints an unmatched '(', ^0 highlights back to that unmatched paren in the error output. There are really two bugs here: 1) looking for a '(' before the first one found, and 2) crossing the boundary between output and following input.

Debugging this appears to involve the separate pyparse module. A deep solution would be separate text widgets for code input and resulting output but I have only started a draft issue for this.

What may be a factor is that Shell treat responses to executed input('prompt') statements, as with help>, as code input -- colorizing and displaying calltips in response to '('s. This is a known bug but has so far merely been annoying.

[SectorCorruptor]

I've found a potential fix: I've replaced line 1744 in pydoc.py to be like this:

            raise ImportError('No Python documentation found for %r.\nUse help() to get the interactive help utility.\nUse help(str) for help on the str class.' % thing)

It seems to be fixed now. All calltips are now working properly after causing an incorrect input

[SectorCorruptor]

Actually, these kind of issues can be easily reproduced with code similar to the following:

>>> class Y:
              def __call__(self, ...):
                  if ...:
                      raise Exception('''...
                       ...''')
>>> x = Y()
>>> x()
        Traceback (most recent call last):
          File "<pyshell#12>", line 1, in <module>
            x()
          File "<pyshell#10>", line 4, in __call__
            raise Exception('''...
        Exception: ...
        ...

After this, every calltip gets replaced with Exception's __doc__.

Apparently, calltips need to be modified, not just help.

[terryjreedy]

Thank you. The help() bug is fixed, so a reproducer is needed to experiment and to test an fix in the main branch. The reduced code with comments.

# killtip.py -- reproducer for https://github.com/python/cpython/issues/105689
# Run this and calltips will show Exception docstring
class Y:
    def __call__(self):
        # Bug requires opening ''' without closing ''' on same line.
        raise Exception('''
            ''')  # Closing ''' and ) not shown on traceback.
x = Y()()

Given that the fence-finder goes back too far, I think the unclosed ''' causes int( or whatever to be seen as part of an unfinished multiline string, making the unmatched ( in the traceback to seem like the closest opener.

More experiments: print('("' + '""') disables calltips because ( is preceded by nothing. Repeating restores calltips because the 2nd printed """ closed the first. print('int("' + '""') makes calltips the int calltip.

Please leave the title as I rewrote it.

The best solution before splitting Shell as discussed above is to make the fence-finder in Shell start at the beginning of the statement. This should be possible now. But with help() fixed, I don't consider this a high priority.

[SectorCorruptor]

About the title, I think that'll be okay for now.

It turns out the minimal required code to reproduce this is

# killtip.py -- reproducer for https://github.com/python/cpython/issues/105689
# Running this will mess up calltips
raise ValueError("""
""")

I would consider this a somewhat midline priority thing to fix because programs that raise multiline errors can mess up calltips for programs in interactive mode afterward (this bug is ONLY reproducible in interactive mode).

[terryjreedy]

Leave this bug open until fixed.