Override default readline via python -m override_readline

[ludwigschwardt]

Over the years we have tried and debated many ways to override the default readline module in Python so that tab completion can be fixed in the standard interactive interpreter (see e.g. #26 and #27).

The original solution was easy_install readline: ugly but effective. In the pip era this option has fallen away. The current installation route of pip install gnureadline leaves it up to the user to override sys.modules['readline'] or to import gnureadline as readline.

The problem is that those tricks have to happen before the interpreter even prints its startup banner. Python 3 roughly starts up like this:

• run site.py:

  • set up sys.__interactivehook__ [but DON'T import readline yet]
  • run sitecustomize.py
  • run usercustomize.py

• import rlcompleter [imports readline!]

• output:

  Python 3.11.3 (main, May  9 2023, 18:38:16) [...] on darwin
  Type "help", "copyright", "credits" or "license" for more information.

• run PYTHONSTARTUP
• run sys.__interactivehook__() [also imports readline fwiw]
• output:

  >>>

PYTHONSTARTUP is a bit late to the party and will need to redo the interactive hook as well (see for example this useful comment). This makes sitecustomize and usercustomize more convenient, as they only need to override readline itself.

Here is a rough comparison of the various customization routes:

Route	Affected versions if on Python 3.11.3	Affects non-interactive scripts too
sitecustomize	3.11.3	yes
usercustomize	3.11.x	yes
PYTHONSTARTUP	all	no

It feels a bit iffy when an installed package modifies those customization modules but, in mitigation, the user has to push the button and the override tries to explain what it does in some detail.

When you run python -m override_readline, the script first targets usercustomize.py because then you don't have to update it after every patch release (third number in version triplet), just like the gnureadline package itself. If you don't have permission or you are in a virtualenv, it falls back to sitecustomize.py. If you want to go straight to sitecustomize.py, simply add the standard -s flag and run python -s -m override_readline.

It does sys.modules['readline'] = gnureadline and also lets sys.__interactivehook__ print a reminder that readline is overridden (and where the override is). This is better than a direct print() which also pops up in non-interactive Python invocations like pip, etc. Here is an example:

Python 3.12.2 (main, Apr 17 2024, 20:25:57) [Clang 15.0.0 (clang-1500.0.40.1)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
Using GNU readline instead of the default readline (see sitecustomize.py)
>>>

The override is reasonably foolproof and won't crash even if gnureadline is uninstalled and the override inadvertently remains behind. And best of all, it all works in Python 2.7 too because site customization has been around forever 😛 The nice thing about running a module as a script is that it inherently uses the appropriate python interpreter, which avoids confusion.

I've also updated the tests to confirm that override_readline works. The original test involving the alternative readline.py module is downplayed. I don't want to give anyone bad ideas 😅

I also refreshed the main README to mention things currently in use, instead of easy_install and the old pyreadline, and incorporated @keeely's very useful suggestion.

[ludwigschwardt]

I've now verified that @onlynone's pythonrc works as expected. I checked it by replacing the line

readline.parse_and_bind('tab: complete')

with silly completion code customised for each library:

if "libedit" in readline.__doc__:
    readline.parse_and_bind("bind e rl_complete")
else:
    readline.parse_and_bind("r: complete")

So if thee key does tab completion, you are definitely using libedit, while tab completion via the r key confirms the presence of GNU readline. 😁

This approach only requires the PYTHONSTARTUP environment variable to be set to ~/.pythonrc, and there is no need to run override_readline as well.

Thanks Steven for the very useful example!