curses "insstr" does not handle newline properly on Mac OS

[yzhanglbto]

Bug report

import curses

def bug(scr):
    scr.addstr("first line\n")
    scr.addstr("second line")
    scr.refresh()
    scr.getch()
    scr.insstr(0, 5, "X\nXX")
    scr.refresh()
    scr.getch()

curses.wrapper(bug)

Using Python 3.11.3 and 3.11.4 on Apple M2 Mac (Mac OS Ventura 13.4, iTerm and Terminal.app), Python 3.10.6 on Intel Mac (Mac OS Monterey 12.6, iTerm) produce:

Python 3.11.2 on Intel Ubuntu 22.04 (in GNOME terminal) produces correct behavior. An equivalent C program using ncurses directly on the above Macs (in the same terminal) also produces correct behavior.

[ronaldoussoren]

How did you install Python (python.org installer, homebrew, ...)?

I get the output in the second screenshot when using Python 3.10.11 or 3.11.4 on macOS 13.4 when using the installer on python.org.

[yzhanglbto]

Thanks! I installed python on both Mac and Ubuntu using "pyenv". Maybe that's the problem? I'll try using a homebrew installed Python on my Mac.

[yzhanglbto]

I just tried on my M2 Mac. I have homebrew installed Python 3.9.17, 3.10.12, 3.11.4, and the OS default installed 3.9.6. e.g.,:

$ /usr/bin/python3 --version
Python 3.9.6
$ /usr/bin/python3 curses_bug.py # This uses the default OS python 3.9.6, where curses_bug.py is the above example.

All had (incorrect) results as in the first screenshot.

[ronaldoussoren]

This could be due to the particular version of the ncurses library used by the CPython extension module.

The script below prints the version of ncurses, for me this prints b'ncurses 5.9.20120616'.

import curses
import ctypes

fn = ctypes.CDLL(None).curses_version
fn.restype = ctypes.c_char_p
print(fn())

Also: What's the value of the environment variable TERM on your system? For me it is xterm-256color (which should be the default for Terminal.app on macOS).

And finally, could you check which ncurses library your Python build links to. The following command should show this information:

$ otool -vL $(python3.10 -c 'import _curses; print(_curses.__file__)')  
/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/lib-dynload/_curses.cpython-310-darwin.so (architecture x86_64):
	/Library/Frameworks/Python.framework/Versions/3.10/lib/libncursesw.5.dylib (compatibility version 5.0.0, current version 5.0.0)
	time stamp 2 Thu Jan  1 01:00:02 1970
	/usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1311.0.0)
	time stamp 2 Thu Jan  1 01:00:02 1970
/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/lib-dynload/_curses.cpython-310-darwin.so (architecture arm64):
	/Library/Frameworks/Python.framework/Versions/3.10/lib/libncursesw.5.dylib (compatibility version 5.0.0, current version 5.0.0)
	time stamp 2 Thu Jan  1 01:00:02 1970
	/usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1311.0.0)
	time stamp 2 Thu Jan  1 01:00:02 1970

On my machine this shows that _curses is linked to a copy of ncurses shipped with our installer. The system copy of ncurses is a lot older (b'ncurses 5.7.20081102') and might contain bugs.

[yzhanglbto]

Thank you for your comments.

My TERM variable is xterm-256color (in both iTerm and Terminal.app).

Here is the output of the little Python program you provided:

Python 3.11.3 (main, May 13 2023, 16:34:49) [Clang 14.0.3 (clang-1403.0.22.14.1)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import curses
>>> import ctypes
>>> fn = ctypes.CDLL(None).curses_version
>>> fn.restype = ctypes.c_char_p
>>> print(fn())
b'ncurses 5.7.20081102'

And here is the output of the otool command:

$ otool -vL $(python -c 'import _curses; print(_curses.__file__)')    
~/.pyenv/versions/3.11.3/lib/python3.11/lib-dynload/_curses.cpython-311-darwin.so:
	/usr/lib/libncurses.5.4.dylib (compatibility version 5.4.0, current version 5.4.0)
	time stamp 2 Wed Dec 31 17:00:02 1969
	/usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1319.100.3)
	time stamp 2 Wed Dec 31 17:00:02 1969

So it looks like the pyenv installed (and also homebrew installed) Python links to the system provided ncurses library (I think it is from the xcode command line tools installation).

I think when I do pyenv install 3.11.3. It downloaded the source and built Python locally on my machine. Maybe I should report the problem to pyenv? so they are aware of the dated ncurses library?

However as I mentioned, I also tried a C program:

#include <curses.h>

int main()
{
  WINDOW* scr = initscr();
  waddstr(scr, "first line\n");
  waddstr(scr, "second line");
  getch();
  mvwinsstr(scr, 0, 5, "X\nXX");
  getch();
  endwin();
}

Call this file curses_bug.c:

$ gcc --version
Apple clang version 14.0.3 (clang-1403.0.22.14.1)
Target: arm64-apple-darwin22.5.0
Thread model: posix
InstalledDir: /Library/Developer/CommandLineTools/usr/bin

$ gcc curses_bug.c -Wall -o curses_bug -lncurses
$ otool -vL ./curses_bug
./curses_bug:
	/usr/lib/libncurses.5.4.dylib (compatibility version 5.4.0, current version 5.4.0)
	time stamp 2 Wed Dec 31 17:00:02 1969
	/usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1319.100.3)
	time stamp 2 Wed Dec 31 17:00:02 1969

It appears the C program is linking with the same ncurses library as the pyenv installed Python 3.11.3. And yet this C program behaves correctly (i.e., as in the second screenshot I provided).

[yzhanglbto]

I checked out the cpython git repository and did a local build on my Mac.

The first build (inside the cpython repository directory):

$ ./configure
$ make
$ ./python.exe

Then:

Python 3.13.0a0 (heads/main:810d5d87d9, Jul 29 2023, 00:40:02) [Clang 14.0.3 (clang-1403.0.22.14.1)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import curses
>>> import ctypes
>>> fn = ctypes.CDLL(None).curses_version
>>> fn.restype = ctypes.c_char_p
>>> print(fn())
b'ncurses 5.7.20081102'

otool -vL $(./python.exe -c 'import _curses; print(_curses.__file__)')
~/tmp/cpython/build/lib.macosx-13.4-arm64-3.13/_curses.cpython-313-darwin.so:
	/usr/lib/libncurses.5.4.dylib (compatibility version 5.4.0, current version 5.4.0)
	time stamp 2 Wed Dec 31 17:00:02 1969
	/usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1319.100.3)
	time stamp 2 Wed Dec 31 17:00:02 1969

So it is the same as before and it still uses the system ncurses library. However the above small program I created with insstr now behaves correctly (results as shown in the second screenshot).

My second build:

$ make distclean
$ git checkout 3.12
$ ./configure
$ make
$ ./python.exe
Python 3.12.0b4+ (heads/3.12:80aebd54c8, Jul 29 2023, 01:00:29) [Clang 14.0.3 (clang-1403.0.22.14.1)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>>

The rest of the outputs are the same as above (still uses the system ncurses). And this version also behaves correctly on the little insstr test program.

Third build:

$ make distclean
$ git checkout 3.11
$ ./configure
$ make
$ ./python.exe
Python 3.11.4+ (heads/3.11:4049c5d6f8, Jul 29 2023, 01:15:27) [Clang 14.0.3 (clang-1403.0.22.14.1)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>>

I am skipping the other outputs as they are still the same (still uses the system ncurses library). However this version behaves incorrectly on the little insstr test program. It behaves exactly as my pyenv installed Python 3.11.3.

So it seems something has happened between 3.11 and 3.12 and that caused the different behavior on the insstr method in the curses module.

[ronaldoussoren]

@ned-deily and @erlend-aasland, any idea what might be going on here?

In summary: there appears to be something wrong with the curses extension module in 3.11 but not in 3.12 when linking with the system install of ncurses. Out installers work fine (but link with our own copy of that library). A small C program linking with the system install works fine.

There are some differences between 3.10 and 3.11 in _cursesmodule, but none seem relevant (3.12...3.10).

[ronaldoussoren]

@yzhanglbto : thanks for your extensive testing, this should make it easier to pinpoint the problem.

[ned-deily]

After a quick look at this, I think the difference in behavior is due to a difference between 3.11 and 3.12 in how the Python readline module is being built.

In 3.11 and earlier, there was a bunch of code in setup.py to determine whether readline should try to link with a curses library or not. When building 3.11 from source, without a newer version of ncursesw like the python.org binaries do, the resulting readline module is linked with the old Apple-supplied ncurses. However, for 3.12 (and 3.13), it appears that the new replacement for setup.py is not choosing to link readline with ncurses and, in this case, that makes the test pass either when building from source or when using the python.org binaries. I'm not at all familiar with the interdependencies of our readline module, libedit or GNU readline, and the various curses/ncurses/ncursesw libs in use but I assume there was a good reason for attempting to link readline with the curses library when possible. So this is a difference in 3.12 that might be considered a bug. Another question is whether this change in behavior is only on macOS.

@erlend-aasland, does this ring any bells?

For reference, otool -L of the readline modules in the most recent python.org 3.11 and 3.12 releases:

readline.cpython-311-darwin.so:
	/usr/lib/libedit.3.dylib (compatibility version 2.0.0, current version 3.0.0)
	/Library/Frameworks/Python.framework/Versions/3.11/lib/libncursesw.5.dylib (compatibility version 5.0.0, current version 5.0.0)
	/usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1311.0.0)

readline.cpython-312-darwin.so:
	/usr/lib/libedit.3.dylib (compatibility version 2.0.0, current version 3.0.0)
	/usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1311.0.0)

[erlend-aasland]

[...] does this ring any bells?

@ned-deily, yes. Christian and I deliberately simplified some of the third-party dependency checks. I'll have a look at what we did and why we did it, but I won't be able to find time for it until a week or two.

Thanks for the heads-up, though!

[erlend-aasland]

I browsed through configure.ac regarding some other readline issues today, and I came across this comment, added by Christian in e925241 (gh-90005, gh-94452):

dnl NOTE: In the past we checked if readline needs an additional termcap
dnl library (tinfo ncursesw ncurses termcap). We now assume that libreadline
dnl or readline.pc provide correct linker information.

[asottile]

fwiw I believe this may be a recent regression (and not in python?) -- perhaps with the macos updates to ncurses given the latest CVE

I updated from (unknown version) to 13.5.2 yesterday and this quirk breaks my text editor babi which seems to be exhibiting the same bug described here (but was working before I upgraded)