System hijacks failing when using python 3.7

[malcolmr]

With Vim 8.1.2269, running Vroom over it's own tests fails with some odd doubled-up outputs for hijacked syscalls:

$ vroom examples/system.vroom
examples/system.vroom
FAILED on line 075: Expected "Testing." in verbatim mode.

Found:
1       Testing.echo Testing. <<<<

Failed command on line 075:
Testing.

Ran 5 tests in examples/system.vroom. 4 passing, 0 errored, 1 failed.
$ vroom examples/continuations.vroom
examples/continuations.vroom
FAILED on line 47: Expected "Captured" in verbatim mode.

Found:
1       CapturedCaptured <<<<

Failed command on line 47:
Captured

Ran 5 tests in examples/continuations.vroom. 4 passing, 0 errored, 1 failed.

I believe this is caused by a change in a recent version of Vim, probably sometime after 8.1.0875.

[malcolmr]

Actually, it doesn't look like this is Vim at all. I've built back to v7.4.2330, and I'm still seeing the same problem.

I am using vroom 0.12.0, but that shouldn't matter here.

[malcolmr]

I wonder if this is actually recent versions of Python? (I have 3.7.5rc1 in the place it's failing).

[dbarnett]

Yep, I can repro and it fails on 3.7 but not on 3.5. Until the issue is tracked down and fixed, is it easy to just avoid python 3.7 for now in the places you're running into the failures, or not that simple?

[dbarnett]

Possibly related: looks like doctests fail against python 3.7 as well, with a bunch of mismatch errors showing a trailing comma in the string representation of thrown exceptions. Example:

======================================================================
FAIL: Doctest: vroom.actions.ActionLine
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/lib/python3.7/doctest.py", line 2196, in runTest
    raise self.failureException(self.format_failure(new.getvalue()))
AssertionError: Failed doctest test for vroom.actions.ActionLine
  File "/home/dbarnett/projects/vroom/vroom/actions.py", line 63, in ActionLine

----------------------------------------------------------------------
File "/home/dbarnett/projects/vroom/vroom/actions.py", line 86, in vroom.actions.ActionLine
Failed example:
    try: ActionLine('  @nope')
    except vroom.ParseError as e: e
Expected:
    ParseError('Unrecognized directive "nope"',)
Got:
    ParseError('Unrecognized directive "nope"')

[dbarnett]

Doctests issue was unrelated. I fixed it in 379eefc. Still looking into the syscalls.

[dbarnett]

Looks like there was a strange behavior change in re.sub. In 3.5.4rc1:

>>> re.sub('.*', 'Goodbye', 'Hello')
'Goodbye'

In 3.7.5rc1:

>>> re.sub('.*', 'Goodbye', 'Hello')
'GoodbyeGoodbye'

Looks pretty simple to work around. We ought to also report upstream however that works, and should be a pretty simple unit test you could write in cpython to bisect when the behavior changed.

[dbarnett]

I bisected it and tracked it down to https://bugs.python.org/issue32308. It's an intentional change, apparently. I can't wrap my head around why it would make sense, but it should be an easy workaround changing .* to ^.*$.