Add in preliminary PyPy support

[ShadowJonathan]

Pull Request Checklist

• Pull request is based on the develop branch
• Pull request includes a changelog file. The entry should:
  • Be a short description of your change which makes sense to users. "Fixed a bug that prevented receiving messages from other servers." instead of "Moved X method from EventStore to EventWorkerStore.".
  • Use markdown where necessary, mostly for code blocks.
  • End with either a period (.) or an exclamation mark (!).
  • Start with a capital letter.
• Pull request includes a sign off
• Code style is correct (run the linters)

Related to #8888
Fixes #5054

Signed-off-by: Jonathan de Jong <jonathan@automatia.nl>

[ShadowJonathan]

Blocked by pallets/markupsafe#176 (Unicode C API messing up), because this affects jinja2.escape, which is pretty vital to jinja templating, and so I wouldn't continue on with this if i wasn't sure this affected other areas of the server.

After they release version 2 (with hopefully this fixed), and jinja2 is verified to work with v2, I think adding "markupsafe>=2 ; platform_python_implementation != 'PyPy'" to dependencies will fix that issue.

[mattip]

Can I see the failing log for PyPy? I want to exactly replicate the environment to debug the PyPy c-api call failure.

[ShadowJonathan]

@mattip it's local, but the command is pypy3 -m tox -re pypy3 -- tests.rest.client.v1.test_login.MultiSSOTestCase.test_multi_sso_redirect, after installing tox on pypy3(.6), to invoke that failing test.

[mattip]

It does not reproduce passes locally for me. What version of PyPy are you using?

[ShadowJonathan]

@mattip

jonathan@jon:~$ pypy3 --version
Python 3.6.9 (7.3.1+dfsg-4ubuntu1, Sep 17 2020, 15:27:03)
[PyPy 7.3.1 with GCC 10.2.0]

[mattip]

Ahh. Yes, there are unicode c-api bugs in that version. I guess that is the one available via apt get? The current version (7.3.3) would be the one you want to use. I see you are using a docker on circleCI for CI? You might want to use the docker images that cibuildwheel provides, they have up-to-date versions of python including pypy. Or you could move to github actions, they also provide up-to-date version of pypy. If you want to do the minimum change, you can use the release tarballs from https://downloads.python.org/pypy/. They are portable and should work on any machine that supports manylinux2010.

pushd /tmp
wget https://downloads.python.org/pypy/pypy3.6-v7.3.3-linux64.tar.bz2
tar -xf pypy3.6-v7.3.3-linux64.tar.bz2
export PATH=/tmp/pypy3.6-v7.3.3-linux64.tar/bin;$PATH
ln -s /tmp/pypy3.6-v7.3.3-linux64.tar/bin/python /tmp/pypy3.6-v7.3.3-linux64.tar/bin/pypy3
python -m ensurepip
python -m install tox 

or if you prefer conda

conda create -n pypy36 pypy3.6
conda activate pypy36
conda install -y tox

[ShadowJonathan]

Docker image pypy:3.6 is PyPy 7.3.3, I think we'll be alright there.

Thanks for the headsup about the unicode bugs in that pypy version. 👍

I see you are using a docker on circleCI for CI?

Docker on buildkite, with boxes provided by matrix.org themselves, see the pipeline for more details.

[ShadowJonathan]

Some test results (-postgres using psycopg2):

-j 2
py36: Ran 1501 tests in 875.411s
pypy3: Ran 1501 tests in 4378.433s

-j 3
py36-postgres: Ran 1501 tests in 706.022s
pypy3-postgres: Ran 1501 tests in 2548.308s

😐

[ShadowJonathan]

https://www.pypy.org/performance.html tells that (perfect unit) tests and short-ran scripts don't get time to be optimised by the JIT

So tests are severely affected by the JIT this way, i'm going to look for ways to front-load this problem, and instead "warm up" the JIT to make it possible for tests to run more smoothly.

[ShadowJonathan]

Looking further into it, i get the distinct impression pypy might have a C-API performance issue, this gist shows the performance difference between python and pypy on psycopg2.

I found pg8000 which is written in pure python, this might be counter-intuitive to use, but it's possible that with a JIT-optimized pypy, this library can outperform psycopg2, as with the tests, i saw them being done at blazing speeds when i removed a stray gc.collect() call somewhere, so this might be worth looking into.

[mattip]

pypy might have a C-API performance issue

Definitely, emulating the C-API is slow. There is work ongoing to improve this via a new c-api (HPy) that will layer this on top of the current c-api.

Writing simple python code (as little dynamic class creation as possible, keeping class heirarchies shallow) will be fast on PyPy, for interfacing with C we recommend CFFI and for C++ we recommend cppyy.

[ShadowJonathan]

Looking at https://morepypy.blogspot.com/2018/09/inside-cpyext-why-emulating-cpython-c.html, i got the idea to actually go use pg8000 (a pure-python DB API v2-complicit postgres driver). I'm currently running the tests again (at -j 3) using this driver, so we'll see how good the performance is, the early results (test OKs breezing past my screen) are promising.

Edit: Oh, i didnt see your comment @mattip, thanks! :D

[ShadowJonathan]

Just finished running the new pypy3-postgres suite with -j 3 and pg8000 as driver: Ran 1501 tests in 763.756s :D

[ShadowJonathan]

To reiterate: This PR will only add in some simple support for PyPy. In it's current form, a running server would be abysmally slow, due to C-API interaction with it's internal sqlite3 package, and C-API interaction with psycopg2cffi.

pg8000 will be added in another PR, and additional DB-cleaning-up PRs after that (that make adding extra adapters/drivers/engines easier, especially with the prospect of PEP 249 existing, and will seek to clean up the codebase to start to use with engine.new_connection() as conn: context managers and such). I'll also think about making a "postgres" virtual adapter then, for reverse compatibility, all "psycopg2"-specified driver configurations will keep using psycopg2, and wont suddenly switch over to pg8000 then, with "postgres" as "engine identifier", it'll switch between psycopg2 and pg8000 depending if the current running python is pypy or python. I think fancy things such as replicated writers will have to wait in pypy, as i first want to vet pg8000 before that happens.

I'll add the -prof cProfileing tox environment in another PR. (#9293)

(I'll open this PR once all tests pass)

[callahad]

In it's current form, a running server would be abysmally slow, due to C-API interaction with it's internal sqlite3 package, and C-API interaction with psycopg2cffi.

I believe you're misunderstanding; psycopg2cffi, per the name, uses CFFI, as mattip recommends. This is in contrast to CPython's native C extension API documented at https://docs.python.org/3/extending/extending.html.

[ShadowJonathan]

It's still slow, the above test results (pypy3-postgres: Ran 1501 tests in 2548.308s) are using psycopg2cffi, so even if cffi is supposed to give a performance boost (or mitigate pypy's C API translation), I can't find evidence of it.

[clokep]

Looks pretty reasonable overall! To be clear -- with these changes tests will pass? Did you run sytests by any chance?

[mattip]

Thanks for doing this. It would be interesting to see how this plays out in a production environment where the JIT can actually make things faster.

When thinking about which version of PyPy to test (assuming you will not test both python3.6 and python3.7), I would suggest you go with 3.7 not 3.6. Going forward, people are starting to drop 3.6 since it is no longer supported, and when underlying dependencies are updated they may not be available for 3.6.

[ShadowJonathan]

To be clear -- with these changes tests will pass? Did you run sytests by any chance?

I ran tox -e pypy3 tests, but i haven't yet ran sytest tests, i'll do that in a second.