Python 3 migration

[dham]

TODO

• Update install script to use Python 3 including venv.
• Delete legacy crud from install script.
• Ensure Python 3 compatibility of Firedrake by debugging all test case fails.
• Ensure Python 3 compatibility and best practice style in all demos.
• Update installation and dependency documentation.
• Document transition pathway for existing user code.
• Merge to master and inform users.
• Set PYTHONHASHSEED=0 for python < 3.6 Not doing this, instead have fixed pytest.
• de-six/future
• python3-isse firedrake-zenodo
• Ensure firedrake-update works
• Version checks for firedrake-update
• Reminder, PETSc/SLEPc build still requires python2 (because their configure system is not p3 compatible)

[miklos1]

Someone suggested:

export PYTHONHASHSEED=$(shuf -i 1-4294967295 -n 1)

[wence-]

Turns out we don't think we need the PYTHONHASHSEED thing. There was an actual bug in PyOP2 related to kernel caching which I fixed. All of the pytest-xdist issues are complete red-herrings, it turns out pytest does not have deterministic fixture collection, I have proposed a fix in pytest-dev/pytest#2617.

[wence-]

Full test suite including demos now passes.

[wence-]

Demos appear to be py3 compatible now. Not sure if it's "best practise", but it's no worse than previously.

[wence-]

Proposed upgrade path text:

Dear all,

as of XXX, Firedrake now only supports Python 3. In particular, we
require Python >= 3.

Rationale

The Python development team have formally end-of-lifed Python 2.
Python 2.7 will be the last ever Python in the 2 series (see
https://www.python.org/dev/peps/pep-0373/#update).

Similarly, many large Python packages have stopped new feature
releases compatible with Python 2. For example, the most recent
IPython releases are Python 3 only. The Scientific Python community
is also commited to dropping support for Python 2 by 2020. See
http://www.python3statement.org/ for details.

What does this mean for you?

Upgrading existing installations

For technical reasons to do with how we package a firedrake
installation, upgrading an existing (Python 2) install is not
possible. You will instead have to reinstall from scratch. Should
you attempt to update your old install, you will get an error.

Fresh installations

The installation procedure is as before (but remember to download a
new copy of the firedrake-install script). See
http://firedrakeproject.org/download.html for details.

Porting your code to Python 3

Most Python code that you have been writing will be compatible with
Python 3 already, especially if you have been writing:

from future import division, print_function, absolute_import

at the top of your Python files.

To find out what to do in detail, we recommend the advice on
http://python-future.org/, in particular, their cheat sheet:

http://python-future.org/compatible_idioms.html

You of course do not need to write compatible code, but you can use
this to see how to convert your Python 2isms to Python 3.

The most common gotchas are:

. print is no longer a statement, but a function.

You cannot write:

print "foo"

But instead should write:

print("foo")

This is enforced in Python 2 with "from future import print_function"

. division of integers returns a floating point number.

In Python 2:

1 / 2 => 0

In Python 3:

1 / 2 => 0.5

If you actually want integer division, write:

1 // 2 => 0

This is enforced in Python 2 with "from future import division"

. Iteration constructs now return generators, not lists

In Python 2, calls like dict.keys(), or range, return a list
(i.e. they are eager).

In Python 3, they return a generator that may be iterated over
(i.e. they are lazy).

You must explicitly convert them to a list if you want that.

As ever, if you have any problems installing or updating, please do
get in touch with bug reports.

[wence-]

Should we mention dict iteration in parallel?

[wence-]

We finally have a green jenkins build: https://jenkins.ese.ic.ac.uk:1080/blue/organizations/jenkins/firedrakeproject%2Ffiredrake/detail/python3/46/pipeline/

[wence-]

One of the things to think about is how to ensure uses aren't running an out-of-date firedrake-install. I don't think we can fix copies in the wild, but should we add something similar to the firedrake-update self-rebuild for the install case (downloading a new version of ourselves and re-execing)?

[miklos1]

I don't think we can fix copies in the wild, but should we add something similar to the firedrake-update self-rebuild for the install case (downloading a new version of ourselves and re-execing)?

I would rather not. Too much magic.

[wence-]

Is it more magic than what we currently do for firedrake-update?

[miklos1]

No, but its utility is much smaller, it won't help with the current problem, and makes testing new install scripts harder.

[wence-]

OK.

[wence-]

Done in #1114.