absl/base/dynamic_annotations.h conflict with Python3.4 Python.h header

[Mizux]

When building google/or-tools python package (using SWIG and abseil-cpp as deps) for Python 3.4
it can't compile since absl/base/dynamic_annotations.h conflict with Python 3.4 symbols:

abseil-cpp/absl/base/dynamic_annotations.h

Lines 253 to 256 in 111ca70

	void AnnotateBenignRaceSized(const char *file, int line,
	const volatile void *address,
	size_t size,
	const char *description);

CPython 3.4: Python/dynamic_annotations.c:

void AnnotateBenignRaceSized(const char *file, int line,
                             const volatile void *mem,
                             long size,
                             const char *description) {}

src: https://github.com/python/cpython/blob/e42b705188271da108de42b55d9344642170aa2b/Python/dynamic_annotations.c#L96-L99

here a trace in our docker jobs

mkdir -p objs/swig
g++ -fPIC -std=c++11 -fwrapv -O4 -DNDEBUG -I. -Iortools/gen -I/root/or-tools/dependencies/install/include -I/root/or-tools/dependencies/install/include -I/root/or-tools/dependencies/install/include -I/root/or-tools/dependencies/install/include -I/root/or-tools/dependencies/install/include/coin -I/root/or-tools/dependencies/install/include -I/root/or-tools/dependencies/install/include/coin -I/root/or-tools/dependencies/install/include -I/root/or-tools/dependencies/install/include/coin -DUSE_CLP -I/root/or-tools/dependencies/install/include -I/root/or-tools/dependencies/install/include/coin -I/root/or-tools/dependencies/install/include -I/root/or-tools/dependencies/install/include/coin -DUSE_CBC -Wno-deprecated -DUSE_GLOP -DUSE_BOP     -I/usr/include/python3.4m -I/usr/lib/python3.4  -I/usr/include/python3.4m -I/usr/include/x86_64-linux-gnu/python3.4m   -DPY3 \
 -c ortools/gen/ortools/algorithms/knapsack_solver_python_wrap.cc \
 -o objs/swig/knapsack_solver_python_wrap.o
In file included from /root/or-tools/dependencies/install/include/absl/container/fixed_array.h:45:0,
                 from /root/or-tools/dependencies/install/include/absl/hash/internal/hash.h:43,
                 from /root/or-tools/dependencies/install/include/absl/hash/hash.h:64,
                 from /root/or-tools/dependencies/install/include/absl/container/internal/hash_function_defaults.h:55,
                 from /root/or-tools/dependencies/install/include/absl/container/flat_hash_map.h:39,
                 from ./ortools/util/time_limit.h:24,
                 from ./ortools/algorithms/knapsack_solver.h:70,
                 from ortools/gen/ortools/algorithms/knapsack_solver_python_wrap.cc:3139:
/root/or-tools/dependencies/install/include/absl/base/dynamic_annotations.h:256:48: error: declaration of C function 'void AnnotateBenignRaceSized(const char*, int, const volatile void*, size_t, const char*)' conflicts with
                         const char *description);
                                                ^
In file included from /usr/include/python3.4m/pyatomic.h:8:0,
                 from /usr/include/python3.4m/Python.h:53,
                 from ortools/gen/ortools/algorithms/knapsack_solver_python_wrap.cc:173:
/usr/include/python3.4m/dynamic_annotations.h:422:6: error: previous declaration 'void AnnotateBenignRaceSized(const char*, int, const volatile void*, long int, const char*)' here
 void AnnotateBenignRaceSized(const char *file, int line,
      ^
make: *** [objs/swig/knapsack_solver_python_wrap.o] Error 1
The command '/bin/sh -c make detect_python && make python' returned a non-zero code: 2
make: *** [Makefile:123: export/ubuntu-14.04/docker.tar] Error 2

note:

• Ubuntu 14.04 LTS (codename Trusty) python3-dev package point to python 3.4
• Usually people using Travis-CI target the default Trusty linux vm...
  • note: Travis-CI "just" release Xenial (16.04 LTS) vm on November 2018
• Seems to work against python last release 2.7, 3.5, 3.6 and 3.7 include headers -> only python3.4 (at least last release) is impacted...
• Also spotted in Cmake links all abseil components and results in a link time error #225 but it deserve to have its own issue to track it (please reference the github issue in the commit message once fixed)
  see: https://help.github.com/articles/closing-issues-using-keywords/

[bstaletic]

It might be worth pointing out that Python3.4 will reach its end of life in 2019-03-16.

[Mizux]

And Ubuntu 14.04.5 LTS which is based on, end of life in April 2019

[mbxx]

This issue is also essentially a duplicate of #122.

Am I correct in inferring that this is only an issue in Python 3.4 and not an issue in later versions?

[bstaletic]

Am I correct in inferring that this is only an issue in Python 3.4 and not an issue in later versions?

Unfortunately no. I've just finished bisecting the CPython history. The patch that allows abseil and CPython to cooperate wasn't present until Python 3.5.1rc1.

The specific CPython commit that allowed everything to work is python/cpython@6df29ad and is referring to a seemingly unrelated bug.

[Mizux]

I would say yes, since the current official Python 3.5 is 3.5.6
see https://docs.python.org/3.5/whatsnew/changelog.html#python-3-5-6

Also using the "official" manylinux docker image, I manage to compile against the python cp35-cp35m provided.
side note: it was the manylinux2010 proposal docker image based on manylinux PR #252 since manylinux1 is based on glibc 2.5 while abseil requires glibc >= 2.12 (fortunately manylinux2010 PEP 571 is based on centos 6 i.e. glibc 2.12)

note: Ubuntu Xenial 16.04 LTS python3 package is based on python3.5.1 see https://packages.ubuntu.com/xenial/python3-dev i.e. it should include the patch of the 3.5.1rc1

[bstaletic]

So the only way for someone to get 3.5.0 is to use pyenv and specifically request for either 3.5 or 3.5.0 in the newly created virtual environment.

For our project, we did have a user who requested 2.7 only to find out there's a bug in CPython that prevents 2.7.0 to work with our project, so we had to increase the minimum required Python version to 2.7.1. Not really a big deal, just pointing out that there may be users who are actually running, deliberately or not, 3.5.0.

note: Ubuntu Xenial 16.04 LTS python3 package is based on python3.5.1 see https://packages.ubuntu.com/xenial/python3-dev i.e. it should include the patch of the 3.5.1rc1

Yes, Ubuntu 16.04 python3 works already.

[OlafvdSpek]

So the only way for someone to get 3.5.0 is to use pyenv and specifically request for either 3.5 or 3.5.0 in the newly created virtual environment.

Shouldn't 3.5 cause the latest 3.5 release to be chosen rather than the first?

[bstaletic]

Shouldn't 3.5 cause the latest 3.5 release to be chosen rather than the first?

Whether it should is one thing, but what actually happens is that asking pyenv to creat a virtual environment of version X.Y, creates an X.Y.0 environment. That's how ycmd realized 2.7.0 does not actually work.

[JonathanDCohen]

Hey there everyone! Sorry for the delay, I've been on vacation and nobody else has picked this up. Question about the end of life dates -- are these deadlines for us to push a fix or dates after which we shouldn't care about this bug? Or maybe both? I'm unfamiliar with the Ubuntu and Python support policies, but a quick look through suggests that this is an end-of-support date for Python 3.4. Given the potential magnitude of the fix for this bug, it seems to me like the best solution is to wait for the bad version of Python to sunset and maybe try to provide a warning somewhere. Any thoughts on this?

[bstaletic]

The dates, 2019-03-16 for Python and 2019-04-XX for Ubuntu 14.04, is when they reach end of life. Ubuntu 14.04 is relevant, because it's the Ubuntu long term support with Python 3.4 as default Python 3. After Python 3.4 reaches end of life, the only stable version of Python affected will be 3.5.0, as 3.5.1 already has a patch that solves this problem.

[bstaletic]

For the record, it's May 2019. Python 3.4 reached end of life as well as Ubuntu 14.04. The only way for someone to hit this problem is if that someone installs 3.5.0 explicitly.