Xlib linking error on macOS Catalina [SOLVED (anaconda issue); keeping open to improve CMake reporting, likely related to #416 - CMAKE_PREFIX_PATH]

[saweiss]

I am using macOS Catalina Version 10.15.4. My cmake command is:

cmake .. -DCMAKE_CXX_COMPILER=clang++ \
-DCMAKE_BUILD_TYPE=Release \
-DCMAKE_CXX_FLAGS_RELEASE="-Xpreprocessor -Wno-return-type" \
"-Wno-switch -Wno-format -O3 -fno-math-errno" \
"-msse2 -march=native -DNDEBUG" \
-DHDF=OFF -DREADLINEDIR=/usr/local/opt/readline/

When I run make, I get a bunch of errors on the files "gdlxstream.hpp", "gdlxstream.cpp", and "devicex.hpp". All of the errors are "use of undeclared identifier", and all of the undeclared identifiers seem to be from the X11/Xlib.h header, e.g., "XOpenDisplay". Is this maybe a linking error with X11? The configure script showed that it correctly located the XQuartz header and lib folders, so it's a bit odd to me that it's not finding those identifiers.

Does anyone have an idea why I am getting this error?

[alaingdl]

Heu, obviously, you are doing that on the Git version of today ?!

I do not have access to a Catalina but I can compile without such problem on a Mojave, with very similar CMake options. (just I cannot activate OpenMP here but it is not the question !)

Concerning X11, on my side, all the X11 aspect (include, lib, ...) is prefixed by
/usr/X11R6/include, e.g.

~/GDL/gdl-1.0.0rc2-git200607CMake/build/]$ grep -ni  CMakeCache.txt
542:X11_X11_INCLUDE_PATH:PATH=/usr/X11R6/include
545:X11_X11_LIB:FILEPATH=/usr/X11R6/lib/libX11.dylib
[...]

(not sure this is related but do you have xcode around ? I have :

$  xcode-select -v
xcode-select version 2354.

[saweiss]

Yes, I tested on the most recent master branch. My CMakeCache.txt file shows the same exact paths for X11 as you posted. When I ran xcode-select -v, I got version 2373.

For what it's worth, I initially thought my problem was caused by the relocated items issue. Essentially, upon upgrading to Catalina, certain files were relocated to a folder for security reasons. Annoyingly, this breaks XQuartz. Had I known about it, I would have uninstalled XQuartz (using this gist) before upgrading to Catalina and then reinstall XQuartz. Fixing the issue, otherwise, involved booting into recovery mode and deleting the folder that holds the relocated files. Long story short, it seems like XQuartz is working now and the /usr/X11 and /usr/X11R6 simlinks are fixed. Despite that, the same compiler errors remain.

[saweiss]

In order to troubleshoot, I made a bare-bones test program that calls XOpenDisplay and XCloseDisplay.

#include <stdio.h>
#include <stdlib.h>
#include <X11/Xlib.h>

int main(void)
{
    Display* display = XOpenDisplay(NULL);
    if (display == NULL)
    {
        fprintf(stderr, 
                "%s:%d: failed to open X11 display\n",
                __FILE__, __LINE__);
        exit(EXIT_FAILURE);
    }
    XCloseDisplay(display);
    return 0;
}

I compiled and ran it with:

$ clang main.c -I/usr/X11R6/include -L/usr/X11R6/lib -lX11 -o test_x11
$ ./test_x11

This successfully opened XQuartz, so it seems like there is no issue with linking to the Xlib library and using some of its functions.

[alaingdl]

two tests I would make

• removing the whole build/ (esp. the CmakeCache files), or create a new one, and re-run
• provide the path to X11 to CMake

Which version of CMake do you use ?

A.

[saweiss]

I had already been removing the entire contents of the build directory before each test. I just tried manually providing the X11 paths to Cmake with the flags -DX11_INCLUDE_DIR=/usr/X11R6/include and -DX11_LIBRARIES=/usr/X11R6/lib. This didn't change anything. My CMake version is 3.17.3.

Here is the very first error that comes up when compiling, for reference.

In file included from [...]/gdl/src/gdlxstream.cpp:22:
[...]/gdl/src/gdlxstream.hpp:32:34: error: use of undeclared
      identifier 'XOpenDisplay'; did you mean 'XwDisplay'?
    : GDLGStream( nx, ny, "xwin",XOpenDisplay(NULL)==NULL?":0":NULL) //I...
                                 ^
/usr/local/include/plplot/plxwd.h:63:3: note: 'XwDisplay' declared here
} XwDisplay;
  ^

Could this have anything to do with plplot? I know Xlib is included from plxwd.h, but I don't see anything obviously wrong with my plplot. Is there a recommended way to install plplot in order to work with GDL (ideally with wxWidgets)? I compiled the most recent release of plplot using the CMake command here.

[alaingdl]

Just to check whether it is due to recent changes, could you check with an older version ? rc2 or rc1
https://github.com/gnudatalanguage/gdl/releases/tag/v1.0.0-rc.1
https://github.com/gnudatalanguage/gdl/releases/tag/v1.0.0-rc.2

On my side, I will try to have the same Cmake version on my 10.14.4, and I asked to some colleagues to access to Catalina systems, without answers up to now.

[saweiss]

Thanks for the continued help. I tried both of those versions with the same results.

[GillesDuvert]

Hm, this may come from plplot includes.
gdlxstream inherits <plplot/plxwd.h>. this one on my linux #include <X11/Xlib.h>.
Would it be possible that it is not the case on MacOS? Or that Xlib.h is not found at this location?
Adding #include <X11/Xlib.h> in gdlxstream.cpp before #include <plplot/plxwd.h> may help at least compile it --- not a full answer yet.

[saweiss]

I did try adding that include to that file a couple days ago, but it didn't change anything. I know that the include works on my Mac because of the test program I posted above. I've tried the plplot from homebrew and compiled from source, as well.

[alaingdl]

What you can try too is to run the script/minimum_script4gdl.sh because it uses its own plplot

[saweiss]

I ran the script, and it compiled fully. When I went to execute, I got this:

[...]/gdl/build $ ./gdl-0.9.9/quick_start_GDL.sh
Running current GDL with ./src/pro and ./testsuite in GDL_PATH
+[...]/gdl/build/gdl-0.9.9/src/pro/:+[...]/gdl/build/gdl-0.9.9/testsuite/
Assertion failed: (semaphore_info != (SemaphoreInfo *) NULL), function LockSemaphoreInfo, file magick/semaphore.c, line 601.
gdl-0.9.9/quick_start_GDL.sh: line 91: 13445 Abort trap: 6           ${Absolute_PATH}/src/gdl

It seems like this issue has already been encountered with #611, but the issue was solved there by changing the FreeBSD ports to use ImageMagick instead of GraphicsMagick. I tried setting DGRAPHICSMAGICK=OFF, but then CMake couldn't find ImageMagick. I then tried specifying the path using DMAGICKDIR, but it still couldn't find it. I have ImageMagick version 7.0.10-16 installed by homebrew, for what it's worth.

[alaingdl]

Great. OK. My point with the script is maybe we can test if something is wrong with the plplot you have before we re-compile it locally. (a way to circumvent the OSX system/mess)
I assume X11 is OK and Plplot provided by the system is KO.

Now I would say you should succeed on the GDL git version but linking with the local plplot

mkdir build
cd build
cmake .. (all the options -DOPENMP=off ...) -DPLPLOTDIR=/Users/saweiss/Path/to/plplot/Compilation

[saweiss]

Right, got distracted ;) ... I used this:

[...]/gdl/build $ cmake .. -DREADLINEDIR=/usr/local/opt/readline/ \
-DOPENMP=OFF -DWXWIDGETS=OFF -DGEOTIFF=OFF -DHDF=OFF \
-DUDUNITS2=OFF -DPSLIB=OFF -DGRIB=off -DGLPK=OFF \
-DPLPLOTDIR=[...]/gdl/build2/plplot-5.13.0/Compilation

I still got the same errors when compiling.

[alaingdl]

Strange ! Do we made such a change ?

Last idea for today : trying on 1.0.0rc1 here https://github.com/gnudatalanguage/gdl/releases/tag/v1.0.0-rc.1

[saweiss]

I tried it, yet still got the same errors.

[GillesDuvert]

Sorry, I'm confused:

• the initial error was at compile time, with 0.9.9, last OSX, on XOpenDisplay.
• It seems solved by compiling from scratch with script/minimum_script4gdl.sh ... compilation OK
• But execution fails on gdl-0.9.9 against latest graphicsmagick-1.3.32 #611
• Since gdl-0.9.9 against latest graphicsmagick-1.3.32 #611 is deemed solved, using 10.0.0rc1 or above should get a working GDL

If not, then #611 is not solved 🤕
In any case, 0.9.9 is not going to pass on new OSX, which is probably OK if 1.0.xx do pass.
Am I right?

[slayoo]

Google seem to suggest that such problem exists on OSX when Anaconda is installed (along with custom Xlib.h, custom include paths, etc), do you happen to use it? Does switching off Python support in GDL change anything? (Alain's script has -DPYTHON=OFF)

[alaingdl]

Ha, would be great !
Yes take care Anaconda don't have some of its paths prepended around. We had the problem before.
(in Bash : printenv PATH)

[saweiss]

Oy vey! I can't believe that's what I was. I had come across something mentioning the bad interplay between Anaconda and CMake, but I tried -DPYTHON=OFF, so I thought I had ruled it out. Turns out Anaconda is a nasty little bugger. I had to completely remove it from my path to solve it. I suppose, in order to close this issue, it should be noted in the MacOS/Linux install part of the wiki that Anaconda cannot be on the path at all.

[slayoo]

A note in a wiki/README can easily get outdated and cause more trouble than help. It would be great to make CMake detect and report the situation. After all, it should be a simple X11-hello-world test for availability of XOpenDisplay...

[alaingdl]

Well done @slayoo ! It is not the first time we had such a problem, see comments stating at
#752 (comment)

A note, on my system when I manually may switch Anaconda, when I switch it on, then running CMake, I got a serious warning !

[...]
  Cannot generate a safe runtime search path for target gdl because files in
  some directories may conflict with libraries in implicit directories:

    runtime library [libncurses.so] in /usr/lib/x86_64-linux-gnu may be hidden by files in:
      /obs/alaingdl/anaconda3/envs/miricle.devel/lib
[...] 

I searched around for a way to detect then clean-up the PATH if Conda/MiniConda/AnaConda prepend it in the PATH, but nothing clear. When activated once, Conda sets up several CONDA_ variables, included CONDA_PREFIX, CONDA_SHLVL and CONDA_EXE. When doing :

conda deactivate

change the status of CONDA_SHLVL and remove CONDA_PREFIX but CONDA_EXE still exist, and unfortunatlly on my env, the PATH is not clean-up.

May I suggested we had at the beginning of _CMake a strong test on that ?

if ((DEFINED ENV{CONDA_PREFIX}) OR (DEFINED ENV{CONDA_EXE}))
  message(FATAL_ERROR "{Ana|mini|}conda around ! You must removed it from you $PATH")
endif()

[slayoo]

I doubt it's a good idea - for many users, Anaconda is the preferred way of obtaining Python.
What about making Python OFF by default? It's non-IDL-compliant and has a super little user base as far as I can imagine...

[alaingdl]

I tried but, as long as Anaconda is in the PATH, it did not change the problem (and the warning message I have at the end of CMake).

May be we just need to say they have to temporarily removed it in the PATH ?

[GillesDuvert]

OK, so #611 is solved.
Anaconda prevents GDL (and probably other innocent software) to run correctly because of a slitherin PATH. It is not at compile time in Cmake that we'll catch it, as it can happen afterwards in a distro.
Our magic wand for such run-time problems that must be trapped (and probably cured if we are clever) lies in a 'gdl' that would be a script, not directly the excutable.

[GillesDuvert]

Hm, probably I misunderstood the situation.
Conda modified Path must be updated at compile time apparently.
This does not change the fact that an intermediate executable script can chech the availability of files we depend that are not dectedable at compile time.

[alaingdl]

Sorry, my messages were unclear : I think we can survive with Conda prepended in the PATH after the compilation (Conda don't alter LD_LIBRARY_PATH !), but during CMake run, at compilation time, the PATH should not point to Conda libs. first, this is not what we want to compile GDL.

[GillesDuvert]

Still, adding a note in the Wiki is better than nothing and will not harm. We'll remove it when it's not usfeul.

[alaingdl]

And what about a big warning at the end of the running of CMake ??

[slayoo]

I'm wondering if this is not related: #416

If we would append all the include paths on the way in CMakeLists.txt, then the find_package(X11) should fail in @saweiss's case. As of now, since the Python paths are not used by find_package(X11), but are used during compilation, one is not warned by CMake of inability to work with X11, but the compilation fails later on. Right?

[saweiss]

I realize this is not a solution for GDL's CMake routine, but perhaps it would be helpful in the mean time for those using Anaconda to do what I did. I completely uninstalled Anaconda and installed pyenv instead. This is a much more friendly python manager that won't conflict with GDL (or other programs that use Xlib). I can still install any version of Anaconda or Miniconda, but if I need to compile something that may conflict, I can switch the python to a non-Anaconda version with a simple command, and Andaconda would no longer be on my path.

So far, this is the most full-featured CMake command that I've been able to run and then compile successfully:

~/gdl/build $ cmake .. -DCMAKE_CXX_COMPILER=clang++ \
-DCMAKE_BUILD_TYPE=Release \
-DCMAKE_CXX_FLAGS_RELEASE="-Wno-return-type -Wno-switch -Wno-format -O3 -fno-math-errno -msse2 -march=native -DNDEBUG" \
-DHDF=OFF -DREADLINEDIR=/usr/local/opt/readline/ -DOPENMP=OFF \
-DPSLIB=OFF -DPYTHONDIR=$HOME/.pyenv/versions/3.8.2 \
-DPYTHON_INCLUDE_DIRS=$HOME/.pyenv/versions/3.8.2/include/python3.8

With this configuration, only 5 of the make check tests failed. I think only pslib and open-mp are the last major additions that I would need to see if I could get working.

EDIT:
I should note, if you would like to use pyenv with GDL, you must install python with shared libraries enabled. Otherwise, GDL's CMake won't find the library for python.

$ env PYTHON_CONFIGURE_OPTS="--enable-shared" pyenv install 3.8.2
$ pyenv global 3.8.2
$ ln -s ~/.pyenv/versions/3.8.2/lib/libpython3.8.dylib ~/.pyenv/versions/3.8.2/lib/libpython.dylib
$ pip install numpy

[slayoo]

@GillesDuvert A propos: what is the current status regarding the need for using pslib in GDL - from earlier exchanges I understood that it is to be completely dropped, right?

[GillesDuvert]

Indeed, as I said, pslib is not really used and certainly not necessary for the simple need it was used.
Besides, using PSLIB, more than adding yet another dependency, complicates solving postscript related issues.

[slayoo]

Indeed, as I said, pslib is not really used and certainly not necessary for the simple need it was used.
Besides, using PSLIB, more than adding yet another dependency, complicates solving postscript related issues.

@GillesDuvert Please check if this would be OK then: #781 Thanks!

[alaingdl]

OK. For me this CMAKE_PREFIX_PATH is not clear at all now.

I did not resist to put a large conda warning at the end of CMakeLists.txt in #791. I found it is a good trade off and may help not to loose time later. Comments welcome. (Please keep in mind I help end-users to compile GDL from scratch. It will help me !)

Should we take notes on Conda and pyenv in the Wiki ?

[slayoo]

Let me close this one, and open a new issue on the CMAKE_PREFIX_PATH: #794

Re Wiki: if it's already in the CmakeLists.txt, I guess it's enough? ... but I don't have a clear vision what kind of contents in the Wiki would be useful.

[GillesDuvert]

I already mentioned something in the wiki (Known Issues)