GLIBC Version

[chibberto]

Hi, i'm trying to run mmlspark (specifically lightgbm) but am getting the following error with glibc

/lib64/libm.so.6: version `GLIBC_2.23' not found (required by /hdfsdata/yarn/nm/usercache/ultra-job/appcache/application_1529615984505_0659/container_1529615984505_0659_01_000002/tmp/mml-natives4539490715051782423/lib_lightgbm.so)

We are currently running glibc version 2.17, and there is no appetite for our IT dept to upgrade due to support etc.
I was wondering whether :
a) I would be able to build the jars myself, and this would work with an older glibc version?
b) This could be incorporated into the build?

[imatiach-msft]

@chibberto it looks like there are a number of issues related to different environments where the native .so (shared object) files from lightgbm don't work. I haven't yet figured out the best way to fix them - maybe I could publish multiple lightgbm packages to maven and somehow conditionally build our mmlspark package against them.

[rohanmonga]

I'm having the same problem. I tried this on dataproc 1.2.40 (gcp) and EMR 5.14.0 (aws). They both use glibc 2.17. I think having a build against that would make it easier to experiment on at least emr & dataproc

[rohanmonga]

@chibberto I tried upgrading my cluster manually to GLIBC 2.23 by upgrading libc6 libc6-dev libc6-dbg libstdc++6 libstdc++-6-dev all these. I started getting SIGSEGVs in random places :(

[julienchamp]

Hi guys,

I have the same problem : on my computer everything is fine as I have glibc 2.23, however on my hadoop/spark cluster I have a glibc 2.17 because of centos 7.

Several questions :

1. Is it possible to build mmlspark with the glibc 2.17 without problems ?
2. Is there another solution to solve our problem ?

Thanks for all this work, and ( I hope ) for answers ;)
Julien

[julienchamp]

I'm currently trying to build mmlspark on a centos 7 but it does not seem easy ^^

[imatiach-msft]

@VALEK I don't think that will work - the binary in lightgbm needs to be packaged and rebuilding mmlspark won't change this, otherwise you should be able to use the non-native parts of mmlspark out of the box as they are platform agnostic (the java bits) and you don't need to rebuild it

[julienchamp]

Thanks for your answer !

You're right @imatiach-msft in fact I meant I was trying to build cntk on centos 7 so as to be able to run mmlspark, but it does not seem the right / appropriate approach

Now I'm trying something else : use a specific LD_LIBRARY_PATH for my yarn jobs with a more recent version of glibc. Do you think this may be a solution @imatiach-msft ?

[jepma]

Did you get this to work @VALEK? We are running on EMR 5.15.0 and here the glibc library is also stuck at 2.17. I did manage to get the LightGBM package to work on its own on EMR, but when running an EMR step will still use the binary shipped with the PIP plugin, unfortunately.

[imatiach-msft]

@jepma @VALEK @rohanmonga @chibberto one possibility would be to have a flag when running lightgbm to specify the version that was built with one or the other GLIBC, but then this would be difficult to maintain. Another option would be to include the newer version of GLIBC binaries with the lib_lightgbm.so and lib_lightgbm_swig.so, I'm not quite sure how to do this though, specifically, I don't know which binaries are needed.

[rohanmonga]

I was able to compile lightgbm by hand on the cluster and ship it in a jar with my spark code. It seems to be working for me.

[friyal]

@rohanmonga Could you share your solution please? I can't solve the problem in my spark cluster. Thank you very much.

[jaraujoduarte]

@friyal together with @jepma we achieved something like this by compiling lightgbmlib within an amazonlinux based docker container, then added that new jar as a dependency for mmlspark and also built that ourselves.

** We had to comment a few lines in the build script from mmlspark to package all dependencies in a single jar via the assembly plugin.

Once the above is done it's just a matter of shipping it with your Spark job code to the cluster.

[acontry]

@jaraujoduarte @jepma what changes did you make in the build script to get it to work? I've been able to get both lightgbm and mmlspark to compile in a docker container, but I'm having trouble in getting lightgbm jar included in mmlspark fat jar. Thanks!

[jaraujoduarte]

@acontry look for assemblyOption in build.scala and comment out the few lines that prevent the dependencies to be part of the final jar.

[acontry]

@jaraujoduarte Thanks that did the trick! Now I'm dealing with "connection refused" errors on my cluster...

[jaraujoduarte]

@acontry check this out: #371

[troszok]

@jaraujoduarte works also for me, thanks mate.

[guolinke]

refer to microsoft/LightGBM#1718

[land1725]

@imatiach-msft hello:
I am sorry to tell that I got the same error
/lib64/libm.so.6: version `GLIBC_2.23' not found (required by /hdfsdata/yarn/nm/usercache/ultra-job/appcache/application_1529615984505_0659/container_1529615984505_0659_01_000002/tmp/mml-natives4539490715051782423/lib_lightgbm.so)

We are currently running glibc version 2.17, and there is no appetite for our IT dept to upgrade due to support etc.

1:I rebuild the lightgbmlib.jar like below to make a jar named lightgbmlib.jar

git clone --recursive https://github.com/Microsoft/LightGBM ; cd LightGBM
mkdir build ; cd build
cmake -DUSE_SWIG=ON ..
make -j4

then I use exclusion to delete the lightgbmlib.jar in the mmlspark
and add it that I build myself to my local maven lib and add dependency

 <dependency>
        <groupId>Azure</groupId>
        <artifactId>mmlspark</artifactId>
        <version>0.14</version>
        <exclusions>
            <exclusion>  <!-- declare the exclusion here -->
                <groupId>com.microsoft.ml.lightgbm</groupId>
                <artifactId>lightgbmlib</artifactId>
            </exclusion>
        </exclusions>
    </dependency>
    <dependency>
        <groupId>com.microsoft.ml.lightgbm</groupId>
        <artifactId>lightgbmlib</artifactId>
        <version>2.1.250</version>
    </dependency>

but Now I'm dealing with "connection refused" errors on my cluster... and can not resolved
2:I want to rebuild lightgbm as a so and add to maven,but I do not konw how to add so in maven

git clone --recursive https://github.com/Microsoft/LightGBM ; cd LightGBM
mkdir build ; cd build
cmake ..
make -j4

Could you give me some advises that 1 or 2 is the right thing?

[chibberto]

@imatiach-msft My issues have been solved on rhel 7.5 by
#391
and
#399

I'm happy to close the issue if you are. Thanks for all your help - looking forward to using it.

[Luo-Jin]

was this solved in #391? i can not download the "lightgbmlib-2.2.100.jar" by using following option in spark-shell
--packages com.microsoft.ml.spark:mmlspark_2.11:0.14.dev17+1.g409034eb and --repositories https://mmlspark.azureedge.net/maven.

@chibberto would you please advise me ?

thank you.

[chibberto]

@ostrich-jin I built using code at bottom of this page :
https://lightgbm.readthedocs.io/en/latest/Installation-Guide.html

[kunguang]

@land1725 您那个问题解决了没。这个lightgbm.so 我还是找不到，按照你那个最新的pom配置

[kunguang]

@imatiach-msft ,hello, i got same error :
lib_lightgbm.so: /lib64/libstdc++ .so.6: version `GLIBCXX_3.4.20' not found.
i use config in pom.xml like:

com.microsoft.ml.lightgbm
lightgbmlib
2.2.200

it's also not work. and i find in my cluster. the biggest version is 3.4.19

[Geeshang]

with --packages com.microsoft.ml.spark:mmlspark_2.11:0.15.dev2+1.g3ac4123 --repositories https://mmlspark.azureedge.net/maven/

got lib_lightgbm.so: /lib64/libstdc++ .so.6: version GLIBCXX_3.4.20' not found.`

@imatiach-msft seems not fixed centos 7 problems.

[land1725]

@land1725 您那个问题解决了没。这个lightgbm.so 我还是找不到，按照你那个最新的pom配置

解决了

    <dependency>
        <groupId>com.microsoft.ml.spark</groupId>
        <artifactId>mmlspark_2.11</artifactId>
        <version>0.14.dev14+1.g1bb0985b</version>

[wangwei420625]

[wangwei420625]

Is there any solution?

[imatiach-msft]

@wangwei420625 this should already be fixed in the latest master

[sherrywang31]

Hi @imatiach-msft , I'm not sure if it's related, but I got /usr/lib64/libstdc++.so.6: version CXXABI_1.3.7' not found` error when running lightgbm with mmlspark 0.17 on spark cluster. Would you mind giving some suggestion?

[imatiach-msft]

@sherry0531 can you try latest master instead of v0.17? You can also try the recent but not latest versions suggested here with the fix:
#533

closing as now we use the official linux .so files produced from Microsoft/LightGBM build, which uses ubuntu 14.04 docker that does not have the glibc issue.
This was fixed with the PR:
#526
Which updates the lightgbm version to:
"com.microsoft.ml.lightgbm" % "lightgbmlib" % "2.2.350"
It should be available in next release. For now, you can use the latest builds from master, eg the build for that PR was:
--packages
com.microsoft.ml.spark:mmlspark_2.11:0.17.dev1+1.g5e0b2a0
and --repositories
https://mmlspark.azureedge.net/maven

[sherrywang31]

Thanks for the quick response! I tried that and I'm now having lib_lightgbm.so: /lib64/libc.so.6: version GLIBC_2.14' not found`...

[imatiach-msft]

@sherry0531 what OS and version are you using? The required GLIBC is quite old so I'm suprised you don't have at least that version, maybe you can update libc on your machines? eg see here:
https://askubuntu.com/questions/421642/libc-so-6-version-glibc-2-14-not-found

That means the program was compiled against glibc version 2.14, and it requires that version to run, but your system has an older version installed. You'll need to either recompile the program against the version of glibc that's on your system, or install a newer version of glibc (the "libc6" package in Debian).

[sherrywang31]

@imatiach-msft oooh yeah we have glibc 2.12. Will try updating glibc. Thanks a lot!