Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Fixes #462 - Static linking issue with multiple libraries #463

Open
wants to merge 1 commit into
base: master
Choose a base branch
from
Open

Fixes #462 - Static linking issue with multiple libraries #463

wants to merge 1 commit into from

Conversation

timlegge
Copy link

This may not be the best fix happy to make changes

@Leont
Copy link
Member

Leont commented Nov 30, 2024

I'm very confused why you would want to install multiple static libraries? And I'm a bit worried it's completely wrong (but I'm not sure it is).

@timlegge
Copy link
Author

@Leont that is completely possible. And it is possible that the static build of IO::Compress::Brotli is doing something wrong:

make veryclean; perl Makefile.PL LINKTYPE=static
rm -f \
  blib/script/bro-perl 
rm -f \
  Brotli.bso Brotli.def \
  Brotli.exp Brotli.x \
  Brotli.bs blib/arch/auto/IO/Compress/Brotli/extralibs.all \
  blib/arch/auto/IO/Compress/Brotli/extralibs.ld Makefile.aperl \
  *.a *.o \
  *perl.core Brotli.base \
  Brotli.bs Brotli.bso \
  Brotli.c Brotli.def \
  Brotli.exp Brotli.o \
  Brotli_def.old MYMETA.json \
  MYMETA.yml blibdirs.ts \
  core core.*perl.*.? \
  core.[0-9] core.[0-9][0-9] \
  core.[0-9][0-9][0-9] core.[0-9][0-9][0-9][0-9] \
  core.[0-9][0-9][0-9][0-9][0-9] libBrotli.def \
  mon.out perl \
  perl perl.exe \
  perlmain.c pm_to_blib \
  pm_to_blib.ts so_locations \
  tmon.out 
rm -rf \
  blib brotli/CMakeCache.txt \
  brotli/CMakeFiles/* brotli/CTestTestfile.cmake \
  brotli/DartConfiguration.tcl brotli/Makefile \
  brotli/brotli* brotli/cmake_install.cmake \
  brotli/libbrotlicommon.a brotli/libbrotlicommon.pc \
  brotli/libbrotlidec.a brotli/libbrotlidec.pc \
  brotli/libbrotlienc.a brotli/libbrotlienc.pc 
mv Makefile Makefile.old > /dev/null 2>&1
rm -f \
  Makefile Makefile.old \
  Brotli.o 
rm -rf \
  IO-Compress-Brotli-0.018 
rm -f *~ */*~ *.orig */*.orig *.bak */*.bak *.old */*.old
WARNING: EXTRALIBS is not a known parameter.
Checking if your kit is complete...
Looks good
'EXTRALIBS' is not a known MakeMaker parameter name.
Generating a Unix-style Makefile
Writing Makefile for IO::Compress::Brotli
Writing MYMETA.yml and MYMETA.json
tim@vivobook:~/code/perl-IO-Compress-Brotli$ make test_static
"/home/tim/perl5/perlbrew/perls/perl-5.38.2-nothread/bin/perl" "/home/tim/perl5/perlbrew/perls/perl-5.38.2-nothread/lib/5.38.2/ExtUtils/xsubpp"  -typemap '/home/tim/perl5/perlbrew/perls/perl-5.38.2-nothread/lib/5.38.2/ExtUtils/typemap' -typemap '/home/tim/code/perl-IO-Compress-Brotli/typemap'  Brotli.xs > Brotli.xsc
mv Brotli.xsc Brotli.c
cc -c  -Ibrotli/c/include -fwrapv -fno-strict-aliasing -pipe -fstack-protector-strong -I/usr/local/include -D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64 -O2   -DVERSION=\"0.018\" -DXS_VERSION=\"0.018\" -fPIC "-I/home/tim/perl5/perlbrew/perls/perl-5.38.2-nothread/lib/5.38.2/x86_64-linux/CORE"   Brotli.c
In file included from Brotli.xs:6:
ppport.h:4385: warning: "WIDEST_UTYPE" redefined
 4385 | # define WIDEST_UTYPE U64TYPE
      | 
In file included from /home/tim/perl5/perlbrew/perls/perl-5.38.2-nothread/lib/5.38.2/x86_64-linux/CORE/perl.h:3327,
                 from Brotli.xs:3:
/home/tim/perl5/perlbrew/perls/perl-5.38.2-nothread/lib/5.38.2/x86_64-linux/CORE/handy.h:1422: note: this is the location of the previous definition
 1422 | #define WIDEST_UTYPE PERL_UINTMAX_T
      | 
cd brotli && "/home/tim/perl5/perlbrew/perls/perl-5.38.2-nothread/lib/site_perl/5.38.2/x86_64-linux/auto/share/dist/Alien-cmake3/bin/cmake" -DCMAKE_MAKE_PROGRAM=make -DBUILD_SHARED_LIBS=OFF -DCMAKE_POSITION_INDEPENDENT_CODE=ON -DCMAKE_BUILD_TYPE=Release -DCMAKE_INSTALL_PREFIX=./installed . && make
-- The C compiler identification is GNU 13.2.0
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Check for working C compiler: /usr/bin/cc - skipped
-- Detecting C compile features
-- Detecting C compile features - done
-- Build type is 'Release'
-- Performing Test BROTLI_EMSCRIPTEN
-- Performing Test BROTLI_EMSCRIPTEN - Failed
-- Compiler is not EMSCRIPTEN
-- Looking for log2
-- Looking for log2 - not found
-- Looking for log2
-- Looking for log2 - found
-- Configuring done (0.3s)
-- Generating done (0.0s)
-- Build files have been written to: /home/tim/code/perl-IO-Compress-Brotli/brotli
make[1]: Entering directory '/home/tim/code/perl-IO-Compress-Brotli/brotli'
make[2]: Entering directory '/home/tim/code/perl-IO-Compress-Brotli/brotli'
make[3]: Entering directory '/home/tim/code/perl-IO-Compress-Brotli/brotli'
make[3]: Leaving directory '/home/tim/code/perl-IO-Compress-Brotli/brotli'
make[3]: Entering directory '/home/tim/code/perl-IO-Compress-Brotli/brotli'
[  2%] Building C object CMakeFiles/brotlicommon.dir/c/common/constants.c.o
[  5%] Building C object CMakeFiles/brotlicommon.dir/c/common/context.c.o
[  8%] Building C object CMakeFiles/brotlicommon.dir/c/common/dictionary.c.o
[ 11%] Building C object CMakeFiles/brotlicommon.dir/c/common/platform.c.o
[ 13%] Building C object CMakeFiles/brotlicommon.dir/c/common/shared_dictionary.c.o
[ 16%] Building C object CMakeFiles/brotlicommon.dir/c/common/transform.c.o
[ 19%] Linking C static library libbrotlicommon.a
make[3]: Leaving directory '/home/tim/code/perl-IO-Compress-Brotli/brotli'
[ 19%] Built target brotlicommon
make[3]: Entering directory '/home/tim/code/perl-IO-Compress-Brotli/brotli'
make[3]: Leaving directory '/home/tim/code/perl-IO-Compress-Brotli/brotli'
make[3]: Entering directory '/home/tim/code/perl-IO-Compress-Brotli/brotli'
[ 22%] Building C object CMakeFiles/brotlidec.dir/c/dec/bit_reader.c.o
[ 25%] Building C object CMakeFiles/brotlidec.dir/c/dec/decode.c.o
[ 27%] Building C object CMakeFiles/brotlidec.dir/c/dec/huffman.c.o
[ 30%] Building C object CMakeFiles/brotlidec.dir/c/dec/state.c.o
[ 33%] Linking C static library libbrotlidec.a
make[3]: Leaving directory '/home/tim/code/perl-IO-Compress-Brotli/brotli'
[ 33%] Built target brotlidec
make[3]: Entering directory '/home/tim/code/perl-IO-Compress-Brotli/brotli'
make[3]: Leaving directory '/home/tim/code/perl-IO-Compress-Brotli/brotli'
make[3]: Entering directory '/home/tim/code/perl-IO-Compress-Brotli/brotli'
[ 36%] Building C object CMakeFiles/brotlienc.dir/c/enc/backward_references.c.o
[ 38%] Building C object CMakeFiles/brotlienc.dir/c/enc/backward_references_hq.c.o
[ 41%] Building C object CMakeFiles/brotlienc.dir/c/enc/bit_cost.c.o
[ 44%] Building C object CMakeFiles/brotlienc.dir/c/enc/block_splitter.c.o
[ 47%] Building C object CMakeFiles/brotlienc.dir/c/enc/brotli_bit_stream.c.o
[ 50%] Building C object CMakeFiles/brotlienc.dir/c/enc/cluster.c.o
[ 52%] Building C object CMakeFiles/brotlienc.dir/c/enc/command.c.o
[ 55%] Building C object CMakeFiles/brotlienc.dir/c/enc/compound_dictionary.c.o
[ 58%] Building C object CMakeFiles/brotlienc.dir/c/enc/compress_fragment.c.o
[ 61%] Building C object CMakeFiles/brotlienc.dir/c/enc/compress_fragment_two_pass.c.o
[ 63%] Building C object CMakeFiles/brotlienc.dir/c/enc/dictionary_hash.c.o
[ 66%] Building C object CMakeFiles/brotlienc.dir/c/enc/encode.c.o
[ 69%] Building C object CMakeFiles/brotlienc.dir/c/enc/encoder_dict.c.o
[ 72%] Building C object CMakeFiles/brotlienc.dir/c/enc/entropy_encode.c.o
[ 75%] Building C object CMakeFiles/brotlienc.dir/c/enc/fast_log.c.o
[ 77%] Building C object CMakeFiles/brotlienc.dir/c/enc/histogram.c.o
[ 80%] Building C object CMakeFiles/brotlienc.dir/c/enc/literal_cost.c.o
[ 83%] Building C object CMakeFiles/brotlienc.dir/c/enc/memory.c.o
[ 86%] Building C object CMakeFiles/brotlienc.dir/c/enc/metablock.c.o
[ 88%] Building C object CMakeFiles/brotlienc.dir/c/enc/static_dict.c.o
[ 91%] Building C object CMakeFiles/brotlienc.dir/c/enc/utf8_util.c.o
[ 94%] Linking C static library libbrotlienc.a
make[3]: Leaving directory '/home/tim/code/perl-IO-Compress-Brotli/brotli'
[ 94%] Built target brotlienc
make[3]: Entering directory '/home/tim/code/perl-IO-Compress-Brotli/brotli'
make[3]: Leaving directory '/home/tim/code/perl-IO-Compress-Brotli/brotli'
make[3]: Entering directory '/home/tim/code/perl-IO-Compress-Brotli/brotli'
[ 97%] Building C object CMakeFiles/brotli.dir/c/tools/brotli.c.o
[100%] Linking C executable brotli
make[3]: Leaving directory '/home/tim/code/perl-IO-Compress-Brotli/brotli'
[100%] Built target brotli
make[2]: Leaving directory '/home/tim/code/perl-IO-Compress-Brotli/brotli'
make[1]: Leaving directory '/home/tim/code/perl-IO-Compress-Brotli/brotli'
rm -f "blib/arch/auto/IO/Compress/Brotli/Brotli.a"
cp brotli/libbrotlienc.a brotli/libbrotlidec.a brotli/libbrotlicommon.a blib/arch/auto/IO/Compress/Brotli
/usr/bin/ar cr "blib/arch/auto/IO/Compress/Brotli/Brotli.a" Brotli.o
: "blib/arch/auto/IO/Compress/Brotli/Brotli.a"
chmod 755 blib/arch/auto/IO/Compress/Brotli/Brotli.a
cp lib/IO/Compress/Brotli.pm blib/lib/IO/Compress/Brotli.pm
cp lib/IO/Uncompress/Brotli.pm blib/lib/IO/Uncompress/Brotli.pm
cp bin/bro-perl blib/script/bro-perl
"/home/tim/perl5/perlbrew/perls/perl-5.38.2-nothread/bin/perl" -MExtUtils::MY -e 'MY->fixin(shift)' -- blib/script/bro-perl
Writing "Makefile.aperl" for this perl
WARNING: EXTRALIBS is not a known parameter.
'EXTRALIBS' is not a known MakeMaker parameter name.
Generating a Unix-style Makefile.aperl
Writing Makefile.aperl for IO::Compress::Brotli
Writing MYMETA.yml and MYMETA.json
make -f Makefile.aperl perl
make[1]: Entering directory '/home/tim/code/perl-IO-Compress-Brotli'
Writing perlmain.c
mv perlmain.ct perlmain.c
cc -c  -Ibrotli/c/include -fwrapv -fno-strict-aliasing -pipe -fstack-protector-strong -I/usr/local/include -D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64 -O2   -DVERSION=\"0.018\" -DXS_VERSION=\"0.018\" -fPIC "-I/home/tim/perl5/perlbrew/perls/perl-5.38.2-nothread/lib/5.38.2/x86_64-linux/CORE"   perlmain.c
cc -c  -Ibrotli/c/include -fwrapv -fno-strict-aliasing -pipe -fstack-protector-strong -I/usr/local/include -D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64 -O2   -DVERSION=\"0.018\" -DXS_VERSION=\"0.018\" -fPIC "-I/home/tim/perl5/perlbrew/perls/perl-5.38.2-nothread/lib/5.38.2/x86_64-linux/CORE"   Brotli.c
In file included from Brotli.xs:6:
ppport.h:4385: warning: "WIDEST_UTYPE" redefined
 4385 | # define WIDEST_UTYPE U64TYPE
      | 
In file included from /home/tim/perl5/perlbrew/perls/perl-5.38.2-nothread/lib/5.38.2/x86_64-linux/CORE/perl.h:3327,
                 from Brotli.xs:3:
/home/tim/perl5/perlbrew/perls/perl-5.38.2-nothread/lib/5.38.2/x86_64-linux/CORE/handy.h:1422: note: this is the location of the previous definition
 1422 | #define WIDEST_UTYPE PERL_UINTMAX_T
      | 
rm -f "blib/arch/auto/IO/Compress/Brotli/Brotli.a"
cp brotli/libbrotlienc.a brotli/libbrotlidec.a brotli/libbrotlicommon.a blib/arch/auto/IO/Compress/Brotli
/usr/bin/ar cr "blib/arch/auto/IO/Compress/Brotli/Brotli.a" Brotli.o
: "blib/arch/auto/IO/Compress/Brotli/Brotli.a"
chmod 755 blib/arch/auto/IO/Compress/Brotli/Brotli.a
cat blib/arch/auto/IO/Compress/Brotli/extralibs.ld >> blib/arch/auto/IO/Compress/Brotli/extralibs.all
cc -fstack-protector-strong -L/usr/local/lib -Wl,-E Brotli.o -O2 ./perlmain.o -o perl "blib/arch/auto/IO/Compress/Brotli/Brotli.a" "/home/tim/perl5/perlbrew/perls/perl-5.38.2-nothread/lib/5.38.2/x86_64-linux/CORE/libperl.a" `cat blib/arch/auto/IO/Compress/Brotli/extralibs.all` -lpthread -ldl -lm -lcrypt -lutil -lc 
To install the new 'perl' binary, call
    make -f Makefile.aperl inst_perl MAP_TARGET=perl
    make -f Makefile.aperl map_clean
make[1]: Leaving directory '/home/tim/code/perl-IO-Compress-Brotli'
PERL_DL_NONLAZY=1 "/home/tim/code/perl-IO-Compress-Brotli/perl" "-Iblib/arch" "-Iblib/lib" "-I/home/tim/perl5/perlbrew/perls/perl-5.38.2-nothread/lib/5.38.2/x86_64-linux" "-I/home/tim/perl5/perlbrew/perls/perl-5.38.2-nothread/lib/5.38.2" "-MExtUtils::Command::MM" "-MTest::Harness" "-e" "undef *Test::Harness::Switches; test_harness(0, 'blib/lib', 'blib/arch')" t/*.t
t/00-load.t ........ ok   
t/01-uncompress.t .. ok       
t/02-roundtrip.t ... ok       
All tests successful.
Files=3, Tests=260,  4 wallclock secs ( 0.03 usr  0.00 sys +  3.83 cusr  0.19 csys =  4.05 CPU)
Result: PASS

@timlegge timlegge mentioned this pull request Nov 30, 2024
Open
@Leont
Copy link
Member

Leont commented Nov 30, 2024

MakeMaker is assuming here that MYEXTLIB is exactly one library. What it does for static linking is copy that library to the target location, and then add the XS object file to it. This way you have one new static library that includes the XS code. For dynamic linking it just happens to work by accident.

So I think that what you're doing is wrong within the currently existing paradigm. But what you want to do is not reasonable, just complicated.

For dynamic links you want it to link these static libraries into the loadable object (which either MYEXTLIB or LIBS or should do), but static links you want to copy them to the archlib (so that they are found when gathering a new static perl that included this code) AND mark them as "these should be linked in". Otherwise they will be lost again the next time someone links a perl with a new library. This will need the use of EXTRALIBS (a subset of LIBS) instead of MYEXTLIB.

Possibly an Alien:: type approach makes more sense here (but inconveniently the Alien::Brotli namespace seems to be taken already). And also Alien::Base isn't very static perl friendly.

There's a reason few people do this, it's really hard to get it right.

@Leont
Copy link
Member

Leont commented Nov 30, 2024 •
edited
Loading

Using LIBS instead of MYEXTLIB it almost works right, it builds fine but it stores the wrong path (build dir instead of install dir) so on the next rebuild it won't be able to find the brotli libs. Something Alien-like would definitely be recommended here (that way you'd have a stable library path).

LIBS             => '-Lbrotli -lbrotlienc -lbrotlidec -lbrotlicommon',

@timlegge
Copy link
Author

@Leont

https://github.com/timlegge/perl-IO-Compress-Brotli/blob/8b44c83b23bb4658179e1494af4b725a1bc476bc/Makefile.PL#L44

I use EXTRALIBS here which I did need in order to get it to work. However, if I remove MYEXTLIB the build fails as you note above I think

@timlegge
Copy link
Author

However, that does allow me to link to the system install library

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@timlegge @Leont