Skip to content

Commit

Permalink
perlPackages: Add cross-compilation support.
Browse files Browse the repository at this point in the history
This involved:

* Installing miniperl as $dev/bin/perl
* Setting miniperl to take INC from
  lib/perl5/{site_perl/,}cross_perl/${version} as well as
  lib/perl5/{site_perl/,}/${version}/${runtimeArch}, in that
  order. miniperl taking from runtimeArch is not really correct, but
  it works in some pure-perl cases (e.g. Config.pm) and can be
  overridden with the cross_perl variant.
* Installing perl-cross's stubs into
  $dev/lib/perl5/cross_perl/${version}
* Patching MakeMaker.pm to gracefully degrade (very slightly) if B.pm
  can't be loaded, which it can't in cross-compilation.
* Passing the right build-time and runtime perls to Makefile.PL
  • Loading branch information
shlevy committed Feb 28, 2018
1 parent 8e65205 commit 306d5cd
Show file tree
Hide file tree
Showing 6 changed files with 81 additions and 7 deletions.
13 changes: 13 additions & 0 deletions doc/languages-frameworks/perl.xml
Original file line number Diff line number Diff line change
Expand Up @@ -177,5 +177,18 @@ you need it.</para>

</section>

<section xml:id="ssec-perl-cross-compilation"><title>Cross-compiling modules</title>

<para>Nixpkgs has experimental support for cross-compiling Perl
modules. In many cases, it will just work out of the box, even for
modules with native extensions. Sometimes, however, the Makefile.PL
for a module may (indirectly) import a native module. In that case,
you will need to make a stub for that module that will satisfy the
Makefile.PL and install it into
<filename>lib/perl5/site_perl/cross_perl/${perl.version}</filename>.
</para>

</section>

</section>

17 changes: 17 additions & 0 deletions pkgs/development/interpreters/perl/MakeMaker-cross.patch
Original file line number Diff line number Diff line change
@@ -0,0 +1,17 @@
diff -Naur a/cpan/ExtUtils-MakeMaker/lib/ExtUtils/MakeMaker.pm b/cpan/ExtUtils-MakeMaker/lib/ExtUtils/MakeMaker.pm
--- a/cpan/ExtUtils-MakeMaker/lib/ExtUtils/MakeMaker.pm 2017-06-30 17:03:20.000000000 -0400
+++ b/cpan/ExtUtils-MakeMaker/lib/ExtUtils/MakeMaker.pm 2018-02-28 10:06:37.031237946 -0500
@@ -1267,7 +1267,12 @@
my $value = shift;
return $value if $UNDER_CORE;
my $tvalue = '';
- require B;
+ eval {
+ require B;
+ };
+ if ($@) {
+ return $tvalue;
+ }
my $sv = B::svref_2object(\$value);
my $magic = ref($sv) eq 'B::PVMG' ? $sv->MAGIC : undef;
while ( $magic ) {
39 changes: 35 additions & 4 deletions pkgs/development/interpreters/perl/default.nix
Original file line number Diff line number Diff line change
@@ -1,4 +1,6 @@
{ lib, stdenv, fetchurlBoot, buildPackages, enableThreading ? stdenv ? glibc, fetchpatch }:
{ lib, stdenv, fetchurlBoot, buildPackages
, enableThreading ? stdenv ? glibc, fetchpatch, makeWrapper
}:

with lib;

Expand Down Expand Up @@ -29,7 +31,8 @@ let
};

# TODO: Add a "dev" output containing the header files.
outputs = [ "out" "man" "devdoc" ];
outputs = [ "out" "man" "devdoc" ] ++
stdenv.lib.optional crossCompiling "dev";
setOutputFlags = false;

patches =
Expand All @@ -45,7 +48,8 @@ let
})
++ optional stdenv.isSunOS ./ld-shared.patch
++ optional stdenv.isDarwin ./cpp-precomp.patch
++ optional (stdenv.isDarwin && versionAtLeast version "5.24") ./sw_vers.patch;
++ optional (stdenv.isDarwin && versionAtLeast version "5.24") ./sw_vers.patch
++ optional crossCompiling ./MakeMaker-cross.patch;

postPatch = ''
pwd="$(type -P pwd)"
Expand Down Expand Up @@ -117,6 +121,28 @@ let
if stdenv.cc.cc or null != null then stdenv.cc.cc else "/no-such-path"
}" /no-such-path \
--replace "$man" /no-such-path
'' + stdenv.lib.optionalString crossCompiling
''
mkdir -p $dev/lib/perl5/cross_perl/${version}
for dir in cnf/{stub,cpan}; do
cp -r $dir/* $dev/lib/perl5/cross_perl/${version}
done
mkdir -p $dev/bin
install -m755 miniperl $dev/bin/perl
export runtimeArch="$(ls $out/lib/perl5/site_perl/${version})"
# wrapProgram should use a runtime-native SHELL by default, but
# it actually uses a buildtime-native one. If we ever fix that,
# we'll need to fix this to use a buildtime-native one.
#
# Adding the arch-specific directory is morally incorrect, as
# miniperl can't load the native modules there. However, it can
# (and sometimes needs to) load and run some of the pure perl
# code there, so we add it anyway. When needed, stubs can be put
# into $dev/lib/perl5/cross_perl/${version}.
wrapProgram $dev/bin/perl --prefix PERL5LIB : \
"$dev/lib/perl5/cross_perl/${version}:$out/lib/perl5/${version}:$out/lib/perl5/${version}/$runtimeArch"
''; # */

meta = {
Expand All @@ -139,7 +165,7 @@ let
sha256 = "1gh8w9m5if2s0lrx2x8f8grp74d1l6d46m8jglpjm5a1kf55j810";
};

depsBuildBuild = [ buildPackages.stdenv.cc ];
depsBuildBuild = [ buildPackages.stdenv.cc makeWrapper ];

postUnpack = ''
unpackFile ${perl-cross-src}
Expand All @@ -150,6 +176,11 @@ let
'';

configurePlatforms = [ "build" "host" "target" ];

inherit version;

# TODO merge setup hooks
setupHook = ./setup-hook-cross.sh;
});
in rec {
perl = perl524;
Expand Down
12 changes: 12 additions & 0 deletions pkgs/development/interpreters/perl/setup-hook-cross.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1,12 @@
addPerlLibPath () {
addToSearchPath PERL5LIB $1/lib/perl5/site_perl/@version@
addToSearchPath PERL5LIB $1/lib/perl5/site_perl/cross_perl/@version@
# Adding the arch-specific directory is morally incorrect, as
# miniperl can't load the native modules there. However, it can
# (and sometimes needs to) load and run some of the pure perl
# code there, so we add it anyway. When needed, stubs can be put
# into $1/lib/perl5/site_perl/cross_perl/@version@
addToSearchPath PERL5LIB $1/lib/perl5/site_perl/@version@/@runtimeArch@
}

addEnvHooks "$targetOffset" addPerlLibPath
2 changes: 1 addition & 1 deletion pkgs/development/perl-modules/generic/builder.sh
Original file line number Diff line number Diff line change
Expand Up @@ -22,7 +22,7 @@ preConfigure() {
fi
done

perl Makefile.PL PREFIX=$out INSTALLDIRS=site $makeMakerFlags
perl Makefile.PL PREFIX=$out INSTALLDIRS=site $makeMakerFlags PERL=$(type -P perl) FULLPERL=$perl/bin/perl
}


Expand Down
5 changes: 3 additions & 2 deletions pkgs/development/perl-modules/generic/default.nix
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
perl:

{ buildInputs ? [], name, ... } @ attrs:
{ nativeBuildInputs ? [], name, ... } @ attrs:

perl.stdenv.mkDerivation (
{
Expand All @@ -27,6 +27,7 @@ perl.stdenv.mkDerivation (
{
name = "perl-" + name;
builder = ./builder.sh;
buildInputs = buildInputs ++ [ perl ];
nativeBuildInputs = nativeBuildInputs ++ [ (perl.dev or perl) ];
inherit perl;
}
)

11 comments on commit 306d5cd

@vcunat
Copy link
Member

@vcunat vcunat commented on 306d5cd Mar 19, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This commit broke perlPackages.YAMLLibYAML (and maybe some others). I don't yet know why exactly. Any immediate ideas?

make[1]: Entering directory '/tmp/nix-build-perl-YAML-LibYAML-0.59.drv-0/YAML-LibYAML-0.59/LibYAML'
PERL_DL_NONLAZY=1 "/nix/store/q1l26alrqk0q23qz0mbvd27y5szvclv3-perl-5.24.3/bin/perl "-I../blib/lib" "-I../blib/arch" test.pl
/nix/store/3h5ws22zd5cszwxajms8f5hkv4v3pvzc-bash-4.4-p12/bin/bash: -c: line 0: unexpected EOF while looking for matching `"'

@vcunat
Copy link
Member

@vcunat vcunat commented on 306d5cd Mar 19, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

BTW, I got to this because it's one of channel blockers, and we haven't had nixos-unstable bump for almost two weeks now.

@shlevy
Copy link
Member Author

@shlevy shlevy commented on 306d5cd Mar 19, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ugh I've seen this before and it makes no sense to me. See 89700d7 for an example. Any ideas on the root cause would be greatly appreciated, if you're unable to dig in can you open an issue and assign me?

@orivej
Copy link
Contributor

@orivej orivej commented on 306d5cd Mar 19, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I have bisected MakeMaker to find that this behavior was introduced in Perl-Toolchain-Gang/ExtUtils-MakeMaker@5dd48b9#diff-ff7d608c3c78066201e5518f660406eaR627:
https://github.com/Perl-Toolchain-Gang/ExtUtils-MakeMaker/blob/5dd48b9ca77e3b8982205dd168b2a0e28526a901/lib/ExtUtils/MakeMaker.pm#L624-L627 (and has not changed since).
It seems that they expect that you pass the value quoted, i.e. perl Makefile.PL FULLPERL=\"$perl/bin/perl\". Nevertheless, this looks like a bug.

@shlevy
Copy link
Member Author

@shlevy shlevy commented on 306d5cd Mar 19, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ah, I see. @vcunat Can you test with the quotes passed? If not I'll try to take a look this evening. In any case we should open up a bug report.

Thanks @orivej !

@shlevy
Copy link
Member Author

@shlevy shlevy commented on 306d5cd Mar 20, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@vcunat Fixed on staging: ab42a75

@orivej
Copy link
Contributor

@orivej orivej commented on 306d5cd Mar 20, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

/cc @mohawk2 as the author of that commit who may be interested in this issue.

@mohawk2
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The expectation in EUMM is that all the PERL variables will be double-quote-wrapped, in case they have spaces in. Perhaps this should be documented. In any case, if you choose to override the value that EUMM creates (I don't know why you would but I assume a good reason), then you should pass in a double-quote-wrapped value.

@shlevy
Copy link
Member Author

@shlevy shlevy commented on 306d5cd Mar 20, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@mohawk2 Thanks! We're overriding to support cross-compilation, where perl in $PATH is a build-native perl but we want to link against a host-native perl

@mohawk2
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@shlevy The first one. It's the quotes that are double, not the wrapping :-)

@shlevy
Copy link
Member Author

@shlevy shlevy commented on 306d5cd Mar 20, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah, realized after I posted and edited my comment 😁

Please sign in to comment.