Is there interest in a jbuilder port?

[rgrinberg]

Hi,

It would be great to have this library built with jbuilder (correct installation of artifacts, windows support, easily part of other jbuilder projects, no non-ocaml dependencies). If you'd still like to maintain the parallel make based build system for your own use, that's fine too. But for opam, it would be really nice if ssl built with jbuilder. If there's interest, I can create a port fairly quickly.

[rgrinberg]

cc @toots

[toots]

Hey!

Can a jbuilder port support cross-compilation with the usual toolchains a-la https://github.com/ocaml-cross/opam-cross-windows ? That would be my only concern. Maybe @smimram has others, though.

[rgrinberg]

Romain Beauxis <notifications@github.com> writes:

Can a `jbuilder` port support cross-compilation with the usual toolchains a-la https://github.com/ocaml-cross/opam-cross-windows ? That would be my only concern. Maybe @smimram has others, though.

Yeah, it supports cross compilation for all the other packages - why not this one? Btw, I never rushed to port this package before I realized that I would need to use the configurator utility which would introduce extra deps. I knew that's not acceptable, but now configurator will be available in dune (since the next version) so this is no longer a problem.

[toots]

I've never used jbuilder to cross-compile packages with C dependencies. Things can get a lil tricky in this area, with pkg-config, linker configuration, etc. But otherwise, I like y'all tools so if it works I'm all for using them.

[rgrinberg]

Alright, regressions regarding cross compilation are certainly unacceptable to us. I admit, we haven't tested cross compilation of c bindings very thoroughly. But we are keen to make it work and definitely made configurator aware of the problem. Okay, I'll prepare an initial port and we can see how cross compilation works. How does the current build system handle these cross compilation issues?

…

On Sun, Apr 22, 2018, 9:46 PM Romain Beauxis ***@***.***> wrote: I've never used jbuilder to cross-compile packages with C dependencies. Things can get a lil tricky in this area, with pkg-config, linker configuration, etc. But otherwise, I like y'all tools so if it works I'm all for using them. — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#35 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAIe-x0HBFzePekxK08LQb2Kuj8tdztfks5trJfhgaJpZM4R4L1f> .

[toots]

We're using the standard autoconf paradigm, so when passing --host=<triplet> option, all compilation binaries are prefixed with <triplet>, i.e. gcc becomes <triplet>-gcc, pkg-config becomes <triplet>-pkg-config etc. In most situations this is enough to make it work. Also, autoconf knows to disable checks such as running cross-compiled binaries and such.

[rgrinberg]

Romain Beauxis <notifications@github.com> writes:

We're using the standard `autoconf` paradigm, so when passing `--host=<triplet>` option, all compilation binaries are prefixed with `<triplet>`, i.e. `gcc` becomes `<triplet>-gcc`, `pkg-config` becomes `<triplet>-pkg-config` etc. In most situations this is enough to make it work. Also, `autoconf` knows to disable checks such as running cross-compiled binaries and such.

I see. In jbuilder, the approach is a bit different. The idea is that the target ocamlopt would have the correct CC set (via ocamlc -config). This convention is also assumed by configurator. As for pkg-config, we definitely do not handle selecting the pkg-config binary this way. Though it seems like something that should be fairly easy to add. I'll create a dune issue for this pkg-config thing.

[toots]

Sounds good. Using ocamlc configuration might be tricky for the linker, however. On windows, in particular, the linker for OCaml code is flexlink and will prolly not work as expected for straight-up C targets.

[rgrinberg]

Thanks @toots. Made a ticket to address these issues: ocaml/dune#715

[rgrinberg]

@toots there's now support for arbirary env variables for cross compilation. The only thing that is left to replace somehow is:

if [ -d "/usr/local/opt/openssl/" ]; then
  # Homebrew
  ./configure --prefix "${PREFIX}" CPPFLAGS="-I/usr/local/opt/openssl/include" LDFLAGS="-L/usr/local/opt/openssl/lib"
else
  # MacPorts
  ./configure --prefix "${PREFIX}" CPPFLAGS="-I/opt/local/include" LDFLAGS="-L/opt/local/lib"
fi

It seems like this bit is also better handled via pkg-config though. I know homebrew's openssl includes a .pc file that we can use to adjust the pkg-config search path. Do you know anything about macports perhaps?

Also, we might need some sort of mechanism to disable this kind of auto discovery in some situations (they'd like to build against some local openssl fork for example). Perhaps if the user sets PKG_CONFIG_PATH manually then we don't overwrite?

[rgrinberg]

It looks like there's some special support for no threads in ssl. In jbuilder, no threads isn't supported and the mt predicate is always picked I believe. I also think that vmthreads are being completely deprecated anyway. Do you think it would be ok to drop support for this in openssl as well?

[toots]

It's great that homebrew ships with pkg-config files. However, it's also important to be able to deal with cases without pkg-config as they also will arise. Guess openssl was a good use-case..

In the case above, I believe setting LDFLAGS and CFLAGS to both directories should be functionally equivalent though not as clean.

A more clean situation would perhaps consist in having a homebrew and a macport toolchain..? Can a default toolchain also be aliased to a named toolchain, for instance default OSX toolchain also referred to as homebrew? In this case the environment variable mechanism already in place would work just fine:

CPPFLAGS(homebrew,env)="-I/usr/local/opt/openssl/include"
CPPFLAGS(macport,env)="-I/opt/local/include"

As for threads, I'll let @smimram chime in but I think your assessment is indeed correct.

[rgrinberg]

It's great that homebrew ships with pkg-config files. However, it's also important to be able to deal with cases without pkg-config as they also will arise. Guess openssl was a good use-case..

The current way to deal with this case is to simply default to searching some hard coded paths. That should be sufficient and quite similar to how things are done today.

A more clean situation would perhaps consist in having a homebrew and a macport toolchain..? Can a default toolchain also be aliased to a named toolchain, for instance default OSX toolchain also referred to as homebrew? In this case the environment variable mechanism already in place would work just fine:

Yeah, that's possible. But it seems like this is only useful if you want to cross compile to OSX. What we'd like is to make the $ opam install ssl use case work for the average user with minimal hassle. Unfortunately, I think that does require making some educated guesses about the user's system.

[toots]

Yeah, unfortunately we're stuck with the intrinsic complications of the platform. It seems to me that the toolchain abstraction would also apply nicely to this kind of situation. However, I'm not sure if there's a way to set this up seamlessly for the user.

[rgrinberg]

Toolchain seem quite useful here because they're currently the only way we have to do switch specific configuration of builds. We could just as easily make these options as opam switch variables as well btw. I think the main advantage of piggy backing on findlib toolchains is the ability easily port packages that just use the ocamlfind binary. Since ocamlfind isn't really aware of CPPFLAGS, this method doesn't seem advantageous over the other ones.

I do agree that it would be quite nice to bulk configuration of builds. Doing this with all opam packages seems like a lost cause - the limitations are basically what you can do through findlib predicates and toolchains. But if tackle the much less difficult problem of configuring just dune packages (via opam or the straight from the source) in a global way, we gain flexibility to do quite a lot more.

[rgrinberg]

@toots I believe this is now done. If you have some time, please make a release :)

[toots]

Yes! I've been super busy with the liquidsoap release but I'll queue that ASAP. Thks for the hard work!

[toots]

Done!