Add a CI job to test dub interoperability

[PetarKirov]

Currently, due to issue #1819 you can't build most dub projects without specifying --compiler=ldmd2, and even that does help in many situations. Since dub is an integral part of the ecosystem, the latest available release on various Linux distributions, 1.1.0-beta3, gives a very bad first impression to new comers.

The only way to ensure good interoperability with DUB is to make sure that no LDC pull request is accepted without verifying that it doesn't break DUB.

Probably good candidates for testing are popular/successful and actively developed/supported D projects such as vibe.d and mir.

[kinke]

and even that does help in many situations

Doesn't help. ;) What's that issue? Is it fixed in master?

[kinke]

The vibe.d issue is #1741, which is probably a duplicate of #1237 and #1022, and isn't fixed in master; just tested on Linux x64. Are there any dub-related issues with --compiler=ldmd2 we don't know about yet?

[PetarKirov]

What's that issue?

The fact that you were unaware of the problem, as is probably the rest of the LDC team makes it obvious that without integrating DUB into the testing process such issues would continue to appear and persist until a user reports them. Please don't take it personally, you and the rest of the team are doing a great job, it's just that DUB needs to get first class support, because it is a vital part of D's ecosystem.

The vibe.d issue is #1741...

No, it's a different problem, which looks highly related to, if not the same as #1819:

$ uname -a
Linux zombinedev-desktop 4.5.0-040500-generic #201603140130 SMP Mon Mar 14 05:32:22 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux

$ lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description:    Ubuntu 16.04.1 LTS
Release:    16.04
Codename:   xenial

$ dpkg -s ldc
Package: ldc
Status: install ok installed
Priority: optional
Section: devel
Installed-Size: 37678
Maintainer: Ubuntu Developers <ubuntu-devel-discuss@lists.ubuntu.com>
Architecture: amd64
Version: 1:1.1.0+b3-1build1~ubuntu16.04.1
Provides: d-compiler, d-v2-compiler
Depends: libphobos2-ldc-dev (= 1:1.1.0+b3-1build1~ubuntu16.04.1), libc6 (>= 2.15), libconfig9, libgcc1 (>= 1:3.4), libphobos2-ldc68 (>= 1:0.17.0~beta2-1), libstdc++6 (>= 5.2), libtinfo5 (>= 6), zlib1g (>= 1:1.2.0)
Conffiles:
 /etc/ldc2.conf 72aa33160083c199bfd88eddb829705c
Description: LLVM D Compiler
 LDC is a compiler for the D programming Language. It is based on the
 latest DMD frontend and uses LLVM as backend.
 .
 This is the actual ldc compiler
Original-Maintainer: Debian D Language Group <pkg-d-devel@lists.alioth.debian.org>
Homepage: https://github.com/ldc-developers/ldc

$ ldc2 --version
LDC - the LLVM D compiler (1.0.0):
  based on DMD v2.071.2 and LLVM 3.8.0
  built with LDC - the LLVM D compiler (0.17.1)
  Default target: x86_64-pc-linux-gnu
  Host CPU: ivybridge
  http://dlang.org - http://wiki.dlang.org/LDC

  Registered Targets:
    aarch64    - AArch64 (little endian)
    aarch64_be - AArch64 (big endian)
    amdgcn     - AMD GCN GPUs
    arm        - ARM
    arm64      - ARM64 (little endian)
    armeb      - ARM (big endian)
    bpf        - BPF (host endian)
    bpfeb      - BPF (big endian)
    bpfel      - BPF (little endian)
    cpp        - C++ backend
    hexagon    - Hexagon
    mips       - Mips
    mips64     - Mips64 [experimental]
    mips64el   - Mips64el [experimental]
    mipsel     - Mipsel
    msp430     - MSP430 [experimental]
    nvptx      - NVIDIA PTX 32-bit
    nvptx64    - NVIDIA PTX 64-bit
    ppc32      - PowerPC 32
    ppc64      - PowerPC 64
    ppc64le    - PowerPC 64 LE
    r600       - AMD GPUs HD2XXX-HD6XXX
    sparc      - Sparc
    sparcel    - Sparc LE
    sparcv9    - Sparc V9
    systemz    - SystemZ
    thumb      - Thumb
    thumbeb    - Thumb (big endian)
    x86        - 32-bit X86: Pentium-Pro and above
    x86-64     - 64-bit X86: EM64T and AMD64
    xcore      - XCore

$ cat /etc/ldc2.conf 
// This configuration file uses libconfig.
// See http://www.hyperrealm.com/libconfig/ for syntax details.

// The default group is required
default:
{
    // 'switches' holds array of string that are appends to the command line
    // arguments before they are parsed.
    switches = [
        "-I/usr/include/d",
        "-I/usr/lib/ldc/x86_64-linux-gnu/include/d/ldc",
        "-I/usr/lib/ldc/x86_64-linux-gnu/include/d",
        "-L-L/usr/lib", 
        "-defaultlib=phobos2-ldc,druntime-ldc",
        "-debuglib=phobos2-ldc-debug,druntime-ldc-debug"
    ];
};

LDC - the LLVM D compiler (1.0.0):
based on DMD v2.071.2 and LLVM 3.8.0

This looks like a packaging issue.

---

-n is for non-interactive dub init mode, not relevant to the problem,
just simplifies the steps to reproduce.

$ dub init -n -t vibe.d

$ ls
dub.json  public  source  views

$ dub --compiler=ldmd2
The determined compiler type "ldc" doesn't match the expected type "dmd". This will probably result in build errors.
Fetching libevent 2.0.1+2.0.16 (getting selected version)...
Placing libevent 2.0.1+2.0.16 to /home/zombinedev/.dub/packages/...
Fetching diet-ng 1.1.0 (getting selected version)...
Placing diet-ng 1.1.0 to /home/zombinedev/.dub/packages/...
Fetching openssl 1.1.5+1.0.1g (getting selected version)...
Placing openssl 1.1.5+1.0.1g to /home/zombinedev/.dub/packages/...
Fetching memutils 0.4.8 (getting selected version)...
Placing memutils 0.4.8 to /home/zombinedev/.dub/packages/...
Fetching vibe-d 0.7.30 (getting selected version)...
Placing vibe-d 0.7.30 to /home/zombinedev/.dub/packages/...
Fetching libasync 0.7.9 (getting selected version)...
Placing libasync 0.7.9 to /home/zombinedev/.dub/packages/...
Performing "debug" build using ldmd2 for x86_64.
vibe-d:utils 0.7.30: building configuration "library"...
vibe-d:data 0.7.30: building configuration "library"...
vibe-d:core 0.7.30: building configuration "libevent"...
diet-ng 1.1.0: building configuration "library"...
Error: Output file 'html.o' for module 'diet.internal.html' collides with previous module 'diet.html'. See the -oq option
ldmd2 failed with exit code 1.

ldmd2 litters the project directory with object files, which doesn't happen with dmd.

$ ls
all.o             driver.o             message.o        stdio.o         url.o
args.o            dub.json             mimetypes.o      stream.o        urltransfer.o
array.o           dub.selections.json  multicast.o      string.o        utils.o
base64.o          file.o               native.o         sync.o          validation.o
botan.o           funcattr.o           net.o            task.o          views
bson.o            hashmap.o            openssl.o        taskpipe.o      webform.o
codegen.o         json.o               operations.o     threadedfile.o  win32.o
concurrency.o     libasync.o           passwordhash.o   timerqueue.o    winrt.o
connectionpool.o  libevent2.o          path.o           tls.o           wrapper.o
core.o            libevent2_tcp.o      public           traits.o        zlib.o
counting.o        log.o                rangeutil.o      typetuple.o
cryptorand.o      markdown.o           serialization.o  uda.o
dictionarylist.o  memory.o             source           urlencode.o

The determined compiler type "ldc" doesn't match the expected type "dmd". This will probably result in build errors.

Another red flag.

# Make sure that the ldmd2 error is not just a debian packaging issue
$ rm -rf ~/.dub/
$ rm -rf .dub/
$ sudo apt remove ldc
$ sudo apt remove libphobos2-ldc*
$ wget -O ../ldc2-1.1.0-beta3-linux-x86_64.tar.xz https://github.com/ldc-developers/ldc/releases/download/v1.1.0-beta3/ldc2-1.1.0-beta3-linux-x86_64.tar.xz
$ tar -C ../ -xf ../ldc2-1.1.0-beta3-linux-x86_64.tar.xz
$ ../ldc2-1.1.0-beta3-linux-x86_64/bin/ldmd2 --version
LDC - the LLVM D compiler (1.1.0-beta3):
  based on DMD v2.071.2 and LLVM 3.9.0
  built with LDC - the LLVM D compiler (1.1.0-beta3)
  Default target: x86_64-unknown-linux-gnu
  Host CPU: ivybridge
  http://dlang.org - http://wiki.dlang.org/LDC

  Registered Targets:
    x86    - 32-bit X86: Pentium-Pro and above
    x86-64 - 64-bit X86: EM64T and AMD64

$ dub --compiler=../ldc2-1.1.0-beta3-linux-x86_64/bin/ldmd2
The determined compiler type "ldc" doesn't match the expected type "dmd". This will probably result in build errors.
Fetching libevent 2.0.1+2.0.16 (getting selected version)...
Placing libevent 2.0.1+2.0.16 to /home/zombinedev/.dub/packages/...
Fetching diet-ng 1.1.0 (getting selected version)...
Placing diet-ng 1.1.0 to /home/zombinedev/.dub/packages/...
Fetching openssl 1.1.5+1.0.1g (getting selected version)...
Placing openssl 1.1.5+1.0.1g to /home/zombinedev/.dub/packages/...
Fetching memutils 0.4.8 (getting selected version)...
Placing memutils 0.4.8 to /home/zombinedev/.dub/packages/...
Fetching vibe-d 0.7.30 (getting selected version)...
Placing vibe-d 0.7.30 to /home/zombinedev/.dub/packages/...
Fetching libasync 0.7.9 (getting selected version)...
Placing libasync 0.7.9 to /home/zombinedev/.dub/packages/...
Performing "debug" build using ../ldc2-1.1.0-beta3-linux-x86_64/bin/ldmd2 for x86_64.
vibe-d:utils 0.7.30: building configuration "library"...
vibe-d:data 0.7.30: building configuration "library"...
vibe-d:core 0.7.30: building configuration "libevent"...
diet-ng 1.1.0: building configuration "library"...
Error: Output file 'html.o' for module 'diet.internal.html' collides with previous module 'diet.html'. See the -oq option
../ldc2-1.1.0-beta3-linux-x86_64/bin/ldmd2 failed with exit code 1.

[kinke]

Please don't take it personally

Of course not, I expect our users to file issues as they pop up, that's the least they can do to bring LDC forward. I only use D when hacking on LDC, so I don't need dub.

This looks like a packaging issue.

Yep and the version has been incremented by now.

The determined compiler type "ldc" doesn't match the expected type "dmd". This will probably result in build errors.

Another red flag.

Please file a dub issue then, it doesn't have anything to do with LDC per se.

As to the actual issue you are having (colliding object files and no objects cleanup when generating a static lib) - that has been fixed in master for LDMD (for DMD compatibility), but the fix was actually only a side-product of the fix for #1819, as nobody cared to file a bug for that in all that time, and it luckily popped up during that fix. The dub guys were probably unaware of those issues too, otherwise they could have filed an LDC issue or worked around it with cmdline flags.

So while we'll eventually add some basic dub tests, LDC will keep on relying on the community to file issues.

[kinke]

I must correct myself, the LDMD dub issue wrt. colliding object files was actually a regression of beta 3, as previously -op was implied when building a lib.

[kinke]

As soon as vibe-d/vibe.d#1611 and vibe-d/vibe.d#1629 are fixed in an official vibe.d release, I think building a sample vibe.d project (dub init <dir> -t vibe.d) with dub as part of all CI jobs (possibly right after building dub itself) would definitely make sense and not cost that much time (1-3 minutes I'd guess).

[dnadlinger]

I think we would want to create a more extensive pre-release test suite with various more well-known D projects, like Martin's extended Jenkins setup.

[PetarKirov]

@kinke @klickverbot I'm very glad to hear that!!!