Adding pkgsStatic: a fully static overlay #48803

matthewbauer · 2018-10-21T22:42:59Z

Motivation for this change

This uses the cross compilation infrastructure to build a statically linked Musl-based package set. Some basic things seem to build okay... Working on getting things working as much as possible.

You can access them from the pkgsStatic attribute set. Basic things like pkgsStatic.hello seem to work okay.

As part of the implementation this adds a new type of overlay called "crossOverlays". These are similar to ordinary overlays except for one critical difference. They only apply to the /target/ package set. To accomplish this, we use the cross stdenv to clearly separate out build, host, and target.

In addition, I have added a custom "static" overlay that will enable static by modifying the stdenv and modifying other packages as necessary. Additional packages can be enabled later on here.

/cc @nh2 @dtzWill

dtzWill · 2018-10-21T23:08:34Z

On Sun, 21 Oct 2018 15:43:08 -0700, Matthew Bauer ***@***.***> wrote: ###### Motivation for this change This uses the cross compilation infrastructure to build a statically linked Musl-based package set. Some basic things seem to build okay... Working on getting things working as much as possible. /cc @nh2 @dtzWill You can view, comment on, or merge this pull request online at:

Big 👍 from me! At least in terms of the high-level goal. Nothing against specifics 😁 but won't be able to look at them properly for at least a few weeks :( so can't necessarily vouch for them either hehe. Apologies, will look through this eagerly as soon as I'm able! :<3:

#48803 -- Commit Summary -- * Add crossOverlays * Cleanup stdenv adapters * Cleanup static overlay * Use musl by default for pkgsStatic -- File Changes -- M pkgs/stdenv/adapters.nix (10) M pkgs/stdenv/cross/default.nix (17) M pkgs/stdenv/darwin/default.nix (4) M pkgs/stdenv/default.nix (4) M pkgs/stdenv/linux/default.nix (2) M pkgs/top-level/default.nix (4) M pkgs/top-level/stage.nix (18) A pkgs/top-level/static.nix (50) -- Patch Links -- https://github.com/NixOS/nixpkgs/pull/48803.patch https://github.com/NixOS/nixpkgs/pull/48803.diff -- You are receiving this because you were mentioned. Reply to this email directly or view it on GitHub: #48803

Non-text part: text/html

GrahamcOfBorg · 2018-10-21T23:53:51Z

Success on x86_64-linux (full log)

Attempted: ncurses, zlib

Partial log (click to expand)

/nix/store/iirmif7qgp7pgbv80z5x1sj3hbay893d-ncurses-6.1
/nix/store/di0d17idlw6fxf3qwpfyxciw94ywi664-zlib-1.2.11

GrahamcOfBorg · 2018-10-21T23:53:58Z

Success on x86_64-darwin (full log)

Attempted: ncurses, zlib

Partial log (click to expand)

/nix/store/snmcl9nfjj49h2j0s1qpvmsajhm91141-ncurses-6.1
/nix/store/v8w9k5zzfidw4ik5rjxbpf9cfvbrj9s3-zlib-1.2.11

GrahamcOfBorg · 2018-10-21T23:56:11Z

Success on aarch64-linux (full log)

Attempted: ncurses, zlib

Partial log (click to expand)

/nix/store/wydzkhfm25cgbr1nfq28nqvgk0mk7518-ncurses-6.1
/nix/store/x0aj2fsd3hskn1fckk3w4b8bl6mx8m1s-zlib-1.2.11

nh2 · 2018-10-25T13:13:58Z

@matthewbauer @dtzWill Are any of you at NixCon by any chance?

dtzWill · 2018-10-25T14:11:25Z

No :(. Watching the stream, though!

…

On Thu, 25 Oct 2018 13:14:05 +0000 (UTC), Niklas Hambüchen ***@***.***> wrote: @matthewbauer @dtzWill Are any of you at NixCon by any chance? -- You are receiving this because you were mentioned. Reply to this email directly or view it on GitHub: #48803 (comment) part: text/html

matthewbauer · 2018-10-25T16:05:27Z

Nope but @Ericson2314 should be

Ericson2314 · 2018-10-26T16:11:43Z

Actually I won't be :(. I was too slow, and no tdpace left.

Ericson2314 · 2018-10-26T16:12:37Z

Maybe next year I'll finally learn not to procrastinate.

Ericson2314 · 2018-10-26T16:13:16Z

pkgs/development/libraries/ncurses/default.nix

@@ -28,7 +29,7 @@ stdenv.mkDerivation rec {
  setOutputFlags = false; # some aren't supported

  configureFlags = [
-    "--with-shared"
+    (if enableShared then "--with-shared" else "--without-shared")


Can use lib.withFeature for this

GrahamcOfBorg · 2018-10-29T19:42:17Z

Success on aarch64-linux (full log)

Attempted: ncurses, zlib

Partial log (click to expand)

/nix/store/wydzkhfm25cgbr1nfq28nqvgk0mk7518-ncurses-6.1
/nix/store/x0aj2fsd3hskn1fckk3w4b8bl6mx8m1s-zlib-1.2.11

GrahamcOfBorg · 2018-10-29T19:42:25Z

Success on x86_64-darwin (full log)

Attempted: ncurses, zlib

Partial log (click to expand)

/nix/store/snmcl9nfjj49h2j0s1qpvmsajhm91141-ncurses-6.1
/nix/store/v8w9k5zzfidw4ik5rjxbpf9cfvbrj9s3-zlib-1.2.11

matthewbauer · 2018-10-29T19:42:54Z

@GrahamcOfBorg build pkgsStatic.hello

GrahamcOfBorg · 2018-10-29T19:43:02Z

Success on x86_64-linux (full log)

Attempted: ncurses, zlib

Partial log (click to expand)

/nix/store/iirmif7qgp7pgbv80z5x1sj3hbay893d-ncurses-6.1
/nix/store/di0d17idlw6fxf3qwpfyxciw94ywi664-zlib-1.2.11

GrahamcOfBorg · 2018-10-29T19:43:48Z

Success on aarch64-linux (full log)

Attempted: pkgsStatic.hello

Partial log (click to expand)

post-installation fixup
shrinking RPATHs of ELF executables and libraries in /nix/store/2zxr3jgljngkcmv3dkxcja3698pjrx1s-hello-2.10-aarch64-unknown-linux-musl
shrinking /nix/store/2zxr3jgljngkcmv3dkxcja3698pjrx1s-hello-2.10-aarch64-unknown-linux-musl/bin/hello
cannot find section .dynamic
gzipping man pages under /nix/store/2zxr3jgljngkcmv3dkxcja3698pjrx1s-hello-2.10-aarch64-unknown-linux-musl/share/man/
aarch64-unknown-linux-musl-strip is /nix/store/bqvzvb5b9c7rl4k5zara50gwnm5mmasr-aarch64-unknown-linux-musl-binutils-2.30/bin/aarch64-unknown-linux-musl-strip
stripping (with command aarch64-unknown-linux-musl-strip and flags -S) in /nix/store/2zxr3jgljngkcmv3dkxcja3698pjrx1s-hello-2.10-aarch64-unknown-linux-musl/lib  /nix/store/2zxr3jgljngkcmv3dkxcja3698pjrx1s-hello-2.10-aarch64-unknown-linux-musl/bin
patching script interpreter paths in /nix/store/2zxr3jgljngkcmv3dkxcja3698pjrx1s-hello-2.10-aarch64-unknown-linux-musl
checking for references to /build in /nix/store/2zxr3jgljngkcmv3dkxcja3698pjrx1s-hello-2.10-aarch64-unknown-linux-musl...
cannot find section .dynamic

GrahamcOfBorg · 2018-10-29T19:44:04Z

Success on x86_64-darwin (full log)

Attempted: pkgsStatic.hello

Partial log (click to expand)

 /nix/store/acnfbicd84bnya2l2dq868b5l482qihw-coreutils-8.30/bin/mkdir -p '/nix/store/fvdzx3bmx2mxcavg4m6lja21h4r6bv09-hello-2.10/share/man/man1'
 /nix/store/acnfbicd84bnya2l2dq868b5l482qihw-coreutils-8.30/bin/install -c -m 644 hello.1 '/nix/store/fvdzx3bmx2mxcavg4m6lja21h4r6bv09-hello-2.10/share/man/man1'
make[3]: Leaving directory '/private/tmp/nix-build-hello-2.10.drv-0/hello-2.10'
make[2]: Leaving directory '/private/tmp/nix-build-hello-2.10.drv-0/hello-2.10'
make[1]: Leaving directory '/private/tmp/nix-build-hello-2.10.drv-0/hello-2.10'
post-installation fixup
gzipping man pages under /nix/store/fvdzx3bmx2mxcavg4m6lja21h4r6bv09-hello-2.10/share/man/
strip is /nix/store/g5r4apl0za012ffs6ladinwa5w0m1l3k-cctools-binutils-darwin/bin/strip
stripping (with command strip and flags -S) in /nix/store/fvdzx3bmx2mxcavg4m6lja21h4r6bv09-hello-2.10/bin
patching script interpreter paths in /nix/store/fvdzx3bmx2mxcavg4m6lja21h4r6bv09-hello-2.10

GrahamcOfBorg · 2018-10-29T19:44:14Z

Success on x86_64-linux (full log)

Attempted: pkgsStatic.hello

Partial log (click to expand)

post-installation fixup
shrinking RPATHs of ELF executables and libraries in /nix/store/7fmc0g5wkc3z0rxrr36fag5hgg5bmry5-hello-2.10-x86_64-unknown-linux-musl
shrinking /nix/store/7fmc0g5wkc3z0rxrr36fag5hgg5bmry5-hello-2.10-x86_64-unknown-linux-musl/bin/hello
cannot find section .dynamic
gzipping man pages under /nix/store/7fmc0g5wkc3z0rxrr36fag5hgg5bmry5-hello-2.10-x86_64-unknown-linux-musl/share/man/
x86_64-unknown-linux-musl-strip is /nix/store/ygcd42hl4maam2h96svkay90p20070ij-x86_64-unknown-linux-musl-binutils-2.30/bin/x86_64-unknown-linux-musl-strip
stripping (with command x86_64-unknown-linux-musl-strip and flags -S) in /nix/store/7fmc0g5wkc3z0rxrr36fag5hgg5bmry5-hello-2.10-x86_64-unknown-linux-musl/lib  /nix/store/7fmc0g5wkc3z0rxrr36fag5hgg5bmry5-hello-2.10-x86_64-unknown-linux-musl/bin
patching script interpreter paths in /nix/store/7fmc0g5wkc3z0rxrr36fag5hgg5bmry5-hello-2.10-x86_64-unknown-linux-musl
checking for references to /build in /nix/store/7fmc0g5wkc3z0rxrr36fag5hgg5bmry5-hello-2.10-x86_64-unknown-linux-musl...
cannot find section .dynamic

GrahamcOfBorg · 2018-10-29T19:52:53Z

Success on x86_64-linux (full log)

Attempted: ncurses, zlib

Partial log (click to expand)

/nix/store/iirmif7qgp7pgbv80z5x1sj3hbay893d-ncurses-6.1
/nix/store/di0d17idlw6fxf3qwpfyxciw94ywi664-zlib-1.2.11

GrahamcOfBorg · 2018-10-29T19:52:54Z

Success on x86_64-darwin (full log)

Attempted: ncurses, zlib

Partial log (click to expand)

/nix/store/snmcl9nfjj49h2j0s1qpvmsajhm91141-ncurses-6.1
/nix/store/v8w9k5zzfidw4ik5rjxbpf9cfvbrj9s3-zlib-1.2.11

GrahamcOfBorg · 2018-10-29T19:52:54Z

Success on aarch64-linux (full log)

Attempted: ncurses, zlib

Partial log (click to expand)

/nix/store/wydzkhfm25cgbr1nfq28nqvgk0mk7518-ncurses-6.1
/nix/store/x0aj2fsd3hskn1fckk3w4b8bl6mx8m1s-zlib-1.2.11

These aren’t needed with dontDisableStatic

crossOverlays only apply to the packages being built, not the build packages. It is useful when you don’t care what is used to build your packages, just what is being built. The idea relies heavily on the cross compiling infrastructure. Using this implies that we need to create a cross stdenv.

Nixpkgs is meant to link everything dynamically. We don’t want to expose static packages at the top level. If some package needs statically built binaries, it should use a custom override.

- makeStaticBinaries don’t work on Darwin (no stable ABI!) - Need to make sure NIX_CFLAGS_LINK appends - isStatic is not used anymore

The conditional was incorrect - postInstall script should only hsppen when enableShared = true.

This is kind of a mess, but basically: - static=true, shared=true means to build statically but move it to the static output - static=true, shared=false means to build statically and leave it in the main output - static=false, shared=true means to not build static at all Confusingly, the old default was static=true, shared=true even though static=false? Still can’t figure out what was meant by that.

when enableShared = false, we set --without-shared flag.

Adds the static overlay that can be used to build Nixpkgs statically. Can be used like: nix build pkgsStatic.hello Not all packages build, as some rely on dynamic linking.

roberth · 2018-12-05T10:50:53Z

pkgs/top-level/default.nix


 , # Allow a configuration attribute set to be passed in as an argument.
  config ? {}

 , # List of overlays layers used to extend Nixpkgs.
  overlays ? []

+, # List of overlays to apply to target packages only.
+  crossOverlays ? []


This looks like a great improvement. As @Ericson2314 mentioned, this brings us much closer to build and host terminology. So what about changing this to hostOverlays, because then it can be applied to non-cross as well.

With non cross, the build and host packages are the same, so it's not possible to just override in one case. crossOverrides actually will turn on cross if no crossSystem (to be renamed :)) was given to make things work.

One reason that crossOverlays makes sense is that you will be using the "cross" stdenv here. This could be a little surprising, for instance, if you use crossOverlays on macOS, you will get CC=gcc instead of CC=clang.

Yeah we should add LLVM-based cross. That is the easier part of iOS host-is-darwin cross, which is context in which I last worked on that. (i.e. I should have been worked on the lower hanging fruit but didn't.)

@matthewbauer so that's the host or run CC, right? So why not hostOverlays or runOverlays? I'd say "cross" is a property of the entire package set: was it cross-compiled? yes or no, boolean. When describing which architecture some variable applies to, it should be build or host/run, because that is precise.

Now I might be entirely wrong about this, but this, crossOverlays, seems very similar to the way in which I accidentally wrote pkgs.extend to be a bit broken.

I'm trying to do something at the edge of my understanding of Nixpkgs internals here on a limited time budget. My initial goal was to achieve greater consistency, mainly between the Nixpkgs function and NixOS, so that it will be as simple as possible to understand the interface and work with it. I hope that a new-ish perspective is useful; if not, I should focus on other things.

@Ericson2314 I seem to have lost the original conversation - but I think there is a way to disable the Clang dependency in cctools. You just have to define DISABLE_CLANG_AS here:

https://github.com/tpoechtrager/cctools-port/blob/f53c6186393915ba9bff287f18769f9e14ec02e6/cctools/as/driver.c#L285

The builtin assembler should work fine then.

@robeth That looks like okay logic. We definitely want to unify the build, host, target & local, cross, eventually. But I still am thinking crossOverlays is good here because it mirrors crossSystem. Right now, this is how I see it-

{build,host,target}Platform - used internally in Nixpkgs as the "immediate" values of build, host, target. These are given to you, not set directly. Sliding windows algorithm handles the details...

{local,cross}System - external arguments of Nixpkgs describing where you are and where you want to go. This is what you want to set to get from point a to point b.

If cross-compiling is a graph, then each node is a set of packages. We activate a kind of greedy pathfinding algorithm in moving from localSystem to crossSystem. Outside of Nixpkgs, we shouldn't care about the exact path that is taken, just that we end up with a package built by "localSystem", for "crossSystem". In the same way, when you set "crossOverlays", you are saying to give me a package set that is statically built, but using my localSystem as is.

dylex · 2019-01-08T02:44:55Z

Since this change, I get an error about passing an unknown crossOverlays argument to stdenv when using a custom stdenv. 2ee9522a6c76e28949b6237928d2cc74fb9edfc2 seems to fix it fine as far as I can tell, similar to what was done to the other stdenvs. Does this seem right?

matthewbauer · 2019-01-08T02:56:15Z

Thanks! I forgot to do some of the extra stdenv's. We could also just use ... because we aren't using crossOverlays. Merged in 04beae0.

danbst · 2019-01-21T07:24:36Z

pkgs/stdenv/adapters.nix

+      if stdenv.hostPlatform.isDarwin
+      then throw "Cannot build fully static binaries on Darwin/macOS"
+      else stdenv.mkDerivation (args // {
+        NIX_CFLAGS_LINK = toString (args.NIX_CFLAGS_LINK or "") + "-static";


@matthewbauer "-static" -> " -static", right?

Yeah probably. The original version left out the space so I just left it as is

jb55 · 2019-02-18T19:33:44Z

I've been playing with pkgsStatic today, I tried to build something which eventually required `meson` and then I hit this: $ nix build nixpkgs.pkgsStatic.surf --show-trace error: while evaluating the attribute 'buildInputs' of the derivation 'surf-2.0-x86_64-unknown-linux-musl' at /home/jb55/nixpkgs/pkgs/stdenv/generic/make-derivation.nix:187:11: while evaluating the attribute 'buildInputs' of the derivation 'webkitgtk-2.22.6-x86_64-unknown-linux-musl' at /home/jb55/nixpkgs/pkgs/stdenv/generic/make-derivation.nix:187:11: while evaluating the attribute 'nativeBuildInputs' of the derivation 'gst-plugins-base-1.14.4-x86_64-unknown-linux-musl' at /home/jb55/nixpkgs/pkgs/stdenv/generic/make-derivation.nix:187:11: while evaluating the attribute 'out.outPath' at /home/jb55/nixpkgs/lib/customisation.nix:142:13: while evaluating the attribute 'crossFile' of the derivation 'meson-0.49.0-x86_64-unknown-linux-musl' at /home/jb55/nixpkgs/pkgs/stdenv/generic/make-derivation.nix:187:11: while evaluating the attribute 'text' of the derivation 'cross-file.conf' at /home/jb55/nixpkgs/pkgs/build-support/trivial-builders.nix:7:14: while evaluating the attribute 'stdenv.targetPlatform.parsed.kernel.name' at /home/jb55/nixpkgs/pkgs/stdenv/booter.nix:111:5: attribute 'targetPlatform' missing, at /home/jb55/nixpkgs/pkgs/development/tools/build-managers/meson/default.nix:70:17 it seems to be complaining about this section in meson's derivation: [host_machine] system = '${targetPackages.stdenv.targetPlatform.parsed.kernel.name}' cpu_family = '${targetPackages.stdenv.targetPlatform.parsed.cpu.family}' cpu = '${targetPackages.stdenv.targetPlatform.parsed.cpu.name}' endian = ${if targetPackages.stdenv.targetPlatform.isLittleEndian then "'little'" else "'big'"} For some reason stdenv.targetPlatform is missing. Any ideas? I'm not too familiar with the cross compilation infrastructure to fix this. Cheers, Will

matthewbauer · 2019-02-20T03:15:54Z

Yeah meson looks to be broken right now.

matthewbauer requested review from Ericson2314 and nbp as code owners October 21, 2018 22:43

GrahamcOfBorg added 6.topic: stdenv Standard environment 10.rebuild-darwin: 0 10.rebuild-linux: 0 labels Oct 21, 2018

Ericson2314 reviewed Oct 26, 2018

View reviewed changes

GrahamcOfBorg added the 2.status: merge conflict label Oct 26, 2018

matthewbauer force-pushed the crossOverlays branch from 5befec5 to f348ed7 Compare October 29, 2018 19:30

GrahamcOfBorg removed the 2.status: merge conflict label Oct 29, 2018

matthewbauer changed the title ~~Experiments in pkgsStatic: a fully static overlay~~ Adding pkgsStatic: a fully static overlay Oct 30, 2018

matthewbauer requested a review from dtzWill October 30, 2018 16:48

GrahamcOfBorg added the 2.status: merge conflict label Dec 5, 2018

matthewbauer force-pushed the crossOverlays branch 3 times, most recently from a0e0d77 to 51ddaf2 Compare December 5, 2018 02:08

GrahamcOfBorg removed the 2.status: merge conflict label Dec 5, 2018

matthewbauer force-pushed the crossOverlays branch from 51ddaf2 to 5e15b96 Compare December 5, 2018 02:21

matthewbauer added 7 commits December 4, 2018 20:58

c-ares: don’t set configure flags

9d8de9f

These aren’t needed with dontDisableStatic

treewide: remove static packages

b966d3c

Nixpkgs is meant to link everything dynamically. We don’t want to expose static packages at the top level. If some package needs statically built binaries, it should use a custom override.

stdenv/adapters.nix: fixup makeStaticBinaries

8726f6a

- makeStaticBinaries don’t work on Darwin (no stable ABI!) - Need to make sure NIX_CFLAGS_LINK appends - isStatic is not used anymore

darwin/libiconv: fix static/shared logic

5e06294

The conditional was incorrect - postInstall script should only hsppen when enableShared = true.

ncurses: make shared linking optinoal

53a6789

when enableShared = false, we set --without-shared flag.

matthewbauer force-pushed the crossOverlays branch from 5e15b96 to b321ee9 Compare December 5, 2018 03:18

top-level/stage.nix: add static overlay

6d90a8b

Adds the static overlay that can be used to build Nixpkgs statically. Can be used like: nix build pkgsStatic.hello Not all packages build, as some rely on dynamic linking.

matthewbauer force-pushed the crossOverlays branch from b321ee9 to 6d90a8b Compare December 5, 2018 03:56

matthewbauer merged commit ceee97c into NixOS:master Dec 5, 2018

roberth reviewed Dec 5, 2018

View reviewed changes

nh2 mentioned this pull request Dec 5, 2018

Fully static Haskell executables - overview issue #43795

Open

60 tasks

Ericson2314 mentioned this pull request Dec 13, 2018

pkgsStatic for 18.09 #51957

Merged

10 tasks

nightkr mentioned this pull request Dec 26, 2018

Consider building images with statical linking Etimo/photo-garden#88

Open

andrew-d mentioned this pull request Jan 2, 2019

stdenv/native: add missing argument to fix build failure #53219

Merged

10 tasks

vaibhavsagar mentioned this pull request Jan 2, 2019

WIP DO NOT MERGE: Statically linked Linux binaries kadena-io/pact#347

Closed

danbst reviewed Jan 21, 2019

View reviewed changes

nh2 mentioned this pull request Mar 18, 2019

musl Haskell cross-build: <command line>: can't load .so/.DLL for: libgmp.so #57757

Closed

nomeata mentioned this pull request Jul 19, 2019

Publish asc.tar as build product dfinity/motoko#580

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Adding pkgsStatic: a fully static overlay #48803

Adding pkgsStatic: a fully static overlay #48803

matthewbauer commented Oct 21, 2018 •

edited

Loading

dtzWill commented Oct 21, 2018 via email

GrahamcOfBorg commented Oct 21, 2018

GrahamcOfBorg commented Oct 21, 2018

GrahamcOfBorg commented Oct 21, 2018

nh2 commented Oct 25, 2018

dtzWill commented Oct 25, 2018 via email

matthewbauer commented Oct 25, 2018

Ericson2314 commented Oct 26, 2018 •

edited

Loading

Ericson2314 commented Oct 26, 2018

Ericson2314 Oct 26, 2018

GrahamcOfBorg commented Oct 29, 2018

GrahamcOfBorg commented Oct 29, 2018

matthewbauer commented Oct 29, 2018

GrahamcOfBorg commented Oct 29, 2018

GrahamcOfBorg commented Oct 29, 2018

GrahamcOfBorg commented Oct 29, 2018

GrahamcOfBorg commented Oct 29, 2018

GrahamcOfBorg commented Oct 29, 2018

GrahamcOfBorg commented Oct 29, 2018

GrahamcOfBorg commented Oct 29, 2018

roberth Dec 5, 2018

Ericson2314 Dec 5, 2018

matthewbauer Dec 5, 2018

Ericson2314 Dec 5, 2018 •

edited

Loading

roberth Dec 5, 2018 •

edited

Loading

matthewbauer Dec 5, 2018

matthewbauer Dec 5, 2018 •

edited

Loading

dylex commented Jan 8, 2019

matthewbauer commented Jan 8, 2019

danbst Jan 21, 2019

matthewbauer Jan 22, 2019

jb55 commented Feb 18, 2019 via email

matthewbauer commented Feb 20, 2019

Adding pkgsStatic: a fully static overlay #48803

Adding pkgsStatic: a fully static overlay #48803

Conversation

matthewbauer commented Oct 21, 2018 • edited Loading

Motivation for this change

dtzWill commented Oct 21, 2018 via email

GrahamcOfBorg commented Oct 21, 2018

GrahamcOfBorg commented Oct 21, 2018

GrahamcOfBorg commented Oct 21, 2018

nh2 commented Oct 25, 2018

dtzWill commented Oct 25, 2018 via email

matthewbauer commented Oct 25, 2018

Ericson2314 commented Oct 26, 2018 • edited Loading

Ericson2314 commented Oct 26, 2018

Ericson2314 Oct 26, 2018

Choose a reason for hiding this comment

GrahamcOfBorg commented Oct 29, 2018

GrahamcOfBorg commented Oct 29, 2018

matthewbauer commented Oct 29, 2018

GrahamcOfBorg commented Oct 29, 2018

GrahamcOfBorg commented Oct 29, 2018

GrahamcOfBorg commented Oct 29, 2018

GrahamcOfBorg commented Oct 29, 2018

GrahamcOfBorg commented Oct 29, 2018

GrahamcOfBorg commented Oct 29, 2018

GrahamcOfBorg commented Oct 29, 2018

roberth Dec 5, 2018

Choose a reason for hiding this comment

Ericson2314 Dec 5, 2018

Choose a reason for hiding this comment

matthewbauer Dec 5, 2018

Choose a reason for hiding this comment

Ericson2314 Dec 5, 2018 • edited Loading

Choose a reason for hiding this comment

roberth Dec 5, 2018 • edited Loading

Choose a reason for hiding this comment

matthewbauer Dec 5, 2018

Choose a reason for hiding this comment

matthewbauer Dec 5, 2018 • edited Loading

Choose a reason for hiding this comment

dylex commented Jan 8, 2019

matthewbauer commented Jan 8, 2019

danbst Jan 21, 2019

Choose a reason for hiding this comment

matthewbauer Jan 22, 2019

Choose a reason for hiding this comment

jb55 commented Feb 18, 2019 via email

matthewbauer commented Feb 20, 2019

matthewbauer commented Oct 21, 2018 •

edited

Loading

Ericson2314 commented Oct 26, 2018 •

edited

Loading

Ericson2314 Dec 5, 2018 •

edited

Loading

roberth Dec 5, 2018 •

edited

Loading

matthewbauer Dec 5, 2018 •

edited

Loading