Use high-resolution timestamps #1219

QuLogic · 2016-12-31T09:28:03Z

I tried to follow #337, but it was too tangled to understand and just wrote it from scratch. Only the other-OS struct fields from there were used. #337 also tried to increase the resolution of the clocks for timing; I didn't bother with this change here.

Fixes #371.

evmar

Thanks for taking this on! I only have minor remarks.

evmar · 2016-12-31T18:08:02Z

src/deps_log.h

        : mtime(mtime), node_count(node_count), nodes(new Node*[node_count]) {}
    ~Deps() { delete [] nodes; }
-    int mtime;
+    int64_t mtime;


I guess this ought to use the TimeStamp type?

evmar · 2016-12-31T18:09:10Z

src/disk_interface.cc

@@ -60,11 +60,11 @@ int MakeDir(const string& path) {
 TimeStamp TimeStampFromFileTime(const FILETIME& filetime) {
  // FILETIME is in 100-nanosecond increments since the Windows epoch.
  // We don't much care about epoch correctness but we do want the
-  // resulting value to fit in an integer.
+  // resulting value to fit in a 64-bit integer.


https://msdn.microsoft.com/en-us/library/windows/desktop/ms724284(v=vs.85).aspx
indicates that perhaps the raw time is just a 64-bit integer, so perhaps we could just use the time directly. Another way of saying this is that we could drop lines 66 and 67 I think (and then adjust the relevant comments).

Oh this doc is better:
https://msdn.microsoft.com/en-us/library/windows/desktop/ms724290(v=vs.85).aspx

According to the first link, you shouldn't treat FILETIME as a 64-bit integer directly because of possible alignment issues.

evmar · 2016-12-31T18:10:14Z

src/timestamp.h

-// Windows we use a different value.  Both fit in an int.
-typedef int TimeStamp;
+// real time.  On POSIX we use timespec (seconds&nanoseconds since epoch)
+// and on Windows we use a different value.  Both fit in an int64.


(...with the Windows change, this would just say something like "File Time":
https://msdn.microsoft.com/en-us/library/windows/desktop/ms724290(v=vs.85).aspx
)

evmar · 2016-12-31T18:10:39Z

src/util.h

@@ -18,6 +18,7 @@
 #ifdef _WIN32
 #include "win32port.h"
 #else
+#include <inttypes.h>


What is this for?

win32port.h defines int64_t, etc., and related printf-specifiers. This header defines the print specifiers on non-Windows. However, based on the CI failing, I probably need to define something to enable those on older systems, or since this is C++ code, maybe try cinttypes.

evmar · 2016-12-31T18:12:36Z

src/deps_log.cc

-      int mtime = deps_data[1];
-      deps_data += 2;
-      int deps_count = (size / 4) - 2;
+      int64_t mtime = reinterpret_cast<int64_t*>(&deps_data[1])[0];


Are there potential alignment issues here?

Probably, I was trying to avoid thinking about endianness, but I'll have to see if I can implement this in some way that's not annoying.

QuLogic · 2017-01-01T23:52:24Z

src/deps_log.cc

+    valid_header = false;
+  } else if (fread(&version, 4, 1, f) < 1) {
+    valid_header = false;
+  } else if (version != kCurrentVersion) {


BTW, I can attempt to read version 3 if wanted; it's not a major change.

QuLogic · 2017-01-28T06:47:54Z

Ping?

maximuska · 2017-02-15T07:23:40Z

src/deps_log.cc

+    valid_header = false;
+  } else if (fread(&buf, 1, 4, f) < 4) {
+    valid_header = false;
+  }


IMHO, the original code was slightly more readable, I wouldn't change it (&see below the question about endianess handling).

maximuska · 2017-02-15T07:36:55Z

src/deps_log.cc

+      } else {
+        mtime = (TimeStamp)((uint64_t)deps_data[1] << 32 |
+                            (uint64_t)deps_data[2]);
+      }


Wouldn't casting to uint64_t* and dereference, instead of having to handle endianess be good enough? Does anyone really care about builds on platforms with misaligned access issues?

Also, an alternative could be writing mtime in two parts (low32, high32), so you could combine these in the same way w/o considering platform endianess.

The Windows link above says not to do that because of alignment problems; I think that's a large enough platform to worry about.

maximuska · 2017-02-15T07:41:10Z

+1, I've just hit timestamp issue in my environment after adding some automatic tests to our metabuild system, could be nice to put it behind w/o adding sleeps.

maximuska · 2017-02-16T07:07:40Z

@QuLogic, I believe MSDN guys had itanium (IA64) hardware in mind when writing the article.

x86/amd64 hardware handles misaligned access w/o involving OS, even performance impact vanished with the years, see e.g., here.

It would be nice to use a wrapper macro getUint64(ptr) for misaligned access, which can expand to a simple cast & dereference on x86 hardware.

fooishbar · 2017-02-16T07:33:12Z

@maximuska ARM (unsure about v8, but certainly v5-7) requires natural alignment, or an OS exception handle and fixup.

maximuska · 2017-02-16T07:45:11Z

Yeah, I know. Anyone here uses ARM build servers?

…

On Feb 16, 2017 9:33 AM, "Daniel Stone" ***@***.***> wrote: @maximuska <https://github.com/maximuska> ARM (unsure about v8, but certainly v5-7) requires natural alignment, or an OS exception handle and fixup. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#1219 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAFq-swBR53dOoLKn2_bfiSNpA15NgdAks5rc_u4gaJpZM4LYhpA> .

fooishbar · 2017-02-16T09:07:44Z

@maximuska Yes, I do, partly because for smaller projects at least, Ninja-backed build systems can make it quicker to build on-device (or on an ARM build server; there are many vendors shipping blade-style servers) than to set up cross-compiling.

Maratyszcza · 2017-04-10T03:13:45Z

@fooishbar ARM doesn't require natural alignment since ARMv6. The ARMv6 architecture introduced the first hardware support for unaligned accesses. ARM11 and Cortex-A/R processors can deal with unaligned accesses in hardware, removing the need for software routines. source

QuLogic · 2017-04-21T23:40:29Z

I rebased this and changed it up a bit to use (low, high) instead of dealing with endian issues in the old manner.

QuLogic · 2017-05-10T08:00:23Z

Turns out this delay was somewhat useful because I managed to find a bug in re-reading the saved timestamp.

Apparently, casting from a signed narrow type to an unsigned wide type does sign extension, producing a completely incorrect result out of the lower 32 bits, but it's kind of random because it depends on the 32nd bit.

nico · 2017-06-20T15:57:59Z

Thanks for the patch! Sorry about the slow turnaround time.

Generally looks great.

Did you check how this affects empty build time? The larger log size might impact incremental build speeds, it'd be good to know by how much.

nirbheek · 2017-06-20T17:58:18Z

As an aside, Meson has a PR that runs this patch on our testsuite on macOS, Windows (mingw, cygwin, native), and Linux (Ubuntu): mesonbuild/meson#1931. Also contains a test for the high-res changes. It's running a full build on all platforms right now.

QuLogic · 2017-07-01T08:02:08Z

I rebased this to fix the conflicts, and modified some bits so that it works correctly on Windows, Cygwin, and MinGW.

nirbheek · 2017-07-17T11:06:03Z

Small caveat about macOS: if you're using HFS+, these patches will do nothing because the filesystem timestamp resolution is 1s. However, I tested on AppleFS and the patches work perfectly.

@nico

Did you check how this affects empty build time? The larger log size might impact incremental build speeds, it'd be good to know by how much.

So I tested full, incremental, and empty builds on Linux (Fedora 26, 64-bit) with a rotating disk and ext4 relatime,data=ordered

I compared ninja before and after these patches on gst-build, which is a huge mostly-C project with ~3000 targets and a 7.3MB build.ninja file.

The log file size increased from 312K to 360K with these patches.
Hot cache:

Full builds, incremental builds, and empty builds took the same time for both.
Cold cache echo 2 > /proc/sys/vm/drop_caches empty builds:

I was getting strange results (it was faster with the patches than without), and after doing a bunch of testing and statistical analysis, I figured out that it was all noise. Both take the same time.

So all in all, these patches have negligible or no effect on the build times.

This prepares it for higher-resolution timestamps.

This uses nanoseconds on POSIX (±~292 years) and 100-ns increments on Windows (±~29247 years). The fallbacks to different structure fields is the only thing grabbed from ninja-build#337, with a slight modification in implementation.

Not sure why the old way works in newer compilers; maybe they just pre-define these types by defaulting to a newer standard.

This is needed on older compilers/stdlibs such as on Ubuntu Precise which is used on Travis.

Read and write the timestamp as two separate 32-bit integers in a fixed order to prevent any issues with alignment or byte order.

This uses the macros as defined by the man page, which, as noted in the comments, are defined correctly on as many libc's that I could check.

Add parentheses so that constant does not overflow; include inttypes.h when using MinGW to get the proper macros.

This prevents overflow on Windows where 'long' is not 64-bit.

QuLogic · 2017-09-16T09:35:52Z

This is rebased and I think I fixed the issue on Windows with the second-last commit. @nirbheek will try it out on the Meson test suite when he has a chance.

ClausKlein · 2017-09-19T07:42:15Z

what is about merging to next?

QuLogic · 2017-09-26T20:29:38Z

We are now using this patch for testing downstream: mesonbuild/meson#2328.

nico · 2017-10-20T17:18:09Z

src/build_log.cc

@@ -353,7 +353,7 @@ BuildLog::LogEntry* BuildLog::LookupByOutput(const string& path) {
 }

 bool BuildLog::WriteEntry(FILE* f, const LogEntry& entry) {
-  return fprintf(f, "%d\t%d\t%d\t%s\t%" PRIx64 "\n",
+  return fprintf(f, "%d\t%d\t%" PRId64 "\t%s\t%" PRIx64 "\n",


Unrelated to this change, but older ninjas will be pretty confused if they read build logs created by the new ninja. This isn't a new problem, but the if (log_version < kOldestSupportedVersion above should probably use != instead. Not in this change, mostly talking to myself.)

nico · 2017-10-20T17:18:55Z

Alright, it's early in the cycle, so let's merge this and see what (if anything) breaks :-) Do you want to send a short note to the list asking people to test this out? (If not, I can do it.)

nico · 2017-10-20T17:19:41Z

...and thanks everyone who contributed to this patch and helped with measuring performance. Much appreciated, and thanks for your patience.

Use high-resolution timestamps process thread fix? attr work thread

nirbheek · 2018-08-23T05:23:47Z

There hasn't been a release with this fix yet, and users still regularly report bugs to Meson that are already fixed by this. Most recently, this caused Valve to accidentally ship debug-only unoptimized DLLs for their Steam Play launch.

@nico do you have any plans you would like to share about a new release? :)

Thanks!

This reverts commit 52de615. Reason for revert: The Rust builder issue should be fixed by https://crrev.com/c/4154013 We found the timestamp format mismatch in .ninja_log. ninja-build/ninja#1219 This causes an issue that ninja skips some actions unexpectedly. https://crbug.com/1406628 .ninja_log will be removed by build/landmines.py to avoid this issue. Original change's description: > Revert "Update ninja version to 1.11.1" > > This reverts commit 511967a. > > Reason for revert: It breaks rust builders that generates invalid depfiles because the the new ninja version checks depfile more strictly. > https://ci.chromium.org/ui/p/chromium/builders/ci/linux-rust-x64-dbg/55612/blamelist > > Original change's description: > > Update ninja version to 1.11.1 > > > > There are no noticeable regressions. > > > > # Build performance tests on builders. > > build-perf-android: https://ci.chromium.org/swarming/task/5f0dbe428f9fe510 > > build-perf-linux: https://ci.chromium.org/swarming/task/5f0dbe28bb7a4310 > > build-perf-windows: https://ci.chromium.org/swarming/task/5f0dbd8f5b7a3310 > > > > # Full build performance tests on developer machines > > There is no notable change. See the following results for more details. > > Full builds: https://crbug.com/931218#c142 > > Noop builds: https://crbug.com/931218#c143 > > > > Bug: 931218 > > Change-Id: I8f70055713efd90569bc49f5691a672b182b8ba8 > > Reviewed-on: https://chromium-review.googlesource.com/c/chromium/src/+/4091014 > > Reviewed-by: Takuto Ikuta <tikuta@chromium.org> > > Commit-Queue: Junji Watanabe <jwata@google.com> > > Cr-Commit-Position: refs/heads/main@{#1091149} > > Bug: 931218 > Change-Id: If506d902c91c5b0342beb26f68eeed32bfbb3ef2 > No-Presubmit: true > No-Tree-Checks: true > No-Try: true > Reviewed-on: https://chromium-review.googlesource.com/c/chromium/src/+/4152932 > Reviewed-by: Takuto Ikuta <tikuta@chromium.org> > Commit-Queue: Junji Watanabe <jwata@google.com> > Cr-Commit-Position: refs/heads/main@{#1091202} Bug: 931218, 1406628 Change-Id: I174d6c958d54378ee1bc393a14c04d496390bb3b Reviewed-on: https://chromium-review.googlesource.com/c/chromium/src/+/4164097 Reviewed-by: Takuto Ikuta <tikuta@chromium.org> Commit-Queue: Junji Watanabe <jwata@google.com> Cr-Commit-Position: refs/heads/main@{#1093201}

This reverts commit 52de615745c3aba2ab7339248c749d4be96b8693. Reason for revert: The Rust builder issue should be fixed by https://crrev.com/c/4154013 We found the timestamp format mismatch in .ninja_log. ninja-build/ninja#1219 This causes an issue that ninja skips some actions unexpectedly. https://crbug.com/1406628 .ninja_log will be removed by build/landmines.py to avoid this issue. Original change's description: > Revert "Update ninja version to 1.11.1" > > This reverts commit 511967a00a7bdf882a3cd2315307c017a2d7fad0. > > Reason for revert: It breaks rust builders that generates invalid depfiles because the the new ninja version checks depfile more strictly. > https://ci.chromium.org/ui/p/chromium/builders/ci/linux-rust-x64-dbg/55612/blamelist > > Original change's description: > > Update ninja version to 1.11.1 > > > > There are no noticeable regressions. > > > > # Build performance tests on builders. > > build-perf-android: https://ci.chromium.org/swarming/task/5f0dbe428f9fe510 > > build-perf-linux: https://ci.chromium.org/swarming/task/5f0dbe28bb7a4310 > > build-perf-windows: https://ci.chromium.org/swarming/task/5f0dbd8f5b7a3310 > > > > # Full build performance tests on developer machines > > There is no notable change. See the following results for more details. > > Full builds: https://crbug.com/931218#c142 > > Noop builds: https://crbug.com/931218#c143 > > > > Bug: 931218 > > Change-Id: I8f70055713efd90569bc49f5691a672b182b8ba8 > > Reviewed-on: https://chromium-review.googlesource.com/c/chromium/src/+/4091014 > > Reviewed-by: Takuto Ikuta <tikuta@chromium.org> > > Commit-Queue: Junji Watanabe <jwata@google.com> > > Cr-Commit-Position: refs/heads/main@{#1091149} > > Bug: 931218 > Change-Id: If506d902c91c5b0342beb26f68eeed32bfbb3ef2 > No-Presubmit: true > No-Tree-Checks: true > No-Try: true > Reviewed-on: https://chromium-review.googlesource.com/c/chromium/src/+/4152932 > Reviewed-by: Takuto Ikuta <tikuta@chromium.org> > Commit-Queue: Junji Watanabe <jwata@google.com> > Cr-Commit-Position: refs/heads/main@{#1091202} Bug: 931218, 1406628 Change-Id: I174d6c958d54378ee1bc393a14c04d496390bb3b Reviewed-on: https://chromium-review.googlesource.com/c/chromium/src/+/4164097 Reviewed-by: Takuto Ikuta <tikuta@chromium.org> Commit-Queue: Junji Watanabe <jwata@google.com> Cr-Commit-Position: refs/heads/main@{#1093201} CrOS-Libchrome-Original-Commit: 448c9f1d052713d56d401b8cba90e9135a8b29f0

This reverts commit 52de615745c3aba2ab7339248c749d4be96b8693. Reason for revert: The Rust builder issue should be fixed by https://crrev.com/c/4154013 We found the timestamp format mismatch in .ninja_log. ninja-build/ninja#1219 This causes an issue that ninja skips some actions unexpectedly. https://crbug.com/1406628 .ninja_log will be removed by build/landmines.py to avoid this issue. Original change's description: > Revert "Update ninja version to 1.11.1" > > This reverts commit 511967a00a7bdf882a3cd2315307c017a2d7fad0. > > Reason for revert: It breaks rust builders that generates invalid depfiles because the the new ninja version checks depfile more strictly. > https://ci.chromium.org/ui/p/chromium/builders/ci/linux-rust-x64-dbg/55612/blamelist > > Original change's description: > > Update ninja version to 1.11.1 > > > > There are no noticeable regressions. > > > > # Build performance tests on builders. > > build-perf-android: https://ci.chromium.org/swarming/task/5f0dbe428f9fe510 > > build-perf-linux: https://ci.chromium.org/swarming/task/5f0dbe28bb7a4310 > > build-perf-windows: https://ci.chromium.org/swarming/task/5f0dbd8f5b7a3310 > > > > # Full build performance tests on developer machines > > There is no notable change. See the following results for more details. > > Full builds: https://crbug.com/931218#c142 > > Noop builds: https://crbug.com/931218#c143 > > > > Bug: 931218 > > Change-Id: I8f70055713efd90569bc49f5691a672b182b8ba8 > > Reviewed-on: https://chromium-review.googlesource.com/c/chromium/src/+/4091014 > > Reviewed-by: Takuto Ikuta <tikuta@chromium.org> > > Commit-Queue: Junji Watanabe <jwata@google.com> > > Cr-Commit-Position: refs/heads/main@{#1091149} > > Bug: 931218 > Change-Id: If506d902c91c5b0342beb26f68eeed32bfbb3ef2 > No-Presubmit: true > No-Tree-Checks: true > No-Try: true > Reviewed-on: https://chromium-review.googlesource.com/c/chromium/src/+/4152932 > Reviewed-by: Takuto Ikuta <tikuta@chromium.org> > Commit-Queue: Junji Watanabe <jwata@google.com> > Cr-Commit-Position: refs/heads/main@{#1091202} Bug: 931218, 1406628 Change-Id: I174d6c958d54378ee1bc393a14c04d496390bb3b Reviewed-on: https://chromium-review.googlesource.com/c/chromium/src/+/4164097 Reviewed-by: Takuto Ikuta <tikuta@chromium.org> Commit-Queue: Junji Watanabe <jwata@google.com> Cr-Commit-Position: refs/heads/main@{#1093201} NOKEYCHECK=True GitOrigin-RevId: 448c9f1d052713d56d401b8cba90e9135a8b29f0

QuLogic mentioned this pull request Dec 31, 2016

Race condition in project reconfiguration. mesonbuild/meson#1267

Closed

evmar reviewed Dec 31, 2016

View reviewed changes

QuLogic force-pushed the resolution branch from 68823b8 to a0e873f Compare January 1, 2017 08:52

QuLogic commented Jan 1, 2017

View reviewed changes

maximuska reviewed Feb 15, 2017

View reviewed changes

QuLogic force-pushed the resolution branch from 0a056b7 to 45b9d35 Compare April 21, 2017 23:38

QuLogic force-pushed the resolution branch from 45b9d35 to ad52a78 Compare May 10, 2017 06:02

QuLogic force-pushed the resolution branch from ad52a78 to 303d1d4 Compare July 1, 2017 07:59

QuLogic added 8 commits September 16, 2017 03:54

Make TimeStamp 64-bit.

5fcdcf9

This prepares it for higher-resolution timestamps.

Read file timestamps in higher resolution.

f50a525

This uses nanoseconds on POSIX (±~292 years) and 100-ns increments on Windows (±~29247 years). The fallbacks to different structure fields is the only thing grabbed from ninja-build#337, with a slight modification in implementation.

Move #include for type definition to correct place.

9fa250d

Not sure why the old way works in newer compilers; maybe they just pre-define these types by defaulting to a newer standard.

Add #define to get printf-format specifiers.

eb6cea2

This is needed on older compilers/stdlibs such as on Ubuntu Precise which is used on Travis.

Use 64-bit-alignment-safe timestamp reading.

bf4fb03

Read and write the timestamp as two separate 32-bit integers in a fixed order to prevent any issues with alignment or byte order.

Update checks for new stat fields.

d2bf82f

This uses the macros as defined by the man page, which, as noted in the comments, are defined correctly on as many libc's that I could check.

Fix some Windows troubles.

f117cc1

Add parentheses so that constant does not overflow; include inttypes.h when using MinGW to get the proper macros.

Use strtoll when reading mtime from build log.

df19c5a

This prevents overflow on Windows where 'long' is not 64-bit.

QuLogic force-pushed the resolution branch from 303d1d4 to df19c5a Compare September 16, 2017 09:14

Fix minor typo.

0c42653

nico reviewed Oct 20, 2017

View reviewed changes

nico merged commit e234a7b into ninja-build:master Oct 20, 2017

QuLogic deleted the resolution branch October 20, 2017 20:27

jdrouhard mentioned this pull request Oct 22, 2017

Record most recent input's mtime in log file for correct dirty status computation #1165

Closed

atetubou pushed a commit to atetubou/ninja that referenced this pull request Dec 18, 2017

Merge pull request ninja-build#1219 from QuLogic/resolution

9aa164c

Use high-resolution timestamps process thread fix? attr work thread

nirbheek mentioned this pull request Mar 12, 2018

Issue running Meson via Python subprocess.run() mesonbuild/meson#3230

Closed

nirbheek mentioned this pull request Aug 8, 2018

Build definition not regenerated after meson configure mesonbuild/meson#3992

Closed

nirbheek mentioned this pull request Aug 23, 2018

DXVK DLLs are compiled in debug mode ValveSoftware/Proton#192

Closed

mathstuf mentioned this pull request Oct 3, 2018

Build dependency races #1460

Closed

Twey mentioned this pull request Oct 9, 2018

Needs release #1477

Closed

Jwata mentioned this pull request Jan 17, 2023

Increment ninja_log version #2240

Merged

This pull request was closed.

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Use high-resolution timestamps #1219

Use high-resolution timestamps #1219

QuLogic commented Dec 31, 2016

evmar left a comment

evmar Dec 31, 2016

evmar Dec 31, 2016

QuLogic Dec 31, 2016

evmar Dec 31, 2016

evmar Dec 31, 2016

QuLogic Dec 31, 2016

evmar Dec 31, 2016

QuLogic Dec 31, 2016

QuLogic Jan 1, 2017

QuLogic commented Jan 28, 2017

maximuska Feb 15, 2017

maximuska Feb 15, 2017 •

edited

Loading

QuLogic Feb 16, 2017

maximuska commented Feb 15, 2017

maximuska commented Feb 16, 2017

fooishbar commented Feb 16, 2017

maximuska commented Feb 16, 2017 via email

fooishbar commented Feb 16, 2017

Maratyszcza commented Apr 10, 2017

QuLogic commented Apr 21, 2017

QuLogic commented May 10, 2017 •

edited

Loading

nico commented Jun 20, 2017

nirbheek commented Jun 20, 2017

QuLogic commented Jul 1, 2017

nirbheek commented Jul 17, 2017 •

edited

Loading

QuLogic commented Sep 16, 2017 •

edited

Loading

ClausKlein commented Sep 19, 2017

QuLogic commented Sep 26, 2017

nico Oct 20, 2017

nico commented Oct 20, 2017

nico commented Oct 20, 2017

nirbheek commented Aug 23, 2018 •

edited

Loading

Use high-resolution timestamps #1219

Use high-resolution timestamps #1219

Conversation

QuLogic commented Dec 31, 2016

evmar left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

QuLogic commented Jan 28, 2017

Choose a reason for hiding this comment

maximuska Feb 15, 2017 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

maximuska commented Feb 15, 2017

maximuska commented Feb 16, 2017

fooishbar commented Feb 16, 2017

maximuska commented Feb 16, 2017 via email

fooishbar commented Feb 16, 2017

Maratyszcza commented Apr 10, 2017

QuLogic commented Apr 21, 2017

QuLogic commented May 10, 2017 • edited Loading

nico commented Jun 20, 2017

nirbheek commented Jun 20, 2017

QuLogic commented Jul 1, 2017

nirbheek commented Jul 17, 2017 • edited Loading

QuLogic commented Sep 16, 2017 • edited Loading

ClausKlein commented Sep 19, 2017

QuLogic commented Sep 26, 2017

Choose a reason for hiding this comment

nico commented Oct 20, 2017

nico commented Oct 20, 2017

nirbheek commented Aug 23, 2018 • edited Loading

maximuska Feb 15, 2017 •

edited

Loading

QuLogic commented May 10, 2017 •

edited

Loading

nirbheek commented Jul 17, 2017 •

edited

Loading

QuLogic commented Sep 16, 2017 •

edited

Loading

nirbheek commented Aug 23, 2018 •

edited

Loading