Update JAX and fix aarch64-darwin build #219778

samuela · 2023-03-06T01:31:36Z

Description of changes

This PR does a few things:

Update jaxlib(-bin): 0.3.22 -> 0.4.4
Update jax: 0.4.1 -> 0.4.5
Fix the rev that JAX was building on which was messed up by 0d6a071 which was merged without approval.
Unbreak JAX on aarch64-darwin. The jaxlib source build still doesn't work on this platform but that doesn't mean we can use the jaxlib-bin package in its place, esp. considering it's only a checkPhase dependency of jax.

Things done

samuela · 2023-03-06T02:11:33Z

There are a few TODOs before this can be merged:

jaxlib is currently failing to build on x86_64-linux due to Build failure on x86_64-linux: error: 'RawCompressFromIOVec' is not a member of 'snappy' jax-ml/jax#14795.
We'll need the fetchAttrs hashes for platforms that I do not have access to: https://github.com/NixOS/nixpkgs/pull/219778/files#diff-8906f9e778af6b3dc2ab4057b61c97ee82f0f35b3d02f49a5beabb4de4d64c23R252-R253. @divanorama previously updated these hashes, but I only have access to x86_64-linux and aarch64-darwin machines. I'm leaning towards (temporarily?) dropping support for the other platforms since I have no way of testing/building on them as a maintainer. @uri-canva IIRC you previously worked on the x86_64-darwin builds. Do you still rely on these builds? Would you be able to test builds on that platform? Update: I went ahead and removed support for failing platforms.

samuela · 2023-03-06T02:13:21Z

I should add that although we may drop support for from-source jaxlib builds for platforms, they don't necessarily need to be dropped from jaxlib-bin since that's easier to maintain. So support would not be dropped entirely.

samuela · 2023-03-09T18:33:21Z

This PR should be ready to go now. The JAX team was kind enough to hook us up with a new snappy release in order to fix the build (s/o @hawkinsp! 🚀)

I've tested that jaxlib builds with and without CUDA support. I'm running a nixpkgs-review now...

samuela · 2023-03-09T18:39:38Z

cc @NixOS/cuda-maintainers @uri-canva @ndl

uri-canva · 2023-03-09T21:27:04Z

Unfortunately I now do not have access to x86_64-darwin anymore either.

uri-canva · 2023-03-09T21:27:42Z

You can run the build on ofborg with fake shas to get the hash of the fetch derivations.

samuela · 2023-03-09T21:41:12Z

If we don't have anyone with access to a machine to test it out conveniently I vote that we drop support from the source build and x86_64-darwin users can use the jaxlib-bin package instead. Otherwise the maintenance load is just too high IMHO.

SomeoneSerge · 2023-03-10T23:17:02Z

Feels reasonable. For the sake of documentation, I suggest that if one later wants to reintroduce x8664Darwin support they also set up a GitHub Actions workflow that runs over a matrix of platforms and tries to update the derivation (version and hashes) via something like nix-update. And I don't expect nix-update to just work here. The whole business of conditional hashes is too bearing

uri-canva · 2023-03-11T01:03:26Z

We could also fix it so the fetch hashes are not system dependent, though I'm not sure how hard it would be. I'll take a look. Are there any other reasons to keep jax both built from source and as a binary?

samuela · 2023-03-11T01:51:18Z

We could also fix it so the fetch hashes are not system dependent, though I'm not sure how hard it would be. I'll take a look. Are there any other reasons to keep jax both built from source and as a binary?

jaxlib-bin comes in handy since the builds are so simple and fast. So eg that makes it a lot easier to support more architectures and lets users more easily override to get different versions of jaxlib if necessary. The full jaxlib source builds are just so looooooong

samuela · 2023-03-11T01:53:08Z

snappy is fairly popular, so this actually results in a few hundred rebuilds. I was hoping to run a nixpkgs-review and be done with it, but I actually ran out of disk space on my 200Gb drive. Would someone else be able to nixpkgs-review it? If not, I could create a snappy-only PR targeting staging.

uri-canva · 2023-03-11T02:18:54Z

I hit another error in snappy on aarch64-darwin:

/tmp/nix-build-snappy-1.1.10.drv-0/source/snappy.cc:1293:37: error: comparison of integers of different signs: 'unsigned long' and 'ptrdiff_t' (aka 'long') [-Werror,-Wsign-compare]
             (op + deferred_length) < op_limit_min_slop);
              ~~~~~~~~~~~~~~~~~~~~  ^ ~~~~~~~~~~~~~~~~~
/tmp/nix-build-snappy-1.1.10.drv-0/source/snappy.cc:1293:37: error: comparison of integers of different signs: 'unsigned long' and 'ptrdiff_t' (aka 'long') [-Werror,-Wsign-compare]
             (op + deferred_length) < op_limit_min_slop);
              ~~~~~~~~~~~~~~~~~~~~  ^ ~~~~~~~~~~~~~~~~~
/tmp/nix-build-snappy-1.1.10.drv-0/source/snappy.cc:1405:15: note: in instantiation of function template specialization 'snappy::DecompressBranchless<char *>' requested here
              DecompressBranchless(reinterpret_cast<const uint8_t*>(ip),
              ^
/tmp/nix-build-snappy-1.1.10.drv-0/source/snappy.cc:1600:17: note: in instantiation of function template specialization 'snappy::SnappyDecompressor::DecompressAllTags<snappy::SnappyIOVecWriter>' requested here
  decompressor->DecompressAllTags(writer);
                ^
/tmp/nix-build-snappy-1.1.10.drv-0/source/snappy.cc:1587:10: note: in instantiation of function template specialization 'snappy::InternalUncompressAllTags<snappy::SnappyIOVecWriter>' requested here
  return InternalUncompressAllTags(&decompressor, writer, r->Available(),
         ^
/tmp/nix-build-snappy-1.1.10.drv-0/source/snappy.cc:1936:10: note: in instantiation of function template specialization 'snappy::InternalUncompress<snappy::SnappyIOVecWriter>' requested here
  return InternalUncompress(compressed, &output);
         ^
/tmp/nix-build-snappy-1.1.10.drv-0/source/snappy.cc:1293:37: error: comparison of integers of different signs: 'unsigned long' and 'ptrdiff_t' (aka 'long') [-Werror,-Wsign-compare]
             (op + deferred_length) < op_limit_min_slop);
              ~~~~~~~~~~~~~~~~~~~~  ^ ~~~~~~~~~~~~~~~~~
/tmp/nix-build-snappy-1.1.10.drv-0/source/snappy.cc:1405:15: note: in instantiation of function template specialization 'snappy::DecompressBranchless<unsigned long>' requested here
              DecompressBranchless(reinterpret_cast<const uint8_t*>(ip),
              ^
/tmp/nix-build-snappy-1.1.10.drv-0/source/snappy.cc:1600:17: note: in instantiation of function template specialization 'snappy::SnappyDecompressor::DecompressAllTags<snappy::SnappyDecompressionValidator>' requested here
  decompressor->DecompressAllTags(writer);
                ^
/tmp/nix-build-snappy-1.1.10.drv-0/source/snappy.cc:1587:10: note: in instantiation of function template specialization 'snappy::InternalUncompressAllTags<snappy::SnappyDecompressionValidator>' requested here
  return InternalUncompressAllTags(&decompressor, writer, r->Available(),
         ^
/tmp/nix-build-snappy-1.1.10.drv-0/source/snappy.cc:2101:10: note: in instantiation of function template specialization 'snappy::InternalUncompress<snappy::SnappyDecompressionValidator>' requested here
  return InternalUncompress(&reader, &writer);
         ^

uri-canva · 2023-03-11T02:19:37Z

Yeah we should probably do the snappy upgrade separately first.

uri-canva · 2023-03-11T02:20:28Z

I can try to nixpkgs-review the snappy PR once you put it up.

uri-canva · 2023-03-11T02:22:27Z

jaxlib-bin comes in handy since the builds are so simple and fast. So eg that makes it a lot easier to support more architectures and lets users more easily override to get different versions of jaxlib if necessary. The full jaxlib source builds are just so looooooong

Are there things the source build supports that the binary doesn't? I'm confused why we need both, I thought we had both as a workaround for some issue that prevented us from building from source, but it doesn't sound like it's a transitional thing. Do we still need both?

uri-canva · 2023-03-11T10:52:02Z

I worked around the snappy break with:

diff --git a/pkgs/development/libraries/snappy/default.nix b/pkgs/development/libraries/snappy/default.nix
index 24c3e7bb7df..d33e5755084 100644
--- a/pkgs/development/libraries/snappy/default.nix
+++ b/pkgs/development/libraries/snappy/default.nix
@@ -29,6 +29,8 @@ stdenv.mkDerivation rec {

   nativeBuildInputs = [ cmake ];

+  env.NIX_CFLAGS_COMPILE = "-Wno-sign-compare";
+
   cmakeFlags = [
     "-DBUILD_SHARED_LIBS=${if static then "OFF" else "ON"}"
     "-DSNAPPY_BUILD_TESTS=OFF"

but I'm having trouble getting the diff of the fetch derivation, I'll have to come back to it another time.

samuela · 2023-03-14T20:47:47Z

Spun off the snappy change into #221215. @uri-canva could you ptal?

We should be able to rebase and merge this once that chance lands on master.

samuela · 2023-03-14T20:49:26Z

but I'm having trouble getting the diff of the fetch derivation, I'll have to come back to it another time.

wdym? it built fine for me on aarch64-darwin

Are there things the source build supports that the binary doesn't? I'm confused why we need both, I thought we had both as a workaround for some issue that prevented us from building from source, but it doesn't sound like it's a transitional thing. Do we still need both?

Are you proposing removing the source build or removing the binary build? i think the main selling point for the source build is just that nixpkgs generally prefers source builds 🤷

uri-canva · 2023-03-14T22:10:25Z

wdym? it built fine for me on aarch64-darwin

Yeah but I want to build it on a different system too so I can diff them. Usually if different systems need different shas it means the deps have system specific files in them, which we can delete and rebuild during the build. Ideally the output of the fetch only has system independent files that were downloaded.

Are you proposing removing the source build or removing the binary build? i think the main selling point for the source build is just that nixpkgs generally prefers source builds 🤷

I'm proposing keeping only one of the two, so we don't duplicate the maintenance effort. Ideally it would be the source one, that's why I'm trying to figure out what is causing the diff in the deps shas, so we can develop the derivation for all systems more easily.

samuela · 2023-03-14T22:16:04Z

Yeah but I want to build it on a different system too so I can diff them. Usually if different systems need different shas it means the deps have system specific files in them, which we can delete and rebuild during the build. Ideally the output of the fetch only has system independent files that were downloaded.

Ohhhh this is in regards to fetchAttrs? Yeah I don't really know anything about that. It'd be really great if we could get all platforms on the same fetchAttrs hash!

I'm proposing keeping only one of the two, so we don't duplicate the maintenance effort. Ideally it would be the source one, that's why I'm trying to figure out what is causing the diff in the deps shas, so we can develop the derivation for all systems more easily.

There are still benefits to having the binary build around. It's a lot easier to hack on, and users (like myself) occasionally need to override things in order to upgrade/downgrade versions, play around with settings, etc. Luckily the maintenance burden of the binary build is much lower.

uri-canva · 2023-03-14T22:33:13Z

There are still benefits to having the binary build around. It's a lot easier to hack on, and users (like myself) occasionally need to override things in order to upgrade/downgrade versions, play around with settings, etc. Luckily the maintenance burden of the binary build is much lower.

Right then maybe we should keep only the binary build?

samuela · 2023-03-14T23:21:14Z

Right then maybe we should keep only the binary build?

I would be sympathetic to such a change, but IIUC there are a number of people who really like to have the source build around

uri-canva · 2023-03-14T23:45:04Z

Ok 😢 .

samuela · 2023-04-07T15:39:48Z

Note to self: We should prob merge #221390 before this

GaetanLepage · 2023-04-18T16:27:16Z

Thank you for this updating effort !
I would suggest that we directly go to jaxlib v0.4.7 and jax v0.4.8.

I think that we might need to change the prefetch.sh script as is:

version="$1"
nix hash to-sri --type sha256 "$(nix-prefetch-url https://storage.googleapis.com/jax-releases/nocuda/jaxlib-${version}-cp310-cp310-manylinux2014_x86_64.whl)"
nix hash to-sri --type sha256 "$(nix-prefetch-url https://storage.googleapis.com/jax-releases/nocuda/jaxlib-${version}-cp311-cp311-manylinux2014_x86_64.whl)"
nix hash to-sri --type sha256 "$(nix-prefetch-url https://storage.googleapis.com/jax-releases/mac/jaxlib-${version}-cp310-cp310-macosx_10_14_x86_64.whl)"
nix hash to-sri --type sha256 "$(nix-prefetch-url https://storage.googleapis.com/jax-releases/mac/jaxlib-${version}-cp311-cp311-macosx_10_14_x86_64.whl)"
nix hash to-sri --type sha256 "$(nix-prefetch-url https://storage.googleapis.com/jax-releases/mac/jaxlib-${version}-cp310-cp310-macosx_11_0_arm64.whl)"
nix hash to-sri --type sha256 "$(nix-prefetch-url https://storage.googleapis.com/jax-releases/mac/jaxlib-${version}-cp311-cp311-macosx_11_0_arm64.whl)"
nix hash to-sri --type sha256 "$(nix-prefetch-url https://storage.googleapis.com/jax-releases/cuda12/jaxlib-${version}+cuda12.cudnn88-cp310-cp310-manylinux2014_x86_64.whl)"
nix hash to-sri --type sha256 "$(nix-prefetch-url https://storage.googleapis.com/jax-releases/cuda12/jaxlib-${version}+cuda12.cudnn88-cp311-cp311-manylinux2014_x86_64.whl)"

SomeoneSerge · 2023-04-18T16:34:53Z

pkgs/development/python-modules/jaxlib/bin.nix

@@ -50,21 +50,21 @@ let
  cpuSrcs = {
    "x86_64-linux" = fetchurl {
      url = "https://storage.googleapis.com/jax-releases/nocuda/jaxlib-${version}-cp310-cp310-manylinux2014_x86_64.whl";


cp310-cp310

Should we also account for pythonVersion?

mm not a bad idea....

samuela · 2023-04-18T19:46:46Z

Every time I try to run nixpkgs-review it hangs, apparently on the cvxpy test suite. Not sure why as the cvxpy build does not seem to hang either on this branch or on master.

SomeoneSerge · 2023-04-18T19:53:26Z

Every time I try to run nixpkgs-review it hangs, apparently on the cvxpy test suite. Not sure why as the cvxpy build does not seem to hang either on this branch or on master.

I've been continuously suffering from cvxpy tests when running parallel builds, and I haven't quite figured out why. Upstream github actions reveal that their tests take at most two minutes

GaetanLepage · 2023-04-18T20:05:43Z

Every time I try to run nixpkgs-review it hangs, apparently on the cvxpy test suite. Not sure why as the cvxpy build does not seem to hang either on this branch or on master.

Could you try to update this branch ? I can give it a shot on my machine if you want.

SomeoneSerge · 2023-04-18T20:12:33Z

@samuela if you get bored, I think it's OK to

❯ sudo kill -9 $(pgrep -f cvxpy)

The failures we're looking for are likely unrelated to cvxpy...

samuela · 2023-04-18T20:32:45Z

I've been continuously suffering from cvxpy tests when running parallel builds, and I haven't quite figured out why. Upstream github actions reveal that their tests take at most two minutes

Glad to hear I'm not alone here!

If you get bored, I think it's OK to [...]

Brilliant! I'll see how far I can get with that. I really wish nixpkgs-review had an option to skip packages...

GaetanLepage · 2023-04-18T21:02:54Z

@samuela When trying to build jaxlib 0.4.7 from source I faced a build failure at the preBuild phase.
More precisely, linking the protobuf lib in ../output/external/org_tensorflow was failing because this folder doesn't exist.
I removed this patch and it built succesfully.

It seems to come from this upstream commit.

samuela · 2023-04-18T21:06:47Z

Result of nixpkgs-review pr 219778 run on x86_64-linux 1

43 packages failed to build:

python310Packages.arviz
python310Packages.arviz.dist
python310Packages.bambi
python310Packages.bambi.dist
python310Packages.blackjax
python310Packages.blackjax.dist
python310Packages.dalle-mini
python310Packages.dalle-mini.dist
python310Packages.distrax
python310Packages.distrax.dist
python310Packages.dm-sonnet
python310Packages.dm-sonnet.dist
python310Packages.elegy
python310Packages.elegy.dist
python310Packages.flax
python310Packages.flax.dist
python310Packages.jaxopt
python310Packages.jaxopt.dist
python310Packages.numpyro
python310Packages.numpyro.dist
python310Packages.pymc
python310Packages.pymc.dist
python310Packages.pytensor
python310Packages.pytensor.dist
python310Packages.rlax
python310Packages.rlax.dist
python310Packages.tensorflow-datasets
python310Packages.tensorflow-datasets.dist
python310Packages.treex
python310Packages.treex.dist
python310Packages.vqgan-jax
python310Packages.vqgan-jax.dist
python311Packages.blackjax
python311Packages.blackjax.dist
python311Packages.dm-haiku
python311Packages.dm-haiku.dist
python311Packages.dm-haiku.testsout
python311Packages.jaxopt
python311Packages.jaxopt.dist
python311Packages.jmp
python311Packages.jmp.dist
python311Packages.numpyro
python311Packages.numpyro.dist

45 packages built:

python310Packages.aeppl
python310Packages.aeppl.dist
python310Packages.aesara
python310Packages.aesara.dist
python310Packages.augmax
python310Packages.augmax.dist
python310Packages.chex
python310Packages.chex.dist
python310Packages.dm-haiku
python310Packages.dm-haiku.dist
python310Packages.dm-haiku.testsout
python310Packages.jax
python310Packages.jax.dist
python310Packages.jaxlib (python310Packages.jaxlib-build ,python310Packages.jaxlibWithoutCuda)
python310Packages.jaxlib-bin
python310Packages.jaxlib-bin.dist
python310Packages.jaxlib.dist (python310Packages.jaxlib-build.dist ,python310Packages.jaxlibWithoutCuda.dist)
python310Packages.jaxlibWithCuda
python310Packages.jaxlibWithCuda.dist
python310Packages.jmp
python310Packages.jmp.dist
python310Packages.objax
python310Packages.objax.dist
python310Packages.optax
python310Packages.optax.dist
python310Packages.optax.testsout
python310Packages.tensorflow-bin
python310Packages.tensorflow-bin.dist
python310Packages.treeo
python310Packages.treeo.dist
python311Packages.augmax
python311Packages.augmax.dist
python311Packages.chex
python311Packages.chex.dist
python311Packages.jax
python311Packages.jax.dist
python311Packages.jaxlib (python311Packages.jaxlib-build ,python311Packages.jaxlibWithoutCuda)
python311Packages.jaxlib.dist (python311Packages.jaxlib-build.dist ,python311Packages.jaxlibWithoutCuda.dist)
python311Packages.jaxlibWithCuda
python311Packages.jaxlibWithCuda.dist
python311Packages.optax
python311Packages.optax.dist
python311Packages.optax.testsout
python311Packages.treeo
python311Packages.treeo.dist

samuela · 2023-04-18T21:49:08Z

Failed derivations

tensorflow_probability-0.19.0-py2.py3-none-any.whl.drv (derivation hash: m5hnwrjlfw35c8m12glr6f5vhqs47g1l)
- https://gist.github.com/samuela/764dac33adcc2c3cfe20050af6cfe125
python3.10-flax-0.6.5.drv (derivation hash: 4nzqqvf0ni6yfcv8b5gf7wqfdsd1qwrh)
- https://gist.github.com/samuela/78fe1a8c60b527583d6384cdaaa16679
python3.11-jmp-unstable-2021-10-03.drv (derivation hash: bqn5sdpw8fll1y0fz3pdkdzz72b14p1a)
- https://gist.github.com/samuela/2fca6aa1344494ed52d60f39b3261749
python3.10-tensorflow-datasets-4.8.2.drv (derivation hash: rmw0l6014xm0pkyskm88wlzb956r7fy6)
- https://gist.github.com/samuela/8d00e41b98fe7351ed11473e6c7b22ef
python3.10-cvxpy-1.3.0.drv (derivation hash: n2ig5g2fbvgrirqcnmmvba0pz1fj2kw3)
- https://gist.github.com/samuela/3a45e122d31de4139c108d9deedbb588
python3.11-cvxpy-1.3.0.drv (derivation hash: xyyrmk6kxpy1i05f6vyca7g48wfr2vb9)
- https://gist.github.com/samuela/1a012a96311e79847292bd45031202c1
python3.10-pytensor-2.10.1.drv (derivation hash: pvnd0l1vjrnxvnmj4ssf4c5laa469zay)
- https://gist.github.com/samuela/dcaafc5bc8319550513fed6332a5a18e
python3.11-numpyro-0.11.0.drv (derivation hash: 6mb3rzixdwr4nr0s8rdhdc4rrck7fsk0)
- https://gist.github.com/samuela/f33cd1c0be79dce552809863f62e9471
python3.10-numpyro-0.11.0.drv (derivation hash: 1piw5qwr4lswd04lw3aglr302704dbdp)
- https://gist.github.com/samuela/d1ca6da8b6aa43f466154a9764ee4e24

samuela · 2023-04-18T21:53:33Z

Doesn't look like any of these failures are JAX version related AFAICT. I propose we go ahead and merge this and then address the python version-specific downloads and 0.4.7/8 upgrade in separate PR(s).

SomeoneSerge · 2023-04-19T00:34:55Z

We're still at 80| src = if !cudaSupport then cpuSrcs."${stdenv.hostPlatform.system}" else gpuSrc;?

samuela · 2023-04-19T01:31:46Z

We're still at 80| src = if !cudaSupport then cpuSrcs."${stdenv.hostPlatform.system}" else gpuSrc;?

You mean we're not yet branching on the python version? Or something else?

SomeoneSerge · 2023-04-19T01:53:54Z

You mean we're not yet branching on the python version? Or something else?

No, the ofborg thing?

See https://discourse.nixos.org/t/ofborg-does-not-respect-meta-platforms/27019 for more info.

samuela · 2023-04-19T15:59:35Z

Ohhh, I totally forgot about that! Should be addressed in 3fa9f1f.

samuela · 2023-04-19T19:53:55Z

Looks like all checks are green now! proceeding to merge

ofborg bot added the 6.topic: darwin Running or building packages on Darwin label Mar 6, 2023

github-actions bot added the 6.topic: python label Mar 6, 2023

samuela force-pushed the samuela/jax branch from 88f6b99 to 0d8d83f Compare March 6, 2023 01:56

samuela mentioned this pull request Mar 6, 2023

Build failure on x86_64-linux: error: 'RawCompressFromIOVec' is not a member of 'snappy' jax-ml/jax#14795

Closed

samuela force-pushed the samuela/jax branch from 212ee95 to 28c0e59 Compare March 9, 2023 18:29

samuela marked this pull request as ready for review March 9, 2023 18:31

samuela mentioned this pull request Mar 14, 2023

snappy: 1.1.9 -> 1.1.10 #221215

Merged

12 tasks

samuela added 3 commits April 17, 2023 15:56

python3Packages.jaxlib-bin: 0.3.22 -> 0.4.4

0ab0cbb

python3Packages.jax: 0.4.1 -> 0.4.5 and fix aarch64-darwin build

a6c0e4d

python3Packages.jaxlib: 0.3.22 -> 0.4.4

11687d0

samuela force-pushed the samuela/jax branch from aaf0d0a to 11687d0 Compare April 17, 2023 23:42

SomeoneSerge reviewed Apr 18, 2023

View reviewed changes

uri-canva approved these changes Apr 18, 2023

View reviewed changes

python3Packages.jaxlib-bin: fix ofborg evaluation

3fa9f1f

See https://discourse.nixos.org/t/ofborg-does-not-respect-meta-platforms/27019 for more info.

ofborg bot added the 8.has: clean-up label Apr 19, 2023

ofborg bot requested a review from ndl April 19, 2023 16:16

ofborg bot added 10.rebuild-darwin: 11-100 10.rebuild-linux: 11-100 labels Apr 19, 2023

samuela merged commit 8faef6d into NixOS:master Apr 19, 2023

samuela deleted the samuela/jax branch April 19, 2023 19:54

GaetanLepage mentioned this pull request Apr 20, 2023

cudaPackages: bump default cudaPackages_11_7 -> cudaPackages_11_8 #224927

Closed

4 tasks

Update JAX and fix aarch64-darwin build #219778

Update JAX and fix aarch64-darwin build #219778

Conversation

samuela commented Mar 6, 2023 • edited Loading

Description of changes

Things done

samuela commented Mar 6, 2023 • edited Loading

samuela commented Mar 6, 2023

samuela commented Mar 9, 2023

samuela commented Mar 9, 2023

uri-canva commented Mar 9, 2023

uri-canva commented Mar 9, 2023

samuela commented Mar 9, 2023

SomeoneSerge commented Mar 10, 2023

uri-canva commented Mar 11, 2023

samuela commented Mar 11, 2023

samuela commented Mar 11, 2023

uri-canva commented Mar 11, 2023

uri-canva commented Mar 11, 2023

uri-canva commented Mar 11, 2023

uri-canva commented Mar 11, 2023

uri-canva commented Mar 11, 2023

samuela commented Mar 14, 2023

samuela commented Mar 14, 2023

uri-canva commented Mar 14, 2023

samuela commented Mar 14, 2023

uri-canva commented Mar 14, 2023

samuela commented Mar 14, 2023

uri-canva commented Mar 14, 2023

samuela commented Apr 7, 2023

GaetanLepage commented Apr 18, 2023

SomeoneSerge Apr 18, 2023 • edited Loading

Choose a reason for hiding this comment

samuela Apr 18, 2023

Choose a reason for hiding this comment

samuela commented Apr 18, 2023

SomeoneSerge commented Apr 18, 2023

GaetanLepage commented Apr 18, 2023

SomeoneSerge commented Apr 18, 2023

samuela commented Apr 18, 2023

GaetanLepage commented Apr 18, 2023

samuela commented Apr 18, 2023

samuela commented Apr 18, 2023 • edited Loading

Failed derivations

samuela commented Apr 18, 2023

SomeoneSerge commented Apr 19, 2023

samuela commented Apr 19, 2023

SomeoneSerge commented Apr 19, 2023

samuela commented Apr 19, 2023

samuela commented Apr 19, 2023

samuela commented Mar 6, 2023 •

edited

Loading

samuela commented Mar 6, 2023 •

edited

Loading

SomeoneSerge Apr 18, 2023 •

edited

Loading

samuela commented Apr 18, 2023 •

edited

Loading