feat: add Linux aarch64/arm64 support for bioconda-utils #866

Yikun · 2023-04-06T07:25:19Z

This PR try to add Linux aarch64/arm64 support for bioconda-utils:

Support BASE_IMAGE build args in Dockerfile: the devs can specify BASE_IMAGE to switch base image.
bioconda_utils/bioconda_utils-requirements.txt: Upgrade packages version to support Linux aarch64.
Add --docker-base-image image to help users specify docker base image.
Add Linux aarch64 image build to release-please.yml and build-image.yml workflow
bioconda_utils/docker_utils.py: create conda build subdir according to arch.

After this PR, users can build the bioconda package by using same cmd with x86 by specifing --docker-base-image to build the package.

Note that the mulled-test are still not supported, there are a circle dependency between bioconda-containers and bioconda-utils, we'd better to support bioconda-utils aarch64 image first.

Related: #706

Test step:

$ uname -m
aarch64

# Activate a conda env
conda create -n biotest
conda activate biotest

git clone https://github.com/Yikun/bioconda-utils.git
cd bioconda-utils
git checkout aarch64

# Build docker image with embed involucro for bioconda
docker build --build-arg BASE_IMAGE=quay.io/condaforge/linux-anvil-aarch64 -t ghcr.io/yikun/bioconda-utils-build-env-cos7-aarch64 -f Dockerfile .

# Install all bioconda-utils
conda install --file bioconda_utils/bioconda_utils-requirements.txt -c conda-forge -c bioconda

# Install the bioconda-utils with this PR
pip install -e .

# Test
bioconda-utils -h

# Build a package to test
cd bioconda-recipes

bioconda-utils build --docker --packages bamstats --docker-base-image ghcr.io/yikun/bioconda-utils-build-env-cos7-aarch64 --force

After this PR merged:

conda install -c bioconda bioconda-utils

bioconda-utils build --docker --packages bamstats --docker-base-image ghcr.io/yikun/bioconda-utils-build-env-cos7-aarch64

Dockerfile.aarch64

Yikun · 2023-04-10T03:50:40Z

Test passed in my local fork: Yikun#2

bgruening · 2023-04-10T09:53:24Z

This one needs a rebase.

Yikun · 2023-04-10T14:37:28Z

A question about divergent_build:

bioconda-utils/bioconda_utils/utils.py

Lines 1136 to 1140 in 1923d24

    
           new_metas, existing_metas, divergent_builds = ( 
        
               _filter_existing_packages(metas, check_channels)) 
        
           if divergent_builds: 
        
               raise DivergentBuildsError(*sorted(divergent_builds))

With this PR, all aarch64 pkg without verison bump (even with the --force) will rasie the exception here, and finally failed on:

bioconda-utils/bioconda_utils/build.py

Lines 324 to 327 in 1923d24

    
           except utils.DivergentBuildsError as exc: 
        
               logger.error('BUILD ERROR: packages with divergent build strings in repository ' 
        
                            'for recipe %s. A build number bump is likely needed: %s', 
        
                            recipe, exc)

I'm not sure what's the meaning this exception? What's the meaning of divergent build strings?

cc @bgruening @epruesse Is this behavior normal? Would you give some idea on here?

(I also dump the values compared to x86 to see how divergent_builds work, I still can't understand well)

bioconda-utils/bioconda_utils/utils.py

Line 1119 in 1923d24

for divergent_build in (existing_pkg_builds - set(build_meta.keys())):

# existing_pkg_builds - set(build_meta.keys())
# aarch64: {Pandas(subdir='linux-64', build='he881be0_4')} - {('linux-aarch64', 'h8865c2f_4')}
# x86_64: {Pandas(subdir='linux-64', build='he881be0_4')} - {('linux-64', 'he881be0_4')}

Yikun · 2023-04-11T09:34:01Z

I think I understand the meaning of divergent_build, it's for detecting changes on build but no version bump for specific arch/noarch. So I beliebe we need changes like: 9e7b0fd#diff-79a86610eedbe72fc010927f2fec251188ef01ec79e21f8ddef67d912cda4295 .

And it's ready for review!

bgruening · 2023-04-11T09:36:58Z

I think I understand the meaning of divergent_build, it's for detecting changes on build but no version bump for specific arch/noarch. So I beliebe we need changes like: 9e7b0fd#diff-79a86610eedbe72fc010927f2fec251188ef01ec79e21f8ddef67d912cda4295 .

Yes, see in matrix :)

What is your deployment strategy for the entire repo? Should we have an initial allow list and migrate single recipes one by one?

Yikun · 2023-04-11T15:54:01Z

What is your deployment strategy for the entire repo? Should we have an initial allow list and migrate single recipes one by one?

@bgruening I thought the strategy might be:

Step 1: Support bioconda-utils on Linux aarch64. (This PR)

pip install bioconda-utils: User can install the bioconda-utils package directly in Linux aarch64
bioconda-utils build --docker --packages XXX --docker-base-image bioconda-utils-build-env-cos7-aarch64:latest --force: User can build the bioconda package

Step 2: Support bioconda-utils mulled test on Linux aarch64 (depends on bioconda/bioconda-containers#55 and this PR)

bioconda-utils build --docker --mulled-test --packages XXX --docker-base-image bioconda-utils-build-env-cos7-aarch64:latest --force: User can build and test bioconda package.

(After step1 / step2 completed, users can build their wanted packages in a local aarch64 env)

Step 3: Run bulk test on Linux aarch64 and get build/test results

bioconda-utils build recipes config.yml --worker-offset 0 --n-workers N --docker --mulled-test --force: build and run all tests and find all failure packages. Then we will have an initial allow list candidate package. And in this step we might get a page to show which packages build passed or failure, the potential style might be something like what we done in bioconductor report or conda-forge team migration page.
Fix the package one by one, and the results page will generated peridically to guide us which packages should be fixed.

Step 4: Upload all passed package and enable Linux aarch64 CI.

This step required the Linux aarch64 resource (need resource donation) or qemu on github action (might slow).
We could select a few and then dozens of packages as initial allow list to validate the linux aarch64 build/test/upload workflow
After complete the initial validation, we can change allow list to block list, or full list and enable Linux aarch64 CI.

This is just an initial idea, there is no doubt that the bioconda team has a more professional perspective, so we'd also like to hear your suggestions, any thoughts?

Signed-off-by: Yikun Jiang <yikunkero@gmail.com>

Yikun · 2023-04-13T09:36:01Z

I pick two packages randomly to test the build / test / push step:

bamstats
- bioconda-utils build --docker --mulled-test --packages bamstats --docker-base-image ghcr.io/yikun/bioconda-utils-build-env-cos7-aarch64 --mulled-conda-image ghcr.io/martin-g/create-env-aarch64 --anaconda-upload --force
- https://anaconda.org/yikunkero/bamstats
py2bit
- bioconda-utils build --docker --mulled-test --packages bioconda-utils --docker-base-image ghcr.io/yikun/bioconda-utils-build-env-cos7-aarch64 --mulled-conda-image ghcr.io/martin-g/create-env-aarch64 --anaconda-upload --force
- https://anaconda.org/yikunkero/py2bit

All is well!

Feel free to tell me if any other preworks should be done before merged.

Yikun · 2023-04-18T01:36:37Z

@bgruening Is there any way to run all package build/test without failfast but also consider dependency?

# This will break after first package failed
bioconda-utils build recipes config.yml --worker-offset 0 --n-workers 1 --docker biocontainers

I hope there is a way like:

ll /root/bio/bioconda-recipes | xargs -n1 bioconda-utils build --docker --mulled-test --packages

but also consider the dependency, otherwise we have to run above command multiple times when publish aarch64 at first time.

bgruening · 2023-04-18T12:30:02Z

@Yikun yes this should be possible. We are doing this on the bulk branch. I think if you try your rebuild on this branch it should work. Background is that we use this branch to bulk-rebuild the entire tree against new version-pins and 100 of package-updates at once.

Yikun · 2023-04-18T13:02:25Z

@bgruening OK, thanks for your reply. Do you think it's time to get this PR merge? This will help to setup arm CI and also help to build bioconda on aarch64, otherwise we have to apply the change and do local build in each CI env. Besides bioconda_utils-requirements.txt, all changes are isolation with original version (x86).

martin-g · 2023-04-18T13:15:56Z

What command one should use to build multiple packages ? In parallel or sequentially, but without stopping on the first failed package.

I tried

bioconda-utils build recipes config.yml --worker-offset 0 --n-workers 5000 --docker --mulled-test --anaconda-upload --docker-base-image ghcr.io/yikun/bioconda-utils-build-env-cos7-aarch64 --mulled-conda-image ghcr.io/martin-g/create-env-aarch64

on both master and bulk branches but it always fails on:

13:03:48 BIOCONDA INFO (OUT)      Running `rustc --crate-name rust_htslib --edition=2018 /opt/conda/conda-bld/10x_bamtofastq_1681822376637/_build_env/.cargo/registry/src/git.luolix.top-1ecc6299db9ec823/rust-htslib-0.38.2/src/lib.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts,future-incompat --crate-type lib --emit=dep-info,metadata,link -C opt-level=3 -C linker-plugin-lto -C debuginfo=1 --cfg 'feature="bzip2"' --cfg 'feature="libdeflate"' --cfg 'feature="lzma"' -C metadata=21249687d7a2ff42 -C extra-filename=-21249687d7a2ff42 --out-dir /opt/conda/conda-bld/10x_bamtofastq_1681822376637/work/target/release/deps -C linker=/opt/conda/conda-bld/10x_bamtofastq_1681822376637/_build_env/bin/aarch64-conda-linux-gnu-cc -L dependency=/opt/conda/conda-bld/10x_bamtofastq_1681822376637/work/target/release/deps --extern bio_types=/opt/conda/conda-bld/10x_bamtofastq_1681822376637/work/target/release/deps/libbio_types-b44ae3b4e5106054.rmeta --extern byteorder=/opt/conda/conda-bld/10x_bamtofastq_1681822376637/work/target/release/deps/libbyteorder-3fa2d67d6d18ade5.rmeta --extern custom_derive=/opt/conda/conda-bld/10x_bamtofastq_1681822376637/work/target/release/deps/libcustom_derive-ffaeddde1e18b352.rmeta --extern derive_new=/opt/conda/conda-bld/10x_bamtofastq_1681822376637/work/target/release/deps/libderive_new-ad983a2555eabb00.so --extern hts_sys=/opt/conda/conda-bld/10x_bamtofastq_1681822376637/work/target/release/deps/libhts_sys-99c9abb0944d285f.rmeta --extern ieee754=/opt/conda/conda-bld/10x_bamtofastq_1681822376637/work/target/release/deps/libieee754-075bde6610681de0.rmeta --extern lazy_static=/opt/conda/conda-bld/10x_bamtofastq_1681822376637/work/target/release/deps/liblazy_static-1464c69ac8c9cd33.rmeta --extern libc=/opt/conda/conda-bld/10x_bamtofastq_1681822376637/work/target/release/deps/liblibc-9ed5c1a1ce845d01.rmeta --extern linear_map=/opt/conda/conda-bld/10x_bamtofastq_1681822376637/work/target/release/deps/liblinear_map-e477efefa8fc49d6.rmeta --extern newtype_derive=/opt/conda/conda-bld/10x_bamtofastq_1681822376637/work/target/release/deps/libnewtype_derive-d631dfb49616692a.rmeta --extern regex=/opt/conda/conda-bld/10x_bamtofastq_1681822376637/work/target/release/deps/libregex-dffaa8dd80e1dd48.rmeta --extern thiserror=/opt/conda/conda-bld/10x_bamtofastq_1681822376637/work/target/release/deps/libthiserror-054997a83c8358b5.rmeta --extern url=/opt/conda/conda-bld/10x_bamtofastq_1681822376637/work/target/release/deps/liburl-9ab6a5e77ff5d9aa.rmeta --cap-lints allow -L native=/opt/conda/conda-bld/10x_bamtofastq_1681822376637/work/target/release/build/hts-sys-3bb7368342e96f5e/out -L native=/opt/conda/conda-bld/10x_bamtofastq_1681822376637/work/target/release/build/bzip2-sys-86a1869615adb9cc/out/lib -L native=/opt/conda/conda-bld/10x_bamtofastq_1681822376637/work/target/release/build/libdeflate-sys-169cab9152541b0c/out/lib -L native=/opt/conda/conda-bld/10x_bamtofastq_1681822376637/work/target/release/build/libz-sys-869a20e7c98a6611/out/lib -L native=/opt/conda/conda-bld/10x_bamtofastq_1681822376637/work/target/release/build/lzma-sys-c093ea58537dae4b/out`
13:03:49 BIOCONDA INFO (OUT) error[E0308]: mismatched types
13:03:49 BIOCONDA INFO (OUT)    --> /opt/conda/conda-bld/10x_bamtofastq_1681822376637/_build_env/.cargo/registry/src/git.luolix.top-1ecc6299db9ec823/rust-htslib-0.38.2/src/bam/record.rs:152:16
13:03:49 BIOCONDA INFO (OUT)     |
13:03:49 BIOCONDA INFO (OUT) 152 |             s: sam_copy.as_ptr() as *mut i8,
13:03:49 BIOCONDA INFO (OUT)     |                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ expected `u8`, found `i8`
13:03:49 BIOCONDA INFO (OUT)     |
13:03:49 BIOCONDA INFO (OUT)     = note: expected raw pointer `*mut u8`
13:03:49 BIOCONDA INFO (OUT)                found raw pointer `*mut i8`
13:03:49 BIOCONDA INFO (OUT)

for 10x_bamtofastq package.
I've requested a new release of this package with a fix - 10XGenomics/bamtofastq#136.
What command I could use until then ?

Yikun · 2023-04-18T13:28:45Z

What command one should use to build multiple packages ? In parallel or sequentially, but without stopping on the first failed package.

I think it should be:

ls /root/bio/bioconda-recipes/recipes/ | xargs -n1 bioconda-utils build --docker --mulled-test --docker-base-image ghcr.io/yikun/bioconda-utils-build-env-cos7-aarch64 --mulled-conda-image ghcr.io/martin-g/create-env-aarch64 --anaconda-upload --packages

johanneskoester · 2023-04-26T13:59:40Z

.github/workflows/release-please.yml

@@ -28,12 +37,19 @@ jobs:
          tag=${{ steps.release.outputs.tag_name }}
          printf %s "::set-output name=tag::${tag#v}"

-      - name: Build Image
-        if: ${{ steps.release.outputs.release_created }}


Why did you remove this?

Oh, good catch. It was the bad copy paste from build-image.yml.

Resolved!

Signed-off-by: Yikun Jiang <yikunkero@gmail.com>

julien-faye · 2023-06-09T10:37:07Z

.github/workflows/release-please.yml

+      matrix:
+        include:
+          - arch: arm64
+            image: bioconda-utils-build-env-cos7-aarch64


https://quay.io/repository/bioconda/bioconda-utils-build-env-cos7-aarch64 gives me 404
Is there a release after the merge of this PR ?

Seems only repos are created:
https://quay.io/repository/bioconda/bioconda-utils-build-env-cos7-aarch64?tab=tags&tag=latest

But docker image fail to push due to auth reason:
https://github.com/bioconda/bioconda-utils/actions/runs/5200214305/jobs/9378654544

⏳ Pushing "bioconda-utils-build-env-cos7-aarch64:latest, bioconda-utils-build-env-cos7-aarch64:2.3.1" to "***/bioconda-utils-build-env-cos7-aarch64:latest, ***/bioconda-utils-build-env-cos7-aarch64:2.3.1" respectively as "***" /usr/bin/podman push --quiet --digestfile bioconda-utils-build-env-cos7-aarch64-latest_digest.txt bioconda-utils-build-env-cos7-aarch64:latest ***/bioconda-utils-build-env-cos7-aarch64:latest --tls-verify=true --creds=***:*** Error: writing blob: initiating layer upload to /v2/bioconda/bioconda-utils-build-env-cos7-aarch64/blobs/uploads/ in quay.io: unauthorized: access to the requested resource is not authorized Removing temporary Podman image storage for pulling from Docker daemon /usr/bin/podman --root /tmp/podman-from-docker-KJX7Zt rmi -a -f (node:12371) [DEP0147] DeprecationWarning: In future versions of Node.js, fs.rmdir(path, { recursive: true }) will be removed. Use fs.rm(path, { recursive: true }) instead (Use `node --trace-deprecation ...` to show where the warning was created) Error: podman exited with code 125 Error: writing blob: initiating layer upload to /v2/bioconda/bioconda-utils-build-env-cos7-aarch64/blobs/uploads/ in quay.io: unauthorized: access to the requested resource is not authorized

@johanneskoester Would you mind taking a look? Maybe we need some configure on Quay bioconda repo?

also cc @bgruening

https://github.com/bioconda/bioconda-utils/actions/runs/5471986195/jobs/9963670995 Would you mind taking a look?

Friendly ping!
Could someone please push the new Docker image for aarch64 ? Thank you!

milot-mirdita · 2023-06-16T05:55:57Z

Does this work already?

I would like to make my recipes available for linux arm64. They would need a different cmake flag thought.
Would uname -m be the recommended mechanism to set arm64 specific flags?

Yikun · 2023-06-16T06:36:51Z

Now you can build the aarch64 package in your linux arm64 env locally by below cmd:

bioconda-utils build --docker --packages YOUR_PACKAGE_NAME --docker-base-image ghcr.io/yikun/bioconda-utils-build-env-cos7-aarch64 --force

We still need to support [1][2] before pulish the bioconda packages.

[1] Support mulled test: bioconda/bioconda-containers#55
[2] Offcial publish: bioconda/bioconda-recipes#40550

Would uname -m be the recommended mechanism to set arm64 specific flags?

uname -m is a good way to set linux aarch64 flags. I guess you might also interested on config.guess.

milot-mirdita · 2023-06-16T06:54:42Z

Thanks for the update! I'll wait until the two steps are merged/completed.

In the bioconda#866 we introduced the `--docker-base-image` to help user to specify customized build docker image (such as Linux aarch64 build). But we noticed that it doesn’t work after the bioconda#894 , because a local var introduced and the `docker-base-image` arguments is ignored: https://github.com/bioconda/bioconda-utils/blame/9a85115ae306f58c8b4e65e5f92f6cbdb5b68f04/bioconda_utils/cli.py#L468 This PR try to recover it, if user specify the `--docker-base-image`, it will be used directly otherwise use the local var. Signed-off-by: Yikun Jiang <yikunkero@gmail.com>

In the #866 we introduced the `--docker-base-image` to help user to specify customized build docker image (such as Linux aarch64 build). But we noticed that it doesn’t work after the #894 , because a local var introduced and the `docker-base-image` arguments is ignored. This PR try to recover it, if user specify the `--docker-base-image`, it will be used directly otherwise use the local var. Test: ``` [root@kunpeng bioconda-recipes]# bioconda-utils build --docker --packages bamstats --docker-base-image ghcr.io/yikun/bioconda-utils-build-env-cos7-aarch64 11:50:12 BIOCONDA INFO Considering total of 1 recipes (bamstats). 11:50:12 BIOCONDA INFO Processing 1 recipes (bamstats). 11:50:12 BIOCONDA WARNING Using tag 2.3.4 for docker image, since there is no image for a not yet release version (2.3.4+0.g9a85115.dirty). 11:50:12 BIOCONDA INFO Using docker image ghcr.io/yikun/bioconda-utils-build-env-cos7-aarch64 for building. // ... ... ``` Signed-off-by: Yikun Jiang <yikunkero@gmail.com>

Yikun changed the title ~~Add Linux aarch64 support for bioconda-utils~~ feat: add Linux aarch64 support for bioconda-utils Apr 6, 2023

Yikun changed the title ~~feat: add Linux aarch64 support for bioconda-utils~~ feat: add Linux aarch64/arm64 support for bioconda-utils Apr 6, 2023

Yikun force-pushed the aarch64 branch 8 times, most recently from 67d4efe to c193c38 Compare April 7, 2023 07:31

martin-g reviewed Apr 7, 2023

View reviewed changes

Dockerfile.aarch64 Outdated Show resolved Hide resolved

martin-g reviewed Apr 7, 2023

View reviewed changes

Dockerfile.aarch64 Outdated Show resolved Hide resolved

Yikun force-pushed the aarch64 branch from c193c38 to 7cfb9cf Compare April 10, 2023 01:26

Yikun mentioned this pull request Apr 10, 2023

Support other architectures also supported by conda-forge (e.g. Mac arm64) #706

Closed

Yikun marked this pull request as ready for review April 10, 2023 07:12

Yikun force-pushed the aarch64 branch from 2d66e0a to 21114d6 Compare April 10, 2023 11:34

Yikun force-pushed the aarch64 branch from 21114d6 to 6802b8a Compare April 11, 2023 01:48

martin-g mentioned this pull request Apr 11, 2023

Make base images multiarch bioconda/bioconda-containers#55

Closed

Yikun force-pushed the aarch64 branch from 6802b8a to 9e7b0fd Compare April 11, 2023 09:30

Yikun force-pushed the aarch64 branch 2 times, most recently from ff3869f to 178c720 Compare April 11, 2023 11:35

Add Linux aarch64 support for bioconda-utils

e275d0f

Signed-off-by: Yikun Jiang <yikunkero@gmail.com>

Yikun force-pushed the aarch64 branch from 178c720 to e275d0f Compare April 11, 2023 16:03

johanneskoester mentioned this pull request Apr 21, 2023

feat: add initial support for arm builds bioconda/bioconda-recipes#40550

Merged

7 tasks

martin-g mentioned this pull request Apr 25, 2023

Suggestion: Consider making Aarch64 builts ? bioconda/bioconda-recipes#23454

Closed

johanneskoester requested changes Apr 26, 2023

View reviewed changes

Yikun force-pushed the aarch64 branch from 06c5c51 to 6c2e4e3 Compare April 27, 2023 01:28

Recover release-please.yml condition

f83374d

Signed-off-by: Yikun Jiang <yikunkero@gmail.com>

Yikun force-pushed the aarch64 branch from 6c2e4e3 to f83374d Compare April 27, 2023 01:29

Yikun requested a review from johanneskoester April 27, 2023 11:12

johanneskoester approved these changes Apr 28, 2023

View reviewed changes

johanneskoester merged commit 794ec06 into bioconda:master Apr 28, 2023

julien-faye reviewed Jun 9, 2023

View reviewed changes

Yikun mentioned this pull request Jul 25, 2023

fix: recover --docker-base-image feature #906

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: add Linux aarch64/arm64 support for bioconda-utils #866

feat: add Linux aarch64/arm64 support for bioconda-utils #866

Yikun commented Apr 6, 2023 •

edited

Loading

Yikun commented Apr 10, 2023 •

edited

Loading

bgruening commented Apr 10, 2023

Yikun commented Apr 10, 2023 •

edited

Loading

Yikun commented Apr 11, 2023

bgruening commented Apr 11, 2023

Yikun commented Apr 11, 2023 •

edited

Loading

Yikun commented Apr 13, 2023

Yikun commented Apr 18, 2023 •

edited

Loading

bgruening commented Apr 18, 2023

Yikun commented Apr 18, 2023

martin-g commented Apr 18, 2023

Yikun commented Apr 18, 2023

johanneskoester Apr 26, 2023

Yikun Apr 27, 2023 •

edited

Loading

julien-faye Jun 9, 2023

Yikun Jun 9, 2023

Yikun Jun 9, 2023

Yikun Jul 13, 2023

martin-g Aug 23, 2023

milot-mirdita commented Jun 16, 2023

Yikun commented Jun 16, 2023

milot-mirdita commented Jun 16, 2023

feat: add Linux aarch64/arm64 support for bioconda-utils #866

feat: add Linux aarch64/arm64 support for bioconda-utils #866

Conversation

Yikun commented Apr 6, 2023 • edited Loading

Yikun commented Apr 10, 2023 • edited Loading

bgruening commented Apr 10, 2023

Yikun commented Apr 10, 2023 • edited Loading

Yikun commented Apr 11, 2023

bgruening commented Apr 11, 2023

Yikun commented Apr 11, 2023 • edited Loading

Yikun commented Apr 13, 2023

Yikun commented Apr 18, 2023 • edited Loading

bgruening commented Apr 18, 2023

Yikun commented Apr 18, 2023

martin-g commented Apr 18, 2023

Yikun commented Apr 18, 2023

johanneskoester Apr 26, 2023

Choose a reason for hiding this comment

Yikun Apr 27, 2023 • edited Loading

Choose a reason for hiding this comment

julien-faye Jun 9, 2023

Choose a reason for hiding this comment

Yikun Jun 9, 2023

Choose a reason for hiding this comment

Yikun Jun 9, 2023

Choose a reason for hiding this comment

Yikun Jul 13, 2023

Choose a reason for hiding this comment

martin-g Aug 23, 2023

Choose a reason for hiding this comment

milot-mirdita commented Jun 16, 2023

Yikun commented Jun 16, 2023

milot-mirdita commented Jun 16, 2023

Yikun commented Apr 6, 2023 •

edited

Loading

Yikun commented Apr 10, 2023 •

edited

Loading

Yikun commented Apr 10, 2023 •

edited

Loading

Yikun commented Apr 11, 2023 •

edited

Loading

Yikun commented Apr 18, 2023 •

edited

Loading

Yikun Apr 27, 2023 •

edited

Loading