How to run on Mac

[erarin]

Hello!

I want to install and run Clair3 on my Mac, but run_clair3.sh does not work.
If I want to run on a Mac, which do you recommend, the Intel version or the M1 version?
I am studying on a sequence with Nanopore long reads, but I am not yet familiar to scripts .
So, I would appreciate it if you could give me advice.

Many thanks!

[aquaskyline]

Clair3 runs on Apple Silicon. Clair3 uses the GPU on M1/M2, and it runs fast. 50x HG003 Guppy5 ONT data took around 80 minutes on a 8P2E24G Mac Studio M1 Max. But the installation is not as simple, and it subjects to updates in the future because brew and the conda apple channel both remove old packages from time to time. If you use Clair3 frequently on Mac, I think it is worth the time to configure it.

The instructions are as follows:

Run Clair3 on Mac M1 (MacOS Ventura 13.0.1)

Install prerequisitions

Install brew from https://brew.sh/
brew install parallel
brew install zlib
brew install autoconf
brew install automake
brew install gnu-getopt
brew install bash
echo 'export PATH="/opt/homebrew/opt/gnu-getopt/bin:$PATH"' >> $HOME/.bash_profile

brew install miniforge
conda create -n clair3-arm64 python=3.9
conda activate clair3-arm64

conda install -c apple tensorflow-deps=2.8.0
conda install libdeflate
python -m pip install tensorflow-macos
python -m pip install tensorflow-metal
pip install cffi
pip reinstall numpy==1.23

Get Clair3

cd $HOME
git clone https://github.com/HKU-BAL/Clair3.git
cd Clair3

Install pypy (in $HOME/Clair3)

curl -o pypy3.9-v7.3.8-osx64.tar.bz2 https://downloads.python.org/pypy/pypy3.9-v7.3.8-osx64.tar.bz2
tar -zxf pypy3.9-v7.3.8-osx64.tar.bz2
./pypy3.9-v7.3.8-osx64/bin/pypy -m ensurepip
./pypy3.9-v7.3.8-osx64/bin/pypy -m pip install mpmath==1.2.1

Install samtools (in $HOME/Clair3)

curl -o 1.15.1.tar.gz https://github.com/samtools/samtools/archive/refs/tags/1.15.1.tar.gz
tar -zxf 1.15.1.tar.gz
cd samtools-1.15.1
curl -o htslib-1.15.1.tar.bz2 https://github.com/samtools/htslib/releases/download/1.15.1/htslib-1.15.1.tar.bz2
tar -zxf htslib-1.15.1.tar.bz2
autoheader
autoconf -Wno-syntax
./configure
make -j
cd ../

build Clair3 (in $HOME/Clair3)

python3 build.py
If an error ld: unknown options: -rpath=... shows, comment line extra_link_args = ['-Wl,-rpath={}/lib'.format(conda_path)] in build.py and try again.

Install longphase (in $HOME/Clair3)

wget https://github.com/twolinin/longphase/archive/refs/tags/v1.5.tar.gz
tar -zxf v1.5.tar.gz
cd longphase-1.5/
export CC=/opt/homebrew/opt/llvm/bin/clang
autoreconf -i
./configure
make -j

Download the models (in $HOME/Clair3)

mkdir models
curl -o clair3_models.tar.gz http://www.bio8.cs.hku.hk/clair3/clair3_models/clair3_models.tar.gz
tar -zxf clair3_models.tar.gz -C ./models

Install whatshap

pip install --user whatshap

Clair3 should be ready on M1 by now, when running Clair3 using run_clair3.sh, please always include the following binary paths

--samtools=$HOME/Clair3/samtools-1.15.1/samtools --pypy=$HOME/Clair3/pypy3.9-v7.3.8-osx64/bin/pypy --parallel=/opt/homebrew/bin/parallel --whatshap=$HOME/.local/bin/whatshap --longphase=$HOME/Clair3/longphase-1.5/longphase --python="python"

Before running Clair3, after entering a terminal environment, you need below commands to make the environment ready

export SHELL="/opt/homebrew/bin/bash"
/usr/bin/env bash
source ~/.bash_profile
conda activate clair3-arm64

As an example, after opening a terminal, I ran following commands to get Clair3 started running

export SHELL="/opt/homebrew/bin/bash"
/usr/bin/env bash
source ~/.bash_profile
conda activate clair3-arm64
cd Clair3
./run_clair3.sh --bam_fn=/Volumes/MSE/bam/HG003_GM24143_guppy506_50x.bam --ref_fn=/Volumes/MSE/GRCh38_no_alt_analysis_set.fasta --threads=8 --platform="ont" --model_path=$HOME/Clair3/models/r941_prom_sup_g5014 --output=/Volumes/MSE/clair3Testing/1 --samtools=$HOME/Clair3/samtools-1.15.1/samtools --pypy=$HOME/Clair3/pypy3.9-v7.3.8-osx64/bin/pypy --parallel=/opt/homebrew/bin/parallel --whatshap=$HOME/.local/bin/whatshap --longphase=$HOME/Clair3/longphase-1.5/longphase --python="python"

[erarin]

I greatly appreciate your prompt and detailed reply!
I will try with Apple silicon Mac following your instructions.

[t3kawano]

Recently, I successfully installed Clair3 on an M2 MacBook Pro/macOS 14.5 by following the instructions. During the installation process, I encountered several steps that required tweaks to work properly.

• I am using anaconda, so I did not want to use miniforge to avoid confusion. In conclusion, the instruction steps work with anaconda. I did not run the command brew install miniforge However, I installed llvm with brew install llvm to be able to use clang/llvm.
• curl -o 1.15.1.tar.gz https://github.com/samtools/samtools/archive/refs/tags/1.15.1.tar.gz This command did not work. Instead, I manually downloaded the following file from the Samtools repository and placed it into the folder. https://github.com/samtools/samtools/releases/download/1.15.1/samtools-1.15.1.tar.bz2 Also, this file seems to include htslib-1.15.1, so I did not run curl -o htslib-1.15.1.tar.bz2.
• wget https://github.com/twolinin/longphase/archive/refs/tags/v1.5.tar.gz also did not work, so I manually downloaded and processed it as mentioned above. In this case, https://github.com/twolinin/longphase/archive/refs/tags/v1.5.tar.gz failed to open with tar, so I used the zipped file instead: https://github.com/twolinin/longphase/archive/refs/tags/v1.5.zip.
• pip reinstall numpy==1.23 may be a typo. pip install numpy==1.23 seems to work. However, it caused an error saying: tensorflow 2.16.2 requires numpy<2.0.0,>=1.23.5. So, I installed numpy==1.23.5 additionally.
• As seen above, the version of TensorFlow installed by the instruction command seems high, causing the error: Keras 3 only supports V3 .keras and .weights.h5 files, or legacy V1/V2 .h5 files. To circumvent this, run pip install tf-keras~=2.16 and set the environment variable with export TF_USE_LEGACY_KERAS=1 as described in https://github.com/tensorflow/tensorflow/releases/tag/v2.16.1.
• If I understand correctly, I need to use the --include_all_ctgs option to work with non-human genomes. If this is accurate, it might be useful to include this in the manual.

I don't know if my changes cause problems, but I hope this may helps others trying to install Clair3 on Mac.

[aquaskyline]

Use Mac Docker, enable virtualization, install via Docker.

[t3kawano]

Hi, Thank you for reply.
As you said using docker makes it slow, right?
#312 (comment)
10 hrs for 40MB reads seems not practical in my situation.
In may case, clair3 installed via above way handled a 7GB bam file within 10 minutes.
But if Docker provides reasonable processing time, that would be nice.

[aquaskyline]

After that issue, I however, tried on my M2 Pro with a 30x human genome dataset and got only 20% slower than a powerful Linux server with the same number of cores.

[t3kawano]

That sounds great. I might give Docker a try later. Thank you for the information.

Edit: I tested the Docker version, and it took five times longer to process than the native version in my case. It may depend on how much GPU-dependent calculation is required for your data, because Docker doesn't support Metal Performance Shaders currently.