Building Yari(-sys) on macOS arm64 #67
Comments
|
Hi Nils, thanks for showing interest in helping with this issue. It seems that You can locally delete this |
|
Great, that worked! As the build seems to work for Windows and Linux, I marked the impl block for there OS only and I was able to compile However, I'm running into another issue now, as I want to create the python bindings: |
|
Very nice! I am afraid bare Even though we don't use it yet, you can take inspiration from step in Github Action. Here is the procedure I use for Windows And here are the steps for Linux. |
|
Needed to trick a bit with the library path, but I got it to work: 📦 Built wheel for CPython 3.12 to /<path>/yari/target/wheels/yari_py-0.1.6-cp312-cp312-macosx_11_0_arm64.whl >>> import yari
>>> c = yari.Context()
>>> c.eval("time.now()")
1704194493So, the next step would be finding out how to implement this as Github action? |
|
Excellent! You are right, the next step would be to try to create a GitHub Action that will be able to build portable wheels and publish them as artifacts. Here we need to make sure that the wheels are truly portable and don't depend on something from your system (statically linked). Then the next GitHub Action after that will be able to follow and release those artifacts to PyPI. Your help is greatly appreciated 🙏 . It would be nice to document the procedure you used to setup environment and build the wheels. |
|
As a rough guideline:
diff --git a/yari-sys/build.rs b/yari-sys/build.rs
index 82b58bd..fcbe8c4 100644
--- a/yari-sys/build.rs
+++ b/yari-sys/build.rs
@@ -43,6 +43,15 @@ fn link_linux() {
link_lib("z");
}
+#[cfg(target_os = "macos")]
+fn link_macos() {
+ println!("cargo:rustc-link-lib=static=yara");
+ link_lib("crypto");
+ link_lib("magic");
+ link_lib("jansson");
+ link_lib("z");
+}
+
fn main() {
let ignored_macros = IgnoreMacros(
vec![
@@ -60,6 +69,9 @@ fn main() {
#[cfg(target_os = "linux")]
link_linux();
+ #[cfg(target_os = "macos")]
+ link_macos();
+
#[cfg(target_os = "windows")]
link_windows();
diff --git a/yari-sys/src/bindings.rs b/yari-sys/src/bindings.rs
index 7b5cf34..339ab70 100644
--- a/yari-sys/src/bindings.rs
+++ b/yari-sys/src/bindings.rs
@@ -23,6 +23,19 @@ pub struct SIZED_STRING {
pub c_string: __IncompleteArrayField<::std::os::raw::c_char>,
}
+#[cfg(target_os = "linux")]
+impl Debug for YR_SCAN_CONTEXT {
+ fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::result::Result<(), std::fmt::Error> {
+ // Basic implementation just to satisfy derive Debug
+ // We don't need all of the fields for now
+ f.debug_struct("YR_SCAN_CONTEXT")
+ .field("file_size", &self.file_size)
+ .field("entry_point", &self.entry_point)
+ .finish()
+ }
+}
+
+#[cfg(target_os = "windows")]
impl Debug for YR_SCAN_CONTEXT {
fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::result::Result<(), std::fmt::Error> {
// Basic implementation just to satisfy derive Debug
#!/usr/bin/env bash
set -e
# Envs
export CFLAGS="-I/opt/homebrew/include $CFLAGS"
export LDFLAGS="-L/opt/homebrew/lib $LDFLAGS"
export LIBRARY_PATH="/opt/homebrew/lib:$LIBRARY_PATH"
pushd yari-sys/yara
./bootstrap.sh
CFLAGS="-fPIC $CFLAGS" ./configure --enable-debug --disable-shared --enable-static --enable-magic --enable-dotnet --with-crypto
make clean && make
popd
pushd yari-sys
cargo build
popd
pushd yari-cli
cargo build
popd
pushd yari-py
maturin build -i python3.12
popd
I tried creating a naive Github Actions implementation in https://github.com/3c7/yari/blob/feature/macos_arm64/.github/workflows/test-macos.yml and then realized you're not able to build on Apple Silicon for free. 😬 Update: |
That's a bummer :( I was not aware of that, maybe we have something enabled in https://github.com/avast organization. I will be able to try it tomorrow.
Yup there is one version of the bindings.rs that is manually generated. You can find it here. I usually generate them just by coping the bindings.rs that is generated during the compilation. In this case maybe we will need to have two different versions. The reason for pre-generated bindings.rs is that not all systems where we use YARI have recent enough clang and LLVM (bindgen dependencies) to be able to generate bindings. |
|
But maybe another good news: example run for macos intel x64 where everything is working: https://github.com/3c7/yari/actions/runs/7388115313 done with this workflow: name: Rust
on:
push:
branches: ["feature/macos_arm64"]
env:
CARGO_TERM_COLOR: always
jobs:
build:
runs-on: macos-latest
steps:
- name: Checkout sources
uses: actions/checkout@v3
with:
submodules: 'true'
- name: Install Dependencies
run: brew install autoconf automake libtool pkg-config jansson libmagic openssl python@3.10 python@3.11 python@3.12
- name: Build Yara
run: cd yari-sys/yara && ./bootstrap.sh && CFLAGS="-fPIC -I$(brew --prefix)/include" LDFLAGS="-L$(brew --prefix)/lib $LDFLAGS" LIBRARY_PATH="$(brew --prefix)/lib:$LIBRARY_PATH" ./configure --enable-debug --disable-shared --enable-static --enable-cuckoo --enable-magic --enable-dotnet --with-crypto && make
- name: Build Yari (yari-sys)
run: cd yari-sys && CFLAGS="-I$(brew --prefix)/include" LDFLAGS="-L$(brew --prefix)/lib $LDFLAGS" LIBRARY_PATH="$(brew --prefix)/lib:$LIBRARY_PATH" cargo build --verbose --release
- name: Build Yari (yari-cli)
run: cd yari-cli && CFLAGS="-I$(brew --prefix)/include" LDFLAGS="-L$(brew --prefix)/lib $LDFLAGS" LIBRARY_PATH="$(brew --prefix)/lib:$LIBRARY_PATH" cargo build --verbose --release
- name: Build Yari (yari-py)
run: cd yari-py && python3.12 -mvenv venv-maturin && venv-maturin/bin/pip install maturin && CFLAGS="-I$(brew --prefix)/include" LDFLAGS="-L$(brew --prefix)/lib $LDFLAGS" LIBRARY_PATH="$(brew --prefix)/lib:$LIBRARY_PATH" venv-maturin/bin/maturin build -i python3.10 -i python3.11 -i python3.12
- name: Run Rust tests
run: CFLAGS="-I$(brew --prefix)/include" LDFLAGS="-L$(brew --prefix)/lib $LDFLAGS" LIBRARY_PATH="$(brew --prefix)/lib:$LIBRARY_PATH" cargo test --verbose |
|
Including all test from the original workflow now. In order to implement the test with bundled bindings, I added the following workaround and added a static binding file from another run: diff --git a/yari-sys/build.rs b/yari-sys/build.rs
index fcbe8c4..c09c5f1 100644
--- a/yari-sys/build.rs
+++ b/yari-sys/build.rs
@@ -131,7 +131,12 @@ fn main() {
let out_dir = env::var("OUT_DIR").unwrap();
let out_path = PathBuf::from(out_dir).join("bindings.rs");
if use_bundled_bindings.is_some() {
+ #[allow(unused_variables)]
let binding_file = "bindings-unix.rs";
+
+ // Use different bindings on macOS.
+ #[cfg(target_os = "macos")]
+ let binding_file = "bindings-macos.rs";
fs::copy(PathBuf::from("bindings").join(binding_file), out_path)
.expect("Could not copy bindings to output directory");
} else if let Some(bindings_file) = option_env!("YARI_BINDINGS_FILE") {Job: https://github.com/3c7/yari/actions/runs/7388404344/job/20099169664 |
|
Played around a bit and hosted a macos arm runner myself. Compilation works, however, there are multiple issues with the tests: I assumed that only one test was failing, but with |
TestsIt seems that the majority of tests fail, we just don't see any traceback or anything. The most precise way to debug this issue would be to compile the tests and run them in you will then be able to run the test and There is also one easier option to try to get logs from tests. For this you need to run tests with [...]
pub fn context() -> Context {
+ let _ = env_logger::builder().is_test(true).try_init();
ContextBuilder::default().build().unwrap()
}
[...]This is because we need to initialize logging system in tests (which is currently not done). Also the function RunnersSituation with the runners might be very tricky, I was not able to find if we have access to apple silicon runners or not. One possible solution that comes to my mind is to cross-compile the wheel on x64 to arm. Based on my previous experience with this, I suspect it will be very painful. If this was a pure Rust project than it would be much easier, but we would have to statically link and cross-compile the whole libyara. |
|
gdb seems not to be available for macOS arm64, I just used the easy way with LLDB and VS Codium. :) Unfortunately Lines 743 to 753 in b062de7 I need to recheck the Yara build, however, I'm pretty sure I included all modules there. Update: I probably messed up something. |
|
Yes, indeed I missed building the cuckoo module. Now all tests passing locally and they should also via Github Actions, I don't really know what happened there - Regarding the Github hosted runners for macOS arm64: I also haven't found an overview of runner which you're able to use beside a general overview, however, there is only a single runner for macOS arm64 ( |
That is a very valuable find 👍 , I will try to improve the situation for the future and panic/return a proper error instead of crashing.
The tests that are running are
Sorry, I should have spotted that and link you how to build libyara https://github.com/avast/yari/blob/master/yari-sys/README.md for YARI.
I will check, maybe we have some capacity here in the organization. The current blocker for the cross compilation was that even if we were to generate some python wheel on x64, we had no way of testing if it works on real arm silicon. |
They locally pass, so I assume the runner is a bit... unstable?
Yeah, of course I built the lib but forgot about cuckoo. :)
Understood. If I can help shoot me a message. :) |
|
I've created an experimental brew formula so it's possible to build the yari binaries with I still fight with ruby and the tests. :) |
Hey, I saw #14 and tried to compile the project on my setup, however, I fail quite at the start. I have just a very basic Rust knowledge, so any Rust error make me panic a bit. :)
After compiling Yara, I tried to compile
yari-syswhich leads to the followingbinding.rsrelated error message:I have no idea about bindgen and the code generating the bindings in
build.rs, but can help debugging. I attached the resulting binding.rs.txt as it may helps finding the issue.The text was updated successfully, but these errors were encountered: