Linkage error when cross-compiling for aarch64-pc-windows-msvc
#453
Comments
|
Thanks for the report. I'll try to reproduce this. For reference, we do have a CI job that tests cross-compiling to |
|
To help get some clarity, I'll narrow my focus to just that first error you listed: From what I can tell, this relates to the inability for the linker to resolve the On Windows, we're specifically including I'm not sure this gets us any close to a root-cause, but for |
@justsmth Thanks for the prompt response! I'm aware that you have a CI workflow for this case, and that was the first thing for me to check out when I encountered this error just in case I'd missed something. However, I realized that you really just built the lib. In our original build log, the linkage failure happened way beyond the point of building |
|
A possible lead -- I see some mingw headers with macro definitions that could create a Perhaps our build is picking up a MINGW header? |
|
Could it be related to this thread? https://learn.microsoft.com/en-us/answers/questions/576680/visual-studio-2017-c-linking-error-lnk2019-unresol At least it does look like this has something to do with how the lib handles CRT. |
Indeed, https://stackoverflow.com/a/3007915 and OpenMathLib/OpenBLAS#2825 also talk about |
|
Thanks for your help diagnosing this! I suspect we should be using I'll put the change into this PR: |
|
@justsmth Thanks! Tomorrow I should be able to add a patched dependency to point to the PR branch and see if your changes work for us. |
|
@justsmth I just tried again with the patched dependency and unfortunately the fix didn't work: https://github.com/rust-lang/rustup/actions/runs/9720192943/job/26831234964?pr=3917 And it's almost the same kind of error... I'm confused. I wonder if it's possible to at least try to build the test binary (so in addition to the library itself) for ARM64 Windows and see if the linkage problem can be reproduced there? |
Good idea. I'll try to set this up tomorrow. |
|
I added Please let me know any guidance you might have related to reproducing these build problems locally. I'll work tomorrow on trying to reproduce this and the other ( |
|
@justsmth Thanks anyway for having tried! I still believe there's a CRT conflict somewhere but I'm not sure what kind of conflict it is... I cross-checked our setups for the Windows compilation tests and frankly speaking I cannot find any significant differences... Maybe @ChrisDenton can shed a light on this issue? (x_x) |
|
I've not looked into this yet but I can perhaps give some more information on the MSVC C runtime. There are essentially four mutually exclusive C runtime libraries. Scroll down that page to the table that lists
All libraries and the final binary should use the same CRT, whether they're compiled in C/C++ or Rust. Stable Rust does not currently support the debug CRTs (the ones ending in In nightly you can use, for example, |
|
@ChrisDenton Aha! Thanks for your "Rust" column! I immediately spotted: env:
RUSTFLAGS: -Ctarget-feature=+crt-staticDoes removing this mean our MSVC build can no longer run if our users don't install MSVCRT externally? Anyway, I removed that and most errors are gone (as we must have used I assume this is due to building I think at least I have some ideas about how to get ourselves out of this situation now. So basically this project's build script (when calling
@justsmth Another crucial difference between our setups is that we use the following linker option on Windows MSVC as well: println!("cargo:rustc-link-arg-bin=rustup-init=/WX");This was originally added by @ChrisDenton to prevent |
Yes. It would either need to be installed be the user or else the vcruntime DLL needs to be distributed with the application (Microsoft prefers people do the former). |
@ChrisDenton Okay... That's rather unfortunate since we won't want to make a Rust dev environment even harder to setup... It's pretty hard for newcomers already :'( |
|
Yes, the proper fix would be to use the C/C++ libraries that are built against the same CRT as rustup wants. But, as I say, I've not looked into how they're built or what control rustup has (or not) over this. |
I think this might be the issue. I did see on Windows that |
@justsmth Thanks a lot for your prompt response! Given that the previous
I'd like to help you with the testing on our side if necessary, just ping me back when it's time! |
|
Latest commit: 37884d0
|
|
Somehow that |
@justsmth Seems to work perfectly for Looks like it's the shell being used! |
🤦🏻 Thanks! 😄 |
|
With the It's different that the one originally reported, but that might be due (in part) to the other changes that were made to the build by the PR. |
|
@justsmth It's pretty late at my place now so I'll take a closer look tomorrow. Anyway, I very much appreciate your efforts and I'm so glad to see things moving forward! |
|
... also I'm curious about what will happen if you duplicate a current Windows MSVC test and set That said, I'm glad to see that the |
|
@justsmth Not sure whether it's getting better or not, but with 63bd01b, I'm now getting this with ... looks like we're back with |
Well... today, the good news is that I managed to maintain my sanity. :-) The problem with our aarch64 build relates to our use of Ninja. But unfortunately, I've not been able to get the build to succeed on "aarch64" w/o it. It seems the default "MSBuild" generator doesn't want to invoke "clang-cl" on the assembly files. The build errors look like this: With Ninja the build completes but when the test binaries link to it, a warning/error is generated relating to "defaultlib 'MSVCRTD' conflicts with use of other libs": I think the best path forward would be for us to eliminate our need for Ninja, but a little more digging is needed. |
|
One thing to note is that I attempted to fix the Ninja build by executing vcvarsall.bat to setup the build environment properly: 63bd01b. (This was something needed for our FIPS build.) However, this seemed to not have any impact on the build errors we were getting. |
@justsmth Oh, I'm sorry to hear that 🙇♀️ Thanks again for your hard work so far!
On our side we don't have a specific deadline ahead for shipping a new version with |
|
Thanks! These types of build issues are difficult to reason about logically and the outcome of a change attempt often seems arbitrary, making it a little challenging to one's "sanity". But I'm glad we're sorting this out. The result will be a better experience for our users. 😃 |
I can relate, as I had also force pushed quite a few times on my PR only to see the CI turning red, thinking that might be my own problem, before I finally decided to report this as an issue 🤝
Definitely! Looking forward to it 🚄 |
Signed-off-by: Bob Weinand <bob.weinand@datadoghq.com>
Signed-off-by: Bob Weinand <bob.weinand@datadoghq.com>
* Bump rustls version to 0.23 We were running into symbol issues with older rustls versions, see briansmith/ring#1808. Note that the new rustls version requires nasm to be installed on windows, so we install it in CI. Signed-off-by: Bob Weinand <bob.weinand@datadoghq.com> * Require at least aws-lc-rs 1.8 Signed-off-by: Bob Weinand <bob.weinand@datadoghq.com> * Use ring instead of aws-lc-sys due to aws/aws-lc-rs#453 Signed-off-by: Bob Weinand <bob.weinand@datadoghq.com> * Use ring for windows only, as aws-lc-sys has link issues on windows. But aws-lc-sys is actually preferable. Signed-off-by: Bob Weinand <bob.weinand@datadoghq.com> --------- Signed-off-by: Bob Weinand <bob.weinand@datadoghq.com>
### Issues: * Addresses: aws/aws-lc-rs#453 ### Description of changes: * When building for Windows/ARM64 with the "Visual Studio" generator (instead of Ninja) the "*.S" assembly source files for libraries are ignored. * This change provides a custom command to assemble the require objects files and adds them to the list of sources for the library. ### Call-outs: * See related aws-lc-rs PR: aws/aws-lc-rs#452 ### Testing * I've added CI tests to build for Windows using the "Visual Studio" generator. By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license and the ISC license.
|
What's the current status for aarch64-pc-windows-msvc building on Windows? A colleague of mine wasted a few days on this after rustls was updated to use aws-lc-rs, and now that I'm back from vacation, I wasted a few hours on it myself, only to be even more confused. Can the MSVC clang-llvm component be used for the the build, or does it truly require the clang-llvm installer from llvm.org? Why is it using a different CMake generator based on the CPU target on Windows? It's unclear what kind of environment variables we need to setup to get it working when it's mixed with clang-cl and the Visual Studio environment variables to find the library and include directories, etc. |
I noticed that while investigating what the GitHub Actions workflow did - we're compiling to Windows x64 + arm64 from Windows x64, nothing fancy. At a bare minimum, compilation on Windows using MSVC should be tested, as it is the most common over Linux cross-compilation using clang-cl |
|
I'm sorry about the difficulty with building our package on Windows. We're working to improve the build experience for Windows users. We have a few build fixes to release this week, and several further improvements are planned for the near future. Feel free to reference any of the GitHub CI configurations we have setup:
We've merged a required Windows/ARM64 build fix upstream, and plan to release the fix as part of
We support MSVC's ClangCL (
Going forward, we will support the "Visual Studio"/"MSBuild" generator for (non-FIPS) MSVC builds on Windows x86/x86_64/aarch64 platforms. The issue is (or was) that the "Visual Studio"/"MSBuild" generator on ARM64 ignores the assembly files required for our build. (Other projects have also noted this issue.) We currently require Ninja as CMake's "generator" for Windows/ARM64, but with this issue now addressed in AWS-LC, that requirement will go away with our v1.8.1 release. |
diff --git a/aws-lc-sys/builder/cmake_builder.rs b/aws-lc-sys/builder/cmake_builder.rs
index b4914d1ebc..9eb4c714a2 100644
--- a/aws-lc-sys/builder/cmake_builder.rs
+++ b/aws-lc-sys/builder/cmake_builder.rs
@@ -3,8 +3,8 @@
use crate::OutputLib::{Crypto, RustWrapper, Ssl};
use crate::{
- cargo_env, emit_warning, execute_command, is_no_asm, option_env, target, target_arch,
- target_env, target_os, target_underscored, target_vendor, OutputLibType,
+ cargo_env, emit_warning, execute_command, is_no_asm, is_prebuilt_nasm, option_env, target,
+ target_arch, target_env, target_os, target_underscored, target_vendor, OutputLibType,
};
use std::env;
use std::ffi::OsStr;
@@ -168,6 +168,23 @@ impl CmakeBuilder {
}
}
+ if target_os() == "windows" && target_arch() == "x86_64" && is_prebuilt_nasm() {
+ emit_warning("!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!");
+ emit_warning("!!! Using pre-built NASM binaries !!!");
+ emit_warning("!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!");
+
+ let script_path = self
+ .manifest_dir
+ .join("builder")
+ .join("prebuilt-nasm.bat")
+ .display()
+ .to_string();
+ let script_path = script_path.replace(r"\", "/");
+
+ cmake_cfg.define("CMAKE_ASM_NASM_COMPILER", script_path.as_str());
+ cmake_cfg.define("CMAKE_VERBOSE_MAKEFILE", "1");
+ }
+
if target_underscored() == "aarch64_pc_windows_msvc" {
cmake_cfg.generator("Ninja");
cmake_cfg.define("CMAKE_C_COMPILER", "clang-cl");
@@ -212,7 +229,11 @@ impl crate::Builder for CmakeBuilder {
let mut missing_dependency = false;
if target_os() == "windows" {
- if target_arch() == "x86_64" && !test_nasm_command() && !is_no_asm() {
+ if target_arch() == "x86_64"
+ && !test_nasm_command()
+ && !is_no_asm()
+ && !is_prebuilt_nasm()
+ {
eprintln!(
"Consider setting `AWS_LC_SYS_NO_ASM` in the environment for development builds.\
See User Guide about the limitations: https://aws.github.io/aws-lc-rs/index.html"
diff --git a/aws-lc-sys/builder/main.rs b/aws-lc-sys/builder/main.rs
index 866d2055b8..d9936718a6 100644
--- a/aws-lc-sys/builder/main.rs
+++ b/aws-lc-sys/builder/main.rs
@@ -313,6 +313,7 @@ static mut AWS_LC_SYS_NO_PREFIX: bool = false;
static mut AWS_LC_SYS_INTERNAL_BINDGEN: bool = false;
static mut AWS_LC_SYS_EXTERNAL_BINDGEN: bool = false;
static mut AWS_LC_SYS_NO_ASM: bool = false;
+static mut AWS_LC_SYS_PREBUILT_NASM: bool = false;
fn initialize() {
unsafe {
@@ -322,6 +323,7 @@ fn initialize() {
AWS_LC_SYS_EXTERNAL_BINDGEN =
env_var_to_bool("AWS_LC_SYS_EXTERNAL_BINDGEN").unwrap_or(false);
AWS_LC_SYS_NO_ASM = env_var_to_bool("AWS_LC_SYS_NO_ASM").unwrap_or(false);
+ AWS_LC_SYS_PREBUILT_NASM = env_var_to_bool("AWS_LC_SYS_PREBUILT_NASM").unwrap_or(false);
}
if !is_external_bindgen() && (is_internal_bindgen() || !has_bindgen_feature()) {
@@ -369,6 +371,10 @@ fn is_no_asm() -> bool {
unsafe { AWS_LC_SYS_NO_ASM }
}
+fn is_prebuilt_nasm() -> bool {
+ unsafe { AWS_LC_SYS_PREBUILT_NASM }
+}
+
fn has_bindgen_feature() -> bool {
cfg!(feature = "bindgen")
}
diff --git a/aws-lc-sys/builder/prebuilt-nasm.bat b/aws-lc-sys/builder/prebuilt-nasm.bat
new file mode 100644
index 0000000000..9c761db12a
--- /dev/null
+++ b/aws-lc-sys/builder/prebuilt-nasm.bat
@@ -0,0 +1,21 @@
+@echo off
+set "ScriptDir=%~dp0"
+set "ScriptDir=%ScriptDir:~0,-1%"
+:loop
+set "arg1=%~1"
+if "%arg1%"=="-o" goto end
+if "%arg1%"=="" goto failure
+shift
+goto loop
+:end
+shift
+set "path=%~1"
+for %%f in ("%path%") do set "filename=%%~nxf"
+copy "%ScriptDir%\prebuilt-nasm\%filename%" "%path%"
+exit 0
+
+:failure
+echo PATH: %path% 1>&2
+echo FILENAME: %filename% 1>&2
+echo ScriptDir: %ScriptDir% 1>&2
+exit 1
\ No newline at end of file
diff --git a/aws-lc-sys/builder/prebuilt-nasm/aes128gcmsiv-x86_64.obj b/aws-lc-sys/builder/prebuilt-nasm/aes128gcmsiv-x86_64.obj
new file mode 100644
index 0000000000..3caa1c64f7
Binary files /dev/null and b/aws-lc-sys/builder/prebuilt-nasm/aes128gcmsiv-x86_64.obj differ
diff --git a/aws-lc-sys/builder/prebuilt-nasm/aesni-gcm-avx512.obj b/aws-lc-sys/builder/prebuilt-nasm/aesni-gcm-avx512.obj
new file mode 100644
index 0000000000..99dff121d2
Binary files /dev/null and b/aws-lc-sys/builder/prebuilt-nasm/aesni-gcm-avx512.obj differ
diff --git a/aws-lc-sys/builder/prebuilt-nasm/aesni-gcm-x86_64.obj b/aws-lc-sys/builder/prebuilt-nasm/aesni-gcm-x86_64.obj
new file mode 100644
index 0000000000..a40c77f839
Binary files /dev/null and b/aws-lc-sys/builder/prebuilt-nasm/aesni-gcm-x86_64.obj differ
diff --git a/aws-lc-sys/builder/prebuilt-nasm/aesni-sha1-x86_64.obj b/aws-lc-sys/builder/prebuilt-nasm/aesni-sha1-x86_64.obj
new file mode 100644
index 0000000000..9b14149bc8
Binary files /dev/null and b/aws-lc-sys/builder/prebuilt-nasm/aesni-sha1-x86_64.obj differ
diff --git a/aws-lc-sys/builder/prebuilt-nasm/aesni-sha256-x86_64.obj b/aws-lc-sys/builder/prebuilt-nasm/aesni-sha256-x86_64.obj
new file mode 100644
index 0000000000..de492cd645
Binary files /dev/null and b/aws-lc-sys/builder/prebuilt-nasm/aesni-sha256-x86_64.obj differ
diff --git a/aws-lc-sys/builder/prebuilt-nasm/aesni-x86_64.obj b/aws-lc-sys/builder/prebuilt-nasm/aesni-x86_64.obj
new file mode 100644
index 0000000000..77ef35b43f
Binary files /dev/null and b/aws-lc-sys/builder/prebuilt-nasm/aesni-x86_64.obj differ
diff --git a/aws-lc-sys/builder/prebuilt-nasm/aesni-xts-avx512.obj b/aws-lc-sys/builder/prebuilt-nasm/aesni-xts-avx512.obj
new file mode 100644
index 0000000000..45e257d9ff
Binary files /dev/null and b/aws-lc-sys/builder/prebuilt-nasm/aesni-xts-avx512.obj differ
diff --git a/aws-lc-sys/builder/prebuilt-nasm/chacha-x86_64.obj b/aws-lc-sys/builder/prebuilt-nasm/chacha-x86_64.obj
new file mode 100644
index 0000000000..37a378b223
Binary files /dev/null and b/aws-lc-sys/builder/prebuilt-nasm/chacha-x86_64.obj differ
diff --git a/aws-lc-sys/builder/prebuilt-nasm/chacha20_poly1305_x86_64.obj b/aws-lc-sys/builder/prebuilt-nasm/chacha20_poly1305_x86_64.obj
new file mode 100644
index 0000000000..05cee5a3d1
Binary files /dev/null and b/aws-lc-sys/builder/prebuilt-nasm/chacha20_poly1305_x86_64.obj differ
diff --git a/aws-lc-sys/builder/prebuilt-nasm/ghash-ssse3-x86_64.obj b/aws-lc-sys/builder/prebuilt-nasm/ghash-ssse3-x86_64.obj
new file mode 100644
index 0000000000..42566f60be
Binary files /dev/null and b/aws-lc-sys/builder/prebuilt-nasm/ghash-ssse3-x86_64.obj differ
diff --git a/aws-lc-sys/builder/prebuilt-nasm/ghash-x86_64.obj b/aws-lc-sys/builder/prebuilt-nasm/ghash-x86_64.obj
new file mode 100644
index 0000000000..3706db4563
Binary files /dev/null and b/aws-lc-sys/builder/prebuilt-nasm/ghash-x86_64.obj differ
diff --git a/aws-lc-sys/builder/prebuilt-nasm/md5-x86_64.obj b/aws-lc-sys/builder/prebuilt-nasm/md5-x86_64.obj
new file mode 100644
index 0000000000..6ebd3e7916
Binary files /dev/null and b/aws-lc-sys/builder/prebuilt-nasm/md5-x86_64.obj differ
diff --git a/aws-lc-sys/builder/prebuilt-nasm/p256-x86_64-asm.obj b/aws-lc-sys/builder/prebuilt-nasm/p256-x86_64-asm.obj
new file mode 100644
index 0000000000..ef2ab1e558
Binary files /dev/null and b/aws-lc-sys/builder/prebuilt-nasm/p256-x86_64-asm.obj differ
diff --git a/aws-lc-sys/builder/prebuilt-nasm/p256_beeu-x86_64-asm.obj b/aws-lc-sys/builder/prebuilt-nasm/p256_beeu-x86_64-asm.obj
new file mode 100644
index 0000000000..98b6522f5a
Binary files /dev/null and b/aws-lc-sys/builder/prebuilt-nasm/p256_beeu-x86_64-asm.obj differ
diff --git a/aws-lc-sys/builder/prebuilt-nasm/rdrand-x86_64.obj b/aws-lc-sys/builder/prebuilt-nasm/rdrand-x86_64.obj
new file mode 100644
index 0000000000..25ecc304a0
Binary files /dev/null and b/aws-lc-sys/builder/prebuilt-nasm/rdrand-x86_64.obj differ
diff --git a/aws-lc-sys/builder/prebuilt-nasm/rsaz-avx2.obj b/aws-lc-sys/builder/prebuilt-nasm/rsaz-avx2.obj
new file mode 100644
index 0000000000..39d98ee7b7
Binary files /dev/null and b/aws-lc-sys/builder/prebuilt-nasm/rsaz-avx2.obj differ
diff --git a/aws-lc-sys/builder/prebuilt-nasm/sha1-x86_64.obj b/aws-lc-sys/builder/prebuilt-nasm/sha1-x86_64.obj
new file mode 100644
index 0000000000..28f049ba7f
Binary files /dev/null and b/aws-lc-sys/builder/prebuilt-nasm/sha1-x86_64.obj differ
diff --git a/aws-lc-sys/builder/prebuilt-nasm/sha256-x86_64.obj b/aws-lc-sys/builder/prebuilt-nasm/sha256-x86_64.obj
new file mode 100644
index 0000000000..88fa74e3cb
Binary files /dev/null and b/aws-lc-sys/builder/prebuilt-nasm/sha256-x86_64.obj differ
diff --git a/aws-lc-sys/builder/prebuilt-nasm/sha512-x86_64.obj b/aws-lc-sys/builder/prebuilt-nasm/sha512-x86_64.obj
new file mode 100644
index 0000000000..258d354f4e
Binary files /dev/null and b/aws-lc-sys/builder/prebuilt-nasm/sha512-x86_64.obj differ
diff --git a/aws-lc-sys/builder/prebuilt-nasm/trampoline-x86_64.obj b/aws-lc-sys/builder/prebuilt-nasm/trampoline-x86_64.obj
new file mode 100644
index 0000000000..3370a1073f
Binary files /dev/null and b/aws-lc-sys/builder/prebuilt-nasm/trampoline-x86_64.obj differ
diff --git a/aws-lc-sys/builder/prebuilt-nasm/vpaes-x86_64.obj b/aws-lc-sys/builder/prebuilt-nasm/vpaes-x86_64.obj
new file mode 100644
index 0000000000..1e24b23b01
Binary files /dev/null and b/aws-lc-sys/builder/prebuilt-nasm/vpaes-x86_64.obj differ
diff --git a/aws-lc-sys/builder/prebuilt-nasm/x86_64-mont.obj b/aws-lc-sys/builder/prebuilt-nasm/x86_64-mont.obj
new file mode 100644
index 0000000000..5a841d3ee3
Binary files /dev/null and b/aws-lc-sys/builder/prebuilt-nasm/x86_64-mont.obj differ
diff --git a/aws-lc-sys/builder/prebuilt-nasm/x86_64-mont5.obj b/aws-lc-sys/builder/prebuilt-nasm/x86_64-mont5.obj
new file mode 100644
index 0000000000..135436db3b
Binary files /dev/null and b/aws-lc-sys/builder/prebuilt-nasm/x86_64-mont5.obj differ
diff --git a/aws-lc-sys/builder/cmake_builder.rs b/aws-lc-sys/builder/cmake_builder.rs
index ce6dec782a..a769d68da1 100644
--- a/aws-lc-sys/builder/cmake_builder.rs
+++ b/aws-lc-sys/builder/cmake_builder.rs
@@ -3,7 +3,7 @@
use crate::OutputLib::{Crypto, RustWrapper, Ssl};
use crate::{
- cargo_env, emit_warning, execute_command, is_crt_static, is_no_asm, option_env, target,
+ cargo_env, emit_warning, execute_command, is_crt_static, is_no_asm, is_prebuilt_nasm, option_env, target,
target_arch, target_env, target_os, target_underscored, target_vendor, OutputLibType,
};
use std::env;
@@ -135,7 +135,7 @@ impl CmakeBuilder {
// See issue: aws#453
if target_os() == "windows" {
- Self::configure_windows(&mut cmake_cfg);
+ self.configure_windows(&mut cmake_cfg);
}
// If the build environment vendor is Apple
@@ -162,13 +162,30 @@ impl CmakeBuilder {
}
if target_env() == "ohos" {
- Self::configure_open_harmony(&mut cmake_cfg);
+ self.configure_open_harmony(&mut cmake_cfg);
}
cmake_cfg
}
- fn configure_windows(cmake_cfg: &mut cmake::Config) {
+ fn configure_windows(&self, cmake_cfg: &mut cmake::Config) {
+ if target_arch() == "x86_64" && is_prebuilt_nasm() {
+ emit_warning("!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!");
+ emit_warning("!!! Using pre-built NASM binaries !!!");
+ emit_warning("!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!");
+
+ let script_path = self
+ .manifest_dir
+ .join("builder")
+ .join("prebuilt-nasm.bat")
+ .display()
+ .to_string();
+ let script_path = script_path.replace(r"\", "/");
+
+ cmake_cfg.define("CMAKE_ASM_NASM_COMPILER", script_path.as_str());
+ cmake_cfg.define("CMAKE_VERBOSE_MAKEFILE", "1");
+ }
+
match (target_env().as_str(), target_arch().as_str()) {
("msvc", "aarch64") => {
cmake_cfg.generator_toolset(format!(
@@ -200,7 +217,7 @@ impl CmakeBuilder {
}
}
- fn configure_open_harmony(cmake_cfg: &mut cmake::Config) {
+ fn configure_open_harmony(&self, cmake_cfg: &mut cmake::Config) {
const OHOS_NDK_HOME: &str = "OHOS_NDK_HOME";
if let Ok(ndk) = env::var(OHOS_NDK_HOME) {
cmake_cfg.define(
@@ -253,7 +270,11 @@ impl crate::Builder for CmakeBuilder {
let mut missing_dependency = false;
if target_os() == "windows" {
- if target_arch() == "x86_64" && !test_nasm_command() && !is_no_asm() {
+ if target_arch() == "x86_64"
+ && !test_nasm_command()
+ && !is_no_asm()
+ && !is_prebuilt_nasm()
+ {
eprintln!(
"Consider setting `AWS_LC_SYS_NO_ASM` in the environment for development builds.\
See User Guide about the limitations: https://aws.github.io/aws-lc-rs/index.html"
diff --git a/aws-lc-sys/builder/main.rs b/aws-lc-sys/builder/main.rs
index 4bf220138b..1241db16b4 100644
--- a/aws-lc-sys/builder/main.rs
+++ b/aws-lc-sys/builder/main.rs
@@ -313,6 +313,7 @@ static mut AWS_LC_SYS_NO_PREFIX: bool = false;
static mut AWS_LC_SYS_INTERNAL_BINDGEN: bool = false;
static mut AWS_LC_SYS_EXTERNAL_BINDGEN: bool = false;
static mut AWS_LC_SYS_NO_ASM: bool = false;
+static mut AWS_LC_SYS_PREBUILT_NASM: bool = false;
fn initialize() {
unsafe {
@@ -322,6 +323,7 @@ fn initialize() {
AWS_LC_SYS_EXTERNAL_BINDGEN =
env_var_to_bool("AWS_LC_SYS_EXTERNAL_BINDGEN").unwrap_or(false);
AWS_LC_SYS_NO_ASM = env_var_to_bool("AWS_LC_SYS_NO_ASM").unwrap_or(false);
+ AWS_LC_SYS_PREBUILT_NASM = env_var_to_bool("AWS_LC_SYS_PREBUILT_NASM").unwrap_or(false);
}
if !is_external_bindgen() && (is_internal_bindgen() || !has_bindgen_feature()) {
@@ -369,6 +371,10 @@ fn is_no_asm() -> bool {
unsafe { AWS_LC_SYS_NO_ASM }
}
+fn is_prebuilt_nasm() -> bool {
+ unsafe { AWS_LC_SYS_PREBUILT_NASM }
+}
+
fn has_bindgen_feature() -> bool {
cfg!(feature = "bindgen")
}
diff --git a/aws-lc-sys/builder/prebuilt-nasm.bat b/aws-lc-sys/builder/prebuilt-nasm.bat
new file mode 100644
index 0000000000..9c761db12a
--- /dev/null
+++ b/aws-lc-sys/builder/prebuilt-nasm.bat
@@ -0,0 +1,21 @@
+@echo off
+set "ScriptDir=%~dp0"
+set "ScriptDir=%ScriptDir:~0,-1%"
+:loop
+set "arg1=%~1"
+if "%arg1%"=="-o" goto end
+if "%arg1%"=="" goto failure
+shift
+goto loop
+:end
+shift
+set "path=%~1"
+for %%f in ("%path%") do set "filename=%%~nxf"
+copy "%ScriptDir%\prebuilt-nasm\%filename%" "%path%"
+exit 0
+
+:failure
+echo PATH: %path% 1>&2
+echo FILENAME: %filename% 1>&2
+echo ScriptDir: %ScriptDir% 1>&2
+exit 1
\ No newline at end of file
diff --git a/aws-lc-sys/builder/prebuilt-nasm/aes128gcmsiv-x86_64.obj b/aws-lc-sys/builder/prebuilt-nasm/aes128gcmsiv-x86_64.obj
new file mode 100644
index 0000000000..3caa1c64f7
Binary files /dev/null and b/aws-lc-sys/builder/prebuilt-nasm/aes128gcmsiv-x86_64.obj differ
diff --git a/aws-lc-sys/builder/prebuilt-nasm/aesni-gcm-avx512.obj b/aws-lc-sys/builder/prebuilt-nasm/aesni-gcm-avx512.obj
new file mode 100644
index 0000000000..99dff121d2
Binary files /dev/null and b/aws-lc-sys/builder/prebuilt-nasm/aesni-gcm-avx512.obj differ
diff --git a/aws-lc-sys/builder/prebuilt-nasm/aesni-gcm-x86_64.obj b/aws-lc-sys/builder/prebuilt-nasm/aesni-gcm-x86_64.obj
new file mode 100644
index 0000000000..a40c77f839
Binary files /dev/null and b/aws-lc-sys/builder/prebuilt-nasm/aesni-gcm-x86_64.obj differ
diff --git a/aws-lc-sys/builder/prebuilt-nasm/aesni-sha1-x86_64.obj b/aws-lc-sys/builder/prebuilt-nasm/aesni-sha1-x86_64.obj
new file mode 100644
index 0000000000..9b14149bc8
Binary files /dev/null and b/aws-lc-sys/builder/prebuilt-nasm/aesni-sha1-x86_64.obj differ
diff --git a/aws-lc-sys/builder/prebuilt-nasm/aesni-sha256-x86_64.obj b/aws-lc-sys/builder/prebuilt-nasm/aesni-sha256-x86_64.obj
new file mode 100644
index 0000000000..de492cd645
Binary files /dev/null and b/aws-lc-sys/builder/prebuilt-nasm/aesni-sha256-x86_64.obj differ
diff --git a/aws-lc-sys/builder/prebuilt-nasm/aesni-x86_64.obj b/aws-lc-sys/builder/prebuilt-nasm/aesni-x86_64.obj
new file mode 100644
index 0000000000..77ef35b43f
Binary files /dev/null and b/aws-lc-sys/builder/prebuilt-nasm/aesni-x86_64.obj differ
diff --git a/aws-lc-sys/builder/prebuilt-nasm/aesni-xts-avx512.obj b/aws-lc-sys/builder/prebuilt-nasm/aesni-xts-avx512.obj
new file mode 100644
index 0000000000..45e257d9ff
Binary files /dev/null and b/aws-lc-sys/builder/prebuilt-nasm/aesni-xts-avx512.obj differ
diff --git a/aws-lc-sys/builder/prebuilt-nasm/chacha-x86_64.obj b/aws-lc-sys/builder/prebuilt-nasm/chacha-x86_64.obj
new file mode 100644
index 0000000000..37a378b223
Binary files /dev/null and b/aws-lc-sys/builder/prebuilt-nasm/chacha-x86_64.obj differ
diff --git a/aws-lc-sys/builder/prebuilt-nasm/chacha20_poly1305_x86_64.obj b/aws-lc-sys/builder/prebuilt-nasm/chacha20_poly1305_x86_64.obj
new file mode 100644
index 0000000000..05cee5a3d1
Binary files /dev/null and b/aws-lc-sys/builder/prebuilt-nasm/chacha20_poly1305_x86_64.obj differ
diff --git a/aws-lc-sys/builder/prebuilt-nasm/ghash-ssse3-x86_64.obj b/aws-lc-sys/builder/prebuilt-nasm/ghash-ssse3-x86_64.obj
new file mode 100644
index 0000000000..42566f60be
Binary files /dev/null and b/aws-lc-sys/builder/prebuilt-nasm/ghash-ssse3-x86_64.obj differ
diff --git a/aws-lc-sys/builder/prebuilt-nasm/ghash-x86_64.obj b/aws-lc-sys/builder/prebuilt-nasm/ghash-x86_64.obj
new file mode 100644
index 0000000000..3706db4563
Binary files /dev/null and b/aws-lc-sys/builder/prebuilt-nasm/ghash-x86_64.obj differ
diff --git a/aws-lc-sys/builder/prebuilt-nasm/md5-x86_64.obj b/aws-lc-sys/builder/prebuilt-nasm/md5-x86_64.obj
new file mode 100644
index 0000000000..6ebd3e7916
Binary files /dev/null and b/aws-lc-sys/builder/prebuilt-nasm/md5-x86_64.obj differ
diff --git a/aws-lc-sys/builder/prebuilt-nasm/p256-x86_64-asm.obj b/aws-lc-sys/builder/prebuilt-nasm/p256-x86_64-asm.obj
new file mode 100644
index 0000000000..ef2ab1e558
Binary files /dev/null and b/aws-lc-sys/builder/prebuilt-nasm/p256-x86_64-asm.obj differ
diff --git a/aws-lc-sys/builder/prebuilt-nasm/p256_beeu-x86_64-asm.obj b/aws-lc-sys/builder/prebuilt-nasm/p256_beeu-x86_64-asm.obj
new file mode 100644
index 0000000000..98b6522f5a
Binary files /dev/null and b/aws-lc-sys/builder/prebuilt-nasm/p256_beeu-x86_64-asm.obj differ
diff --git a/aws-lc-sys/builder/prebuilt-nasm/rdrand-x86_64.obj b/aws-lc-sys/builder/prebuilt-nasm/rdrand-x86_64.obj
new file mode 100644
index 0000000000..25ecc304a0
Binary files /dev/null and b/aws-lc-sys/builder/prebuilt-nasm/rdrand-x86_64.obj differ
diff --git a/aws-lc-sys/builder/prebuilt-nasm/rsaz-avx2.obj b/aws-lc-sys/builder/prebuilt-nasm/rsaz-avx2.obj
new file mode 100644
index 0000000000..39d98ee7b7
Binary files /dev/null and b/aws-lc-sys/builder/prebuilt-nasm/rsaz-avx2.obj differ
diff --git a/aws-lc-sys/builder/prebuilt-nasm/sha1-x86_64.obj b/aws-lc-sys/builder/prebuilt-nasm/sha1-x86_64.obj
new file mode 100644
index 0000000000..28f049ba7f
Binary files /dev/null and b/aws-lc-sys/builder/prebuilt-nasm/sha1-x86_64.obj differ
diff --git a/aws-lc-sys/builder/prebuilt-nasm/sha256-x86_64.obj b/aws-lc-sys/builder/prebuilt-nasm/sha256-x86_64.obj
new file mode 100644
index 0000000000..88fa74e3cb
Binary files /dev/null and b/aws-lc-sys/builder/prebuilt-nasm/sha256-x86_64.obj differ
diff --git a/aws-lc-sys/builder/prebuilt-nasm/sha512-x86_64.obj b/aws-lc-sys/builder/prebuilt-nasm/sha512-x86_64.obj
new file mode 100644
index 0000000000..258d354f4e
Binary files /dev/null and b/aws-lc-sys/builder/prebuilt-nasm/sha512-x86_64.obj differ
diff --git a/aws-lc-sys/builder/prebuilt-nasm/trampoline-x86_64.obj b/aws-lc-sys/builder/prebuilt-nasm/trampoline-x86_64.obj
new file mode 100644
index 0000000000..3370a1073f
Binary files /dev/null and b/aws-lc-sys/builder/prebuilt-nasm/trampoline-x86_64.obj differ
diff --git a/aws-lc-sys/builder/prebuilt-nasm/vpaes-x86_64.obj b/aws-lc-sys/builder/prebuilt-nasm/vpaes-x86_64.obj
new file mode 100644
index 0000000000..1e24b23b01
Binary files /dev/null and b/aws-lc-sys/builder/prebuilt-nasm/vpaes-x86_64.obj differ
diff --git a/aws-lc-sys/builder/prebuilt-nasm/x86_64-mont.obj b/aws-lc-sys/builder/prebuilt-nasm/x86_64-mont.obj
new file mode 100644
index 0000000000..5a841d3ee3
Binary files /dev/null and b/aws-lc-sys/builder/prebuilt-nasm/x86_64-mont.obj differ
diff --git a/aws-lc-sys/builder/prebuilt-nasm/x86_64-mont5.obj b/aws-lc-sys/builder/prebuilt-nasm/x86_64-mont5.obj
new file mode 100644
index 0000000000..135436db3b
Binary files /dev/null and b/aws-lc-sys/builder/prebuilt-nasm/x86_64-mont5.obj differ
|
@justsmth thanks for the detailed update. I've spent some more time on this today, hoping to fix the build issue on our side since the switch from ring-rs to aws-lc-rs in rustls. I spent a fair amount of time helping to get ring-rs fixed upstream for Windows on ARM in the past, and there may be some interesting strategies to reuse for build setup simplification. I almost got a build setup working properly with the current release (0.19.0) used by rustls, the one remaining issue is the lack of proper support for RUSTFLAGS="-C target-feature=+crt-static" (MSVC static runtime linking) that doesn't seem to get propagated properly to the aws-lc-sys native component, causing a runtime linker conflict. I believe you may be trying to fix this as we speak, given you recent commits. Let me summarize the steps I took to get the current version of aws-lc-rs to build locally with MSVC on Windows. In Visual Studio, I installed "MSVC v143 - VS 2022 C++ ARM64 build tools (v14,32 -17.2)", "MSVC v143 - VS 2022 C++ ARM64/ARM64EC build tools (latest)", "C++ ATL for latest v143 build tools (ARM64/ARM64EC)", "C++ Clang Compiler for Windows (17.0.3)" and "MSBuild support for LLVM (clang-cl) toolset". There's probably a few of those that aren't needed, but more testing would be required to find the minimum subset. Install ninja, vswhere and nasm:
To get an easier Visual Studio Developer PowerShell environment, I use my own VsDevShell PowerShell module (much less painful to use than the one that comes with Visual Studio):
Clone the current aws-lc-rs repository with the release tag for 0.19.0: Add nasm to the PATH (not added by default, very annoying, even if it's in a very predictable install path): Set LIBCLANG_PATH to the one from Visual Studio, using vswhere.exe: Enter an arm64 developer shell to get all the environment variables for the cmake Ninja generator to build C/C++ programs properly in Windows. Since clang-cl is installed and loaded automatically in the developer shell, it will likely pick it up as the C/C++ compiler, but it may also use it as its assembler program. For ring-rs, relying on clang for building C and assembly for ARM64 was just simpler due to limitations of the MSVC cl.exe and armasm64.exe. Build aws-lc-sys in release for aarch64-pc-windows-msvc, with very verbose output so you can see the CMake commmands: To get to build to fail with the static crt issue, just set $Env:RUSTFLAGS="-C target-feature=+crt-static" before calling cargo build. Can you give an update on the current status for crt-static support? Will it be fixed in the next release? |
|
I’ll be watching this. |
|
@awakecoding |
|
I tested again in https://github.com/rust-lang/rustup/actions/runs/9967623316/job/27541588108 with the cargo patch set to eeac93b. The CI didn't pass completely but at least this issue regarding Windows ARM64 can be safely closed. @justsmth thank you so much for your investigation and hard work so far! I'll see you again very soon in other follow-up issues 😶 |
@awakecoding - Yeah, our latest changes on main have CI tests using /WX both with and w/o "crt-static". Here's a CI workflow where it's being tested: https://github.com/aws/aws-lc-rs/actions/runs/9962674772/job/27527026533?pr=460 |
|
@justsmth do you build shared libraries or executables that link aws-lc-rs into it? MSVC linker errors due to a runtime mismatch will only cause issues then, so the current tests are probably not enough. I managed to backport crt-static build fixed on 0.19.0, but they go beyond what's currently on the main branch, so I suspect the problem wasn't truly fixed, at least not with the Ninja generator. I've been patching CMake build systems for several years to inject annoying code to search and replace the /MD and /MT flags because they're getting set in the initial flags, and those passed through cmake parameters only get added, which still leads to a conflict. That's what I've been hitting locally. CMake 3.15 added the CMAKE_MSVC_RUNTIME_LIBRARY option which takes a CMake generator expression to set the MSVC runtime properly for debug and release builds, but the current CMakeLists.txt uses cmake_minimum_required(VERSION 3.0) at the beginning, and it needs to be changed to 3.15 at a minimum for this to work. I added the following on my patched branch: Yes, MSVC runtime linker options with CMake are a pain... I know. this is just the simplest solution to get things to build, otherwise a pre-3.15 cmake backport requires custom functions to modify all the flag variables and switch them based on a custom cmake option. I then added this to my project Cargo.toml to patch aws-lc-sys 0.19.0 in my builds of sspi-rs: and I finally have working builds again, with all the CI modifications for the environment setup. It's still going to be painful to build for local development now, but hopefully we can reduce reliance on certain things to be preconfigured, and add automatic detection of some tools in known install paths. I also noticed there might be some pre-generated object code options like what ring-rs does? This simplifies the build setup without an assembler installed. For now, I suggest adding the building of an executable or shared library that links aws-lc-rs to properly test for MSVC runtime linker problems. Is this the case right now? |
@awakecoding AFAIK the new CI is building test binaries (included in
But |
yeah, that was my next question - I looked at the cmake-rs code and it handles the MSVC runtime linker options differently for the Visual Studio and Ninja cmake generators, so if it's working upstream with the Visual Studio generator, it means there's an issue with the Ninja generator support and how it was used with 0.19.0. I got something that works with 0.19.0 with my patch, just know about this limitation if you ever switch to Ninja again, or offer the option to use it instead of the Visual Studio generator. Even after aws-lc-sys 0.20.0 is released, we'll need to wait for Rustls to update, which is why backporting fixes to 0.19.0 was the best solution to get our sspi-rs builds back on track. |
Update CI
diff --git a/.github/workflows/sys-bindings-generator.yml b/.github/workflows/sys-bindings-generator.yml
index 3fba893d80..dc226f9f1b 100644
--- a/.github/workflows/sys-bindings-generator.yml
+++ b/.github/workflows/sys-bindings-generator.yml
@@ -270,3 +270,36 @@ jobs:
run: ./scripts/build/collect_build_src.sh -t ${{ matrix.target }}
- name: Commit & Push changes
run: ./scripts/ci/ci_add_commit_rebase_push.sh "Collected source files for ${{ matrix.target }}"
+ collect-nasm-and-commit:
+ needs: generate-windows-bindings-and-commit
+ if: github.repository == 'aws/aws-lc-rs'
+ runs-on: windows-latest
+ steps:
+ - uses: actions/checkout@v4
+ with:
+ submodules: 'recursive'
+ ref: ${{ github.ref_name }}
+ - uses: dtolnay/rust-toolchain@master
+ id: toolchain
+ with:
+ toolchain: stable
+ targets: "x86_64-pc-windows-msvc,x86_64-pc-windows-gnu"
+ - uses: ilammy/setup-nasm@v1
+ - name: Build aws-lc-sys
+ shell: bash
+ run: AWS_LC_SYS_PREBUILT_NASM=0 cargo build -p aws-lc-sys --release --target x86_64-pc-windows-msvc
+ - name: Collect NASM object files
+ shell: bash
+ run: ./scripts/build/collect_nasm_obj.sh
+ - name: Clean build
+ shell: bash
+ run: cargo clean
+ - name: Test aws-lc-rs for x86_64-pc-windows-msvc
+ shell: bash
+ run: AWS_LC_SYS_PREBUILT_NASM=1 cargo build -p aws-lc-sys --target x86_64-pc-windows-msvc
+ - name: Test aws-lc-sys for x86_64-pc-windows-gnu
+ shell: bash
+ run: AWS_LC_SYS_PREBUILT_NASM=1 cargo build -p aws-lc-sys --target x86_64-pc-windows-gnu
+ - name: Commit & Push changes
+ shell: bash
+ run: ./scripts/ci/ci_add_commit_rebase_push.sh "Collected NASM files"
diff --git a/.github/workflows/tests.yml b/.github/workflows/tests.yml
index a0da3f48b8..71716a64c0 100644
--- a/.github/workflows/tests.yml
+++ b/.github/workflows/tests.yml
@@ -37,7 +37,7 @@ jobs:
- --no-default-features --features non-fips,ring-sig-verify,unstable
- --no-default-features --features non-fips,alloc,unstable
steps:
- - uses: actions/checkout@v3
+ - uses: actions/checkout@v4
with:
submodules: 'recursive'
- uses: dtolnay/rust-toolchain@master
@@ -66,7 +66,7 @@ jobs:
- --no-default-features --features aws-lc-sys,bindgen,unstable
- --release --all-targets --features bindgen,unstable
steps:
- - uses: actions/checkout@v3
+ - uses: actions/checkout@v4
with:
submodules: 'recursive'
- if: ${{ matrix.os == 'macos-13-xlarge' }}
@@ -106,9 +106,10 @@ jobs:
- --no-default-features --features non-fips,ring-io,unstable
- --no-default-features --features non-fips,ring-sig-verify,unstable
- --no-default-features --features non-fips,alloc,unstable
+ env:
+ AWS_LC_SYS_PREBUILT_NASM: 1
steps:
- - uses: ilammy/setup-nasm@v1
- - uses: actions/checkout@v3
+ - uses: actions/checkout@v4
with:
submodules: 'recursive'
- uses: dtolnay/rust-toolchain@master
@@ -125,7 +126,7 @@ jobs:
name: aws-ls-rs coverage
runs-on: ubuntu-latest
steps:
- - uses: actions/checkout@v3
+ - uses: actions/checkout@v4
with:
submodules: 'recursive'
lfs: true
@@ -167,7 +168,7 @@ jobs:
- --no-default-features --features fips,asan
runs-on: ubuntu-latest
steps:
- - uses: actions/checkout@v3
+ - uses: actions/checkout@v4
with:
submodules: 'recursive'
- uses: dtolnay/rust-toolchain@master
@@ -196,7 +197,7 @@ jobs:
os: [ ubuntu-latest, macos-12, macos-13-xlarge ]
static: [ 0, 1 ]
steps:
- - uses: actions/checkout@v3
+ - uses: actions/checkout@v4
with:
submodules: 'recursive'
- uses: dtolnay/rust-toolchain@stable
@@ -205,6 +206,40 @@ jobs:
# See: rust-lang/cargo#8531
run: cargo test -p aws-lc-rs --tests
+ build-env-nasm-test:
+ if: github.repository_owner == 'aws'
+ name: prebuilt NASM verification
+ runs-on: windows-latest
+ strategy:
+ fail-fast: false
+ matrix:
+ target:
+ - 'x86_64-pc-windows-msvc'
+ - 'x86_64-pc-windows-gnu'
+ steps:
+ - uses: actions/checkout@v4
+ with:
+ submodules: 'recursive'
+ - uses: dtolnay/rust-toolchain@stable
+ - name: Install NASM
+ uses: ilammy/setup-nasm@v1
+ - name: Remove NASM artifacts
+ shell: bash
+ run: |
+ cargo clean
+ rm ./aws-lc-sys/builder/prebuilt-nasm/*
+ - name: Run cargo test
+ shell: bash
+ run: AWS_LC_SYS_PREBUILT_NASM=0 cargo test --tests -p aws-lc-rs --release --no-default-features --features aws-lc-sys
+ - name: Collect NASM outputs
+ shell: bash
+ run: ./scripts/build/collect_nasm_obj.sh
+ - name: Flag any NASM changes
+ shell: bash
+ run: |
+ git add .
+ git diff --cached --exit-code HEAD -- aws-lc-sys/builder/prebuilt-nasm/*.txt
+
build-env-external-bindgen-test:
if: github.repository_owner == 'aws'
name: aws-lc-rs FIPS - External bindgen test
@@ -216,14 +251,14 @@ jobs:
matrix:
os: [ ubuntu-latest, macos-12, macos-13-xlarge, windows-latest ]
steps:
- - if: ${{ matrix.os == 'windows-latest' }}
- uses: ilammy/setup-nasm@v1
- - uses: actions/checkout@v3
+ - uses: actions/checkout@v4
with:
submodules: 'recursive'
- uses: dtolnay/rust-toolchain@stable
- - name: Install ninja-build tool
- uses: seanmiddleditch/gha-setup-ninja@v4
+ - if: ${{ matrix.os == 'windows-latest' }}
+ uses: ilammy/setup-nasm@v1
+ - if: ${{ matrix.os == 'windows-latest' }}
+ uses: seanmiddleditch/gha-setup-ninja@v5
- name: Install bindgen-cli
run: cargo install --locked bindgen-cli
- name: Remove bindings
@@ -245,12 +280,10 @@ jobs:
os: [ ubuntu-latest, macos-12, macos-13-xlarge ]
static: [ 0, 1 ]
steps:
- - uses: actions/checkout@v3
+ - uses: actions/checkout@v4
with:
submodules: 'recursive'
- uses: dtolnay/rust-toolchain@stable
- - name: Install ninja-build tool
- uses: seanmiddleditch/gha-setup-ninja@v4
- uses: actions/setup-go@v4
with:
go-version: '>=1.18'
@@ -271,7 +304,7 @@ jobs:
matrix:
os: [ ubuntu-latest, macos-12, macos-13-xlarge, windows-latest ]
steps:
- - uses: actions/checkout@v3
+ - uses: actions/checkout@v4
with:
submodules: 'recursive'
- uses: dtolnay/rust-toolchain@stable
@@ -306,15 +339,14 @@ jobs:
matrix:
os: [ ubuntu-latest, macos-12, macos-13-xlarge, windows-latest ]
steps:
- - uses: actions/checkout@v3
+ - uses: actions/checkout@v4
with:
submodules: 'recursive'
- uses: dtolnay/rust-toolchain@stable
- - name: Install ninja-build tool
- uses: seanmiddleditch/gha-setup-ninja@v4
- uses: actions/setup-go@v4
with:
go-version: '>=1.18'
+ - uses: seanmiddleditch/gha-setup-ninja@v5
- name: Run cargo test
run: cargo test -p aws-lc-rs --tests --no-default-features --features fips
- name: Release build
@@ -346,14 +378,14 @@ jobs:
matrix:
os: [ ubuntu-latest, macos-12, macos-13-xlarge, windows-latest ]
steps:
- - if: ${{ matrix.os == 'windows-latest' }}
- uses: ilammy/setup-nasm@v1
- - uses: actions/checkout@v3
+ - uses: actions/checkout@v4
with:
submodules: 'recursive'
+ - if: ${{ matrix.os == 'windows-latest' }}
+ uses: ilammy/setup-nasm@v1
+ - if: ${{ matrix.os == 'windows-latest' }}
+ uses: seanmiddleditch/gha-setup-ninja@v5
- uses: dtolnay/rust-toolchain@stable
- - name: Install ninja-build tool
- uses: seanmiddleditch/gha-setup-ninja@v4
- uses: actions/setup-go@v4
with:
go-version: '>=1.18'
@@ -378,7 +410,7 @@ jobs:
- macos-12
- macos-13-xlarge
steps:
- - uses: actions/checkout@v3
+ - uses: actions/checkout@v4
with:
submodules: 'recursive'
lfs: true
diff --git a/aws-lc-sys/builder/cmake_builder.rs b/aws-lc-sys/builder/cmake_builder.rs
index e25cecb443..c5069c5341 100644
--- a/aws-lc-sys/builder/cmake_builder.rs
+++ b/aws-lc-sys/builder/cmake_builder.rs
@@ -3,9 +3,9 @@
use crate::OutputLib::{Crypto, RustWrapper, Ssl};
use crate::{
- cargo_env, emit_warning, execute_command, is_crt_static, is_no_asm, option_env, target,
- target_arch, target_env, target_family, target_os, target_underscored, target_vendor,
- OutputLibType,
+ allow_prebuilt_nasm, cargo_env, emit_warning, execute_command, is_crt_static, is_no_asm,
+ option_env, target, target_arch, target_env, target_family, target_os, target_underscored,
+ target_vendor, test_nasm_command, OutputLibType,
};
use std::env;
use std::ffi::OsString;
@@ -22,10 +22,6 @@ fn test_clang_cl_command() -> bool {
execute_command("clang-cl".as_ref(), &["--version".as_ref()]).status
}
-fn test_nasm_command() -> bool {
- execute_command("nasm".as_ref(), &["-version".as_ref()]).status
-}
-
fn find_cmake_command() -> Option<OsString> {
if let Some(cmake) = option_env("CMAKE") {
emit_warning(&format!(
@@ -162,7 +158,7 @@ impl CmakeBuilder {
// See issue: aws#453
if target_os() == "windows" {
- Self::configure_windows(&mut cmake_cfg);
+ self.configure_windows(&mut cmake_cfg);
}
// If the build environment vendor is Apple
@@ -213,7 +209,7 @@ impl CmakeBuilder {
cmake_cfg
}
- fn configure_windows(cmake_cfg: &mut cmake::Config) {
+ fn configure_windows(&self, cmake_cfg: &mut cmake::Config) {
match (target_env().as_str(), target_arch().as_str()) {
("msvc", "aarch64") => {
cmake_cfg.generator_toolset(format!(
@@ -243,6 +239,22 @@ impl CmakeBuilder {
}
_ => {}
}
+ if target_arch() == "x86_64" && Some(true) == allow_prebuilt_nasm() {
+ emit_warning("!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!");
+ emit_warning("!!! Using pre-built NASM binaries !!!");
+ emit_warning("!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!");
+
+ let script_path = self
+ .manifest_dir
+ .join("builder")
+ .join("prebuilt-nasm.bat")
+ .display()
+ .to_string();
+ let script_path = script_path.replace('\\', "/");
+
+ cmake_cfg.define("CMAKE_ASM_NASM_COMPILER", script_path.as_str());
+ cmake_cfg.define("CMAKE_VERBOSE_MAKEFILE", "1");
+ }
}
fn configure_open_harmony(cmake_cfg: &mut cmake::Config) {
@@ -298,10 +310,14 @@ impl crate::Builder for CmakeBuilder {
let mut missing_dependency = false;
if target_os() == "windows" {
- if target_arch() == "x86_64" && !test_nasm_command() && !is_no_asm() {
+ if target_arch() == "x86_64"
+ && !is_no_asm()
+ && !test_nasm_command()
+ && Some(true) != allow_prebuilt_nasm()
+ {
eprintln!(
- "Consider setting `AWS_LC_SYS_NO_ASM` in the environment for development builds.\
- See User Guide about the limitations: https://aws.github.io/aws-lc-rs/index.html"
+ "Consider setting `AWS_LC_SYS_PREBUILT_NASM` in the build environment.\
+ See User Guide: https://aws.github.io/aws-lc-rs/index.html"
);
eprintln!("Missing dependency: nasm");
missing_dependency = true;
diff --git a/aws-lc-sys/builder/main.rs b/aws-lc-sys/builder/main.rs
index 23f0214991..987e1038ca 100644
--- a/aws-lc-sys/builder/main.rs
+++ b/aws-lc-sys/builder/main.rs
@@ -319,6 +319,7 @@ static mut AWS_LC_SYS_NO_PREFIX: bool = false;
static mut AWS_LC_SYS_INTERNAL_BINDGEN: bool = false;
static mut AWS_LC_SYS_EXTERNAL_BINDGEN: bool = false;
static mut AWS_LC_SYS_NO_ASM: bool = false;
+static mut AWS_LC_SYS_PREBUILT_NASM: Option<bool> = None;
fn initialize() {
unsafe {
@@ -328,6 +329,7 @@ fn initialize() {
AWS_LC_SYS_EXTERNAL_BINDGEN =
env_var_to_bool("AWS_LC_SYS_EXTERNAL_BINDGEN").unwrap_or(false);
AWS_LC_SYS_NO_ASM = env_var_to_bool("AWS_LC_SYS_NO_ASM").unwrap_or(false);
+ AWS_LC_SYS_PREBUILT_NASM = env_var_to_bool("AWS_LC_SYS_PREBUILT_NASM");
}
if !is_external_bindgen() && (is_internal_bindgen() || !has_bindgen_feature()) {
@@ -363,6 +365,7 @@ fn is_bindgen_required() -> bool {
|| !has_pregenerated()
}
+#[allow(dead_code)]
fn internal_bindgen_supported() -> bool {
// TODO: internal bindgen creates invalid bindings on FreeBSD
// See: aws#476
@@ -385,6 +388,10 @@ fn is_no_asm() -> bool {
unsafe { AWS_LC_SYS_NO_ASM }
}
+fn allow_prebuilt_nasm() -> Option<bool> {
+ unsafe { AWS_LC_SYS_PREBUILT_NASM }
+}
+
fn has_bindgen_feature() -> bool {
cfg!(feature = "bindgen")
}
@@ -393,6 +400,10 @@ fn has_pregenerated() -> bool {
unsafe { PREGENERATED }
}
+fn test_nasm_command() -> bool {
+ execute_command("nasm".as_ref(), &["-version".as_ref()]).status
+}
+
fn prepare_cargo_cfg() {
// This is supported in Rust >= 1.77.0
// Also remove `#![allow(unexpected_cfgs)]` from src/lib.rs
diff --git a/aws-lc-sys/builder/prebuilt-nasm.bat b/aws-lc-sys/builder/prebuilt-nasm.bat
new file mode 100644
index 0000000000..9c761db12a
--- /dev/null
+++ b/aws-lc-sys/builder/prebuilt-nasm.bat
@@ -0,0 +1,21 @@
+@echo off
+set "ScriptDir=%~dp0"
+set "ScriptDir=%ScriptDir:~0,-1%"
+:loop
+set "arg1=%~1"
+if "%arg1%"=="-o" goto end
+if "%arg1%"=="" goto failure
+shift
+goto loop
+:end
+shift
+set "path=%~1"
+for %%f in ("%path%") do set "filename=%%~nxf"
+copy "%ScriptDir%\prebuilt-nasm\%filename%" "%path%"
+exit 0
+
+:failure
+echo PATH: %path% 1>&2
+echo FILENAME: %filename% 1>&2
+echo ScriptDir: %ScriptDir% 1>&2
+exit 1
\ No newline at end of file
diff --git a/scripts/build/collect_nasm_obj.sh b/scripts/build/collect_nasm_obj.sh
new file mode 100644
index 0000000000..c425e9236f
--- /dev/null
+++ b/scripts/build/collect_nasm_obj.sh
@@ -0,0 +1,29 @@
+#!/usr/bin/env bash
+# Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved.
+# SPDX-License-Identifier: Apache-2.0 OR ISC
+
+set -ex
+set -o pipefail
+
+if [[ ${BASH_VERSINFO[0]} -lt 4 ]]; then
+ echo Must use bash 4 or later: ${BASH_VERSION}
+ exit 1
+fi
+
+SCRIPT_DIR=$(cd -- "$( dirname -- "${BASH_SOURCE[0]}" )" &> /dev/null && pwd)
+REPO_ROOT=$(git rev-parse --show-toplevel)
+SYS_CRATE_DIR="${REPO_ROOT}/aws-lc-sys"
+PREBUILT_NASM_DIR="${SYS_CRATE_DIR}/builder/prebuilt-nasm"
+mkdir -p "${PREBUILT_NASM_DIR}"
+rm -f "${PREBUILT_NASM_DIR}"/*
+
+DUMPBIN="$(find /c/Program\ Files/Microsoft\ Visual\ Studio/ -path "*/Hostx64/x64/*" -name "dumpbin.exe" -print -quit)"
+
+for nasm_file in `find aws-lc-sys/aws-lc/generated-src/win-x86_64/ -name "*.asm"`; do
+ OBJNAME=$(basename "${nasm_file}");
+ NASM_OBJ=$(find target/ -name "${OBJNAME/.asm/.obj}");
+ cp "${NASM_OBJ}" "${PREBUILT_NASM_DIR}"
+ # We remove the '.debug$S' value, which indicates the size of the debug section. This value can change across builds
+ # because it typically contains full source file paths that vary by build environment
+ "${DUMPBIN}" //DISASM "${PREBUILT_NASM_DIR}"/"${OBJNAME/.asm/.obj}" | grep -v '.debug$S' | sed -e "s/^Dump of file.*/Dump of file ${OBJNAME/.asm/.obj}/" > "${PREBUILT_NASM_DIR}"/"${OBJNAME/.asm/}"-disasm.txt
+done
Update CI
diff --git a/.github/workflows/sys-bindings-generator.yml b/.github/workflows/sys-bindings-generator.yml
index 3fba893d80..dc226f9f1b 100644
--- a/.github/workflows/sys-bindings-generator.yml
+++ b/.github/workflows/sys-bindings-generator.yml
@@ -270,3 +270,36 @@ jobs:
run: ./scripts/build/collect_build_src.sh -t ${{ matrix.target }}
- name: Commit & Push changes
run: ./scripts/ci/ci_add_commit_rebase_push.sh "Collected source files for ${{ matrix.target }}"
+ collect-nasm-and-commit:
+ needs: generate-windows-bindings-and-commit
+ if: github.repository == 'aws/aws-lc-rs'
+ runs-on: windows-latest
+ steps:
+ - uses: actions/checkout@v4
+ with:
+ submodules: 'recursive'
+ ref: ${{ github.ref_name }}
+ - uses: dtolnay/rust-toolchain@master
+ id: toolchain
+ with:
+ toolchain: stable
+ targets: "x86_64-pc-windows-msvc,x86_64-pc-windows-gnu"
+ - uses: ilammy/setup-nasm@v1
+ - name: Build aws-lc-sys
+ shell: bash
+ run: AWS_LC_SYS_PREBUILT_NASM=0 cargo build -p aws-lc-sys --release --target x86_64-pc-windows-msvc
+ - name: Collect NASM object files
+ shell: bash
+ run: ./scripts/build/collect_nasm_obj.sh
+ - name: Clean build
+ shell: bash
+ run: cargo clean
+ - name: Test aws-lc-rs for x86_64-pc-windows-msvc
+ shell: bash
+ run: AWS_LC_SYS_PREBUILT_NASM=1 cargo build -p aws-lc-sys --target x86_64-pc-windows-msvc
+ - name: Test aws-lc-sys for x86_64-pc-windows-gnu
+ shell: bash
+ run: AWS_LC_SYS_PREBUILT_NASM=1 cargo build -p aws-lc-sys --target x86_64-pc-windows-gnu
+ - name: Commit & Push changes
+ shell: bash
+ run: ./scripts/ci/ci_add_commit_rebase_push.sh "Collected NASM files"
diff --git a/.github/workflows/tests.yml b/.github/workflows/tests.yml
index a0da3f48b8..71716a64c0 100644
--- a/.github/workflows/tests.yml
+++ b/.github/workflows/tests.yml
@@ -37,7 +37,7 @@ jobs:
- --no-default-features --features non-fips,ring-sig-verify,unstable
- --no-default-features --features non-fips,alloc,unstable
steps:
- - uses: actions/checkout@v3
+ - uses: actions/checkout@v4
with:
submodules: 'recursive'
- uses: dtolnay/rust-toolchain@master
@@ -66,7 +66,7 @@ jobs:
- --no-default-features --features aws-lc-sys,bindgen,unstable
- --release --all-targets --features bindgen,unstable
steps:
- - uses: actions/checkout@v3
+ - uses: actions/checkout@v4
with:
submodules: 'recursive'
- if: ${{ matrix.os == 'macos-13-xlarge' }}
@@ -106,9 +106,10 @@ jobs:
- --no-default-features --features non-fips,ring-io,unstable
- --no-default-features --features non-fips,ring-sig-verify,unstable
- --no-default-features --features non-fips,alloc,unstable
+ env:
+ AWS_LC_SYS_PREBUILT_NASM: 1
steps:
- - uses: ilammy/setup-nasm@v1
- - uses: actions/checkout@v3
+ - uses: actions/checkout@v4
with:
submodules: 'recursive'
- uses: dtolnay/rust-toolchain@master
@@ -125,7 +126,7 @@ jobs:
name: aws-ls-rs coverage
runs-on: ubuntu-latest
steps:
- - uses: actions/checkout@v3
+ - uses: actions/checkout@v4
with:
submodules: 'recursive'
lfs: true
@@ -167,7 +168,7 @@ jobs:
- --no-default-features --features fips,asan
runs-on: ubuntu-latest
steps:
- - uses: actions/checkout@v3
+ - uses: actions/checkout@v4
with:
submodules: 'recursive'
- uses: dtolnay/rust-toolchain@master
@@ -196,7 +197,7 @@ jobs:
os: [ ubuntu-latest, macos-12, macos-13-xlarge ]
static: [ 0, 1 ]
steps:
- - uses: actions/checkout@v3
+ - uses: actions/checkout@v4
with:
submodules: 'recursive'
- uses: dtolnay/rust-toolchain@stable
@@ -205,6 +206,40 @@ jobs:
# See: rust-lang/cargo#8531
run: cargo test -p aws-lc-rs --tests
+ build-env-nasm-test:
+ if: github.repository_owner == 'aws'
+ name: prebuilt NASM verification
+ runs-on: windows-latest
+ strategy:
+ fail-fast: false
+ matrix:
+ target:
+ - 'x86_64-pc-windows-msvc'
+ - 'x86_64-pc-windows-gnu'
+ steps:
+ - uses: actions/checkout@v4
+ with:
+ submodules: 'recursive'
+ - uses: dtolnay/rust-toolchain@stable
+ - name: Install NASM
+ uses: ilammy/setup-nasm@v1
+ - name: Remove NASM artifacts
+ shell: bash
+ run: |
+ cargo clean
+ rm ./aws-lc-sys/builder/prebuilt-nasm/*
+ - name: Run cargo test
+ shell: bash
+ run: AWS_LC_SYS_PREBUILT_NASM=0 cargo test --tests -p aws-lc-rs --release --no-default-features --features aws-lc-sys
+ - name: Collect NASM outputs
+ shell: bash
+ run: ./scripts/build/collect_nasm_obj.sh
+ - name: Flag any NASM changes
+ shell: bash
+ run: |
+ git add .
+ git diff --cached --exit-code HEAD -- aws-lc-sys/builder/prebuilt-nasm/*.txt
+
build-env-external-bindgen-test:
if: github.repository_owner == 'aws'
name: aws-lc-rs FIPS - External bindgen test
@@ -216,14 +251,14 @@ jobs:
matrix:
os: [ ubuntu-latest, macos-12, macos-13-xlarge, windows-latest ]
steps:
- - if: ${{ matrix.os == 'windows-latest' }}
- uses: ilammy/setup-nasm@v1
- - uses: actions/checkout@v3
+ - uses: actions/checkout@v4
with:
submodules: 'recursive'
- uses: dtolnay/rust-toolchain@stable
- - name: Install ninja-build tool
- uses: seanmiddleditch/gha-setup-ninja@v4
+ - if: ${{ matrix.os == 'windows-latest' }}
+ uses: ilammy/setup-nasm@v1
+ - if: ${{ matrix.os == 'windows-latest' }}
+ uses: seanmiddleditch/gha-setup-ninja@v5
- name: Install bindgen-cli
run: cargo install --locked bindgen-cli
- name: Remove bindings
@@ -245,12 +280,10 @@ jobs:
os: [ ubuntu-latest, macos-12, macos-13-xlarge ]
static: [ 0, 1 ]
steps:
- - uses: actions/checkout@v3
+ - uses: actions/checkout@v4
with:
submodules: 'recursive'
- uses: dtolnay/rust-toolchain@stable
- - name: Install ninja-build tool
- uses: seanmiddleditch/gha-setup-ninja@v4
- uses: actions/setup-go@v4
with:
go-version: '>=1.18'
@@ -271,7 +304,7 @@ jobs:
matrix:
os: [ ubuntu-latest, macos-12, macos-13-xlarge, windows-latest ]
steps:
- - uses: actions/checkout@v3
+ - uses: actions/checkout@v4
with:
submodules: 'recursive'
- uses: dtolnay/rust-toolchain@stable
@@ -306,15 +339,14 @@ jobs:
matrix:
os: [ ubuntu-latest, macos-12, macos-13-xlarge, windows-latest ]
steps:
- - uses: actions/checkout@v3
+ - uses: actions/checkout@v4
with:
submodules: 'recursive'
- uses: dtolnay/rust-toolchain@stable
- - name: Install ninja-build tool
- uses: seanmiddleditch/gha-setup-ninja@v4
- uses: actions/setup-go@v4
with:
go-version: '>=1.18'
+ - uses: seanmiddleditch/gha-setup-ninja@v5
- name: Run cargo test
run: cargo test -p aws-lc-rs --tests --no-default-features --features fips
- name: Release build
@@ -346,14 +378,14 @@ jobs:
matrix:
os: [ ubuntu-latest, macos-12, macos-13-xlarge, windows-latest ]
steps:
- - if: ${{ matrix.os == 'windows-latest' }}
- uses: ilammy/setup-nasm@v1
- - uses: actions/checkout@v3
+ - uses: actions/checkout@v4
with:
submodules: 'recursive'
+ - if: ${{ matrix.os == 'windows-latest' }}
+ uses: ilammy/setup-nasm@v1
+ - if: ${{ matrix.os == 'windows-latest' }}
+ uses: seanmiddleditch/gha-setup-ninja@v5
- uses: dtolnay/rust-toolchain@stable
- - name: Install ninja-build tool
- uses: seanmiddleditch/gha-setup-ninja@v4
- uses: actions/setup-go@v4
with:
go-version: '>=1.18'
@@ -378,7 +410,7 @@ jobs:
- macos-12
- macos-13-xlarge
steps:
- - uses: actions/checkout@v3
+ - uses: actions/checkout@v4
with:
submodules: 'recursive'
lfs: true
diff --git a/aws-lc-sys/builder/cmake_builder.rs b/aws-lc-sys/builder/cmake_builder.rs
index e25cecb443..c5069c5341 100644
--- a/aws-lc-sys/builder/cmake_builder.rs
+++ b/aws-lc-sys/builder/cmake_builder.rs
@@ -3,9 +3,9 @@
use crate::OutputLib::{Crypto, RustWrapper, Ssl};
use crate::{
- cargo_env, emit_warning, execute_command, is_crt_static, is_no_asm, option_env, target,
- target_arch, target_env, target_family, target_os, target_underscored, target_vendor,
- OutputLibType,
+ allow_prebuilt_nasm, cargo_env, emit_warning, execute_command, is_crt_static, is_no_asm,
+ option_env, target, target_arch, target_env, target_family, target_os, target_underscored,
+ target_vendor, test_nasm_command, OutputLibType,
};
use std::env;
use std::ffi::OsString;
@@ -22,10 +22,6 @@ fn test_clang_cl_command() -> bool {
execute_command("clang-cl".as_ref(), &["--version".as_ref()]).status
}
-fn test_nasm_command() -> bool {
- execute_command("nasm".as_ref(), &["-version".as_ref()]).status
-}
-
fn find_cmake_command() -> Option<OsString> {
if let Some(cmake) = option_env("CMAKE") {
emit_warning(&format!(
@@ -162,7 +158,7 @@ impl CmakeBuilder {
// See issue: #453
if target_os() == "windows" {
- Self::configure_windows(&mut cmake_cfg);
+ self.configure_windows(&mut cmake_cfg);
}
// If the build environment vendor is Apple
@@ -213,7 +209,7 @@ impl CmakeBuilder {
cmake_cfg
}
- fn configure_windows(cmake_cfg: &mut cmake::Config) {
+ fn configure_windows(&self, cmake_cfg: &mut cmake::Config) {
match (target_env().as_str(), target_arch().as_str()) {
("msvc", "aarch64") => {
cmake_cfg.generator_toolset(format!(
@@ -243,6 +239,22 @@ impl CmakeBuilder {
}
_ => {}
}
+ if target_arch() == "x86_64" && Some(true) == allow_prebuilt_nasm() {
+ emit_warning("!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!");
+ emit_warning("!!! Using pre-built NASM binaries !!!");
+ emit_warning("!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!");
+
+ let script_path = self
+ .manifest_dir
+ .join("builder")
+ .join("prebuilt-nasm.bat")
+ .display()
+ .to_string();
+ let script_path = script_path.replace('\\', "/");
+
+ cmake_cfg.define("CMAKE_ASM_NASM_COMPILER", script_path.as_str());
+ cmake_cfg.define("CMAKE_VERBOSE_MAKEFILE", "1");
+ }
}
fn configure_open_harmony(cmake_cfg: &mut cmake::Config) {
@@ -298,10 +310,14 @@ impl crate::Builder for CmakeBuilder {
let mut missing_dependency = false;
if target_os() == "windows" {
- if target_arch() == "x86_64" && !test_nasm_command() && !is_no_asm() {
+ if target_arch() == "x86_64"
+ && !is_no_asm()
+ && !test_nasm_command()
+ && Some(true) != allow_prebuilt_nasm()
+ {
eprintln!(
- "Consider setting `AWS_LC_SYS_NO_ASM` in the environment for development builds.\
- See User Guide about the limitations: https://aws.github.io/aws-lc-rs/index.html"
+ "Consider setting `AWS_LC_SYS_PREBUILT_NASM` in the build environment.\
+ See User Guide: https://aws.github.io/aws-lc-rs/index.html"
);
eprintln!("Missing dependency: nasm");
missing_dependency = true;
diff --git a/aws-lc-sys/builder/main.rs b/aws-lc-sys/builder/main.rs
index 23f0214991..987e1038ca 100644
--- a/aws-lc-sys/builder/main.rs
+++ b/aws-lc-sys/builder/main.rs
@@ -319,6 +319,7 @@ static mut AWS_LC_SYS_NO_PREFIX: bool = false;
static mut AWS_LC_SYS_INTERNAL_BINDGEN: bool = false;
static mut AWS_LC_SYS_EXTERNAL_BINDGEN: bool = false;
static mut AWS_LC_SYS_NO_ASM: bool = false;
+static mut AWS_LC_SYS_PREBUILT_NASM: Option<bool> = None;
fn initialize() {
unsafe {
@@ -328,6 +329,7 @@ fn initialize() {
AWS_LC_SYS_EXTERNAL_BINDGEN =
env_var_to_bool("AWS_LC_SYS_EXTERNAL_BINDGEN").unwrap_or(false);
AWS_LC_SYS_NO_ASM = env_var_to_bool("AWS_LC_SYS_NO_ASM").unwrap_or(false);
+ AWS_LC_SYS_PREBUILT_NASM = env_var_to_bool("AWS_LC_SYS_PREBUILT_NASM");
}
if !is_external_bindgen() && (is_internal_bindgen() || !has_bindgen_feature()) {
@@ -363,6 +365,7 @@ fn is_bindgen_required() -> bool {
|| !has_pregenerated()
}
+#[allow(dead_code)]
fn internal_bindgen_supported() -> bool {
// TODO: internal bindgen creates invalid bindings on FreeBSD
// See: #476
@@ -385,6 +388,10 @@ fn is_no_asm() -> bool {
unsafe { AWS_LC_SYS_NO_ASM }
}
+fn allow_prebuilt_nasm() -> Option<bool> {
+ unsafe { AWS_LC_SYS_PREBUILT_NASM }
+}
+
fn has_bindgen_feature() -> bool {
cfg!(feature = "bindgen")
}
@@ -393,6 +400,10 @@ fn has_pregenerated() -> bool {
unsafe { PREGENERATED }
}
+fn test_nasm_command() -> bool {
+ execute_command("nasm".as_ref(), &["-version".as_ref()]).status
+}
+
fn prepare_cargo_cfg() {
// This is supported in Rust >= 1.77.0
// Also remove `#![allow(unexpected_cfgs)]` from src/lib.rs
diff --git a/aws-lc-sys/builder/prebuilt-nasm.bat b/aws-lc-sys/builder/prebuilt-nasm.bat
new file mode 100644
index 0000000000..9c761db12a
--- /dev/null
+++ b/aws-lc-sys/builder/prebuilt-nasm.bat
@@ -0,0 +1,21 @@
+@echo off
+set "ScriptDir=%~dp0"
+set "ScriptDir=%ScriptDir:~0,-1%"
+:loop
+set "arg1=%~1"
+if "%arg1%"=="-o" goto end
+if "%arg1%"=="" goto failure
+shift
+goto loop
+:end
+shift
+set "path=%~1"
+for %%f in ("%path%") do set "filename=%%~nxf"
+copy "%ScriptDir%\prebuilt-nasm\%filename%" "%path%"
+exit 0
+
+:failure
+echo PATH: %path% 1>&2
+echo FILENAME: %filename% 1>&2
+echo ScriptDir: %ScriptDir% 1>&2
+exit 1
\ No newline at end of file
diff --git a/scripts/build/collect_nasm_obj.sh b/scripts/build/collect_nasm_obj.sh
new file mode 100644
index 0000000000..c425e9236f
--- /dev/null
+++ b/scripts/build/collect_nasm_obj.sh
@@ -0,0 +1,29 @@
+#!/usr/bin/env bash
+# Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved.
+# SPDX-License-Identifier: Apache-2.0 OR ISC
+
+set -ex
+set -o pipefail
+
+if [[ ${BASH_VERSINFO[0]} -lt 4 ]]; then
+ echo Must use bash 4 or later: ${BASH_VERSION}
+ exit 1
+fi
+
+SCRIPT_DIR=$(cd -- "$( dirname -- "${BASH_SOURCE[0]}" )" &> /dev/null && pwd)
+REPO_ROOT=$(git rev-parse --show-toplevel)
+SYS_CRATE_DIR="${REPO_ROOT}/aws-lc-sys"
+PREBUILT_NASM_DIR="${SYS_CRATE_DIR}/builder/prebuilt-nasm"
+mkdir -p "${PREBUILT_NASM_DIR}"
+rm -f "${PREBUILT_NASM_DIR}"/*
+
+DUMPBIN="$(find /c/Program\ Files/Microsoft\ Visual\ Studio/ -path "*/Hostx64/x64/*" -name "dumpbin.exe" -print -quit)"
+
+for nasm_file in `find aws-lc-sys/aws-lc/generated-src/win-x86_64/ -name "*.asm"`; do
+ OBJNAME=$(basename "${nasm_file}");
+ NASM_OBJ=$(find target/ -name "${OBJNAME/.asm/.obj}");
+ cp "${NASM_OBJ}" "${PREBUILT_NASM_DIR}"
+ # We remove the '.debug$S' value, which indicates the size of the debug section. This value can change across builds
+ # because it typically contains full source file paths that vary by build environment
+ "${DUMPBIN}" //DISASM "${PREBUILT_NASM_DIR}"/"${OBJNAME/.asm/.obj}" | grep -v '.debug$S' | sed -e "s/^Dump of file.*/Dump of file ${OBJNAME/.asm/.obj}/" > "${PREBUILT_NASM_DIR}"/"${OBJNAME/.asm/}"-disasm.txt
+done
Update CI
diff --git a/.github/workflows/sys-bindings-generator.yml b/.github/workflows/sys-bindings-generator.yml
index 3fba893d80..dc226f9f1b 100644
--- a/.github/workflows/sys-bindings-generator.yml
+++ b/.github/workflows/sys-bindings-generator.yml
@@ -270,3 +270,36 @@ jobs:
run: ./scripts/build/collect_build_src.sh -t ${{ matrix.target }}
- name: Commit & Push changes
run: ./scripts/ci/ci_add_commit_rebase_push.sh "Collected source files for ${{ matrix.target }}"
+ collect-nasm-and-commit:
+ needs: generate-windows-bindings-and-commit
+ if: github.repository == 'aws/aws-lc-rs'
+ runs-on: windows-latest
+ steps:
+ - uses: actions/checkout@v4
+ with:
+ submodules: 'recursive'
+ ref: ${{ github.ref_name }}
+ - uses: dtolnay/rust-toolchain@master
+ id: toolchain
+ with:
+ toolchain: stable
+ targets: "x86_64-pc-windows-msvc,x86_64-pc-windows-gnu"
+ - uses: ilammy/setup-nasm@v1
+ - name: Build aws-lc-sys
+ shell: bash
+ run: AWS_LC_SYS_PREBUILT_NASM=0 cargo build -p aws-lc-sys --release --target x86_64-pc-windows-msvc
+ - name: Collect NASM object files
+ shell: bash
+ run: ./scripts/build/collect_nasm_obj.sh
+ - name: Clean build
+ shell: bash
+ run: cargo clean
+ - name: Test aws-lc-rs for x86_64-pc-windows-msvc
+ shell: bash
+ run: AWS_LC_SYS_PREBUILT_NASM=1 cargo build -p aws-lc-sys --target x86_64-pc-windows-msvc
+ - name: Test aws-lc-sys for x86_64-pc-windows-gnu
+ shell: bash
+ run: AWS_LC_SYS_PREBUILT_NASM=1 cargo build -p aws-lc-sys --target x86_64-pc-windows-gnu
+ - name: Commit & Push changes
+ shell: bash
+ run: ./scripts/ci/ci_add_commit_rebase_push.sh "Collected NASM files"
diff --git a/.github/workflows/tests.yml b/.github/workflows/tests.yml
index a0da3f48b8..71716a64c0 100644
--- a/.github/workflows/tests.yml
+++ b/.github/workflows/tests.yml
@@ -37,7 +37,7 @@ jobs:
- --no-default-features --features non-fips,ring-sig-verify,unstable
- --no-default-features --features non-fips,alloc,unstable
steps:
- - uses: actions/checkout@v3
+ - uses: actions/checkout@v4
with:
submodules: 'recursive'
- uses: dtolnay/rust-toolchain@master
@@ -66,7 +66,7 @@ jobs:
- --no-default-features --features aws-lc-sys,bindgen,unstable
- --release --all-targets --features bindgen,unstable
steps:
- - uses: actions/checkout@v3
+ - uses: actions/checkout@v4
with:
submodules: 'recursive'
- if: ${{ matrix.os == 'macos-13-xlarge' }}
@@ -106,9 +106,10 @@ jobs:
- --no-default-features --features non-fips,ring-io,unstable
- --no-default-features --features non-fips,ring-sig-verify,unstable
- --no-default-features --features non-fips,alloc,unstable
+ env:
+ AWS_LC_SYS_PREBUILT_NASM: 1
steps:
- - uses: ilammy/setup-nasm@v1
- - uses: actions/checkout@v3
+ - uses: actions/checkout@v4
with:
submodules: 'recursive'
- uses: dtolnay/rust-toolchain@master
@@ -125,7 +126,7 @@ jobs:
name: aws-ls-rs coverage
runs-on: ubuntu-latest
steps:
- - uses: actions/checkout@v3
+ - uses: actions/checkout@v4
with:
submodules: 'recursive'
lfs: true
@@ -167,7 +168,7 @@ jobs:
- --no-default-features --features fips,asan
runs-on: ubuntu-latest
steps:
- - uses: actions/checkout@v3
+ - uses: actions/checkout@v4
with:
submodules: 'recursive'
- uses: dtolnay/rust-toolchain@master
@@ -196,7 +197,7 @@ jobs:
os: [ ubuntu-latest, macos-12, macos-13-xlarge ]
static: [ 0, 1 ]
steps:
- - uses: actions/checkout@v3
+ - uses: actions/checkout@v4
with:
submodules: 'recursive'
- uses: dtolnay/rust-toolchain@stable
@@ -205,6 +206,40 @@ jobs:
# See: rust-lang/cargo#8531
run: cargo test -p aws-lc-rs --tests
+ build-env-nasm-test:
+ if: github.repository_owner == 'aws'
+ name: prebuilt NASM verification
+ runs-on: windows-latest
+ strategy:
+ fail-fast: false
+ matrix:
+ target:
+ - 'x86_64-pc-windows-msvc'
+ - 'x86_64-pc-windows-gnu'
+ steps:
+ - uses: actions/checkout@v4
+ with:
+ submodules: 'recursive'
+ - uses: dtolnay/rust-toolchain@stable
+ - name: Install NASM
+ uses: ilammy/setup-nasm@v1
+ - name: Remove NASM artifacts
+ shell: bash
+ run: |
+ cargo clean
+ rm ./aws-lc-sys/builder/prebuilt-nasm/*
+ - name: Run cargo test
+ shell: bash
+ run: AWS_LC_SYS_PREBUILT_NASM=0 cargo test --tests -p aws-lc-rs --release --no-default-features --features aws-lc-sys
+ - name: Collect NASM outputs
+ shell: bash
+ run: ./scripts/build/collect_nasm_obj.sh
+ - name: Flag any NASM changes
+ shell: bash
+ run: |
+ git add .
+ git diff --cached --exit-code HEAD -- aws-lc-sys/builder/prebuilt-nasm/*.txt
+
build-env-external-bindgen-test:
if: github.repository_owner == 'aws'
name: aws-lc-rs FIPS - External bindgen test
@@ -216,14 +251,14 @@ jobs:
matrix:
os: [ ubuntu-latest, macos-12, macos-13-xlarge, windows-latest ]
steps:
- - if: ${{ matrix.os == 'windows-latest' }}
- uses: ilammy/setup-nasm@v1
- - uses: actions/checkout@v3
+ - uses: actions/checkout@v4
with:
submodules: 'recursive'
- uses: dtolnay/rust-toolchain@stable
- - name: Install ninja-build tool
- uses: seanmiddleditch/gha-setup-ninja@v4
+ - if: ${{ matrix.os == 'windows-latest' }}
+ uses: ilammy/setup-nasm@v1
+ - if: ${{ matrix.os == 'windows-latest' }}
+ uses: seanmiddleditch/gha-setup-ninja@v5
- name: Install bindgen-cli
run: cargo install --locked bindgen-cli
- name: Remove bindings
@@ -245,12 +280,10 @@ jobs:
os: [ ubuntu-latest, macos-12, macos-13-xlarge ]
static: [ 0, 1 ]
steps:
- - uses: actions/checkout@v3
+ - uses: actions/checkout@v4
with:
submodules: 'recursive'
- uses: dtolnay/rust-toolchain@stable
- - name: Install ninja-build tool
- uses: seanmiddleditch/gha-setup-ninja@v4
- uses: actions/setup-go@v4
with:
go-version: '>=1.18'
@@ -271,7 +304,7 @@ jobs:
matrix:
os: [ ubuntu-latest, macos-12, macos-13-xlarge, windows-latest ]
steps:
- - uses: actions/checkout@v3
+ - uses: actions/checkout@v4
with:
submodules: 'recursive'
- uses: dtolnay/rust-toolchain@stable
@@ -306,15 +339,14 @@ jobs:
matrix:
os: [ ubuntu-latest, macos-12, macos-13-xlarge, windows-latest ]
steps:
- - uses: actions/checkout@v3
+ - uses: actions/checkout@v4
with:
submodules: 'recursive'
- uses: dtolnay/rust-toolchain@stable
- - name: Install ninja-build tool
- uses: seanmiddleditch/gha-setup-ninja@v4
- uses: actions/setup-go@v4
with:
go-version: '>=1.18'
+ - uses: seanmiddleditch/gha-setup-ninja@v5
- name: Run cargo test
run: cargo test -p aws-lc-rs --tests --no-default-features --features fips
- name: Release build
@@ -346,14 +378,14 @@ jobs:
matrix:
os: [ ubuntu-latest, macos-12, macos-13-xlarge, windows-latest ]
steps:
- - if: ${{ matrix.os == 'windows-latest' }}
- uses: ilammy/setup-nasm@v1
- - uses: actions/checkout@v3
+ - uses: actions/checkout@v4
with:
submodules: 'recursive'
+ - if: ${{ matrix.os == 'windows-latest' }}
+ uses: ilammy/setup-nasm@v1
+ - if: ${{ matrix.os == 'windows-latest' }}
+ uses: seanmiddleditch/gha-setup-ninja@v5
- uses: dtolnay/rust-toolchain@stable
- - name: Install ninja-build tool
- uses: seanmiddleditch/gha-setup-ninja@v4
- uses: actions/setup-go@v4
with:
go-version: '>=1.18'
@@ -378,7 +410,7 @@ jobs:
- macos-12
- macos-13-xlarge
steps:
- - uses: actions/checkout@v3
+ - uses: actions/checkout@v4
with:
submodules: 'recursive'
lfs: true
diff --git a/aws-lc-sys/builder/cmake_builder.rs b/aws-lc-sys/builder/cmake_builder.rs
index e25cecb443..c5069c5341 100644
--- a/aws-lc-sys/builder/cmake_builder.rs
+++ b/aws-lc-sys/builder/cmake_builder.rs
@@ -3,9 +3,9 @@
use crate::OutputLib::{Crypto, RustWrapper, Ssl};
use crate::{
- cargo_env, emit_warning, execute_command, is_crt_static, is_no_asm, option_env, target,
- target_arch, target_env, target_family, target_os, target_underscored, target_vendor,
- OutputLibType,
+ allow_prebuilt_nasm, cargo_env, emit_warning, execute_command, is_crt_static, is_no_asm,
+ option_env, target, target_arch, target_env, target_family, target_os, target_underscored,
+ target_vendor, test_nasm_command, OutputLibType,
};
use std::env;
use std::ffi::OsString;
@@ -22,10 +22,6 @@ fn test_clang_cl_command() -> bool {
execute_command("clang-cl".as_ref(), &["--version".as_ref()]).status
}
-fn test_nasm_command() -> bool {
- execute_command("nasm".as_ref(), &["-version".as_ref()]).status
-}
-
fn find_cmake_command() -> Option<OsString> {
if let Some(cmake) = option_env("CMAKE") {
emit_warning(&format!(
@@ -162,7 +158,7 @@ impl CmakeBuilder {
// See issue: aws#453
if target_os() == "windows" {
- Self::configure_windows(&mut cmake_cfg);
+ self.configure_windows(&mut cmake_cfg);
}
// If the build environment vendor is Apple
@@ -213,7 +209,7 @@ impl CmakeBuilder {
cmake_cfg
}
- fn configure_windows(cmake_cfg: &mut cmake::Config) {
+ fn configure_windows(&self, cmake_cfg: &mut cmake::Config) {
match (target_env().as_str(), target_arch().as_str()) {
("msvc", "aarch64") => {
cmake_cfg.generator_toolset(format!(
@@ -243,6 +239,22 @@ impl CmakeBuilder {
}
_ => {}
}
+ if target_arch() == "x86_64" && Some(true) == allow_prebuilt_nasm() {
+ emit_warning("!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!");
+ emit_warning("!!! Using pre-built NASM binaries !!!");
+ emit_warning("!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!");
+
+ let script_path = self
+ .manifest_dir
+ .join("builder")
+ .join("prebuilt-nasm.bat")
+ .display()
+ .to_string();
+ let script_path = script_path.replace('\\', "/");
+
+ cmake_cfg.define("CMAKE_ASM_NASM_COMPILER", script_path.as_str());
+ cmake_cfg.define("CMAKE_VERBOSE_MAKEFILE", "1");
+ }
}
fn configure_open_harmony(cmake_cfg: &mut cmake::Config) {
@@ -298,10 +310,14 @@ impl crate::Builder for CmakeBuilder {
let mut missing_dependency = false;
if target_os() == "windows" {
- if target_arch() == "x86_64" && !test_nasm_command() && !is_no_asm() {
+ if target_arch() == "x86_64"
+ && !is_no_asm()
+ && !test_nasm_command()
+ && Some(true) != allow_prebuilt_nasm()
+ {
eprintln!(
- "Consider setting `AWS_LC_SYS_NO_ASM` in the environment for development builds.\
- See User Guide about the limitations: https://aws.github.io/aws-lc-rs/index.html"
+ "Consider setting `AWS_LC_SYS_PREBUILT_NASM` in the build environment.\
+ See User Guide: https://aws.github.io/aws-lc-rs/index.html"
);
eprintln!("Missing dependency: nasm");
missing_dependency = true;
diff --git a/aws-lc-sys/builder/main.rs b/aws-lc-sys/builder/main.rs
index 23f0214991..987e1038ca 100644
--- a/aws-lc-sys/builder/main.rs
+++ b/aws-lc-sys/builder/main.rs
@@ -319,6 +319,7 @@ static mut AWS_LC_SYS_NO_PREFIX: bool = false;
static mut AWS_LC_SYS_INTERNAL_BINDGEN: bool = false;
static mut AWS_LC_SYS_EXTERNAL_BINDGEN: bool = false;
static mut AWS_LC_SYS_NO_ASM: bool = false;
+static mut AWS_LC_SYS_PREBUILT_NASM: Option<bool> = None;
fn initialize() {
unsafe {
@@ -328,6 +329,7 @@ fn initialize() {
AWS_LC_SYS_EXTERNAL_BINDGEN =
env_var_to_bool("AWS_LC_SYS_EXTERNAL_BINDGEN").unwrap_or(false);
AWS_LC_SYS_NO_ASM = env_var_to_bool("AWS_LC_SYS_NO_ASM").unwrap_or(false);
+ AWS_LC_SYS_PREBUILT_NASM = env_var_to_bool("AWS_LC_SYS_PREBUILT_NASM");
}
if !is_external_bindgen() && (is_internal_bindgen() || !has_bindgen_feature()) {
@@ -363,6 +365,7 @@ fn is_bindgen_required() -> bool {
|| !has_pregenerated()
}
+#[allow(dead_code)]
fn internal_bindgen_supported() -> bool {
// TODO: internal bindgen creates invalid bindings on FreeBSD
// See: aws#476
@@ -385,6 +388,10 @@ fn is_no_asm() -> bool {
unsafe { AWS_LC_SYS_NO_ASM }
}
+fn allow_prebuilt_nasm() -> Option<bool> {
+ unsafe { AWS_LC_SYS_PREBUILT_NASM }
+}
+
fn has_bindgen_feature() -> bool {
cfg!(feature = "bindgen")
}
@@ -393,6 +400,10 @@ fn has_pregenerated() -> bool {
unsafe { PREGENERATED }
}
+fn test_nasm_command() -> bool {
+ execute_command("nasm".as_ref(), &["-version".as_ref()]).status
+}
+
fn prepare_cargo_cfg() {
// This is supported in Rust >= 1.77.0
// Also remove `#![allow(unexpected_cfgs)]` from src/lib.rs
diff --git a/aws-lc-sys/builder/prebuilt-nasm.bat b/aws-lc-sys/builder/prebuilt-nasm.bat
new file mode 100644
index 0000000000..9c761db12a
--- /dev/null
+++ b/aws-lc-sys/builder/prebuilt-nasm.bat
@@ -0,0 +1,21 @@
+@echo off
+set "ScriptDir=%~dp0"
+set "ScriptDir=%ScriptDir:~0,-1%"
+:loop
+set "arg1=%~1"
+if "%arg1%"=="-o" goto end
+if "%arg1%"=="" goto failure
+shift
+goto loop
+:end
+shift
+set "path=%~1"
+for %%f in ("%path%") do set "filename=%%~nxf"
+copy "%ScriptDir%\prebuilt-nasm\%filename%" "%path%"
+exit 0
+
+:failure
+echo PATH: %path% 1>&2
+echo FILENAME: %filename% 1>&2
+echo ScriptDir: %ScriptDir% 1>&2
+exit 1
\ No newline at end of file
diff --git a/scripts/build/collect_nasm_obj.sh b/scripts/build/collect_nasm_obj.sh
new file mode 100644
index 0000000000..c425e9236f
--- /dev/null
+++ b/scripts/build/collect_nasm_obj.sh
@@ -0,0 +1,29 @@
+#!/usr/bin/env bash
+# Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved.
+# SPDX-License-Identifier: Apache-2.0 OR ISC
+
+set -ex
+set -o pipefail
+
+if [[ ${BASH_VERSINFO[0]} -lt 4 ]]; then
+ echo Must use bash 4 or later: ${BASH_VERSION}
+ exit 1
+fi
+
+SCRIPT_DIR=$(cd -- "$( dirname -- "${BASH_SOURCE[0]}" )" &> /dev/null && pwd)
+REPO_ROOT=$(git rev-parse --show-toplevel)
+SYS_CRATE_DIR="${REPO_ROOT}/aws-lc-sys"
+PREBUILT_NASM_DIR="${SYS_CRATE_DIR}/builder/prebuilt-nasm"
+mkdir -p "${PREBUILT_NASM_DIR}"
+rm -f "${PREBUILT_NASM_DIR}"/*
+
+DUMPBIN="$(find /c/Program\ Files/Microsoft\ Visual\ Studio/ -path "*/Hostx64/x64/*" -name "dumpbin.exe" -print -quit)"
+
+for nasm_file in `find aws-lc-sys/aws-lc/generated-src/win-x86_64/ -name "*.asm"`; do
+ OBJNAME=$(basename "${nasm_file}");
+ NASM_OBJ=$(find target/ -name "${OBJNAME/.asm/.obj}");
+ cp "${NASM_OBJ}" "${PREBUILT_NASM_DIR}"
+ # We remove the '.debug$S' value, which indicates the size of the debug section. This value can change across builds
+ # because it typically contains full source file paths that vary by build environment
+ "${DUMPBIN}" //DISASM "${PREBUILT_NASM_DIR}"/"${OBJNAME/.asm/.obj}" | grep -v '.debug$S' | sed -e "s/^Dump of file.*/Dump of file ${OBJNAME/.asm/.obj}/" > "${PREBUILT_NASM_DIR}"/"${OBJNAME/.asm/}"-disasm.txt
+done
Problem:
In short, in rust-lang/rustup#3898 we (the Rustup team) are trying to replace the
ringbackend withaws-lc-rsas a part of Rustup's plan to ship therustlsbackend by default (rust-lang/rustup#3806).When cross-compiling for
aarch64-pc-windows-msvcfrom GitHub's official Windows CI runner however, the following error was generated (see the full error log in the foldable section below):Full Error log
You can also go to https://github.com/rust-lang/rustup/actions/runs/9624188283/job/26547561849 for the full CI log.
In addition, to prove that this actually has nothing to do with
rustls-platform-verifier, another build error without the latter was reproduced in rust-lang/rustup#3917 as https://github.com/rust-lang/rustup/actions/runs/9708177268/job/26794554219.Relevant details
The lockfile includes the following versions for
aws-lc-*dependencies:System information for the official GitHub runner in question:
Another interesting detail is that this seems to work without issues when compiling natively on an ARM64 Windows machine:
Many thanks in advance!
The text was updated successfully, but these errors were encountered: