Skip to content

Commit

Permalink
detect: Support PowerPC64 OpenBSD, prepare support for RISC-V
Browse files Browse the repository at this point in the history
  • Loading branch information
taiki-e committed Aug 31, 2024
1 parent b04b77f commit 09a967b
Show file tree
Hide file tree
Showing 63 changed files with 12,662 additions and 64 deletions.
1 change: 1 addition & 0 deletions .github/.cspell/project-dictionary.txt
Original file line number Diff line number Diff line change
Expand Up @@ -47,6 +47,7 @@ exynos
getauxval
getpid
HWCAP
hwprobe
ifunc
includepath
Iotel
Expand Down
4 changes: 2 additions & 2 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -168,13 +168,13 @@ Note: In this file, do not use the hard wrap in the middle of a sentence for com

- Optimize x86_64 outline-atomics for 128-bit atomics.
- Support outline-atomics for cmpxchg16b on Rust 1.69+ (i.e., on Rust 1.69+, x86_64 128-bit atomics is lock-free on all Intel chips and almost all AMD chips, even if cmpxchg16b is not available at compile-time.). Previously it was only nightly. ([#80](https://github.com/taiki-e/portable-atomic/pull/80))
- portable-atomic no longer enables outline-atomics on target where run-time feature detection is not available. ([#80](https://github.com/taiki-e/portable-atomic/pull/80))
- portable-atomic no longer enables outline-atomics on target where run-time CPU feature detection is not available. ([#80](https://github.com/taiki-e/portable-atomic/pull/80))

- Optimize aarch64 outline-atomics for 128-bit atomics.
- Support more targets and improve performance. ([#63](https://github.com/taiki-e/portable-atomic/pull/63), [#64](https://github.com/taiki-e/portable-atomic/pull/64), [#67](https://github.com/taiki-e/portable-atomic/pull/67), [#69](https://github.com/taiki-e/portable-atomic/pull/69), [#75](https://github.com/taiki-e/portable-atomic/pull/75), [#76](https://github.com/taiki-e/portable-atomic/pull/76), [#77](https://github.com/taiki-e/portable-atomic/pull/77))
See the [`atomic128` module's readme](https://github.com/taiki-e/portable-atomic/blob/HEAD/src/imp/atomic128/README.md#run-time-feature-detection) for a list of platforms that support outline-atomics.
Most of these improvements have already been [submitted and accepted in rust-lang/stdarch](https://github.com/rust-lang/stdarch/pulls?q=is%3Apr+author%3Ataiki-e+std_detect) and will soon be available in `std::arch::is_aarch64_feature_detected`.
- portable-atomic no longer enables outline-atomics on target where run-time feature detection is not available.
- portable-atomic no longer enables outline-atomics on target where run-time CPU feature detection is not available.

- Performance improvements. ([#70](https://github.com/taiki-e/portable-atomic/pull/70), [#81](https://github.com/taiki-e/portable-atomic/pull/81), [6c189ae](https://github.com/taiki-e/portable-atomic/commit/6c189ae1792ce0c08b4f56b6e6c256c223475ce2), [13c92b0](https://github.com/taiki-e/portable-atomic/commit/13c92b015a8e8646a4b885229157547354d03b9e), etc.)

Expand Down
2 changes: 1 addition & 1 deletion DEVELOPMENT.md
Original file line number Diff line number Diff line change
Expand Up @@ -16,7 +16,7 @@ portable-atomic/
├── src/
│ ├── imp/
│ │ ├── atomic128/ -- 128-bit atomic implementations (mainly by asm)
│ │ │ └── detect/ -- Run-time feature detection implementations used for outline-atomics
│ │ │ └── detect/ -- Run-time CPU feature detection implementations used for outline-atomics
│ │ ├── arm_linux.rs -- 64-bit atomic implementation for pre-v6 ARM Linux/Android
│ │ ├── core_atomic.rs -- wrapper for core::sync::atomic types
│ │ ├── fallback/ -- fallback implementation based on global locks
Expand Down
2 changes: 2 additions & 0 deletions src/cfgs.rs
Original file line number Diff line number Diff line change
Expand Up @@ -265,6 +265,7 @@ mod atomic_64_macros {
),
target_os = "android",
target_os = "freebsd",
all(target_os = "openbsd", portable_atomic_outline_atomics),
),
not(any(miri, portable_atomic_sanitize_thread)),
),
Expand Down Expand Up @@ -354,6 +355,7 @@ mod atomic_128_macros {
),
target_os = "android",
target_os = "freebsd",
all(target_os = "openbsd", portable_atomic_outline_atomics),
),
not(any(miri, portable_atomic_sanitize_thread)),
),
Expand Down
4 changes: 2 additions & 2 deletions src/imp/arm_linux.rs
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,7 @@
// 64-bit atomic implementation using kuser_cmpxchg64 on pre-v6 ARM Linux/Android.
//
// Refs:
// - https://www.kernel.org/doc/Documentation/arm/kernel_user_helpers.txt
// - https://github.com/torvalds/linux/blob/v6.10/Documentation/arch/arm/kernel_user_helpers.rst
// - https://github.com/rust-lang/compiler-builtins/blob/0.1.88/src/arm_linux.rs
//
// Note: On Miri and ThreadSanitizer which do not support inline assembly, we don't use
Expand All @@ -20,7 +20,7 @@ use core::{arch::asm, cell::UnsafeCell, mem, sync::atomic::Ordering};

use crate::utils::{Pair, U64};

// https://www.kernel.org/doc/Documentation/arm/kernel_user_helpers.txt
// https://github.com/torvalds/linux/blob/v6.10/Documentation/arch/arm/kernel_user_helpers.rst
const KUSER_HELPER_VERSION: usize = 0xFFFF0FFC;
// __kuser_helper_version >= 5 (kernel version 3.1+)
const KUSER_CMPXCHG64: usize = 0xFFFF0F60;
Expand Down
7 changes: 4 additions & 3 deletions src/imp/atomic128/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -28,11 +28,11 @@ Implementations with inline assembly generate assemblies almost equivalent to th

As 128-bit atomics-related APIs stabilize in the standard library, implementations with inline assembly are planned to be updated to get the benefits of both.

## Run-time feature detection
## Run-time CPU feature detection

[detect](detect) module has run-time feature detection implementations.
[detect](detect) module has run-time CPU feature detection implementations.

Here is the table of targets that support run-time feature detection and the instruction or API used:
Here is the table of targets that support run-time CPU feature detection and the instruction or API used:

| target_arch | target_os/target_env | instruction/API | features | note |
| ----------- | -------------------- | --------------- | -------- | ---- |
Expand All @@ -47,6 +47,7 @@ Here is the table of targets that support run-time feature detection and the ins
| aarch64 | fuchsia | zx_system_get_features | lse | Enabled by default |
| powerpc64 | linux | getauxval | all | Disabled by default |
| powerpc64 | freebsd | elf_aux_info | all | Disabled by default |
| powerpc64 | openbsd | elf_aux_info | all | Disabled by default |

Run-time detection is enabled by default on most targets and can be disabled with `--cfg portable_atomic_no_outline_atomics`.

Expand Down
6 changes: 6 additions & 0 deletions src/imp/atomic128/aarch64.rs
Original file line number Diff line number Diff line change
Expand Up @@ -119,6 +119,12 @@ mod detect;
#[cfg(any(target_os = "linux", target_os = "android", target_os = "freebsd"))]
#[path = "detect/aarch64_aa64reg.rs"]
mod detect_aa64reg;
// TODO: OpenBSD 7.6+
// #[cfg(test)]
// #[cfg(not(portable_atomic_no_outline_atomics))]
// #[cfg(target_os = "openbsd")]
// #[path = "detect/auxv.rs"]
// mod detect_auxv;
#[cfg(test)]
#[cfg(not(portable_atomic_no_outline_atomics))]
#[cfg(target_os = "macos")]
Expand Down
29 changes: 15 additions & 14 deletions src/imp/atomic128/detect/aarch64_aa64reg.rs
Original file line number Diff line number Diff line change
@@ -1,14 +1,14 @@
// SPDX-License-Identifier: Apache-2.0 OR MIT

// Run-time feature detection on aarch64 Linux/FreeBSD/NetBSD/OpenBSD by parsing system registers.
// Run-time CPU feature detection on AArch64 Linux/Android/FreeBSD/NetBSD/OpenBSD by parsing system registers.
//
// As of nightly-2023-01-23, is_aarch64_feature_detected doesn't support run-time detection on NetBSD/OpenBSD.
// https://github.com/rust-lang/stdarch/blob/a0c30f3e3c75adcd6ee7efc94014ebcead61c507/crates/std_detect/src/detect/mod.rs
// https://github.com/rust-lang/stdarch/pull/1374
//
// Refs:
// - https://developer.arm.com/documentation/ddi0601/latest/AArch64-Registers
// - https://www.kernel.org/doc/Documentation/arm64/cpu-feature-registers.txt
// - https://github.com/torvalds/linux/blob/v6.10/Documentation/arch/arm64/cpu-feature-registers.rst
// - https://github.com/rust-lang/stdarch/blob/a0c30f3e3c75adcd6ee7efc94014ebcead61c507/crates/std_detect/src/detect/os/aarch64.rs
//
// Supported platforms:
Expand All @@ -22,9 +22,10 @@
// https://github.com/openbsd/src/commit/d335af936b9d7dd9cf655cae1ce19560c45de6c8
//
// For now, this module is only used on NetBSD/OpenBSD.
// On Linux/FreeBSD, this module is test-only:
// - On Linux, this approach requires a higher kernel version than Rust supports,
// and also does not work with qemu-user (as of QEMU 7.2) and Valgrind.
//
// On Linux/Android/FreeBSD, we use auxv.rs and this module is test-only because:
// - On Linux/Android, this approach requires a higher kernel version than Rust supports,
// and also does not work with qemu-user (as of 7.2) and Valgrind (as of 3.19).
// (Looking into HWCAP_CPUID in auxvec, it appears that Valgrind is setting it
// to false correctly, but qemu-user is setting it to true.)
// - On FreeBSD, this approach does not work on FreeBSD 12 on QEMU (confirmed on
Expand All @@ -49,31 +50,31 @@ fn _detect(info: &mut CpuInfo) {
aa64mmfr2,
} = imp::aa64reg();

// ID_AA64ISAR0_EL1, Instruction Set Attribute Register 0
// https://developer.arm.com/documentation/ddi0601/2023-06/AArch64-Registers/ID-AA64ISAR0-EL1--AArch64-Instruction-Set-Attribute-Register-0?lang=en
// ID_AA64ISAR0_EL1, AArch64 Instruction Set Attribute Register 0
// https://developer.arm.com/documentation/ddi0601/2024-06/AArch64-Registers/ID-AA64ISAR0-EL1--AArch64-Instruction-Set-Attribute-Register-0
let atomic = extract(aa64isar0, 23, 20);
if atomic >= 2 {
if atomic >= 0b0010 {
info.set(CpuInfo::HAS_LSE);
// we currently only use FEAT_LSE and FEAT_LSE2 in outline-atomics.
#[cfg(test)]
{
if atomic >= 3 {
if atomic >= 0b0011 {
info.set(CpuInfo::HAS_LSE128);
}
}
}
// we currently only use FEAT_LSE and FEAT_LSE2 in outline-atomics.
#[cfg(test)]
{
// ID_AA64ISAR1_EL1, Instruction Set Attribute Register 1
// https://developer.arm.com/documentation/ddi0601/2023-06/AArch64-Registers/ID-AA64ISAR1-EL1--AArch64-Instruction-Set-Attribute-Register-1?lang=en
if extract(aa64isar1, 23, 20) >= 3 {
// ID_AA64ISAR1_EL1, AArch64 Instruction Set Attribute Register 1
// https://developer.arm.com/documentation/ddi0601/2024-06/AArch64-Registers/ID-AA64ISAR1-EL1--AArch64-Instruction-Set-Attribute-Register-1
if extract(aa64isar1, 23, 20) >= 0b0011 {
info.set(CpuInfo::HAS_RCPC3);
}
}
// ID_AA64MMFR2_EL1, AArch64 Memory Model Feature Register 2
// https://developer.arm.com/documentation/ddi0601/2023-06/AArch64-Registers/ID-AA64MMFR2-EL1--AArch64-Memory-Model-Feature-Register-2?lang=en
if extract(aa64mmfr2, 35, 32) >= 1 {
// https://developer.arm.com/documentation/ddi0601/2024-06/AArch64-Registers/ID-AA64MMFR2-EL1--AArch64-Memory-Model-Feature-Register-2
if extract(aa64mmfr2, 35, 32) >= 0b0001 {
info.set(CpuInfo::HAS_LSE2);
}
}
Expand Down
2 changes: 1 addition & 1 deletion src/imp/atomic128/detect/aarch64_fuchsia.rs
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
// SPDX-License-Identifier: Apache-2.0 OR MIT

// Run-time feature detection on aarch64 Fuchsia by using zx_system_get_features.
// Run-time CPU feature detection on aarch64 Fuchsia by using zx_system_get_features.
//
// As of nightly-2023-01-23, is_aarch64_feature_detected doesn't support run-time detection on Fuchsia.
// https://github.com/rust-lang/stdarch/blob/a0c30f3e3c75adcd6ee7efc94014ebcead61c507/crates/std_detect/src/detect/mod.rs
Expand Down
2 changes: 1 addition & 1 deletion src/imp/atomic128/detect/aarch64_macos.rs
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
// SPDX-License-Identifier: Apache-2.0 OR MIT

// Run-time feature detection on aarch64 macOS by using sysctl.
// Run-time CPU feature detection on aarch64 macOS by using sysctl.
//
// This module is currently only enabled on tests because aarch64 macOS always supports FEAT_LSE and FEAT_LSE2.
// https://github.com/llvm/llvm-project/blob/llvmorg-18.1.2/llvm/include/llvm/TargetParser/AArch64TargetParser.h#L728
Expand Down
2 changes: 1 addition & 1 deletion src/imp/atomic128/detect/aarch64_windows.rs
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
// SPDX-License-Identifier: Apache-2.0 OR MIT

// Run-time feature detection on aarch64 Windows by using IsProcessorFeaturePresent.
// Run-time CPU feature detection on aarch64 Windows by using IsProcessorFeaturePresent.
//
// As of nightly-2023-01-23, is_aarch64_feature_detected doesn't support run-time detection of FEAT_LSE on Windows.
// https://github.com/rust-lang/stdarch/blob/a0c30f3e3c75adcd6ee7efc94014ebcead61c507/crates/std_detect/src/detect/os/windows/aarch64.rs
Expand Down
Loading

0 comments on commit 09a967b

Please sign in to comment.