Indirect branch tracking

[jcs]

OpenBSD now enforces indirect branch tracking (BTI on arm64, IBT on x86_64) which breaks Crystal on OpenBSD going forward.

I'm not sure whether this would all be handled by llvm and Crystal's compiler just needs to turn it on (in clang it's -fcf-protection=branch) or if there is assembly in Crystal that needs to be modified with the new instructions.

[HertzDevil]

It says that indirect branch tracking is already on by default unless there is an opt-out linker flag, so do we actually need to do anything?

[jcs]

Yes, the opt-out is temporary.

[GeopJr]

OpenBSD enforces it now and Crystal has been marked as broken

[joshturge]

I'd like to help get Crystal supported on OpenBSD, what work needs to be done in order to get IBT support? I'm not sure where to start.

[wecing]

@joshturge maybe check out QBE's implementation

Note that it was reverted due to tooling reasons, but I am trying to do a roll-forward since the blocker seems gone.

UPDATE: or, call linker with -z,nobtcfi (see openbsd/ports@71f21f5)

[ysbaddaden]

We need to tell LLVM to generate the correct instructions on the x86_64 and aarch64 targets —in gcc/clang this is the -fcf-protection=branch option.

I understand we also need to fix the fiber context switch asm for x86_64 and aarch64 to have these too, so we start & resume a fiber?

[ysbaddaden]

Searching for examples, I found out that Rust adds the cf-protection-branch=1 flag when creating a LLVM module when we pass -Z cf-protection=branch to rustc. This is equivalent to -fcf-protection=branch in clang.

See the source code and Rust book.

[ysbaddaden]

The -fcf-protection=branch option enables the control-flow enforcement technology (CET) on x86 only. For AArch64 this is the -mbranch-protection=bti option to enable the branch authentication instructions 😩

Looking at the disassembly of a compiled C program, the x86 option starts all functions with endbr64 while the aarch64 option starts all functions with bti c. We might just need to add these in the ASM of fiber context switches (as the first resume instruction) when the feature is enabled?

There are more specific values to each option, to enable more features. See the links above for details. I don't know about their impact on the ASM of fiber context switches. They're not needed to support OpenBSD; in fact IBT/BTI isn't needed and we could keep it disabled, but it's interesting to follow on the security and see & measure the impact.

[ysbaddaden]

Initial attempt: c973b01

It only works with --single-module otherwise we fail with incompatible LLVM module / function / context / type, which means I fail to set the values on every LLVM module (I guess). I must have missed something.

I also didn't test fibers. Only a few test files from nanolib and they didn't fail on OpenBSD 🤞