SPI Blocking Driver Performance Overhead ~2x

[ProfFan]

On the ESP32-S3 I observed that the following snippet uses double the bit time needed for one transaction:

// Just write some garbage to SPI and see how long it takes
let mut recv_buf = [0u8; 20];
let start_time = embassy_time::Instant::now();
spi.transfer(&mut recv_buf).unwrap();
let time_to_spi = start_time.elapsed().as_micros();

This is on a SPI (36MHz) without CS line. Time needed scales linearly with the number of bits (opt-level=3):

No. bytes	Time (us)	LTO?
20	11	Y
64	25	Y
80	40	Y
80	167	N

Without LTO it takes even longer.

[ProfFan]

@bugadani I can mostly confirm this is caused by the SPI hot functions not in IRAM. After adding #[ram] to the two hot path functions timing variance is down to 1us and little overhead

[MabezDev]

Closing in favour of #946