PKCS and X.509 selftests fail on big-endian PowerPC since 2.16.1

[mrpippy]

Description

• Type: Bug
• Priority: Major

---

Bug

OS
Mac OS X

mbed TLS build:
Version: 2.16.1
OS version: 10.4.11
Configuration: default config.h included with mbedtls
Compiler and options: GCC 4.0.1 (included with Xcode 2.5, the latest available for Mac OS X 10.4)
Additional environment information: Running on a Power Mac G5 (contains big-endian PowerPC 970)

Expected behavior
programs/benchmark and programs/selftest should both pass (which they do in 2.16.0)

Actual behavior
In programs/benchmark the RSA-2048 and RSA-4096 benchmarks fail with BIGNUM - Bad input parameters to function
In programs/selftest the PKCS#1 and X.509 certificate load tests both fail.

Steps to reproduce
gmake -k ; programs/test/benchmark ; programs/test/selftest

I believe the issue is related to the mpi_bigendian_to_host() changes by @hanno-arm (f872007). After hard-coding the big-endian codepath (which does no byte-swapping), the tests and benchmarks complete successfully.

I think there's multiple issues to solve:

• figure out why the default behavior/byteswapping isn't working correctly
• add a compiler #warning when the system endianness cannot be detected at compile time
• possibly add an MBEDTLS_ #define to specify endianness when it can't be detected. Defining __BYTE_ORDER__ to __ORDER_BIG_ENDIAN in my config.h seems to work without any problems but is not the cleanest solution

[hanno-becker]

Hi @mrpippy,

thanks a lot for your report and initial analysis!

Could you check which code-path your system selects for mpi_uint_bigendian_to_host() in the faulty build? Does it use mpi_uint_bigendian_to_host_c()?

Kind regards,
Hanno

[ciarmcom]

ARM Internal Ref: IOTSSL-2843

[mrpippy]

Yes it does use mpi_uint_bigendian_to_host_c(). I stepped through it a few times with gdb (with a -g3 -O0 build) and it is byte-swapping. For input of x=0x6fa98858, tmp=0x5888a96f is returned. x=0x10f2de12 results in tmp=0x12def210, etc.

[hanno-becker]

Hi @mrpippy,

thanks, the function is indeed buggy (it's embarrassing, where shall the platform-specific difference in functional behaviour come from if the function only uses operations whose functional effect is fully specified? Anyway...). I'll open a PR for it.

Thanks again for your report!
Hanno