Shim wishlist

[saethlin]

I run Miri on the most-downloaded crates, with the flags -Zmiri-panic-on-unsupported and --no-fail-fast which basically asks Miri to try to run as many tests as it can. Searching all my build logs, I find these are the most-encountered can't call foreign function: errors:

117310 counts
(  1)    16419 (14.0%, 14.0%): socket
(  2)     7934 ( 6.8%, 20.8%): TLS_method
(  3)     5867 ( 5.0%, 25.8%): getaddrinfo
(  4)     3964 ( 3.4%, 29.1%): OPENSSL_init_ssl
(  5)     3172 ( 2.7%, 31.8%): fchmod
(  6)     2957 ( 2.5%, 34.4%): ring_core_0_17_8_OPENSSL_cpuid_setup
(  7)     2485 ( 2.1%, 36.5%): rust_psm_stack_pointer
(  8)     2432 ( 2.1%, 38.6%): rocksdb_options_create
(  9)     2167 ( 1.8%, 40.4%): getrlimit
( 10)     1880 ( 1.6%, 42.0%): dlopen
( 11)     1875 ( 1.6%, 43.6%): ts_parser_new
( 12)     1800 ( 1.5%, 45.1%): sodium_init
( 13)     1798 ( 1.5%, 46.7%): Py_IsInitialized
( 14)     1619 ( 1.4%, 48.1%): sqlite3_open_v2
( 15)     1415 ( 1.2%, 49.3%): epoll_create
( 16)     1382 ( 1.2%, 50.4%): sqlite3_threadsafe
( 17)     1365 ( 1.2%, 51.6%): GFp_cpuid_setup
( 18)     1293 ( 1.1%, 52.7%): chmod
( 19)     1155 ( 1.0%, 53.7%): SQLAllocHandle
( 20)     1003 ( 0.9%, 54.5%): geteuid
( 21)      745 ( 0.6%, 55.2%): rustsecp256k1_v0_10_0_context_preallocated_size
( 22)      713 ( 0.6%, 55.8%): InitializeMagick
( 23)      699 ( 0.6%, 56.4%): EVP_sha256
( 24)      694 ( 0.6%, 57.0%): ruby_init_stack
( 25)      629 ( 0.5%, 57.5%): access
( 26)      604 ( 0.5%, 58.0%): JS_NewRuntime
( 27)      600 ( 0.5%, 58.5%): PQconnectdb
( 28)      549 ( 0.5%, 59.0%): ioctl
( 29)      527 ( 0.4%, 59.5%): blst_scalar_from_uint64
( 30)      526 ( 0.4%, 59.9%): v8__Platform__NewDefaultPlatform
( 31)      520 ( 0.4%, 60.3%): __bid128_from_string
( 32)      516 ( 0.4%, 60.8%): nettle_yarrow256_init
( 33)      485 ( 0.4%, 61.2%): mdb_env_create
( 34)      482 ( 0.4%, 61.6%): NK_login_enum
( 35)      479 ( 0.4%, 62.0%): mprotect
( 36)      476 ( 0.4%, 62.4%): tree_sitter_tsx
( 37)      474 ( 0.4%, 62.8%): LLVMCreateMemoryBufferWithContentsOfFile
( 38)      440 ( 0.4%, 63.2%): git_libgit2_init
( 39)      426 ( 0.4%, 63.6%): roaring_bitmap_init_with_capacity
( 40)      407 ( 0.3%, 63.9%): getifaddrs
( 41)      380 ( 0.3%, 64.2%): blst_keygen_v4_5
( 42)      380 ( 0.3%, 64.6%): secp256k1_context_create
( 43)      367 ( 0.3%, 64.9%): decContextDefault
( 44)      367 ( 0.3%, 65.2%): ffopen
( 45)      342 ( 0.3%, 65.5%): SQLSetEnvAttr
( 46)      338 ( 0.3%, 65.8%): sigaction
( 47)      334 ( 0.3%, 66.0%): H5dont_atexit
( 48)      320 ( 0.3%, 66.3%): mrb_open
( 49)      312 ( 0.3%, 66.6%): cuInit

There are some pretty obvious biases in this data. For example, crates with a lot of tests get their favorite symbols counted more than crates with fewer tests.

There are also a lot of things in here which I don't think we will ever support via shims. We're not going to have shims for openssl. We're also not going to shim CUDA or JavaScript, and also all those errors are from 3 crates using CUDA and 1 crate doing some JavaScript thing.

But in any case, this is a starting point for prioritizing what shims should be added. Just in case there isn't anyone out there who already has a pet function they want added.

@drmeepster asked me to run some crates with a Windows target. Here's what the first ~2k crates run into:

2911 counts
(  1)     1583 (54.4%, 54.4%):  GetFullPathNameW
(  2)      485 (16.7%, 71.0%):  CreateIoCompletionPort
(  3)      292 (10.0%, 81.1%):  GetFileInformationByHandleEx
(  4)      183 ( 6.3%, 87.4%):  GetModuleFileNameW
(  5)       46 ( 1.6%, 88.9%):  SleepConditionVariableSRW
(  6)       24 ( 0.8%, 89.8%):  GetTickCount64
(  7)       22 ( 0.8%, 90.5%):  CompareStringOrdinal
(  8)       18 ( 0.6%, 91.1%):  AcquireCredentialsHandleA
(  9)       16 ( 0.5%, 91.7%):  WSAStartup
( 10)       15 ( 0.5%, 92.2%):  CreateNamedPipeW
( 11)       14 ( 0.5%, 92.7%):  Sleep
( 12)       13 ( 0.4%, 93.1%):  CryptAcquireContextW
( 13)       13 ( 0.4%, 93.6%):  log1p
( 14)       11 ( 0.4%, 94.0%):  CreateSemaphoreA
( 15)       10 ( 0.3%, 94.3%):  CreatePipe
( 16)       10 ( 0.3%, 94.6%):  SHGetKnownFolderPath
( 17)       10 ( 0.3%, 95.0%):  WakeAllConditionVariable
( 18)       10 ( 0.3%, 95.3%):  zx_vmo_create
( 19)        8 ( 0.3%, 95.6%):  GetCurrentProcess

I don't know what GetFullPathNameW is coming from, but when crates hit it, they usually have tends to hundreds of tests that attempt to use it. Possibly it's from proptest? prost is one of the crates that hits it a bunch of times.

[RalfJung]

__xpg_strerror_r is something that also comes up when just debugging Miri, I think even some of the tests will debug-print the IO error when the test fails, leading to an annoying double-failure (first the test fails with an assertion, then printing that assertion fails with an "unsupported" error).

[RalfJung]

I think __xpg_strerror_r is from crates that encounter a std::io::Error and try to format it. This is pretty common because a lot of crates attempt to access files in their tests that are not packaged (this is also a huge problem for crater).

Wish granted. ;) #2067

Just N-1 to go...

[mejrs]

( 10) 205 ( 1.3%, 52.8%): Py_IsInitialized

This the first CPython api call that Rust programs embedding the python interpreter will do. I would really like to test pyo3 and these programs with Miri.

I know Python's C api is too big to shim, but I would love to see some solution for this.

[CraftSpider]

I have a similar wishlist for testing a PHP extension library. Currently it fails at pipe2, but after that it would start failing at whatever the first call to PHP is. It would be nice if there was some way to use miri to test the Rust parts of my project, and then just actually call out to the relevant PHP stuff as normal.

[jjl]

mmap() is the one i'm currently missing. for my purposes it could just use a boxed slice. would be a little painful on memory, but enough to get miri running.

[bjorn3]

Read-only mmap where you don't write to the backing file shouldn't be too hard to implement. If writes through either the mapping or through the backing file are allowed, it becomes a lot harder to implement unix compatible semantics.

[jjl]

Ah, I see what you mean. Fortunately, I'm only using anonymous mappings.

[LegNeato]

Thanks for the list! I picked the easiest one I could see near the top (realpath):

Is there any way we can get a couple of example crates for each missing shim in the list so implementers can check the before and after with real code?

[saethlin]

I've manually grabbed a list for you for realpath: realpath-crates.txt

I have a very simple website which is just Cloudfront + S3 at https://miri.saethlin.dev (most people care about https://miri.saethlin.dev/ub). I'm not opposed to making simple additions, like a way to download all the raw logs in a .tar.gz. You can comment here or open a new issue: saethlin/crater-at-home#8

[saethlin]

Since @eduardosm has been doing some amazing work on implementing LLVM x86 intrinsics, here's my list of those that are encountered currently in test suites:

(  1)     6867 (74.6%, 74.6%): llvm.x86.subborrow.64
(  2)      539 ( 5.9%, 80.5%): llvm.x86.sse2.psad.bw
(  3)      361 ( 3.9%, 84.4%): llvm.x86.sse.cmp.ps
(  4)      177 ( 1.9%, 86.3%): llvm.x86.sse2.cmp.pd
(  5)      137 ( 1.5%, 87.8%): llvm.x86.sse.add.ss
(  6)       93 ( 1.0%, 88.8%): llvm.x86.aesni.aesenc
(  7)       87 ( 0.9%, 89.8%): llvm.x86.sse.sqrt.ss
(  8)       86 ( 0.9%, 90.7%): llvm.x86.rdtsc
(  9)       74 ( 0.8%, 91.5%): llvm.prefetch
( 10)       64 ( 0.7%, 92.2%): llvm.x86.bmi.pdep.64
( 11)       60 ( 0.7%, 92.9%): llvm.x86.sse2.pmulu.dq
( 12)       57 ( 0.6%, 93.5%): llvm.x86.sse2.pslli.d
( 13)       54 ( 0.6%, 94.1%): llvm.x86.sse.sqrt.ps
( 14)       40 ( 0.4%, 94.5%): llvm.x86.sse.movmsk.ps
( 15)       36 ( 0.4%, 94.9%): llvm.x86.sse.min.ps
( 16)       33 ( 0.4%, 95.3%): llvm.x86.sse.max.ps
( 17)       26 ( 0.3%, 95.6%): llvm.x86.sse2.cvttps2dq
( 18)       24 ( 0.3%, 95.8%): llvm.x86.sse.cmp.ss
( 19)       24 ( 0.3%, 96.1%): llvm.x86.sse2.cmp.sd
( 20)       22 ( 0.2%, 96.3%): llvm.x86.sse2.psll.q
( 21)       19 ( 0.2%, 96.5%): llvm.x86.sse2.psll.d
( 22)       18 ( 0.2%, 96.7%): llvm.x86.sse2.cvtps2dq
( 23)       17 ( 0.2%, 96.9%): llvm.x86.pclmulqdq
( 24)       17 ( 0.2%, 97.1%): llvm.x86.sse2.psrl.d
( 25)       16 ( 0.2%, 97.3%): llvm.x86.sse2.psrl.q
( 26)       15 ( 0.2%, 97.4%): llvm.x86.sse2.storeu.dq
( 27)       10 ( 0.1%, 97.5%): llvm.x86.sse.stmxcsr
( 28)       10 ( 0.1%, 97.6%): llvm.x86.sse2.pmulh.w
( 29)       10 ( 0.1%, 97.8%): llvm.x86.sse2.psll.w
( 30)       10 ( 0.1%, 97.9%): llvm.x86.sse2.pslli.w
( 31)        9 ( 0.1%, 98.0%): llvm.x86.sse2.movmsk.pd
( 32)        8 ( 0.1%, 98.0%): llvm.x86.sse2.psra.d
( 33)        6 ( 0.1%, 98.1%): llvm.x86.sse.rcp.ps
( 34)        6 ( 0.1%, 98.2%): llvm.x86.sse.rsqrt.ps
( 35)        6 ( 0.1%, 98.2%): llvm.x86.sse2.cvtdq2ps
( 36)        6 ( 0.1%, 98.3%): llvm.x86.sse2.max.pd
( 37)        6 ( 0.1%, 98.4%): llvm.x86.sse2.min.pd
( 38)        6 ( 0.1%, 98.4%): llvm.x86.sse2.psra.w
( 39)        6 ( 0.1%, 98.5%): llvm.x86.sse2.psrli.d
( 40)        5 ( 0.1%, 98.6%): llvm.x86.sse2.packsswb.128
( 41)        5 ( 0.1%, 98.6%): llvm.x86.sse2.packuswb.128
( 42)        5 ( 0.1%, 98.7%): llvm.x86.ssse3.pshuf.b.128
( 43)        4 ( 0.0%, 98.7%): llvm.x86.sse.sub.ss
( 44)        4 ( 0.0%, 98.7%): llvm.x86.sse2.comige.sd
( 45)        4 ( 0.0%, 98.8%): llvm.x86.sse2.psrl.w
( 46)        4 ( 0.0%, 98.8%): llvm.x86.sse3.ldu.dq
( 47)        3 ( 0.0%, 98.9%): llvm.x86.sse2.packssdw.128
( 48)        3 ( 0.0%, 98.9%): llvm.x86.sse2.psrai.d
( 49)        3 ( 0.0%, 98.9%): llvm.x86.sse2.psrai.w
( 50)        3 ( 0.0%, 99.0%): llvm.x86.sse2.psrli.q
( 51)        2 ( 0.0%, 99.0%): llvm.ctpop.v16i8
( 52)        2 ( 0.0%, 99.0%): llvm.x86.avx2.pslli.d
( 53)        2 ( 0.0%, 99.0%): llvm.x86.sse.comieq.ss
( 54)        2 ( 0.0%, 99.1%): llvm.x86.sse.comige.ss
( 55)        2 ( 0.0%, 99.1%): llvm.x86.sse.comigt.ss
( 56)        2 ( 0.0%, 99.1%): llvm.x86.sse.comile.ss
( 57)        2 ( 0.0%, 99.1%): llvm.x86.sse.comilt.ss
( 58)        2 ( 0.0%, 99.1%): llvm.x86.sse.comineq.ss
( 59)        2 ( 0.0%, 99.2%): llvm.x86.sse.cvtsi2ss
( 60)        2 ( 0.0%, 99.2%): llvm.x86.sse.cvtss2si
( 61)        2 ( 0.0%, 99.2%): llvm.x86.sse.div.ss
( 62)        2 ( 0.0%, 99.2%): llvm.x86.sse.max.ss
( 63)        2 ( 0.0%, 99.2%): llvm.x86.sse.min.ss
( 64)        2 ( 0.0%, 99.3%): llvm.x86.sse.mul.ss
( 65)        2 ( 0.0%, 99.3%): llvm.x86.sse.rcp.ss
( 66)        2 ( 0.0%, 99.3%): llvm.x86.sse.rsqrt.ss
( 67)        2 ( 0.0%, 99.3%): llvm.x86.sse2.comieq.sd
( 68)        2 ( 0.0%, 99.4%): llvm.x86.sse2.comile.sd
( 69)        2 ( 0.0%, 99.4%): llvm.x86.sse2.comilt.sd
( 70)        2 ( 0.0%, 99.4%): llvm.x86.sse2.comineq.sd
( 71)        2 ( 0.0%, 99.4%): llvm.x86.sse2.cvtpd2dq
( 72)        2 ( 0.0%, 99.4%): llvm.x86.sse2.cvtpd2ps
( 73)        2 ( 0.0%, 99.5%): llvm.x86.sse2.cvtps2pd
( 74)        2 ( 0.0%, 99.5%): llvm.x86.sse2.cvtsd2si
( 75)        2 ( 0.0%, 99.5%): llvm.x86.sse2.cvtsd2si64
( 76)        2 ( 0.0%, 99.5%): llvm.x86.sse2.cvtsd2ss
( 77)        2 ( 0.0%, 99.6%): llvm.x86.sse2.cvtss2sd
( 78)        2 ( 0.0%, 99.6%): llvm.x86.sse2.cvttpd2dq
( 79)        2 ( 0.0%, 99.6%): llvm.x86.sse2.cvttsd2si
( 80)        2 ( 0.0%, 99.6%): llvm.x86.sse2.cvttsd2si64
( 81)        2 ( 0.0%, 99.6%): llvm.x86.sse2.max.sd
( 82)        2 ( 0.0%, 99.7%): llvm.x86.sse2.min.sd
( 83)        2 ( 0.0%, 99.7%): llvm.x86.sse2.pavg.b
( 84)        2 ( 0.0%, 99.7%): llvm.x86.sse2.pavg.w
( 85)        2 ( 0.0%, 99.7%): llvm.x86.sse2.pmadd.wd
( 86)        2 ( 0.0%, 99.7%): llvm.x86.sse2.pmulhu.w
( 87)        2 ( 0.0%, 99.8%): llvm.x86.sse2.pslli.q
( 88)        2 ( 0.0%, 99.8%): llvm.x86.sse2.psrli.w
( 89)        2 ( 0.0%, 99.8%): llvm.x86.sse2.sqrt.sd
( 90)        2 ( 0.0%, 99.8%): llvm.x86.sse2.storeu.pd
( 91)        2 ( 0.0%, 99.9%): llvm.x86.sse3.hadd.ps
( 92)        2 ( 0.0%, 99.9%): llvm.x86.sse41.dpps
( 93)        2 ( 0.0%, 99.9%): llvm.x86.sse41.ptestz
( 94)        1 ( 0.0%, 99.9%): llvm.ctpop.v2i64
( 95)        1 ( 0.0%, 99.9%): llvm.ctpop.v4i32
( 96)        1 ( 0.0%, 99.9%): llvm.ctpop.v8i16
( 97)        1 ( 0.0%, 99.9%): llvm.x86.avx2.permd
( 98)        1 ( 0.0%,100.0%): llvm.x86.avx2.pslli.q
( 99)        1 ( 0.0%,100.0%): llvm.x86.bmi.bzhi.64
(100)        1 ( 0.0%,100.0%): llvm.x86.fma.vfnmadd.pd.256
(101)        1 ( 0.0%,100.0%): llvm.x86.rdrand.64
(102)        1 ( 0.0%,100.0%): llvm.x86.rdseed.64

[saethlin]

Here's an update:

(  1)      214 (40.9%, 40.9%): llvm.x86.sse41.pblendw
(  2)      128 (24.5%, 65.4%): llvm.x86.rdtsc
(  3)       37 ( 7.1%, 72.5%): llvm.x86.sse41.blendvps
(  4)       22 ( 4.2%, 76.7%): llvm.x86.sse41.ptestz
(  5)       19 ( 3.6%, 80.3%): llvm.x86.sse42.crc32.32.8
(  6)       17 ( 3.3%, 83.6%): llvm.x86.sse42.pcmpestri128
(  7)       12 ( 2.3%, 85.9%): llvm.x86.sse.stmxcsr
(  8)       10 ( 1.9%, 87.8%): llvm.x86.sse41.round.ps
(  9)        7 ( 1.3%, 89.1%): llvm.x86.sse41.packusdw
( 10)        6 ( 1.1%, 90.2%): llvm.x86.sse41.round.sd
( 11)        6 ( 1.1%, 91.4%): llvm.x86.sse41.round.ss
( 12)        5 ( 1.0%, 92.4%): llvm.x86.sse41.dpps
( 13)        4 ( 0.8%, 93.1%): llvm.x86.sse41.insertps
( 14)        3 ( 0.6%, 93.7%): llvm.x86.sse41.mpsadbw
( 15)        3 ( 0.6%, 94.3%): llvm.x86.sse41.phminposuw
( 16)        3 ( 0.6%, 94.8%): llvm.x86.sse41.pmuldq
( 17)        3 ( 0.6%, 95.4%): llvm.x86.sse41.ptestc
( 18)        3 ( 0.6%, 96.0%): llvm.x86.sse41.ptestnzc
( 19)        3 ( 0.6%, 96.6%): llvm.x86.sse41.round.pd
( 20)        3 ( 0.6%, 97.1%): llvm.x86.sse42.crc32.32.16
( 21)        3 ( 0.6%, 97.7%): llvm.x86.sse42.crc32.32.32
( 22)        3 ( 0.6%, 98.3%): llvm.x86.sse42.crc32.64.64
( 23)        3 ( 0.6%, 98.9%): llvm.x86.sse42.pcmpestrm128
( 24)        3 ( 0.6%, 99.4%): llvm.x86.sse42.pcmpistri128
( 25)        2 ( 0.4%, 99.8%): llvm.experimental.vector.reduce.add.v4i128
( 26)        1 ( 0.2%,100.0%): llvm.x86.sse41.blendvpd

I do not know what happened to the aesni.aesenc calls. My best guess is that I changed to -Ctarget-cpu=x86_64-v2 and now they're hidden behind some SSE4 functions.

[def-]

Thanks for posting this. We use miri in Materialize and seem to be having a different set of unsupported operations, basically our wishlist:

$ git grep "cfg_attr(miri"|sed -e "s#.*\/\/ *##" | sed -e "s/error: *//" | grep -v "slow"|sort|uniq -c|sort -n -r
     81 unsupported operation: can't call foreign function `TLS_client_method` on OS `linux`
     58 unsupported operation: returning ready events from epoll_wait is not yet implemented
     20 unsupported operation: can't call foreign function `decNumberFromInt32` on OS `linux`
     15 unsupported operation: can't call foreign function `decContextDefault` on OS `linux`
     12 unsupported operation: can't call foreign function `rust_psm_stack_pointer` on OS `linux`
     12 unsupported operation: cannot write to event
     11 unsupported operation: can't call foreign function `OPENSSL_init_ssl` on OS `linux`
      7 unsupported operation: integer-to-pointer casts and `ptr::from_exposed_addr` are not supported with `-Zmiri-strict-provenance`
      6 unsupported operation: inline assembly is not supported
      6 unsupported operation: can't call foreign function `TLS_method` on OS `linux`
      4 unsupported operation: can't call foreign function `socket` on OS `linux`
      4 unsupported operation: can't call foreign function `pipe2` on OS `linux`
      3 unsupported operation: can't call foreign function `rocksdb_create_default_env` on OS `linux`
      3 unsupported operation: can't call foreign function `epoll_wait` on OS `linux`
      2 unsupported operation: can't call foreign function `deflateInit2_` on OS `linux`
      1 unsupported operation: non-default mode 0o600 is not supported
      1 unsupported operation: integer-to-pointer casts and `ptr::from_exposed_addr`
      1 unsupported operation: can't call foreign function `pidfile_open` on OS `linux`

[newpavlov]

My list of desired shims: O_NOFOLLOW flag support for open, flock with all 3 flags, pwrite64, and pread64. It's not a full list, just functions on which our tests are stuck (we can cfg our way out for the former two, but not for the latter two).

[RalfJung]

@saethlin it might be time to update the list for FFI shims as well -- the two first items in the list at the top of this issue are now implemented in Miri. :D

socket is still missing so that will probably be number 1 now. (Or all those pipe2 where actually from process::Command and we'll see a bunch of fork/posix_spawn-like things.)

[saethlin]

I've updated the issue description with all the crates I've run this month, which is the ~32,000 crates with the most downloads in the last 90 days.

I'm quite surprised that we don't see a bunch of process::Command internals as you suggested. Maybe I'll check in on that later?