runtime/cgo: Handles (small integers) cannot be passed as a C function's void* context

[adonovan]

A coworker recently hit trouble when using cgo.NewHandle to create a handle---an opaque reference to a Go value suitable for transiting C code---because the C function being called expected its context to be a void* value. Of course, C doesn't really care what bit pattern the void* value has, and will happily plumb any integer value, but the cgo-generated code at the Go/C boundary expects all pointer-shaped values to be legal Go pointers, whereas a cgo.Handle is a small integer. Converting the handle to a pointer using unsafe.Pointer(h) satisfies the type checker but causes the cgo-generated pointer validity check to fail nondeterministically, because some of these small integer values look to runtime.cgoCheckUnknownPointer like invalid Go pointers.

A workaround is to write a C wrapper function that performs the Handle-to-void* conversion, but this is tedious and easy to forget. Perhaps the cgo package could provide a variant of NewHandle that returns not an integer but an unsafe.Pointer value that is safe to pass to a C function's void* context argument. The implementation could, for example, use C.malloc(uintptr_t), and store the handle value in the heap cell. This would require a corresponding C.free operation, but the cgo.Handle.Delete method already imposes the necessary discipline.

A possible API could be the addition of methods such as:

type VoidPointer struct { cptr unsafe.Pointer } // pointer to C.malloc(uintptr) cell containing a cgo.Handle
func (Handle).ToVoidPointer() VoidPointer
func HandleFromVoidPointer(VoidPointer) Handle

Alternatively, the cgo package could declare a special type of unsafe.Pointer that is not subject to the usual cgo-generated pointer validity check.

[bcmills]

Alternatively, the cgo package could declare a special type of unsafe.Pointer that is not subject to the usual cgo-generated pointer validity check.

Compare #46731. (See in particular https://golang.org/cl/345093, which proposed to add a cgo.Incomplete type.)

(But that is for pointers that are known not to point into the Go heap, not for non-pointers.)

[bcmills]

Of course, C doesn't really care what bit pattern the void* value has, and will happily plumb any integer value,

That may be true of many C environments, but it is not portable. Per the C11 standard, §6.3.2.3, cl. 5 (emphasis mine):

An integer may be converted to any pointer type. Except as previously specified, the result is implementation-defined, might not be correctly aligned, might not point to an entity of the referenced type, and might be a trap representation.

So in general cgo programs must not assume that they can pass integers as type void*. Instead, I suggest passing either a C-allocated *uintptr_t or Go-allocated *cgo.Handle, depending on the lifetime requirements.

[bcmills]

(A C API that does not care whether its argument is a pointer or an integer should accept a uintptr_t, not a void*.)

[adonovan]

That may be true of many C environments, but it is not portable.

Fair point; not all the world's an Intel.

I suggest passing either a C-allocated *uintptr_t or Go-allocated *cgo.Handle

Ah, so you're allowed to pass a Go pointer so long as the variable it points to doesn't contain Go pointers? That's much simpler. Then consider this Issue a request for an example in the documentation of runtime/cgo.

[bcmills]

FWIW, the current documentation for the pointer-passing rules is at https://pkg.go.dev/cmd/cgo#hdr-Passing_pointers.

[adonovan]

I've mailed https://go-review.googlesource.com/c/go/+/364774.

[gopherbot]

Change https://golang.org/cl/364774 mentions this issue: runtime/cgo: add example of Handle with void* parameter

[gopherbot]

Change https://golang.org/cl/366075 mentions this issue: misc/cgo/test: remove erroneous "extern" keyword on forward-declaration