[vm/ffi] Pointer failure after upgrading nullsafety

[Sunbreak]

Reproduce

A little long, check: woodemi/twaindsm.dart#2

Summary

/	before upgrading	after nullsafety
empty struct	✅	✅
opaque	~	~
Uint8 pointer	✅	❌
nested struct	~	❌

• ✅: connect success
• ❌: connect failure
• ~: won't compile

class pTW_IDENTITY extends ffi.Struct {}

var identityPtr = ffi.allocate<pTW_IDENTITY>();
var connect = twainDsm.DSM_Entry(identityPtr,
  nullptr, DG_CONTROL, DAT_PARENT, MSG_OPENDSM, parentPtr.cast<Void>());

[Sunbreak]

When using inlien-array & nested-struct, commenting out ProductFamily & ProductName works. Only commenting out ProductName won't work either.

It could be related to packed struct #38158

class TW_VERSION extends ffi.Struct {
  @ffi.Uint16()
  external int MajorNum;

  @ffi.Uint16()
  external int MinorNum;

  @ffi.Uint16()
  external int Language;

  @ffi.Uint16()
  external int Country;
  
  @ffi.Array(34)
  external ffi.Array<ffi.Int8> Info;
}

class TW_IDENTITY extends ffi.Struct {
  @ffi.Uint32()
  external int Id;

  external TW_VERSION Version;

  @ffi.Uint16()
  external int ProtocolMajor;

  @ffi.Uint16()
  external int ProtocolMinor;

  @ffi.Uint32()
  external int SupportedGroups;

  @ffi.Array(34)
  external ffi.Array<ffi.Int8> Manufacturer;

  // @ffi.Array(34)
  // external ffi.Array<ffi.Int8> ProductFamily;

  // @ffi.Array(34)
  // external ffi.Array<ffi.Int8> ProductName;
}

[Sunbreak]

After investigating more, I've found the key different is ffi.allocate() before null-safety and ffi.calloc() after null-safety

With full inlien-array & nested-struct support, ffi.malloc() solve the problem

class TW_IDENTITY extends ffi.Struct {
  // ...
  @ffi.Array(34)
  external ffi.Array<ffi.Int8> ProductName;
}

var identityPtr = ffi.malloc<TW_IDENTITY>();
var connect = twainDsm.DSM_Entry(identityPtr, nullptr, DG_CONTROL,
        DAT_PARENT, MSG_OPENDSM, parentPtr.cast<Void>());

Maybe we need some best practise notes there.

And I'll dig deeper later why malloc/calloc matters

[Sunbreak]

It turns out the key is whether the memory is zero-initialized.

When I manually zero-initialized the malloc pointer, the operation fails as well.

After fill each field correctly as before nullsafety, everything works

It is related to the TWAIN implementation, not to vm/ffi.

Closing it.