Improvement Meta-list v2

[Gnimuc]

Up-for-grabs

• extract docstrings from comments: Function and other signatures in comments #195
• support vararg functions: WARNING: Skipping VarArg Function CLCursor (Clang.cindex.FunctionDecl)  #146 callback to support possible vararg definitions? #17 (for Julia 1.5+)
  • Ccallmacro JuliaLang/julia#32748
  • Add semantics for varargs of different types to ccall. JuliaLang/julia#32758
  • ccall: support different types for vararg arguments [backend-only] JuliaLang/julia#32800
• better support for bitfield struct(see Improvement Meta-list v2 #228 (comment))
• support documentation dumping
• add more helper functions for libclang C API: Wrap support for getOffsetOf #126
• missing argument names for typedef'd args: missing argument names for Typedef'd args #136

Done

• support non-literal macro definitions: Issues wrapping libLLVM #168 #define a data type #103 function-like macro inappropriately parsed as object-like macro #266 Macro help #255
• struct/union improvements: __attribute__ no method value(::CLPackedAttr) #224
• handle forward-decls: Some header files entirely skipped #240 (comment)
• map missing C types: trouble wraping glib #206 Issues wrapping libLLVM #168 Minor issues when wrapping libav #81 (the ultimate solution would be to make a package like rust's libc)

Long-shot Issues

• wrapper generator for C++: C++ wrapping tutorial #132 (maybe for CxxWrap.jl)
• support va_list ? (not sure about the use-cases)
• handle macro/function definitions in a generic way: Handle header only functions #267

[ViralBShah]

Do we have support for _Generic? The discussion on discourse suggests we do not (but it was a while ago). If not, should it be added to the list above?

Came up in https://discourse.julialang.org/t/clang-jl-wrapper-generation-warnings-missing-parts/34093/7?u=viralbshah and https://twitter.com/DocSparse/status/1345479244466544641

[Gnimuc]

Handing function-like macros with libclang is not easy because macros are processed by the preprocessor, there is actually no much information left in the cursors(AST). One needs to tokenize and retrieve the source code of the macro definition and then manually parse the raw string to do something meaningful.

As macro definitions may contain any valid C code, to support function-like macros in a generic way is equivalent to implement a C-to-Julia source code translator. But it looks like C11's Generic selection macros have a simple fixed pattern which, I think, could be supported in Clang.jl, I'm going to submit a PR later on(I revisited several old macro-related issues, it turned out that the current implementation for handing CLMacroDefinition needs an overhaul, this might take a bit longer).

[jpsamaroo]

Another item for the list: C bitfields. Even though according to Linus Torvalds bitfields are usually a terrible idea, they're still used in the Linux kernel's structs, which I need to be able to manipulate in BPFnative.jl.

Even though the layout of such structs are implementation-dependent, I thing we should try to implement support for these, using the host architecture as a hint as to the layout. We can use the Base.getproperty approach to generate accessors that implement the necessary bit-twiddling operations.

[Gnimuc]

@jpsamaroo Could you elaborate more about how to implement the necessary bit-twiddling operations? libclang can tell the offset of the bitfield and the bitfield width.

UPDATE:

I implemented an ugly version here:

https://github.com/gridap/MiniQhull.jl/blob/db2e9f151b3bdf20ec47afa164a245678124c66f/src/LibQhull.jl#L167-L199

but I feel like there must be a more elegant implementation.

[jpsamaroo]

For a setproperty!: Based on the size of the bitfield, for each byte that the bitfield is a component of, you'd read the byte, left-shift the user-supplied value according to the bitfield's offset into that byte and the length of the bitfield, mask out (with bitwise and) the relevant portion of the byte with zeros, bitwise or the shifted supplied value with the read byte, and write the updated byte back into the struct.

[melonedo]

For a setproperty!: Based on the size of the bitfield, for each byte that the bitfield is a component of, you'd read the byte, left-shift the user-supplied value according to the bitfield's offset into that byte and the length of the bitfield, mask out (with bitwise and) the relevant portion of the byte with zeros, bitwise or the shifted supplied value with the read byte, and write the updated byte back into the struct.

I am afraid the biggest issue is not bit manipulation but the internal layout. The C language does not define a standard layout for bit fields, unless explicit alignment directives (like pack(1) are used. For example:

struct S {
    unsigned b1 : 30;
    unsigned b2 : 4;
    unsigned b3 : 30;
};

how much space does S need? We have to figure out these implementation defined behavior to ensure bit fields are handled correctly.

[Gnimuc]

That info can be queried from libclang, specifically, clang_Type_getSizeOf & clang_Type_getOffsetOf.

[melonedo]

That info can be queried from libclang, specifically, clang_Type_getSizeOf & clang_Type_getOffsetOf.

Thanks! I didn't expect it to return offset in bits instead of bytes. However, as this information is only from Clang, other compilers might have different implementation. Anyway, having them is a good starting ground.

[Gnimuc]

However, as this information is only from Clang, other compilers might have different implementation.

Clang should be ABI compatible with gcc(at least that's the goal). As this package is mainly for generating static bindings for Julia JLL packages where only mingw/gcc/clang is used in the build environment, so it's ok to make such an assumption.

BTW, this is the implementation used in Clang, which may serve as a good reference for the impl of getproperty/setproperty!.

[Gnimuc]

Well... I just give up reading gcc's implementation to check whether it's compatible with Clang's or not.