[Guide] how to create c type bindings to koka

[SchrodingerZhu]

Hi @daanx ,

I am testing porting https://github.com/SchrodingerZhu/ahash-c to koka, and I can write something like

include extern {
  c  file "ahash_koka.c"
}
type hasher :: V

with

typedef struct kk_ahash__ahasher_s {
    struct kk_ahash__hasher_s _base;
    ahasher_t _inner;
} koka_ahasher_t;

static AHASH_FAST_PATH ahasher_t get_real_hasher(struct kk_ahash__hasher_s* _x){
    return ((koka_ahasher_t*)_x)->_inner;
}

However, there are still tons of things to do, like I must create APIs for allocation. And I begin to wonder whether there is a way to keep the things in value type in a third party library?

I see in the kklib the kk_integer_t is actually specialized without a base wrapper; is it possible to create something like that out of the stdlib?

[SchrodingerZhu]

@daanx After reading a bit the source of kklib, I came up with this: https://github.com/SchrodingerZhu/ahash-c/tree/koka/koka.
Though the code quality is not good enough and uint64_t is not ready yet, it seems that it is already a working hash library for Koka.

[SchrodingerZhu]

To play with the lib, the following commands may be helpful:

CC=clang koka --ccargs="-march=native -flto" --cclinkargs="-flto" -O2 -p ./ahash.kk

with the minimal Koka demo:

next-state().hasher().write(0).finish

[SchrodingerZhu]

I also created a benchmark for what we have got:

import ahash
import std/time/timer
import std/num/random
fun bench-hash-string(str: string, count: int, last: int) : <div, ndet> () {
    if (count <= 0) then ()
    else {
        val state = next-state()
        bench-hash-string(str, count - 1, state.single(str) + last)
    }
}

fun bench-hash-int(data: int, count: int) : <div, ndet> () {
    if (count <= 0) then ()
    else {
        val state = next-state()
        bench-hash-int(data +  state.single(data), count - 1)
    }
}

fun bench-hash-list(data: list<char>, count: int, last: int) : <div, ndet> () {
    if (count <= 0) then ()
    else {
        val state = next-state()
        bench-hash-list(data, count - 1, last + state.hasher.write(data).finish)
    }
}

fun bench-hash-int32(data: int32, count: int) : <div, ndet> () {
    if (count <= 0) then ()
    else {
        val state = next-state()
        bench-hash-int32(state.single(data).int32, count - 1)
    }
}

fun random-string(current: list<int>, length: int) : div string {
    if (length < 0)
    then current.show
    else random-string(Cons(length, current), length - 1)
}
fun main() {
    println("Ahash Version: " + ahash-version())
    val data = random-string(Nil, 1000)
    println("benchmark hashing string with length: " + data.list.length.show)
    print-elapsed( { bench-hash-string(data, 1000000, 0) } )
    val int-data = 0xffffffffffffffffffffffffffffffffffffffffffffff 
        + random-int() * random-int() * random-int() 
        * random-int() * random-int() * random-int() 
        * random-int() * random-int() * random-int() 
    println("benchmark hashing int with data: " + int-data.show)
    print-elapsed( { bench-hash-int(int-data, 1000000) } )

    println("benchmark hashing int32 (small data): ")
    print-elapsed( { bench-hash-int32(int-data.int32, 1000000) } )

    println("benchmark hashing char list: ")
    print-elapsed( { bench-hash-list(data.list, 1000000, 0) } )
    
}

with -march=native I get

Ahash Version: x86_64+ssse3+aes
benchmark hashing string with length: 3896
elapsed 0.152s
benchmark hashing int with data: 2326275403387940246143867568439566704479422808157040597076415171640592274546353135
elapsed 0.099s
benchmark hashing int32 (small data): 
elapsed 0.045s
benchmark hashing char list: 
elapsed 21.687s

If this benchmark had no problem, we should be able to hash at speed 23.87 GiB/s (of Koka'a string). Moreover, ahash claimed to be anti-DOS and my re-implementation also passes all tests in SMHasher test suite.

P.S.
The original aHash implementation claims to reach the speed at 50+GiB/s, but it is not the case on my PCs, where both my version and the original one can reach 45GiB/s on my Xeon Laptop and 35GiB/s on my Zen2 Desktop. In fact, my version even runs faster on my computer in some of the SMHasher tests. Perhaps this is because the speed is very computer/microarch sensitive.

[daanx]

Awesome to see this. I agree, I need to write a C interface guide -- there is too much to do :-(

I looked a bit but the code you wrote is too dependent on the how the compiler generates code.. You may look at the std/core/xxx-inline.c files for more inspiration. Some further hints:

• I saw you analyze the internal string representation; try to not do that and use the access functions in kklib/include/string.h; usually you only need const char* kk_string_cbuf_borrow(const kk_string_t str) and size_t kk_decl_pure kk_string_len_borrow(const kk_string_t str). Do not use strlen on these but get the kk_string_len_borrow as it is both faster and safer. Not sure, but I think you can do:

size_t len = kk_string_len_borrow(str);
const char* s = kk_string_cbuf_borrow(str);
h = hash_write(h, s, len);

without knowing about the internal representation. I will add a kk_string_cbuf_len_borrow function that combines these :-)

• Same with dropping the string, use kk_string_drop( kk_string_t str, kk_context_t* ctx) without looking at the internals

• I saw you implemented a datatype by hand in C (like kk_ahash__random_state_wrapper) but that is very fragile. It is much better to just declare the datatype in koka itself; you can include the koka generated header to access it through the generated functions.

• Also, to hash integers also try to avoid analyzing the internal representation (I saw your comment so I know you are aware of it :-) ) but use the public api. Now, I am not sure if that works for you since you are hashing but perhaps we need a better integer C api function then in kklib? Like kk_get_digits( kk_integer_t i, const kk_digit_t* digits, size_t* count) or something.

Ah, lot of comments but I hope it helps -- Koka is still an early project so more work is needed here.

[JonahBranch]

Hello, I'm tacking onto this thread to get some help with c bindings. I am trying to bind the linenoise library but I get an error from the c compiler due to a missing declaration.

helloworld.kk

extern import
  c file "linenoise.h"
  c file "linenoise.c"
  c file "linenoise-inline.c"

extern linenoise(prompt: string) : string
  c  "kk_linenoise"

pub fun main()
  val name = linenoise("What is your name? ")
  ("hello " ++ name).println

linenoise-inline.c

#include "linenoise.h"

kk_string_t kk_linenoise(kk_string_t prompt, kk_context_t* ctx) {
    kk_ssize_t len = kk_string_len(prompt, ctx);
    const char * cstring = kk_string_cbuf_borrow(prompt, &len);
    char * input = linenoise(cstring);
    kk_string_t kk_input = kk_string_alloc_raw(input, true, ctx);
    return kk_input;
}

c compiler errors:

.koka/v2.3.8/gcc-debug/helloworld.c: In function ‘kk_helloworld_linenoise’:
.koka/v2.3.8/gcc-debug/helloworld.c:81:10: warning: implicit declaration of function ‘kk_linenoise’; did you mean ‘linenoise’? [-Wimplicit-function-declaration]
   81 |   return kk_linenoise(prompt,kk_context());
      |          ^~~~~~~~~~~~
      |          linenoise
.koka/v2.3.8/gcc-debug/helloworld.c:81:10: error: incompatible types when returning type ‘int’ but ‘kk_string_t’ {aka ‘struct kk_string_s’} was expected
   81 |   return kk_linenoise(prompt,kk_context());
      |          ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
.koka/v2.3.8/gcc-debug/helloworld.c:82:1: warning: control reaches end of non-void function [-Wreturn-type]
   82 | }
      | ^
failure during code generation:
  error  : command failed (exit code 1)
  command: /usr/bin/gcc -Wall -Wextra -Wno-unknown-pragmas -Wno-unused-parameter -Wno-unused-variable -Wno-unused-value -Wno-missing-field-initializers -Wpointer-arith -Wshadow -Wstrict-aliasing -Wno-unused-but-set-variable -g -O1 -c -I /usr/local/share/koka/v2.3.8/kklib/include -DKK_MIMALLOC=8 -o .koka/v2.3.8/gcc-debug/helloworld.o .koka/v2.3.8/gcc-debug/helloworld.c

So it seems like all the errors are being caused by a missing kk_linenoise declaration, and I checked the generated source code to see if that would give any further clues. What I found was that kk_linenoise was in fact not declared anywhere. When I copied the kk_linenoise into the generated c code, the error went away.

So basically what I want to do is get the compiler to include kk_linenoise on its own if possible.

I looked over a bunch of std library code and the comments above to get the code I have now, but I still seem to be missing something. Thank you all for the help!

[JonahBranch]

Hello everyone. I believe I have figured it out. The import should look like this:

extern import
  c file "linenoise.h"
extern import
  c file "linenoise.c"
extern import
  c file "linenoise-inline.c"

This got all the files included into the compiler. Also btw you shouldn't trust my linenoise-inline function from above. Now that I can test it, it looks like it has many errors 😅
But I am making progress now!

[daanx]

Great to see it is working now for you (although I prefer of course isocline over linenoise ;-) ) . Eech extern import is indeed meant for just one import but can be different for different backends . I will see if it is not too hard to add a warning if one tries to import for the same backend in one declaration (as you first attempt tried where the latter were just ignored)

I need to document the C interface still, but I want to delay a bit until I have libuv intergrated -- that is a complex library and I am sure I will learn things and maybe change some things. It would be great to also support strings in an easier way.

btw, some hints on strings, your function should probably look more like:

kk_string_t kk_linenoise(kk_string_t prompt, kk_context_t* ctx) {
    kk_ssize_t len;
    const char * cstring = kk_string_cbuf_borrow(prompt, &len);
    char * input = linenoise(cstring);
    kk_string_t kk_input = kk_string_alloc_from_qutf8(input, ctx);  // guarantees well-formed utf8 inside koka (unlike raw)
    linenoiseFree(input);
    kk_string_drop(prompt,ctx);
    return kk_input;
}

(untested, just a hint)
(also the std/text/regex may provide a useful example of a somewhat complex binding)

[JonahBranch]

Thanks for the tips! isocline does look pretty good, I haven't heard of it before now. Does a koka binding exist for it yet or has it just been used with Haskell?