Mir driver example compatible with musl or cosmo libc in statically linked mode

[cyrilmhansen]

The standard Linux libc is glibc, which only supports dynamic linking nowadays. cf https://developers.redhat.com/articles/2023/08/31/how-we-ensure-statically-linked-applications-stay-way

Static linking is still popular among people who like lightweight software like mir. Musl and cosmo libc are designed for static linking only on the contrary. https://www.alpinelinux.org is one of the most famous Linux distro using static linking and musl libc for example.

Would it be difficult to provide a mir driver example compatible with such a statically linked libc ? This means removing calls to dlopen in practice as this is not implemented by musl and only through hacks by cosmo.

[Wiguwbe]

Hi,
I'm assuming this is related to MIR generated code finding external symbols/functions (not implemented in MIR bytecode),
As far as I know, and I'm talking mostly about Linux/elf, happens in the linking process (from .o to .exe/.so) when resolving external symbols. These symbols (and their values), can be copied into the final binary (static link) or be kept in a special section for later being solved through the dynamic loader/linker (dlopen and friends),
Since MIR doesn't execute this step, as expected, the driver needs to do this symbol resolution,

The symbols can be solved through dlopen (as they are now), but to be included statically in the driver the linker of the driver needs to include all of libc in the final binary, which I think is just hard to do (I wouldn't even know how to try),

Anyway, as I said, this is just for external symbol resolution, you may not even need to load any libraries if they are not to be used.

Not much of a solution here, just to give more context

[rempas]

Take a look here and the reply after that.

[Wiguwbe]

Alright, I see what you mean, you want the mir-run.c (or equivalent) to remove the dependencies from dlopen, is that right?

[cyrilmhansen]

Yes in very practical terms, I would like to experiment with mir and cosmo libc as both seem to share the same target of lean software and low latency and like with musl libc first step seems to be the removal of all calls to dlopen().

[rofl0r]

https://www.alpinelinux.org/ is one of the most famous Linux distro using static linking and musl libc for example.

alpine does use musl, yet it rarely if at all uses static linking. even its busybox binary is dynamically linked.
in my distro sabotage linux, i use static linking for crucial system binaries, so the system can still boot even if the dynamic linker was removed or has some ABI issues.

... dlopen in practice as this is not implemented by musl and only through hacks by cosmo.

musl supports dlopen() just fine, unless you statically link.

[rofl0r]

Mir driver uses dlopen extensively to call std libc functions.

while it would be nice to be able to generate static linked binaries with a programming language based on mir, it isn't really uncommon to rely on dynamic linking especially in interpreted programming languages, e.g. perl or python. both of these can be compiled as static binaries, but then they're mostly useless as they can't load any modules with code written in C (think crypto or ssl routines that require the speed of C). perl has a feature so you can compile the extensions into the binary, but imo it's not that great when you have to recompile/link your interpreter whenever you got a new module to install. python doesn't offer this functionality, so it's either the barebone interpreter as static binary, never extendable, or the full package but relying on dynamic linking.

[cyrilmhansen]

Mir, equipped with its low-latency C compiler, shapes my perspective on dynamic loading to be more oriented towards source rather than binary. I acknowledge that this viewpoint is quite personal to me.

[rempas]

Mir, equipped with its low-latency C compiler, shapes my perspective on dynamic loading to be more oriented towards source rather than binary. I acknowledge that this viewpoint is quite personal to me.

You mean low latency loader? The C compiler is the c2m program that converts C code to Mir (either binary or textual). Not really sure how this can be called "low latency"...

[cyrilmhansen]

that's how I understand the first row of the table in the Current MIR Performance Data section of the README. Maybe I also misunderstood this.

[rempas]

that's how I understand the first row of the table in the Current MIR Performance Data section of the README. Maybe I also misunderstood this.

It's ok! Mir isn't AOT, but JIT. Meaning that the "binary" Mir presentation isn't really "binary" with the normal kind of sense, but actually bytecode. This row tells you how much time it takes to convert that bytecode to binary compared to other compilers. This is done by the loader at runtime, compared to AOT compilers that do when you compile the program and then you just "execute" that code.

[Wiguwbe]

So I adapted the mir-run.c to remove everything related to dynamic linking, I named it mir-run-static.c. I'm compiling from the mir directory with musl-gcc -o mir-run-static mir-run-static.c -static -I. -L. -lmir.

I also added the printf to the import_resolver just to be easier to understand if thing are happening.

Here is the mir-run-static.c:

#include <stdio.h>
#include <string.h>
#include <stdlib.h>
#include "mir-gen.h"

#define MIR_TYPE_INTERP 1
#define MIR_TYPE_INTERP_NAME "interp"
#define MIR_TYPE_JIT 2
#define MIR_TYPE_JIT_NAME "jit"
#define MIR_TYPE_LAZY 3
#define MIR_TYPE_LAZY_NAME "lazy"

#define MIR_TYPE_DEFAULT MIR_TYPE_INTERP

#define MIR_ENV_VAR_TYPE "MIR_TYPE"

int _mir_printi(int i) {
  return printf("%d\n", i);
}

static void *import_resolver (const char *name) {
  if (!strcmp(name, "printf")) return printf;

  fprintf(stderr, "can not load symbol %s\n", name);
  exit (1);
}

static int get_mir_type(void) {
  const char *type_value = getenv(MIR_ENV_VAR_TYPE);
  if (type_value == NULL || type_value[0] == '\0')
    return MIR_TYPE_DEFAULT;

  if (strcmp(type_value, MIR_TYPE_INTERP_NAME) == 0)
    return MIR_TYPE_INTERP;

  if (strcmp(type_value, MIR_TYPE_JIT_NAME) == 0)
    return MIR_TYPE_JIT;

  if (strcmp(type_value, MIR_TYPE_LAZY_NAME) == 0)
    return MIR_TYPE_LAZY;

  fprintf(stderr, "warning: unknown MIR_TYPE '%s', using default one\n", type_value);
  return MIR_TYPE_DEFAULT;
}

int main (int argc, char **argv, char **envp) {

#ifdef _WITH_BINFMT_MISC
  if (argc < 3) {
    fprintf(stderr, "usage: %s <full-path> <name> [<arg>...]\n", argv[0]);
    return 1;
  }
#else
  if (argc < 2) {
    fprintf(stderr, "usage: %s <bmir-file> [<arg>...]\n", argv[0]);
    return 1;
  }
#endif

  int mir_type = get_mir_type();

  MIR_val_t val;
  int exit_code;

  MIR_item_t main_func = NULL;

  MIR_context_t mctx = MIR_init();

  FILE *mir_file = fopen(argv[1], "r");
  if (!mir_file) {
    fprintf(stderr, "failed to open file '%s'\n", argv[1]);
    return 1;
  }
  MIR_read(mctx, mir_file);

  for (MIR_module_t module = DLIST_HEAD (MIR_module_t, *MIR_get_module_list (mctx)); module; module = DLIST_NEXT (MIR_module_t, module)) {
    for (MIR_item_t func = DLIST_HEAD (MIR_item_t, module->items); func; func = DLIST_NEXT (MIR_item_t, func)) {
      if (func ->item_type != MIR_func_item) continue;
      if (strcmp (func->u.func->name, "main") == 0) main_func = func;
    }
    MIR_load_module (mctx, module);
  }
  if (main_func == NULL) {
    fprintf(stderr, "cannot execute program w/o main function\n");
    return 1;
  }

#ifdef _WITH_BINFMT_MISC
#define _SKIP_ARGS 2
#else
#define _SKIP_ARGS 1
#endif

  if (mir_type == MIR_TYPE_INTERP) {
    MIR_link (mctx, MIR_set_interp_interface, import_resolver);
    MIR_interp (
      mctx, main_func, &val, 3,
      (MIR_val_t){.i = (argc-_SKIP_ARGS)},
      (MIR_val_t){.a = (void*)(argv+_SKIP_ARGS)},
      (MIR_val_t){.a = (void*)envp}
    );
    exit_code = val.i;
  } else {
    MIR_gen_init (mctx, 1);
    MIR_link (mctx,
              mir_type == MIR_TYPE_JIT
                ? MIR_set_gen_interface
                : MIR_set_lazy_gen_interface,
              import_resolver);
    uint64_t (*fun_addr)(int, char**, char**) = MIR_gen(mctx, 0, main_func);
    exit_code = fun_addr(argc-_SKIP_ARGS, argv+_SKIP_ARGS, envp);
    MIR_gen_finish (mctx);
  }

  MIR_finish (mctx);

  return exit_code;
}

I'm testing with the following C file, using the c2m:

int printf(char *fmt, ...);

int main() {
        printf("hello\n");
        printf("world\n");
        return 0;
}