Converting C++ code with MediaPipe library into Web Assembly

[prantoran]

I organized my project in the same workspace as MediaPipe and build my project with Bazel. Now I am looking into converting my project into a wasm Web Assembly file so that I can import it on a static website.

I thought about directly converting using Ecmscript but not sure how to efficiently include all the dependencies of MediaPipe present in the MediaPipe's WORKSPACE file.

I am also wondering how to use Ecmscript when building the MediaPipe project using Bazel.

Also, is it better to include MediaPipe as a separate dependency and organize my project from scratch with a separate WORKSPACE file? If so then how to include MediaPipe as a dependency?

Is there any code demo where MediaPipe example is converted to Web Assembly?

I am new to Bazel and Ecmscript so the questions might seem obvious ^_^

[prantoran]

I tried to it by getting the .bazelrc and WORKSPACE from this blog: https://hackernoon.com/c-to-webassembly-using-bazel-and-emscripten-4him3ymc. I can produce Web Assembly code for simple cc code with no dependencies.

However, I am stuck with 2 types of errors based on what MediaPipe dependency I include in the BUILD file.

Here is my .bazelrc:

# The bazelrc file for MediaPipe OSS.

# Tensorflow needs remote repo
common --experimental_repo_remote_exec

# Basic build settings
build --jobs 128
build --define='absl=1'
build --enable_platform_specific_config

# Linux
build:linux --cxxopt=-std=c++14
build:linux --host_cxxopt=-std=c++14
build:linux --copt=-w

# emscripten

##### WASM #####
# Use our custom-configured c++ toolchain.
build:wasm --crosstool_top=//javascript/toolchain:emscripten

# Use --cpu as a differentiator.
build:wasm --cpu=wasm

# Use the default C++ toolchain to build the tools used during the build.
build:wasm --host_crosstool_top=@bazel_tools//tools/cpp:toolchain

# These compile flags are active no matter which build mode we are in
# (dbg vs opt). For flags specific to build mode, see cc_toolchain_config.bzl.
#build:wasm --cxxopt="-flto" --copt=-flto
#build:wasm --host_cxxopt="-fno-rtti" # disable generation of info of every class for runtime access
build:wasm --host_cxxopt="-fno-exceptions"
#build:wasm --host_cxxopt="-fomit-frame-pointer"

build:wasm --cxxopt=-std=c++14
build:wasm --host_cxxopt=-std=c++14
#build:wasm --copt=-w

# Disable sandbox environment because emsdk caches files by writing to
# home directory.
build:wasm --spawn_strategy=local

I turned off the optimization flags --cxxopt="-flto", --host_cxxopt="-fomit-frame-pointer", --host_cxxopt="-fno-rtti" on purpose. In fact, --host_cxxopt="-fno-rtti" prevented dynamic type inferencing.

Here is the WORKSPACE:

workspace(name = "practice")

load("@bazel_tools//tools/build_defs/repo:http.bzl", "http_archive")
load("@bazel_tools//tools/build_defs/repo:git.bzl", "git_repository")


skylib_version = "0.9.0"
http_archive(
    name = "bazel_skylib",
    type = "tar.gz",
    url = "https://github.com/bazelbuild/bazel-skylib/releases/download/{}/bazel_skylib-{}.tar.gz".format (skylib_version, skylib_version),
    sha256 = "1dde365491125a3db70731e25658dfdd3bc5dbdfd11b840b3e987ecf043c7ca0",
)
load("@bazel_skylib//lib:versions.bzl", "versions")
versions.check(minimum_bazel_version = "2.0.0")


# ABSL cpp library lts_2020_02_25
http_archive(
    name = "com_google_absl",
    urls = [
        "https://github.com/abseil/abseil-cpp/archive/20200225.tar.gz",
    ],
    # Remove after https://github.com/abseil/abseil-cpp/issues/326 is solved.
    patches = [
        "@//third_party:com_google_absl_f863b622fe13612433fdf43f76547d5edda0c93001.diff"
    ],
    patch_args = [
        "-p1",
    ],
    strip_prefix = "abseil-cpp-20200225",
    sha256 = "728a813291bdec2aa46eab8356ace9f75ac2ed9dfe2df5ab603c4e6c09f1c353"
)

http_archive(
    name = "rules_cc",
    strip_prefix = "rules_cc-master",
    urls = ["https://github.com/bazelbuild/rules_cc/archive/master.zip"],
)

# GoogleTest/GoogleMock framework. Used by most unit-tests.
http_archive(
     name = "com_google_googletest",
     urls = ["https://github.com/google/googletest/archive/master.zip"],
     strip_prefix = "googletest-master",
)

# Google Benchmark library.
http_archive(
    name = "com_google_benchmark",
    urls = ["https://github.com/google/benchmark/archive/master.zip"],
    strip_prefix = "benchmark-master",
    build_file = "@//third_party:benchmark.BUILD",
)

# gflags needed by glog
http_archive(
    name = "com_github_gflags_gflags",
    strip_prefix = "gflags-2.2.2",
    sha256 = "19713a36c9f32b33df59d1c79b4958434cb005b5b47dc5400a7a4b078111d9b5",
    url = "https://github.com/gflags/gflags/archive/v2.2.2.zip",
)

# glog v0.3.5
# TODO: Migrate MediaPipe to use com_github_glog_glog on all platforms.
http_archive(
    name = "com_github_glog_glog_v_0_3_5",
    url = "https://github.com/google/glog/archive/v0.3.5.zip",
    sha256 = "267103f8a1e9578978aa1dc256001e6529ef593e5aea38193d31c2872ee025e8",
    strip_prefix = "glog-0.3.5",
    build_file = "@//third_party:glog.BUILD",
    patches = [
        "@//third_party:com_github_glog_glog_9779e5ea6ef59562b030248947f787d1256132ae.diff"
    ],
    patch_args = [
        "-p1",
    ],
)

# 2020-02-16
http_archive(
    name = "com_github_glog_glog",
    strip_prefix = "glog-3ba8976592274bc1f907c402ce22558011d6fc5e",
    sha256 = "feca3c7e29a693cab7887409756d89d342d4a992d54d7c5599bebeae8f7b50be",
    urls = [
        "https://github.com/google/glog/archive/3ba8976592274bc1f907c402ce22558011d6fc5e.zip",
    ],
)

# easyexif
http_archive(
    name = "easyexif",
    url = "https://github.com/mayanklahiri/easyexif/archive/master.zip",
    strip_prefix = "easyexif-master",
    build_file = "@//third_party:easyexif.BUILD",
)

# libyuv
http_archive(
    name = "libyuv",
    urls = ["https://chromium.googlesource.com/libyuv/libyuv/+archive/refs/heads/master.tar.gz"],
    build_file = "@//third_party:libyuv.BUILD",
)

# Note: protobuf-javalite is no longer released as a separate download, it's included in the main Java download.
# ...but the Java download is currently broken, so we use the "source" download.
http_archive(
    name = "com_google_protobuf_javalite",
    sha256 = "a79d19dcdf9139fa4b81206e318e33d245c4c9da1ffed21c87288ed4380426f9",
    strip_prefix = "protobuf-3.11.4",
    urls = ["https://github.com/protocolbuffers/protobuf/archive/v3.11.4.tar.gz"],
)

http_archive(
    name = "com_google_protobuf",
    sha256 = "a79d19dcdf9139fa4b81206e318e33d245c4c9da1ffed21c87288ed4380426f9",
    strip_prefix = "protobuf-3.11.4",
    urls = ["https://github.com/protocolbuffers/protobuf/archive/v3.11.4.tar.gz"],
    patches = [
        "@//third_party:com_google_protobuf_fixes.diff"
    ],
    patch_args = [
        "-p1",
    ],
)

http_archive(
    name = "com_google_audio_tools",
    strip_prefix = "multichannel-audio-tools-master",
    urls = ["https://github.com/google/multichannel-audio-tools/archive/master.zip"],
)

http_archive(
    name = "ceres_solver",
    url = "https://github.com/ceres-solver/ceres-solver/archive/1.14.0.zip",
    patches = [
        "@//third_party:ceres_solver_compatibility_fixes.diff"
    ],
    patch_args = [
        "-p1",
    ],
    strip_prefix = "ceres-solver-1.14.0",
    sha256 = "5ba6d0db4e784621fda44a50c58bb23b0892684692f0c623e2063f9c19f192f1"
)

new_local_repository(
    name = "linux_opencv",
    build_file = "@//third_party:opencv_linux.BUILD",
    path = "/usr/local",
)

new_local_repository(
    name = "linux_ffmpeg",
    build_file = "@//third_party:ffmpeg_linux.BUILD",
    path = "/usr"
)

# Maven dependencies.

RULES_JVM_EXTERNAL_TAG = "3.2"
RULES_JVM_EXTERNAL_SHA = "82262ff4223c5fda6fb7ff8bd63db8131b51b413d26eb49e3131037e79e324af"

http_archive(
    name = "rules_jvm_external",
    strip_prefix = "rules_jvm_external-%s" % RULES_JVM_EXTERNAL_TAG,
    sha256 = RULES_JVM_EXTERNAL_SHA,
    url = "https://github.com/bazelbuild/rules_jvm_external/archive/%s.zip" % RULES_JVM_EXTERNAL_TAG,
)

load("@rules_jvm_external//:defs.bzl", "maven_install")



# Needed by TensorFlow
http_archive(
    name = "io_bazel_rules_closure",
    sha256 = "e0a111000aeed2051f29fcc7a3f83be3ad8c6c93c186e64beb1ad313f0c7f9f9",
    strip_prefix = "rules_closure-cf1e44edb908e9616030cc83d085989b8e6cd6df",
    urls = [
        "http://mirror.tensorflow.org/github.com/bazelbuild/rules_closure/archive/cf1e44edb908e9616030cc83d085989b8e6cd6df.tar.gz",
        "https://github.com/bazelbuild/rules_closure/archive/cf1e44edb908e9616030cc83d085989b8e6cd6df.tar.gz",  # 2019-04-04
    ],
)

#Tensorflow repo should always go after the other external dependencies.
# 2020-05-11
_TENSORFLOW_GIT_COMMIT = "7c09d15f9fcc14343343c247ebf5b8e0afe3e4aa"
_TENSORFLOW_SHA256= "673d00cbd2676ae43df1993e0d28c10b5ffbe96d9e2ab29f88a77b43c0211299"
http_archive(
    name = "org_tensorflow",
    urls = [
      "https://mirror.bazel.build/github.com/tensorflow/tensorflow/archive/%s.tar.gz" % _TENSORFLOW_GIT_COMMIT,
      "https://github.com/tensorflow/tensorflow/archive/%s.tar.gz" % _TENSORFLOW_GIT_COMMIT,
    ],
    patches = [
        "@//third_party:org_tensorflow_compatibility_fixes.diff",
    ],
    patch_args = [
        "-p1",
    ],
    strip_prefix = "tensorflow-%s" % _TENSORFLOW_GIT_COMMIT,
    sha256 = _TENSORFLOW_SHA256,
)

load("@org_tensorflow//tensorflow:workspace.bzl", "tf_workspace")
tf_workspace(tf_repo_name = "org_tensorflow")

load("//javascript/toolchain:cc_toolchain_config.bzl", "emsdk_configure")
emsdk_configure(name = "emsdk")

In particular, I added the lines in the end:

load("//javascript/toolchain:cc_toolchain_config.bzl", "emsdk_configure")
emsdk_configure(name = "emsdk")

The contents of //javascript/toolchain:cc_toolchain_config.bzl:

load("@bazel_tools//tools/build_defs/cc:action_names.bzl", "ACTION_NAMES")
load(
    "@bazel_tools//tools/cpp:cc_toolchain_config_lib.bzl",
    "feature",
    "flag_group",
    "flag_set",
    "tool_path",
    "with_feature_set",
)

def _impl(ctx):
    tool_paths = [
        tool_path(
            name = "gcc",
            path = "emcc.sh",
        ),
        tool_path(
            name = "ld",
            path = "emcc.sh",
        ),
        tool_path(
            name = "ar",
            path = "emar.sh",
        ),
        tool_path(
            name = "cpp",
            path = "false.sh",
        ),
        tool_path(
            name = "gcov",
            path = "false.sh",
        ),
        tool_path(
            name = "nm",
            path = "NOT_USED",
        ),
        tool_path(
            name = "objdump",
            path = "false.sh",
        ),
        tool_path(
            name = "strip",
            path = "NOT_USED",
        ),
    ]
    preprocessor_compile_actions = [
        ACTION_NAMES.c_compile,
        ACTION_NAMES.cpp_compile,
        ACTION_NAMES.linkstamp_compile,
        ACTION_NAMES.preprocess_assemble,
        ACTION_NAMES.cpp_header_parsing,
        ACTION_NAMES.cpp_module_compile,
        ACTION_NAMES.clif_match,
    ]

    all_link_actions = [
        ACTION_NAMES.cpp_link_executable,
        ACTION_NAMES.cpp_link_dynamic_library,
        ACTION_NAMES.cpp_link_nodeps_dynamic_library,
    ]

    all_compile_actions = [
        ACTION_NAMES.c_compile,
        ACTION_NAMES.cpp_compile,
        ACTION_NAMES.linkstamp_compile,
        ACTION_NAMES.assemble,
        ACTION_NAMES.preprocess_assemble,
        ACTION_NAMES.cpp_header_parsing,
        ACTION_NAMES.cpp_module_compile,
        ACTION_NAMES.cpp_module_codegen,
        ACTION_NAMES.clif_match,
        ACTION_NAMES.lto_backend,
    ]
    toolchain_include_directories_feature = feature(
        name = "toolchain_include_directories",
        enabled = True,
        flag_sets = [
            flag_set(
                actions = all_compile_actions,
                flag_groups = [
                    flag_group(
                        flags = [
                            "-isystem",
                            "external/emsdk/emsdk/upstream/emscripten/system/include/libcxx",
                            "-isystem",
                            "external/emsdk/emsdk/upstream/emscripten/system/lib/libcxxabi/include",
                            "-isystem",
                            "external/emsdk/emsdk/upstream/emscripten/system/include/compat",
                            "-isystem",
                            "external/emsdk/emsdk/upstream/emscripten/system/include",
                            "-isystem",
                            "external/emsdk/emsdk/upstream/emscripten/system/include/libc",
                            "-isystem",
                            "external/emsdk/emsdk/upstream/emscripten/system/lib/libc/musl/arch/emscripten",
                            "-isystem",
                            "external/emsdk/emsdk/upstream/emscripten/system/local/include",
                        ],
                    ),
                ],
            ),
        ],
    )

    crosstool_default_flag_sets = [
        # Optimized (opt)
        flag_set(
            actions = preprocessor_compile_actions,
            flag_groups = [flag_group(flags = ["-DNDEBUG"])],
            with_features = [with_feature_set(features = ["opt"])],
        ),
        # In advanced C++ apps, -O3 can break asmjs.
        flag_set(
            actions = all_compile_actions + all_link_actions,
            flag_groups = [flag_group(flags = ["-g0", "-O3"])], 
            with_features = [with_feature_set(features = ["opt"])],
        ),
        # Fastbuild (fastbuild)
        flag_set(
            actions = all_compile_actions + all_link_actions,
            flag_groups = [flag_group(flags = ["-O2"])],
            with_features = [with_feature_set(features = ["fastbuild"])],
        ),
        # Debug (dbg)
        flag_set(
            actions = all_compile_actions + all_link_actions,
            flag_groups = [flag_group(flags = ["-g2", "-O0"])],
            with_features = [with_feature_set(features = ["dbg"])],
        ),
    ]

    features = [
        toolchain_include_directories_feature,
        # These 3 features will be automatically enabled by blaze in the
        # corresponding build mode.
        feature(
            name = "opt",
            provides = ["variant:crosstool_build_mode"],
        ),
        feature(
            name = "dbg",
            provides = ["variant:crosstool_build_mode"],
        ),
        feature(
            name = "fastbuild",
            provides = ["variant:crosstool_build_mode"],
        ),
        feature(
            name = "crosstool_default_flags",
            enabled = True,
            flag_sets = crosstool_default_flag_sets,
        ),
    ]

    return cc_common.create_cc_toolchain_config_info(
        ctx = ctx,
        toolchain_identifier = "wasm-toolchain",
        host_system_name = "i686-unknown-linux-gnu",
        target_system_name = "wasm-unknown-emscripten",
        target_cpu = "wasm",
        target_libc = "unknown",
        compiler = "emscripten",
        abi_version = "unknown",
        abi_libc_version = "unknown",
        tool_paths = tool_paths,
        features = features,
    )

cc_toolchain_config = rule(
    implementation = _impl,
    attrs = {},
    provides = [CcToolchainConfigInfo],
)

def _emsdk_impl(ctx):
    if "EMSDK" not in ctx.os.environ or ctx.os.environ["EMSDK"].strip() == "":
        fail("The environment variable EMSDK is not found. " +
             "Did you run source ./emsdk_env.sh ?")
    path = ctx.os.environ["EMSDK"]
    ctx.symlink(path, "emsdk")
    ctx.file("BUILD", """
filegroup(
    name = "all",
    srcs = glob(["emsdk/**"]),
    visibility = ["//visibility:public"],
)
""")

emsdk_configure = repository_rule(
    implementation = _emsdk_impl,
    local = True,
)

And the dummy cc file:

#include <emscripten/bind.h>
#include <iostream>
#include <sys/syscall.h>



int main() {

    std::cout << "empty inside\n";
}

And the BUILD file used for bazel build:

package(default_visibility = ["//visibility:public"])

DEFAULT_EMSCRIPTEN_LINKOPTS = [
    # "-flto",                        # Specify lto (has to be set on for compiler as well)
    "-O0",
    "--bind",                       # Compiles the source code using the Embind bindings to connect C/C++ and JavaScript
    "--closure 1",                  # Run the closure compiler
    "-s MALLOC=emmalloc",           # Switch to using the much smaller implementation
    "-s ALLOW_MEMORY_GROWTH=1",     # Our example doesn't need memory growth
    "-s USE_PTHREADS=1",            # Disable pthreads
    "-s ASSERTIONS=0",              # Turn off assertions
    "-s EXPORT_ES6=1",              # Export as es6 module, used for rollup
    "-s MODULARIZE=1",              # Allows us to manually invoke the initializatio of wasm
    "-s EXPORT_NAME=createModule",  # Not used, but good to specify
    "-s USE_ES6_IMPORT_META=0",     # Disable loading from import meta since we use rollup
    "-s SINGLE_FILE=1"              # Pack all webassembly into base64
]

WASM_LINKOPTS = [
    "-s WASM=1",                    # Specify wasm output
]


linux_or_darwin_copts = [
    # Disable warnings that exists in glog.
    "-Wno-sign-compare",
    "-Wno-unused-function",
    "-Wno-unused-local-typedefs",
    "-Wno-unused-variable",
    # # Allows src/base/mutex.h to include pthread.h.
    "-DHAVE_PTHREAD",
    # # Allows src/logging.cc to determine the host name.
    "-DHAVE_SYS_UTSNAME_H",
    # For src/utilities.cc.
    "-DHAVE_SYS_SYSCALL_H",
    "-DHAVE_SYS_TIME_H",
    # # Enable dumping stacktrace upon sigaction.
    # "-DHAVE_SIGACTION",
    # # For logging.cc.
    "-DHAVE_PREAD",
    "-DHAVE___ATTRIBUTE__",
]

cc_binary(
    name = "khaliwasm",
    srcs = ["bindings/khali.cpp"],
    linkopts = DEFAULT_EMSCRIPTEN_LINKOPTS + WASM_LINKOPTS + linux_or_darwin_copts,
    deps = [
        "//desktop:main_gpu",
        # "//mediapipe/graphs/hand_tracking:multi_hand_mobile_calculators",
    ],
)

The desktop:main_gpu contains the BUILD contents of the Two Hands GPU example.

The command I used for bazel build:

 bazel build -c opt //javascript:khaliwasm --config=wasm --sandbox_debug --verbose_failures

Now, if I do not include the calculators dependency then I get this main error related to SYS_gettid:

Execution platform: @local_execution_config_platform//:platform
mediapipe/framework/deps/threadpool_pthread_impl.cc:63:64: error: use of undeclared identifier 'SYS_gettid'
      internal::CreateThreadName(thread->name_prefix_, syscall(SYS_gettid));

If I include the calculators dependency then I get the following error:

ERROR: /home/prantoran/.cache/bazel/_bazel_prantoran/53d0985b61cdf3fef140f3aded9a4395/external/cpuinfo/BUILD.bazel:96:11: Configurable attribute "srcs" doesn't match this configuration (would a default condition help?).
Conditions checked:
 @cpuinfo//:linux_x86_64
 @cpuinfo//:linux_arm
 @cpuinfo//:linux_armhf
 @cpuinfo//:linux_aarch64
 @cpuinfo//:macos_x86_64
 @cpuinfo//:windows_x86_64
 @cpuinfo//:android_armv7
 @cpuinfo//:android_arm64
 @cpuinfo//:android_x86
 @cpuinfo//:android_x86_64
 @cpuinfo//:ios_x86_64
 @cpuinfo//:ios_x86
 @cpuinfo//:ios_armv7
 @cpuinfo//:ios_arm64
 @cpuinfo//:ios_arm64e
 @cpuinfo//:watchos_x86_64
 @cpuinfo//:watchos_x86
 @cpuinfo//:watchos_armv7k
 @cpuinfo//:watchos_arm64_32
 @cpuinfo//:tvos_x86_64
 @cpuinfo//:tvos_arm64
ERROR: Analysis of target '//javascript:khaliwasm' failed; build aborted: Analysis failed
INFO: Elapsed time: 1.252s
INFO: 0 processes.
FAILED: Build did NOT complete successfully (2 packages loaded, 3563 targets configured)

I do not have any idea about how to solve the errors

[prantoran]

I was experimenting this again and stumbled onto the following error:

mediapipe/mediapipe/framework/deps/BUILD:314:11: C++ compilation of rule '//mediapipe/framework/deps:threadpool' failed (Exit 1)
mediapipe/framework/deps/threadpool_pthread_impl.cc:63:64: error: use of undeclared identifier 'SYS_gettid'
      internal::CreateThreadName(thread->name_prefix_, syscall(SYS_gettid));
                                                               ^
mediapipe/framework/deps/threadpool_pthread_impl.cc:101:15: error: use of undeclared identifier 'pthread_setname_np'
  int error = pthread_setname_np(name.c_str());

and found a prospective solution which requires using a specific commit of GLog. I was thinking how to add the commit update locally.

[droidlabour]

@prantoran Were you able to fix this? Also the web version seems to be loading a .data binary file.
Do you think it means that we can't deploy our own tflite Model and will have to stick to the model inside that .data file.

[prantoran]

@droidlabour I have not made any new attempts. Last time, I was stuck with GLog version in build files which was later updated in MediaPipe. Currently, I am learning about wasm and pthreads, after which I will attempt again. Not sure about .data binary file though.

[vKrypto]

I was experimenting this again and stumbled onto the following error:

mediapipe/mediapipe/framework/deps/BUILD:314:11: C++ compilation of rule '//mediapipe/framework/deps:threadpool' failed (Exit 1)
mediapipe/framework/deps/threadpool_pthread_impl.cc:63:64: error: use of undeclared identifier 'SYS_gettid'
      internal::CreateThreadName(thread->name_prefix_, syscall(SYS_gettid));
                                                               ^
mediapipe/framework/deps/threadpool_pthread_impl.cc:101:15: error: use of undeclared identifier 'pthread_setname_np'
  int error = pthread_setname_np(name.c_str());

and found a prospective solution which requires using a specific commit of GLog. I was thinking how to add the commit update locally.

@prantoran , I am also stuck at the same, I guess you should try again as MediaPipe launched a release and you already explored a lot. best of luck in advance.

[prantoran]

@vKrypto It is harder than it seems to be, currently I am doing the desktop version but trying to port to web-assembly. I faced the pthread issue, as far as I know the current web-assembly version do not support multithreading natively and I am not sure whether XNNPACK web-assembly version or TensorflowJS was used for web version. There is a MediaPipe pre-trained handpose model available as npm package that uses TensorflowJS so I guess TensorflowJS is compatible.

[tyrmullen]

Hi all-- this is quite a tricky task, so expect to run into a lot of strange build errors along the way. However, a few tips that might be helpful:
(1) I'd recommend (even if building on Desktop) that you use mostly mobile versions of the different MediaPipe components (so mostly iOS/Android C++ code), since mobile code is generally more portable.
(2) I'd also recommend starting simpler with the options, and then once you have things building, you can increase the specificity/difficulty. So for example, leave most things as default (e.g. don't change malloc), and then once things work tweak for optimizations/improvements.
(3) Disable pthreads, and consider enabling assertions.
(4) Make sure you're using the official emscripten bazel rules (I think the TF.js project you indirectly copied from initially wasn't, but a few days ago was updated to do so? Not sure on the details there).
(5) When I tried a build this way, glog was the library I ran into the first BUILD incompatibility with-- so if you're reaching that, then you're off to a good start.

Also, yes, the .data file is for packaging up assets to load at runtime, and is tied to the Emscripten virtual FileSystem. So don't worry too much about that, since it won't really be relevant until you have things building.

[evanbrown3000]

Hi, has anyone made progress building Mediapipe for Javascript?
I am trying to use Objectron

[sgowroji]

Hi @prantoran, Did you get a chance to go through this comment

[prantoran]

@sgowroji I saw the comment, I am working on it. I did make some progress by gradually adding dependencies and fixing bugs.

[google-ml-butler]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you.

[google-ml-butler]

Closing as stale. Please reopen if you'd like to work on this further.

[prantoran]

I merged all the changes that seemed essential to me in a single commit on the latest version of MediaPipe: https://github.com/prantoran/mediapipe/tree/latest_wasm.

The latest_wasm has all the changes in a single commit while the default branch refers to the original project.

[LebronJames0423]

@prantoran @tyrmullen could you tell :
Background of my question:

1. goal to use: mediapipe face mesh solution in js.
2. my environment : PC windows10，CPU: intel-i7-8700, GPU: intel UHD Graphics 630
3. already run:
   1. official mediapipe js face mesh solution and tf-js face mesh model. The performance is 15fps but with 65% GPU Utilization. This high Utlization of gpu leads to few gpus left for other programs to use.
   2. c++ CPU version mediapipe face mesh solution, i can compile it to a dll, and it runs fast in about 3-5ms( only inference part without rendering), and it can meet my demand. so i want converting this c++ CPU mediapipe library into wasm.

MY question:

1. is it possible to converting this c++ CPU version mediapipe library into wasm?
2. if it possible , does the speed of wasm version can be as fast as the c++ cpu version?
3. how can i learn from your project?

Thanks a lot!

[prantoran]

@LebronJames0423 In the latest branch of the project, all the changes in the MediaPipe's internal files are shown in the commits.

One caveat is that I developed using Ubuntu so there might be additional requirements for Windows.

[tyrmullen]

@LebronJames0423 For any of our APIs running on GPU, you can instead force CPU ML inference by setting the useCpuInference option to true in the configuration options. This should work for our FaceMesh API as well. So to answer your first two questions:
(1) Yes-- just use the above advice.
(2) C++ CPU should always be faster, all other things being equal. But the gap between them used to be huge, and now is much smaller, and continuing to narrow (with features like Wasm-SIMD helping tremendously).

[LebronJames0423]

@LebronJames0423 For any of our APIs running on GPU, you can instead force CPU ML inference by setting the useCpuInference option to true in the configuration options. This should work for our FaceMesh API as well. So to answer your first two questions: (1) Yes-- just use the above advice. (2) C++ CPU should always be faster, all other things being equal. But the gap between them used to be huge, and now is much smaller, and continuing to narrow (with features like Wasm-SIMD helping tremendously).
@tyrmullen @prantoran OK，I will try your suggestions，thanks!

[tso996]

@prantoran By refering to your repo, I was able to compile the hello world desktop into webassembly. I was able to load the wasm file in chrome and the cout commands in the cpp file were appearing in the browser console as well. However, the wasm hangs at poller.next(&packet), as is evident from observing the debug flags I had put in the cpp code from the console. I did some digging in the internet and found out that it is hanging because poller.next() waits until the next packet is available. The same code is working properly when run from the terminal as a cpp file but upon converting to wasm something is not working. It would be great if someone could provide more information on why the code behaves as such upon being converted to wasm.

build file for cpp helloworld

cc_binary(
    name = "hello_world",
    srcs = ["hello_world.cc"],
    visibility = ["//visibility:public"],
    deps = [
        "//mediapipe/calculators/core:pass_through_calculator",
        "//mediapipe/framework:calculator_graph",
        "//mediapipe/framework/port:logging",
        "//mediapipe/framework/port:parse_text_proto",
        "//mediapipe/framework/port:status",
    ],
)

cpp helloworld code that runs properly

#include "mediapipe/framework/calculator_graph.h"
#include "mediapipe/framework/port/logging.h"
#include "mediapipe/framework/port/parse_text_proto.h"
#include "mediapipe/framework/port/status.h"

//namespace mediapipe {

int PrintHelloWorld() {
  // Configures a simple graph, which concatenates 2 PassThroughCalculators.
  mediapipe::CalculatorGraphConfig config =
      mediapipe::ParseTextProtoOrDie<mediapipe::CalculatorGraphConfig>(R"pb(
        input_stream: "in"
        output_stream: "out"
        node {
          calculator: "PassThroughCalculator"
          input_stream: "in"
          output_stream: "out1"
        }
        node {
          calculator: "PassThroughCalculator"
          input_stream: "out1"
          output_stream: "out"
        }
      )pb");

  mediapipe::CalculatorGraph graph;
  graph.Initialize(config);
  auto status_or_poller = graph.AddOutputStreamPoller("out");
  mediapipe::OutputStreamPoller poller = std::move(status_or_poller.value());
  graph.StartRun({});
  // Give 10 input packets that contains the same std::string "Hello World!".
  for (int i = 0; i < 10; ++i) {
    graph.AddPacketToInputStream(
        "in", mediapipe::MakePacket<std::string>("Hello World!").At(mediapipe::Timestamp(i)));
  }
  // Close the input stream "in".
  graph.CloseInputStream("in");
  mediapipe::Packet packet;
  // Get the output packets std::string.
  while (poller.Next(&packet)) {
    std::cout << poller.QueueSize()<< std::endl;
    std::cout << packet.Get<std::string>()<<std::endl;
  }
  // return graph.WaitUntilDone();
}
//}  // namespace mediapipe

int main(int argc, char** argv) {
  google::InitGoogleLogging(argv[0]);
  PrintHelloWorld();
  //std::cout << "hello world!!!!!!!!!" << std::endl;
  return 0;
}

output as expected in the terminal

build file for the wasm helloworld:

cc_binary(
    name = "hello-world-web",
    srcs = ["helloWorldWasm.cc"],
    deps = [
        "//mediapipe/calculators/core:pass_through_calculator",
        "//mediapipe/framework:calculator_graph",
        "//mediapipe/framework/port:parse_text_proto",
        "//mediapipe/framework/port:status",
        "//third_party:glog",
        "//mediapipe/framework/port:logging",
    ],
    linkopts = [
        "-s USE_PTHREADS=0",
        "-s ALLOW_MEMORY_GROWTH=1",
        "-s ASSERTIONS=1",
        "-s USE_WEBGL2=1",
        "-s ERROR_ON_UNDEFINED_SYMBOLS=0", 
        "--bind",
    ]
)


wasm_cc_binary(
    name = "hello-world-wasm",
    cc_target = ":hello-world-web",
)

cpp code used to compile the wasm file:

#include "mediapipe/framework/calculator_graph.h"
#include "mediapipe/framework/port/logging.h"
#include "mediapipe/framework/port/parse_text_proto.h"
#include "mediapipe/framework/port/status.h"

#include <emscripten/bind.h>
//#include <emscripten.h>


    int PrintHelloWorld() {
                std::cout << "flag 1"<< std::endl;
                // Configures a simple graph, which concatenates 2 PassThroughCalculators.
                mediapipe::CalculatorGraphConfig config =
        mediapipe::ParseTextProtoOrDie<mediapipe::CalculatorGraphConfig>(R"pb(
            input_stream: "in"
            output_stream: "out"
            node {
            calculator: "PassThroughCalculator"
            input_stream: "in"
            output_stream: "out1"
            }
            node {
            calculator: "PassThroughCalculator"
            input_stream: "out1"
            output_stream: "out"
            }
        )pb");

            mediapipe::CalculatorGraph graph;
            graph.Initialize(config);
            auto status_or_poller = graph.AddOutputStreamPoller("out");
            mediapipe::OutputStreamPoller poller = std::move(status_or_poller.value());
            graph.StartRun({});
            std::cout << "flag 2"<< std::endl;
            // Give 10 input packets that contains the same std::string "Hello World!".
            for (int i = 0; i < 10; ++i) {
                graph.AddPacketToInputStream(
                    "in", mediapipe::MakePacket<std::string>("Hello World!").At(mediapipe::Timestamp(i)));
            }
            // Close the input stream "in".
            graph.CloseInputStream("in");
            std::cout << "flag 3"<< std::endl;
            mediapipe::Packet packet;
            // Get the output packets std::string.
            while (poller.Next(&packet)) {
                std::cout << "flag 4"<< std::endl;
                std::cout << poller.QueueSize()<< std::endl;
                std::cout << packet.Get<std::string>()<<std::endl;
            }
            // return graph.WaitUntilDone();
            std::cout << "flag 5"<< std::endl;
            return 1;
    }
        int main(int argc, char** argv) {
            google::InitGoogleLogging(argv[0]);
            std::cout << "hello world wasm file loaded." << std::endl;
            PrintHelloWorld();
            return 0;
          }


     EMSCRIPTEN_BINDINGS(hello_world_module) {
             emscripten::function("PrintHelloWorld", &PrintHelloWorld);
          
     }

the output from chrome console:

The chrome tab hangs and will need multiple clicks to close upon loading the wasm file. I guess it is not throwing any errors since packet.next() function is behaving as it is intended to behave, however, I would like to understand why the packets aren't in the queue like it is in the cpp example when run in the terminal.
The machine I used to compile the code to wasm is mac m1 running MacOS Montenery.
Any help is much appreciated.

[tyrmullen]

WebAssembly by default is single-threaded, while in your terminal cpp example, it is almost certainly multi-threaded. Therefore, any statements which "wait" for something to happen from another thread will not work. Instead of using OutputStreamPoller, try using ObserveOutputStream instead.

[tso996]

@tyrmullen Hi, thank you for the suggestion. Following your advice I tried to modify the code and compiled it to wasm. This time it is not hanging when it is loaded in chrome. However I'm not getting the "hello world" strings either. I might be doing something silly here.
cpp code compiled to wasm:

#include "mediapipe/framework/calculator_graph.h"
#include "mediapipe/framework/port/logging.h"
#include "mediapipe/framework/port/parse_text_proto.h"
#include "mediapipe/framework/port/status.h"

#include <emscripten/bind.h>
//#include <emscripten.h>

//extern "C"{

          //EMSCRIPTEN_KEEPALIVE
    int PrintHelloWorld() {
                std::cout << "flag 1"<< std::endl;
                // Configures a simple graph, which concatenates 2 PassThroughCalculators.
                mediapipe::CalculatorGraphConfig config =
        mediapipe::ParseTextProtoOrDie<mediapipe::CalculatorGraphConfig>(R"pb(
            input_stream: "in"
            output_stream: "out"
            node {
            calculator: "PassThroughCalculator"
            input_stream: "in"
            output_stream: "out1"
            }
            node {
            calculator: "PassThroughCalculator"
            input_stream: "out1"
            output_stream: "out"
            }
        )pb");

            mediapipe::CalculatorGraph graph;
            graph.Initialize(config);
            // auto status_or_poller = graph.AddOutputStreamPoller("out");
            // mediapipe::OutputStreamPoller poller = std::move(status_or_poller.value());
            graph.StartRun({});
            std::cout << "flag 2"<< std::endl;
            // Give 10 input packets that contains the same std::string "Hello World!".
            for (int i = 0; i < 10; ++i) {
                graph.AddPacketToInputStream(
                    "in", mediapipe::MakePacket<std::string>("Hello World!").At(mediapipe::Timestamp(i)));
            }
            // Close the input stream "in".
            graph.CloseInputStream("in");
            std::cout << "flag 3"<< std::endl;
            //mediapipe::Packet packet;
            // Get the output packets std::string.
            // while (poller.Next(&packet)) {
            //     std::cout << "flag 4"<< std::endl;
            //     std::cout << poller.QueueSize()<< std::endl;
            //     std::cout << packet.Get<std::string>()<<std::endl;
            // }
            std::string a;
            graph.ObserveOutputStream("out", [&](const mediapipe::Packet& p) {
                    a = p.Get<std::string>();
                    std::cout << "flag 4"<< std::endl;
                    std::cout << p.Get<std::string>()<<std::endl;
                    // return 80;
                    return absl::OkStatus();
                });
            std::cout << a << std::endl;
            // return graph.WaitUntilDone();
            std::cout << "flag 5"<< std::endl;
            return 1;
    }
        int main(int argc, char** argv) {
            google::InitGoogleLogging(argv[0]);
            std::cout << "hello world wasm file loaded." << std::endl;
            PrintHelloWorld();
            return 0;
          }


     EMSCRIPTEN_BINDINGS(tracking_module) {
             emscripten::function("PrintHelloWorld", &PrintHelloWorld);
          
     }

console output in chrome when the wasm is loaded:

[tyrmullen]

Try moving ObserveOutputStream to before GraphStart to make sure it's connecting, and add a WaitUntilIdle where you used to have WaitUntilDone. WaitUntilIdle will not actually wait by default on web (despite the name), but rather will force all processing to occur inline; otherwise, you're just queueing up input packets without actually running them through the graph. You may also want to move the std::cout << a << std::endl bit to after the WaitUntilIdle, or else it will still be blank.

[tso996]

@tyrmullen That seems to be what was causing the problems. It is showing the outputs correctly in chrome console now. Thanks again.

[Hrituraj202]

@LebronJames0423 For any of our APIs running on GPU, you can instead force CPU ML inference by setting the useCpuInference option to true in the configuration options. This should work for our FaceMesh API as well. So to answer your first two questions: (1) Yes-- just use the above advice. (2) C++ CPU should always be faster, all other things being equal. But the gap between them used to be huge, and now is much smaller, and continuing to narrow (with features like Wasm-SIMD helping tremendously).

Thanks for your valuable comments @tyrmullen and your work @prantoran
Please explain where and how to use this useCpuInference to make this demo project by [prantoran] faster than this GPU version.

[tyrmullen]

@Hrituraj202 I was saying that non-web C++ CPU should be faster than WebAssembly CPU.

If you're looking just at web apps, then GPU will usually be faster than CPU (although CPU lends itself better to multithreading, so it's not as straightforward a comparison), so most users will not want to use the useCpuInference flag. Also, that specific question was in reference to our JavaScript FaceMesh Solution API, so it doesn't really apply to the rest of this thread.

[tso996]

@JonathanGoorBlink I don't know about all the modules that use OpenCV, but I was able to compile the box tracking cpu module which uses OpenCV to WebAssembly following the information in this thread. My cloned repo is heavily modified but I think it might be useful.
iirc one of the things I did to get it to compile was to use the static OpenCV libraries. By default, the OpenCV BUILD file specified the .dylib files.
I built it on an M1 mac so some of the changes that I made to the various BUILD files in the project adds mac specific changes to the default parameter because Bazel was not identifying my mac as a mac os device and was choosing the default parameter.

[justinduynguyen]

@JonathanGoorBlink I don't know about all the modules that use OpenCV, but I was able to compile the box tracking cpu module which uses OpenCV to WebAssembly following the information in this thread. My cloned repo is heavily modified but I think it might be useful. iirc one of the things I did to get it to compile was to use the static OpenCV libraries. By default, the OpenCV BUILD file specified the .dylib files. I built it on an M1 mac so some of the changes that I made to the various BUILD files in the project adds mac specific changes to the default parameter because Bazel was not identifying my mac as a mac os device and was choosing the default parameter.

Can u do a bootstrap code to use box tracking cpu js ?
Thank u in advance.

[github-actions]

This issue has been marked stale because it has no recent activity since 7 days. It will be closed if no further activity occurs. Thank you.

[github-actions]

This issue was closed due to lack of activity after being marked stale for past 7 days.