Glimpse — Communicating with the richness of our physical selves

We are building a real-time motion capture system for the body using mobile devices such as phones and AR/VR headsets.

It's important to us that this kind of technology not be locked within walled gardens as the ability to interact and express with our bodies is most interesting when that can include everyone.

Our rendering pipeline (for generating training data) is based on open source projects including Blender and Makehuman, and all our code for training and runtime motion capture is under the permissive MIT license.

Status

The early direction of this project has been to implement some of the capabilities demoed in our 'Glimpse — a sneak peek into your creative self' teaser video.

It's early days but we already have an end-to-end MoCap system working, including:

• A rendering pipeline implemented using Blender to create large sets of training data along with a post-processing tool that helps model the artefacts of real-world cameras.
• A custom framework for training a forest of random decision trees that's tailored and optimized for our needs such that we can efficiently train trees with more than 1M images on very modest PC hardware.
• A runtime engine useable on Linux, OSX, Android and iOS supporting:
  • Multiple camera device backends (Kinect, Tango, AVFoundation)
  • Real-time segmentation of people from the background (including support for a moving camera)
  • Real-time inference of per-pixel, body part labels (currently 34)
  • Real-time inference of a 3D skeleton based on body part labels
• Development tooling to capture recordings for offline testing and visualizing the tracking engine state including 3D point cloud views of camera data and wireframes of inferred skeletons.
• A native Unity plugin and editor extensions to facilitate easily creating content based on the skeltal data from Glimpse.

Here's a recording from March 2018 of Glimpse running on an Android, Zenfone AR:

and a screenshot of the Glimpse Viewer debug tool:

Fetching

Along with this repository you'll also need to fetch the Glimpse Training Data and/or the Glimpse Pre-Trained Models as follows:

git clone https://github.com/glimpse-project/glimpse
git clone --depth=1 https://github.com/glimpse-project/glimpse-training-data
git clone --depth=1 https://github.com/glimpse-project/glimpse-models

Note: We recommend you clone the later repositories with --depth=1 since they contain large resources whose history is rarely required so you can save a lot of time/bandwidth this way

Once cloned, please check the respective README.md files in each repo for the latest instructions but it's expected you will need to also run:

cd glimpse-training-data
./unpack.sh
cd blender
./install-addons.sh

(to decompress the CMU motion capture data that we use in our rendering pipeline and to configure Blender with all the required addons for rendering via blender/glimpse-cli.py)

and in the glimpse-models repo run:

cd glimpse-models
./unpack.sh

(to decompress the decision trees)

Environment variables

Further instructions assume the following environment variables are set

GLIMPSE_ASSETS_ROOT is set to an absolute path containing all the data from the glimpse-models repository above. For the glimpse_viewer application, the UI font file is also expected. This path can also contain recordings and motion capture targets, in ViewerRecording and Targets directories, respectively.

Building

Currently we support building and running Glimpse on Linux, OSX and cross-compiling for Android and iOS. If someone wants to help port to Windows, that would be greatly appreciated and probably wouldn't be too tricky.

We're using Meson and Ninja for building.

Since upstream Meson doesn't have sufficient support for Android or iOS and to also ensure consistent build setups we're using a custom branch of meson to build Glimpse which can be installed via:

pip3 install --user --upgrade git+https://github.com/glimpse-project/meson

On OSX you may need to explicitly add locally installed Python binaries to your PATH, E.g. by adding the following line to your ~/.bash_profile:

export PATH=$HOME/Library/Python/3.7/bin:$PATH

(The 3.7 version number may differ, depending on the specific version of Python 3 being used)

The version should have glimpse-devX in the suffix like:

$ meson --version
0.46.0.glimpse-dev2

Debug

By default Meson will compile without optimizations and with debug symbols:

mkdir build-debug
cd build-debug
meson ..
ninja

Release

An optimized build can be compiled as follows:

mkdir build-release
cd build-release
CFLAGS="-march=native -mtune=native" CXXFLAGS="-march=native -mtune=native" meson --buildtype=release ..
ninja

Building for Android

We currently only test building with NDK r18b, though later versions may work.

From here it's assumed you have set up an environment something like:

export ANDROID_NDK_HOME="path/to/android-ndk-r18b"
export ANDROID_HOME="path/to/android-sdk"
export PATH="$ANDROID_HOME/tools:$PATH"
export PATH="$ANDROID_HOME/tools/bin:$PATH"
export PATH="$ANDROID_HOME/platform-tools:$PATH"

For ease of integration with Meson we create a standalone toolchain like so:

$ANDROID_NDK_HOME/build/tools/make_standalone_toolchain.py \
        --install-dir ~/local/android-arm-toolchain-28 \
        --arch arm \
        --api 28 \
        --stl libc++
export PATH=~/local/android-arm-toolchain-28/bin:$PATH

Note: we can't build for arm64 when building the libglimpse-unity-plugin.so since Unity doesn't natively support arm64 on Android Note: while building for 32bit arm we have to use api level >= 24 otherwise we hit build issues with -D_FILE_OFFSET_BITS=64 usage Note: it's not recommended to add the above toolchain path to your default PATH considering that it includes a minimal installation of python2.

The following SDK components should be downloaded:

sdkmanager "platforms;android-28"
sdkmanager "build-tools;28.0.3"
sdkmanager "platform-tools"

Then to compile Glimpse:

mkdir build-android-debug
cd build-android-debug
meson --cross-file ../android-armeabi-v7a-cross-file.txt --buildtype=debug ..
ninja

or release:

mkdir build-android-release
cd build-android-release
meson --cross-file ../android-armeabi-v7a-cross-file.txt --buildtype=release ..
ninja

To build an .apk first check ~/.android/debug.keystore exists and if not create the keystore and and debug signing key with:

keytool -genkey -v \
        -keystore ~/.android/debug.keystore \
        -alias androiddebugkey \
        -storepass android \
        -keypass android \
        -keyalg RSA \
        -keysize 2048 \
        -validity 10000 \
        -dname "CN=Android Debug,O=Android,C=US"

Build an .apk and install with:

ninja viewer_apk
adb install -r glimpse_viewer_apk_stage/glimpse_viewer.apk

Upload all the required assets to /sdcard/GlimpseViewer/:

adb shell mkdir -p /sdcard/GlimpseViewer/ViewerRecording
cd path/to/glimpse-assets
adb push \
        bone-map.json \
        glimpse-config.json \
        joint-dist.json \
        joint-map.json \
        joint-params.json \
        label-map.json \
        tree?.rdt \
        /sdcard/GlimpseViewer/

(assuming .rdt files have been generated with json-to-rdt)

Note: It's currently only possible to use the recordings backend on Android since Google stopped developing Tango we are no longer actively maintaining the Tango backend for Glimpse.

Building for OSX

Assuming you use brew to install third-party tools then the Glimpse build will first require:

brew install pkg-config zlib libpng glfw python ninja git

If you want to use the software with a Kinect device, you will also need libfreenect installed:

brew install libfreenect

It's assumed that you have installed the glimpse branch of the meson build tool as documented earler.

Then to compile Glimpse:

mkdir build-debug
cd build-debug
meson --buildtype=debug ..
ninja

or release:

mkdir build-release
cd build-release
CFLAGS="-march=native -mtune=native" CXXFLAGS="-march=native -mtune=native" meson --buildtype=release ..
ninja

Building for iOS

Firstly you need to make sure that your using the command line tools from the full Xcode installation not the more minimal Command Line Tools package via:

sudo xcode-select -switch /Applications/Xcode.app

If you run xcode-select -p you should see:

/Applications/Xcode.app/Contents/Developer

not:

/Library/Developer/CommandLineTools

If you run xcrun --sdk iphoneos --show-sdk-path you should see something like:

/Applications/Xcode.app/Contents/Developer/Platforms/iPhoneOS.platform/Developer/SDKs/iPhoneOS12.1.sdk

(If not then it's expected that the build of glimpse will fail because the build script uses the same command to locate the correct headers for cross compiling)

Then to compile Glimpse:

mkdir build-ios-debug
cd build-ios-debug
meson --cross-file ../ios-xcode-arm64-cross-file.txt --buildtype=debug --default-library=static ..
ninja

or release:

mkdir build-ios-release
cd build-ios-release
meson --cross-file ../ios-xcode-arm64-cross-file.txt --buildtype=release --default-library=static ..
ninja

Note: When configuring, append the option --default-library=static, as dynamic libraries are not supported on iOS.

Copy the built glimpse_viewer binary to xcode/GlimpseViewer/GlimpseViewer/Glimpse Viewer and open xcode/GlimpseViewer in XCode to build, package and run on an iPhone X.

For the Unity plugin, use ninja install_plugin as with other platform builds.