Tebako: an image packager

Purpose

Tebako is an executable packager. It packages a set of files into a DwarFS file system for read-only purposes.

After packaging the file system into an image, Tebako produces a single executable binary that allows the user to execute a selected file from the packaged software from a point in the file system.

The packaged binary should support:

• Packaging a default DwarFS image inside the binary

• Support signing of the binary on macOS (via notarization)

In the future:

• Downloading new DwarFS images to be stored in the local home directory

• Allowing loading multiple DwarFS images in a stacked way

• Supporting a COW mechanism that the newly written files are stored in a separate image that can be loaded on top of the read-only file systems.

Supported platforms

The Tebako packager is tested on the following platforms:

• Linux: Ubuntu 20.04; Alpine 3.17

• macOS: macOS 11 (Big Sur), 12 (Monterey), 13 (Ventura)

• Windows: TBD

Supported Ruby versions

The Tebako packager supports the following versions of Ruby for packaging:

• 2.7.8

• 3.0.6

• 3.1.4

• 3.2.3

Support of specific version including minor release requires some effort, sometimes extensive but our goal is to be able to package all maintained Ruby releases.

Prerequisites

Ubuntu

GNU C/C 10+ or Clang C/C 12+

CMake version 3.20+

Tebako relies on CMake 3.20+, which may not be available as a default package.

If it is not available as default package it can be set up as follows:

apt-get remove --purge --auto-remove cmake
apt-get update
apt-get install -y software-properties-common lsb-release curl
apt-get clean all
curl https://apt.kitware.com/kitware-archive.sh | bash
apt-get install cmake

Other development tools and libraries

apt-get install -y curl git ruby ruby-dev pkg-config bison flex make autoconf
apt-get install -y binutils-dev libarchive-dev libevent-dev libjemalloc-dev acl-dev \
  libdouble-conversion-dev libiberty-dev liblz4-dev liblzma-dev libssl-dev \
  libboost-context-dev libboost-filesystem-dev libboost-program-options-dev \
  libboost-regex-dev libboost-system-dev libboost-thread-dev \
  libunwind-dev libdwarf-dev libelf-dev libfuse-dev libgoogle-glog-dev \
  libffi-dev libgdbm-dev libyaml-dev libncurses-dev libreadline-dev \
  libfmt-dev

Installation

General

Tebako is distributed as a Ruby gem

gem install tebako

Quick setup on Ubuntu 20.04 on Docker

Launch a container on the target platform:

# For x86_64
docker run -it --platform linux/x86_64 ubuntu bash

# For Apple M1
docker run -it --platform linux/aarch64 ubuntu bash

In the container:

export DEBIAN_FRONTEND=noninteractive
export TZ=Etc/UTC

apt-get update
apt-get install -y software-properties-common
add-apt-repository -y ppa:ubuntu-toolchain-r/test
apt-get install -y gcc-10 g++-10

apt-get install -y curl git ruby ruby-dev pkg-config bison flex make autoconf
curl https://apt.kitware.com/kitware-archive.sh | bash
apt-get install -y cmake

apt-get install -y binutils-dev libarchive-dev libevent-dev libjemalloc-dev acl-dev \
  libdouble-conversion-dev libiberty-dev liblz4-dev liblzma-dev libssl-dev \
  libboost-context-dev libboost-filesystem-dev libboost-program-options-dev \
  libboost-regex-dev libboost-system-dev libboost-thread-dev \
  libunwind-dev libdwarf-dev libelf-dev libfuse-dev libgoogle-glog-dev \
  libffi-dev libgdbm-dev libyaml-dev libncurses-dev libreadline-dev libfmt-dev

gem install tebako

Usage

Commands

Installation

gem install tebako

Press

This command "presses" a Ruby project using the Tebako setup from the Tebako root folder (<tebako-root-folder>). Please note that upon the first invocation of press command tebako collects required packages, builds the and creates packaging environment. This is a lengthly task that can take significant time, up to 1 hour. Upon the next invocation tebako will use previously created packaging environment. The press process itself takes minutes. You can manage setup of packaging environment manually; please refer to description of setup and clean cmmands below.

tebako press     \
  [-p|--prefix=<tebako-root-folder>] \
  [-R|--Ruby=<2.7.8|3.0.6|3.1.4|3.2.3>]   \
  -r|--root=<project-root-folder>     \
  -e|--entry-point=<entry-point>      \
  [-o|--output=<packaged file name>] \
  [-l|--log-level=<error|warn|debug|trace>]

Where:

• <tebako-root-folder>, the Tebako setup folder (optional, defaults to current folder)

• Ruby parameter defines Ruby version that will be packaged (optional, defaults to 3.1.4)

• <project-root>, a folder at the host source file system where project files are located

• <entry-point>, an executable file (binary executable or script) that shall be started when packaged file is called

• output, the output file name (optional, defaults to <current folder>/<entry point base name)

• log-level, the logging level for tebako built-in memory filesystem driver (optional, defaults to error)

tebako press \
  --root='~/projects/myproject' \
  --entry=start.rb \
  --output=/temp/myproject.tebako

Setup

Collects required packages, builds the and creates packaging environment. This is a lengthly task that can take significant time, up to 1 hour. Tebako supports several configurations at a single system given that their root directories differ and nultiple Ruby versions within single configuration

This command is optional, tebako creates packaging environment automatically upon the first invocation of press command. However, if you plan to use tebako in CI/CD environment with caching it is highly recommended to build cache based on tebako setup output. Building cache based on tebako press may create inconsistent environment upon restore.

tebako setup     \
  [-p |--prefix=<tebako-root-folder>] \
  [-R |--Ruby=<2.7.8|3.0.6|3.1.4|3.2.3>]

Where:

• <tebako-root-folder>, the Tebako setup folder (optional, defaults to current folder)

• Ruby parameter defines Ruby version that will be packaged (optional, defaults to 3.1.4)

Clean

This command deletes tebako artifacts created by setup and press commands. Normally you do not need to do it since tebako packager optimizes artifacts lifecycle on its own.

tebako clean \
  [-p|--prefix=<tebako-root-folder>]

Where:

• <tebako-root-folder>, the Tebako setup folder (optional, defaults to current folder)

tebako clean --prefix='~/.tebako'

Clean ruby

This command deletes tebako Ruby artifacts created by setup and press commands. Dwarfs libraries are not cleaned. Normally you do not need to do it since tebako packager optimizes artifacts lifecycle on its own.

tebako clean_ruby
  [-p|--prefix=<tebako-root-folder>]
  [-R|--Ruby=<2.7.8|3.0.6|3.1.4|3.2.3>]

Where:

• <tebako-root-folder>, the Tebako setup folder (optional, defaults to current folder)

• Ruby parameter defines Ruby version that will cleaned (optional, cleans all versions by default)

tebako clean_ruby --prefix='~/.tebako'

Build script hash

Hash command will calculate tebako script hash that may be used as a cache key in CI/CD environment like GitHub Actions

tebako hash

Exit codes

Code	Condition
0	No error
1	Invalid command line
101	tebako setup failed at configuration step
102	tebako setup failed at build step
103	tebako press failed at configuration step
104	tebako press failed at build step
253	Unsupported Ruby version
254	Unsupported operating systems
255	Internal error

Image extraction

Tebako provides an option to an extract filesystem from a package to local folder for verification or execution.

<tebako-packaged-executable> --tebako-extract [<root folder for extracted filesystem>]

Where,

• <root folder for extracted filesystem> is optional and defaults to source_filesystem

Extracting Tebako content from the metanorma package:

metanorma --tebako-extract temp-image

The --tebako-extract option actually runs the following Ruby script:

require 'fileutils'
FileUtils.copy_entry '<in-memory filesystem root>', ARGV[2] || 'source_filesystem'

Ruby packaging specification

This is high-level description of the Tebako Ruby packaging mechanism. This specification was inspired by the ruby-packer approach.

Note	For various reasons, Tebako Ruby is a fully separate implementation, no line of code was copied from ruby-packer.

Depending on the configuration files that are present in the root project folder, the Tebako Ruby packager support five different scenarios:

Scenario	*.gemspec	Gemfile	*.gem
1	No	No	No
2	No	No	One
3	One	No	Any
4	One	One	Any
5	No	One	Any
Error	No	No	Two or more
Error	Two or more	Any	Any

These scenarios differ in what files are packaged and where the entry point is located, as follows:

Scenario	Description	Packaging	Entry point
1	Simple ruby script	Copy <project-root> with all sub-folders to packaged filesystem	<mount_point>/local/<entry_point base name>
2	Packaged gem	Install the gem with gem install to packaged filesystem	<mount_point>/bin/<entry_point base name> (i.e., binstub is expected)
3	Gem source, no bundler	Build the gem using gem build command at the host Install it with gem install to packaged filesystem	<mount_point>/bin/<entry_point base name> (i.e., binstub is expected)
4	Gem source, bundler	Collect dependencies at the host with bundle install Build the gem using gem build command Install it with gem install to packaged file system	<mount_point>/bin/<entry_point base name> (i.e., binstub is expected)
5	Rails project	Deploy project to packaged filesystem using bundle install	<mount_point>/local/<entry_point base name>

Trivia: origin of name

"tamatebako" (玉手箱) is the treasure box given to Urashima Taro in the Ryugu, for which he was asked not to open if he wished to return. He opened the box upon the shock from his return that three hundred years has passed. Apparently what was stored in the box was his age.

This packager was made to store Ruby and its gems, and therefore named after the said treasure box (storing gems inside a treasure box).

Since "tamatebako" is rather long for the non-Japanese speaker, we use "tebako" (手箱, also "tehako") instead, the generic term for a personal box.