Trusted OS and Applet execution

Let us see how two concurrent TamaGo unikernels can be scheduled in different privilege domains.

The monitor package provides privileged support for instantiating bare metal Go in user mode, while the applet package provides applet support.

We call the privileged unikernel the "Trusted OS", running in privileged modes (PL1) while the unprivileged (PL0) unikernel is known as the "Trusted Applet", both running in TrustZone Secure World.

Let us assume that our Trusted OS has the Trusted Applet ELF binary somewhere in memory, we can load it as follows:

ta, err := monitor.Load(appletELF, appletStart, appletSize)

⚠️ while in our example the Trusted Applet is also a TamaGo unikernel, it can really be any program capable of running in user mode and implementing GoTEE API, so it could very well be a freestanding C or Rust program.

The appletStart and appletSize variables must be set with the physical memory start address and size that we want to assign to the applet, the GoTEE example defines the memory layout here.

The returned ta is a monitor.ExecCtx structure which holds the entire execution context state.

We can invoke ta.Print() to see the initial applet state:

        r0:00000000   r1:00000000   r2:00000000   r3:00000000
        r1:00000000   r2:00000000   r3:00000000   r4:00000000
        r5:00000000   r6:00000000   r7:00000000   r8:00000000
        r9:00000000  r10:00000000  r11:00000000  r12:00000000
        sp:00000000   lr:00000000   pc:8206dab8 spsr:000001d0

As expected all user registers are not set at the beginning, while the program counter matches the entry point information found within the ELF binary by Load(). The Saved Program Status Register is initialized which corresponds to user mode.

Using TamaGo dma package the ELF is placed in physical memory between appletStart and appletStart+appletSize which is now reserved exclusively for this context and not directly mapped by the Trusted OS Go runtime.

Using TamaGo arm package ConfigureMMU function, the memory is already flagged with PL0+PL1 privilege level, while the Trusted OS Go runtime remains PL1 only by default.

We can now "run" the applet context to schedule its execution:

ta.Run()

Here we can see the GoTEE example ta running concurrently with the Trusted OS:

00:00:00 PL1 starting mode:USR ns:false sp:0x00000000 pc:0x8206dab8
00:00:00 PL0 tamago/arm (go1.16.4) • TEE user applet (Secure World)
00:00:00 PL0 obtained 16 random bytes from PL1: 2745e86bd483211a7de9672072b01049
00:00:00 PL0 requests echo via RPC: hello
00:00:00 PL0 received echo via RPC: hello
00:00:00 PL0 will sleep for 5 seconds
00:00:01 PL1 says 1 missisipi
...
00:00:05 PL1 says 5 missisipi
00:00:05 PL0 says 5 missisipi

Note that on a single core context "concurrent" means that CPU time is yielded back to the Trusted OS in PL1 until one of the following conditions arises:

• the applet triggers an error exception (e.g. invalid memory access)
• the applet triggers a supervisor exception (e.g. system call)

When either of these conditions apply the Trusted OS Go runtime will get the chance to:

1. schedule goroutines using runtime.Gosched()
2. serve GoTEE system calls using package syscall
3. handle exceptions using TamaGo arm package handlers

The Run() function takes care of all of this automatically, as it will exit only if the Exit() GoTEE system call is invoked by the applet.

The GoTEE default system call handler can be overridden by setting ta.Handler with a custom function, the TamaGo default exception handler can be overridden using the arm.SetVectorTable function.

This API gives flexibility to the Trusted OS implementer to implement the desired supervision for the trusted applet.

The GoTEE example ta triggers an invalid memory access to demonstrate its handling, this is what happens:

PL0 is about to read PL1 memory at 0x80010000
       r0:80010000   r1:828220c0   r2:00000001   r3:00000000
       r1:828220c0   r2:00000001   r3:00000000   r4:00000000
       r5:00000000   r6:00000000   r7:00000000   r8:00000007
       r9:00000037  r10:828000e0  r11:802bd968  r12:00000000
       sp:8284df2c   lr:8215926c   pc:82011374 spsr:600000d0
PL1 stopped mode:USR ns:false sp:0x8284df2c lr:0x8215926c pc:0x82011374 err:exception mode ABT

As you can see as soon as the data abort exception is triggered by the trusted applet, which is attempting to read PL1 memory, the Trusted OS can inspect the applet state and take action.

Next

➞ System Calls