Questions Regarding How LoadSnapshot Performs

[DanielLee343]

Hi,

I have some conceptual questions and hopefully can get some help here :-) I ran TestSnapLoad() that performs end-to-end snapshot saving and loading. I don't use the UPF feature. I noticed that somehow LoadSnapshot() and ResumeVM() both finish really fast. My TestSnapLoad() looks like:

I'm using docker.io/vhiveease/video-analytics-decoder:latest as testImageName, inside a 1GB microVM. Inside StartVM(), I also sleep 10 seconds after task.Start(). So I make sure the function runs for 10 sec before SIGKILL-ed.
Here's my log:

=== RUN   TestSnapLoad1
INFO[2022-11-30T18:10:36.584138230Z] Registering bridges for tap manager          
INFO[2022-11-30T18:10:36.586144850Z] Creating containerd client                   
INFO[2022-11-30T18:10:36.588431345Z] Created containerd client                    
INFO[2022-11-30T18:10:36.588581222Z] Creating firecracker client                  
INFO[2022-11-30T18:10:36.589177473Z] Created firecracker client                   
INFO[2022-11-30T18:10:36.611206824Z] pulling remote image: docker.io/vhiveease/video-analytics-decoder:latest 
INFO[2022-11-30T18:11:01.176271108Z] OpenBLAS WARNING - could not determine the L2 cache size on this system, assuming 256k  vmID=1
INFO[2022-11-30T18:11:01.176576014Z]                                               vmID=1
INFO[2022-11-30T18:11:01.365190345Z] OpenBLAS WARNING - could not determine the L2 cache size on this system, assuming 256k  vmID=1
INFO[2022-11-30T18:11:01.365380447Z]                                               vmID=1
INFO[2022-11-30T18:11:01.422663053Z] ISTRACINGENABLED: False                       vmID=1
INFO[2022-11-30T18:11:01.422934097Z]                                               vmID=1
INFO[2022-11-30T18:11:10.296601182Z] snapshotPath: /var/lib/firecracker-containerd/snapshots/1/mem_file 
INFO[2022-11-30T18:11:40.766279238Z] Loading SS...                                
INFO[2022-11-30T18:11:40.805564620Z] resuming function...                         
INFO[2022-11-30T18:11:40.807073899Z] finish!!!                                    
INFO[2022-11-30T18:11:40.807238577Z] Removing bridges                             
INFO[2022-11-30T18:11:41.003522738Z] Remove bridges done!                         
--- PASS: TestSnapLoad1 (64.42s)

I observed that both LoadSnapshot() and ResumeVM() finish really fast. I suppose without REAP enabled, FC is doing on-demand load with serial page faults. So the function is still running after ResumeVM() returns, but should be before CleanUp() gets called.
I wrote a script to monitor SSD IO per second. Here's part of the logs after ResumeVM() returns. It can be observed that the function is reading from disk on-demand (page faults), but it's really sparse, of course it depends on each function.

INFO[0097] Read duing CSS in MB: 0.078125               
INFO[0097] Write duing CSS in MB: 0                     
INFO[0098] Read duing CSS in MB: 0.06640625             
INFO[0098] Write duing CSS in MB: 0                     
INFO[0099] Read duing CSS in MB: 0                      
INFO[0099] Write duing CSS in MB: 0                     
INFO[0100] Read duing CSS in MB: 0.125                  
INFO[0100] Write duing CSS in MB: 0.02734375            
INFO[0101] Read duing CSS in MB: 0                      
INFO[0101] Write duing CSS in MB: 0                     
INFO[0102] Read duing CSS in MB: 0                      
INFO[0102] Write duing CSS in MB: 0                     
INFO[0103] Read duing CSS in MB: 0                      
INFO[0103] Write duing CSS in MB: 0                     
INFO[0104] Read duing CSS in MB: 1.921875               
INFO[0104] Write duing CSS in MB: 0                     
INFO[0105] Read duing CSS in MB: 0                      
INFO[0105] Write duing CSS in MB: 0                     
INFO[0106] Read duing CSS in MB: 0                      
INFO[0106] Write duing CSS in MB: 0                     
INFO[0107] Read duing CSS in MB: 0                      
INFO[0107] Write duing CSS in MB: 0                     
INFO[0108] Read duing CSS in MB: 0                      
INFO[0108] Write duing CSS in MB: 0                     
INFO[0109] Read duing CSS in MB: 0                      
INFO[0109] Write duing CSS in MB: 0                     
INFO[0110] Read duing CSS in MB: 0                      
INFO[0110] Write duing CSS in MB: 0                     
INFO[0111] Read duing CSS in MB: 0                      
INFO[0111] Write duing CSS in MB: 0                     
INFO[0112] Read duing CSS in MB: 0                      
INFO[0112] Write duing CSS in MB: 0                     
INFO[0113] Read duing CSS in MB: 0.12109375             
INFO[0113] Write duing CSS in MB: 0                     
INFO[0114] Read duing CSS in MB: 0                      
INFO[0114] Write duing CSS in MB: 0                     
INFO[0115] Read duing CSS in MB: 0.05859375             
INFO[0115] Write duing CSS in MB: 0         

If that's correct, my questions are:
--- Is there a way to know when does the function finish, in either StartVM() or ResumeVM()? From the official docs I only see they are sending SIGKILL to the runc inside the container. So is there a why to let the function finishes by itself and returns/ sends a signal to the host?
--- I'm not using vSwarm because Im testing on Snapshot feature. So Im looking at the vhiveease docker hub where there're bunch of FaaS functions can be tested through containerd. I guess the input for most of the functions are located inside the container already, e.g., video-analytics-decoder is 340MB of size. Can you provide some other functions that is also reading input from the image itself, but has a much denser on-demand page-faults?

I appreciate anyone who can help on these :)

[ustiugov]

@DanielLee343 you don't invoke the function so the resumed function doesn't do anything, hence doesn't really touch many pages.