Figure 1: Calls made during labeling in Hawk
- Admin sends configuration message to scouts
- Mission is started across scouts by admin
- ScoutM sends selected tile
- Home sends labels to coordinator
- Coordinator fetches the tile content from ScoutM
- If positive, coordinator sends the labeled tile to other scouts
- Mission may be explicitly stopped by admin
- Admin collects the logs and models trained on scouts for post processing
Scouts are distant unmanned compute units collecting and learning on unlabeled data. The source of unlabeled data may be images stored on the disk or real time video feed. Each scout receives configuration message from home. The configuration message specifies the type of training, selective and retraining strategies to be used for the mission. It also contains the weight of initial DNN model or labeled images needed to train the initial model. The mission is started upon receiving start request fom home.
Once the mission starts, the model in scout processes the input data and a subset of tiles are chosen by the selective strategy for transmission. These tiles are transmitted to home using ZeroMQ. After labeling, home sends back labels to the coordinator. If a tile is labeled positive and is not present in the coordinator, indicated by parentIndex, the coordinator requests the tile content from the parent scout. The coordinator then sends the labeled tile along with its content to other scouts. Ony positives are shared between scouts, if the tile is negative, then the label is only forwarded to the parent scout.
Scouts train a new model when sufficient positive labels are attained as specified by the retrain policy. Learning in scout continues till all input data is exhausted or the mission is explicitly stopped by home.
Home is the base station where human annotators are located. When home process starts, it reads the mission configuration file and launches the necessary labeling interface. It then sends the configuration to the coordinator to start the mission. The configuration file also specifies the path to data directories used in the mission and hyperparameters used for training DNN models. An example configuration file can be found here
Hawk supports three modes of training:
- Scout-based: Training happens on scouts
- Cloud-based: Training happens on the cloud and the model is transmitted to scouts
- Adaptive: The location of training is decided based on bandwidth availability
Home process forks three child processes:
- Admin process: Sends control messages to scouts
- Inbound process: Receives and saves tiles or thumbnails from scouts
- Outbound process: Reads the labels provided by the annotator and send them to the coordinator
Hawk supports two modes of labeling, 1) interactive and 2) scripted. If interactive labeling is specified, a web-browser is launched for labeling. The browser displays the thumbnails saved in the inbound path and saves the labels provided by the user in the outbound path. Otherwise if scripted labeling is chosen, home programmatically saves the labels of thumbnails to the outbound path. The label files from this path are then read by the outbound process to send to the coordinator scout.
Admin module sends control messages to scouts. It reads the configuration file from home and constructs the configuration message, which defines the parameters used during mission. The network is not constricted using FireQos. Post mission, the admin collects the trained models and logs from scouts.
Hawk is a bandwidth-adaptive live learning system, composed of the above components. The components work together to learn DNN models on the fly and effectively reduces human effort in collecting positives of a rare new target. The server process is started on each scout, it waits for the mission definition and start command from home to begin the mission. Before the scouts are launched, we assume LAN connectivity between scouts and home.
Home is the location where the human labeler is located. When the home process is launched it reads the configuration file and starts the necessary labeling interface. It also launches three child processes necessary for controlling and communicating with the scouts. Hawk uses ZeroMQ messaging library for the communication between scouts and home. The messages are serialized using protocol buffers. Between admin and scouts, we use a request-reply pattern. In a reply-request pattern, a client connects to a server, requests info, then receives a reply. The network is not constricted by FireQos.
Once the scout is launched, the connectivity between scouts and home degrades and in extreme cases it may be a few kilobits per second. This is emulated by contricting the network bandwidth using FireQos. At such low bandwidth and high latency network conditions, we may no receive reply in time. For this reason we use a publisher-subscriber message pattern between home and scouts for sending tiles and labels. The tiles selected are published by the scouts. At home, the inbound process collects these results and saves them in the "images" directory. The user specifies a directory for storing mission related data at home. This directory have three sub-directories, namely: images, meta, and labels. The "images" directory stores the tiles received by home, the meta-data associated with the tiles are stored in the "meta" directory. Finally the labels from annotator are saved in the "labels" directory.
The annotator may choose to label some of the tiles received. The labels are saved as a json file in the "labels" directory. The outbound process reads from the "labels" directory and publishes the labels. The coordinator receives the labels and distributes the labels and the corresponding tile content to other scouts.
When enough number of tiles have been labeled as specified by the retrain-policy, a new model is trained. Training can happen on the scouts or the cloud. If the training happens on the scouts, then the new model is immediately deployed. However, if the training happens on the cloud, then the model is first compressed and then transmitted to the scouts. The incoming data is now processed using the new model. The mission finishes when the admin explicitly stops the mission or when the scouts stop capturing input frame.
Figure 1, shows the sequence of calls made during labeling of tiles in Hawk. The configuration file from home is send to the admin, which constructs the configuration message. This message is transmitted to the scouts via a2s_configure_scout call. When a scout receives the configuration message, it initializes all modules needed for the mission which includes training, processing, selective transmission, and so on. The configuration message also includes the initial boostrapping examples or the intial trained model. The admin waits to receive reply from all the participating scouts. On receiving response from all the participating scouts, the admin calls a2s_start_mission to begin the mission. The model processing the incoming tiles and the selective module selects a small subset for transmission. These tiles are publised by the scouts using s2h_send_tiles method. The inbound process receives these tiles along with their meta-data and saves them in "images" and "meta" directory respectively. The labels from the annotator are saved in the "labels" directory. The outbound process reads from this directory and sends the labels to the coordinator. The coordinator calls s2s_get_tile to fetch the tile content from the parent scout if the label is a positive. It then invokes s2s_add_tile_and_label to distribute the labeled tile along with its content to other scouts. This continues till the end of the mission. The admin calls a2s_stop_mission to explicitly stop the mission. Post mission, the admin makes a2s_get_post_mission_archivecall to collect the model generation along with logs from the scouts.