Skip to content
/ IoT-System-Server Public

I have built an IoT system to (1) dynamically deploy IoT analytics on gateways to save upload bandwidth as much as possible and (2) dynamically allocate upload bandwidth for each IoT analytic to maximize overall QoS.

0 stars 0 forks Branches Tags Activity
Star
Notifications You must be signed in to change notification settings

yjwangtw/IoT-System-Server

BranchesTags

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

7 Commits
Implementation
Implementation
 
 
README.md
README.md
 
 
config.env
config.env
 
 

Repository files navigation

IoT-System-Server (Github)

I have built an IoT system to

  • dynamically deploy IoT analytics on gateways to save upload bandwidth as much as possible.
  • dynamically allocate upload bandwidth for each IoT analytic to maximize overall QoS.

Precisely, my system

  • leverages IoT devices, gateways, and cloud servers.
  • packages IoT analytics into Docker containers, which are easily deployed and managed.
  • manages containers and monitors resources of IoT devices with Kubernetes.

Tools: Python, Docker, Kubernetes, TensorFlow, MQTT, FFserver.

This repository is the server side.

The document is here.

Environment

OS: Ubuntu 18.04.1 LTS Docker: 18.09.7 Kubernetes:

Server Version: version.Info{Major:"1", Minor:"11", GitVersion:"v1.11.0", GitCommit:"91e7b4fd31fcd3d5f436da26c980becec37ceefe", GitTreeState:"clean", BuildDate:"2018-06-27T20:08:34Z", GoVersion:"go1.10.2", Compiler:"gc", Platform:"linux/amd64"}
Client Version: version.Info{Major:"1", Minor:"11", GitVersion:"v1.11.0", GitCommit:"91e7b4fd31fcd3d5f436da26c980becec37ceefe", GitTreeState:"clean", BuildDate:"2018-06-27T20:17:28Z", GoVersion:"go1.10.2", Compiler:"gc", Platform:"linux/amd64"}

File Structure

Implementation/
 |____Experiment/
 |   |____generate_request/
 |   |    |____run.py
 |   |
 |   |____log/
 |   |____parsers/
 |   |____yaml/
 |   |____Algorithm.log
 |   |____cplog.sh
 |   |____delete_all_images.py
 |   |____poisson_run.py
 |   |____qos_measurement_audio.py
 |   |____qos_measurement_yolo_image.py
 |   |____qos_measurement_yolo_result.py
 |   |____registry.sh
 |   |____replace_time.input
 |   |____run.sh
 |   |____SubQoSAudio.sh
 |   |____SubQoSYoloImage.sh
 |   |____SubQoSYoloResult.sh
 |
 |____Gateway/
 |   |____ObjectDector/
 |   |____ObjectDector2/
 |   |____ObjectDector3/
 |   |____ObjectDector4/
 |   |____ObjectDector5/
 |
 |____RunDeploy/
 |   |____analytics.input
 |   |____departure_analytics.txt
 |   |____deploy_measure.py
 |   |____exist.input
 |   |____expect.input
 |   |____images_download.log
 |   |____list.txt
 |   |____replace_time.input
 |   |____terminal.input
 |
 |____RunAllocation/
 |   |____Algo/
 |   |    |____allocate_algo.py
 |   |    |____produce_weights.py
 |   |    |____unweighted_algo.py
 |   |    |____weighted_algo.py
 |   |    |____weights.txt
 |   |
 |   |____QoSknob.txt
 |   |____run.py
 |
 |____docker_daemon.json

Folders and Files

Experiment/: Main experiments

  • generate_request/: Input requests and request generations
  • log/: Log files
  • parsers/: Parsers to parse raw log files
  • yaml/: Analytics deployment yaml files
  • cplog.sh: Script to copy log files
  • poisson_run.py: Main process
  • registry.sh: Script to run local registry
  • run.sh: Main process

Gateway/: Including Yolo processes

  • ObjectDector/~ObjectDector5/: Yolo processes

RunDeploy/: Container deployment process

  • deploy_measure.py: Image Download Algorithms:
    • Dynamic programming algorithm ($IDA_D$)
    • $(1-\epsilon)$-approximation algorithm ($IDA_A$)
    • Greedy algorithm ($IDA_G$)

RunAllocation/: Rate allocation process

  • run.py: Rate allocation processes
  • Algo/: Rate Allocation Algorithms:
    • Our proposed algorithm (RAA)
    • Weighted allocation algorithm (WA)
    • Unweighted allocation algorithm (UA)
  • QoSknob.txt: QoS knobs determined by rate allocation algorithms

docker_daemon.json: Local registry setup

Set up

Build Local Registry

Copy Implementation/docker_daemon.json to /etc/docker/daemon.json

cd Implementation
cp docker_daemon.json /etc/docker/daemon.json

Start up

Environment Variables

Set up environment variables in config.env, and then source config.env.

Run Local Registry

  • Go to Experiment/
cd Implementation/Experiment/
  • Run local registry container
bash registry.sh

Generate Requests

  • Go to generate_request/
cd generate_request/
  • Run the generating script run.py
python3 run.py [number_of_requests_per_minute] [minimal_departure_time] [maximal_departure_time] [total_experiment_time] [output_file]

output_file is requests_[number_of_requests_per_minute]_[minimal_departure_time]_[maximal_departure_time]_[total_experiment_time]_[version].txt

Example.

python3 run.py 1 1 10 20 requests_1_1_10_20_1.txt

Set up Parameters of Experiments in run.sh

Parameters Meanings
ida Image download algorithm (IDA) Ex.
DP: $IDA_D$
FPTAS: $IDA_A$
Greedy: $IDA_G$
raa Rate allocation algorithm (RAA) Ex.
RAA: RAA
weighted: WA
unweighted: UA
lrp Layer replacement policy (LRP) Ex. LRU, MRU, LFU, MFU
epsilon Approximation parameter $\epsilon$ of $IDA_A$ Ex. 0.4
alpha Step size $\alpha$ of RAA Ex. 0.01
total_time Total experiment time (unit is minute).
Ex. 40: for a 40-minute long experiment.
duration Time slot duration of IDA (unit is minute) Ex. 5
network Download / upload bandwidth. Ex.
optical: 100Mbps / 5Mbps
up10: 100Mbps / 10Mbps
request_per_m Number of requests per minute Ex. 1
alive_time_min Minimal departure time (unit is minute) Ex. 1
alive_time_max Maximal departure time (unit is minute) Ex. 10
i Requests version
date_time Time stamp

Run Experiments

screen -S run 
bash run.sh > run.log

Check Screen Status

screen -ls

Output should be like this:

There are screens on:
	20519.SubQoSAudio	(08/30/2020 08:51:00 PM)	(Detached)
	20429.SubQoSYoloImage	(08/30/2020 08:50:52 PM)	(Detached)
	20348.SubQoSYoloResult	(08/30/2020 08:50:44 PM)	(Detached)
	19471.run	(08/30/2020 08:49:04 PM)	(Detached)
4 Sockets in /run/screen/S-master.

Parse Log files

Raw Log files

Here is an example

Log files Information of Generated by
DP_RAA_LRU_0.4_0.01_1.log Main process Experiment/poisson_run.py
deploy_optical_DP_RAA_LRU_0.4_0.01_1.log IDA RunDeploy/deploy_measure.py
or
RunDeploy/greedy_measure.py
allocation_optical_DP_RAA_LRU_0.4_0.01_1.log RAA RunAllocation/run.py
Bandwidth_optical_DP_RAA_LRU_0.4_0.01_1.log Experiment/Bandwidth consumption Experiment/poisson_run.py
CPU_optical_DP_RAA_LRU_0.4_0.01_1.log CPU consumption Experiment/poisson_run.py
qos_DP_RAA_LRU_0.4_0.01_1_0830_2049_audio.log Result of audio classifiers Experiment/qos_measurement_audio.py
qos_DP_RAA_LRU_0.4_0.01_1_0830_2049_yolo.log Result of object detectors Experiment/qos_measurement_yolo_result.py

Copy log files to log/

  • Edit Parameters in cplog.sh
    • network: network condition. Ex. optical
    • i: version of requests Ex. 1
  • Run cplog.sh
bash cplog.sh [algorithm] [target_dir]
  • Arguments:
    • algorithm: the name of main process's log file. Ex. DP_RAA_LRU_0.4_0.01_1
    • target_dir: output directory. Ex. log/DP_RAA_MFU_0.4_0.01_1_1_10_20_3GB_optical_Aug_31/
      Notice that ./ will remove log files
  • Example.
bash cplog.sh DP_RAA_LRU_0.4_0.01_1 log/DP_RAA_LRU_0.4_0.01_3_1_10_30_3GB_optical_Aug_7/
  • Check log directory
ls log/DP_RAA_LRU_0.4_0.01_3_1_10_30_3GB_optical_Aug_7

Output:

...
run.log
allocation_optical_DP_RAA_LRU_0.4_0.01_1.log
Bandwidth_optical_DP_RAA_LRU_0.4_0.01_1.log
CPU_optical_DP_RAA_LRU_0.4_0.01_1.log
deploy_optical_DP_RAA_LRU_0.4_0.01_1.log
DP_RAA_LRU_0.4_0.01_1.log
qos_DP_RAA_LRU_0.4_0.01_1_0830_2049_audio.log
qos_DP_RAA_LRU_0.4_0.01_1_0830_2049_yolo.log
...

Using Yolo to Calculate ground truth and Parse Log Files

cd log/DP_RAA_LRU_0.4_0.01_3_1_10_30_3GB_optical_Aug_7
bash ../../parsers/parse.sh [log_dir] [name_of_main_process_log_file] [date_time] [yolo_dir] [request_version]
  • yolo_dir: the directory of yolo.
    Ex. ../../../Gateway/ObjectDector

Example.

bash ../../parsers/parse.sh ./ DP_RAA_LRU_0.4_0.01_1 0830_2049 ../../../Gateway/ObjectDector 1

Parsed Log Files

Log Files Information of
accuracy_DP_RAA_LRU_0.4_0.01_1.txt Overall weighted QoS
deploy_num_DP_RAA_LRU_0.4_0.01_1.txt Number of analytics containers on the gateway
deploy_num_expect_DP_RAA_LRU_0.4_0.01_1.txt Expected number of analytics containers on the gateway
down_DP_RAA_LRU_0.4_0.01_1.txt Download bandwidth consumption
gateway_num_DP_RAA_LRU_0.4_0.01_1.txt Number of analytics containers on the gateway
request_num_DP_RAA_LRU_0.4_0.01_1.txt Number of analytics containers on the cloud server and gateway
saved_bw_DP_RAA_LRU_0.4_0.01_1.txt Saved upload bandwidth
saved_bw_expect_DP_RAA_LRU_0.4_0.01_1.txt Expected saved upload bandwidth
time_allocate_DP_RAA_LRU_0.4_0.01_1.txt Running time of rate alloction algorithm
time_ida_DP_RAA_LRU_0.4_0.01_1.txt Running time of image download algorithm
up_DP_RAA_LRU_0.4_0.01_1.txt Upload bandwidth consumption

About

I have built an IoT system to (1) dynamically deploy IoT analytics on gateways to save upload bandwidth as much as possible and (2) dynamically allocate upload bandwidth for each IoT analytic to maximize overall QoS.

Resources

Readme
Activity

Stars

0 stars

Watchers

1 watching

Forks

0 forks
Report repository

Releases

No releases published

Packages

No packages published

Languages