- Read the information from our official website MMSys Website carefully.
- You should design an algorithm to predict the bandwidth. We provide a Gym to you and here is a reinforcement learning example to demonstrate how to design a bandwidth estimator model by the Gym
You should submit your model and paper to participate the challenge. Please refer the MMSys Website.
-
You should convert your model or algorithm for AlphaRTC to a PyInfer instance. Here is a tiny example of acceptable submission. Meanwhile, you can verify the validation of your model following this section. You implementation will run in the Challenge-Environment that we pre-installed some popular third-parties library in this environment.
-
You should compress all materials of your bandwidth estimator as a zip package. Here is an valid submission example.
-
Submit your materials into our OpenNetLab platform. We provide online evaluation before the deadline.
- Please use the first author's email address to login this system.
- We only store your last submission for evaluation.
- We limit the number of submissions for each participant to 3 times per day.
Please refer the MMSys Website.
We will provide two stages of the evaluation on OpenNetLab, online evaluation and offline evaluation. The goals of the evaluations are different.
When a participant submit a zip, we check the basic functions of the submissions to ensure every submission can work in the offline evaluation. Each submission is scheduled to one pair of the servers randomly. The scores from the online evaluation are only for references.
We will start the offline evaluation after the final deadline. All submissions will be tested in different scenarios (e.g. networks and locations). Then we calculate the final cores and rank the submissions.
The final score is divided into three parts video , audio and network. The specific formula is as follows :
final_score = w1 * video_score + w2 * audio_score + network_score
The different parts of score can be calculated by the method below :
- video_score = 100 * Vmaf score / ground_truth
- audio_score = 100 if DNSMOS score > ground_truth * binarize_bound else 0
- network_score = w3 * delay_score + w4 * receive_rate_score + w5 * loss_score
For each part of the network score, they can be calculated as follows :
- delay_score = max(0, 100 * (max_delay - delay_95th) / (max_delay – min_one_way_delay))
- receive_rate_score = 100 * recv_rate / ground_truth
- loss_score = 100 * (1 - loss_rate)
The ground_truth means the correspond score that can be obtained in an ideal environment (like no loss, high capacity). We use the score of ground_truth as the full marks.
The binarize_bound means the bound that can get a full mark if the score exceeds binarize_bound times of ground_truth. Otherwise it will be 0.
To tune the model, we ran multiple experiments to test the performance of different bandwidth estimator algorithms. The experiments lead us to set the coefficients as follows :
w1, w2, w3, w4, w5 = 0.2, 0.1, 0.2, 0.2, 0.3
- AlphaRTC: https://github.com/OpenNetLab/AlphaRTC
- AlphaRTC Gym: https://github.com/OpenNetLab/gym
- AlphaRTC Gym-example: https://github.com/OpenNetLab/gym-example
- Challenge Runtime Environment: https://github.com/OpenNetLab/Challenge-Environment
- Challenge Submission Example: https://github.com/OpenNetLab/Challenge-Example