-
Notifications
You must be signed in to change notification settings - Fork 7
EuroSys'22 Artifact Evaluation
EuroSys artifact evaluation results: https://sysartifacts.github.io/eurosys2022/results
Dear authors,
The Seventeenth European Conference on Computer Systems (EuroSys'22) artifact evaluation committee is delighted to inform you that your submission #86 has been awarded the artifact available, functional and results reproduced badge.
- Title: Reasoning about Configurable System Performance through the lens of Causality
- Authors: Md Shahriar Iqbal (University of South Carolina); Rahul Krishna (IBM T. J. Watson Research Center, USA); Mohammad Ali Javidian (Purdue University); Baishakhi Ray (Columbia University); Pooyan Jamshidi (University of South Carolina)
- Yes
- Yes
- Yes
This artifact demonstrates how the Unicorn detects the configuration issues based on Casual Performance Model. Given an application, Unicorn profiles the application at first to obtain the basic knowledge of the relationship between configuration and performance events. Then Unicorn refines the model by iteratively sampling more events for the application. At the end, Unicorn can answer if a configuration is set well for given performance metric.
The artifact is public available in the GitHub (available badge). The authors provide clean script and guidance so that I can check the functionality very easily (functional badge). The authors provide additional material for verifying the paper results in a cloud vm. It takes some time due to the unstable network of the vm. Finally, I was able to get the result from their script. And the result matches to the table in the paper (reproduce badge)
Please do fix the command typo in the README/guidance.
- Yes
- Yes
- Yes
The authors present Unicorn, a methodology that enables reasoning about configurable system performance with causal inference and counterfactual reasoning.
Thanks for submitting the artifact for evaluation. I enjoyed reading your paper and look forward to the artifact being released publically. I've provided a detailed summary of my artifact evaluation process in the comments below.
- The artifact is publically available and easily accessible
- The artifact contains a README file containing all the necessary instructions to access it.
- A license is added w.r.t the use of this artifact.
Since it satisfies all the requirements, I am going to recommend awarding the artifacts available badge
- The README contains detailed instruction on how to set up the dependencies and run the first few examples easily.
- I had minor trouble running the examples with root access and would like the authors to add some more instructions to regarding running examples with
sudo. For example, a common error that showed up for me wasPermissionError: [Errno 13] Permission denied: '/root/data/measurement/output/debug_exp.csv' - Overall, I was able to run the examples listed under offline mode with ease.
Given that there were not a whole lot of debugging I had to do to get the artifact running, I would recommend awarding the Artifact's functional badge
- I'd like to thank the authors for updating the artifact instructions to enable access to the online version of Unicorn.
- I was able to reproduce results from Table 2 and Figure 16 as explained in the README guide, and I verified the video recording that the authors uploaded demonstrating the functionality of the online version.
I'd recommend awarding the artifact's reproduced badge as well.
- Yes
- Yes
- Yes
The artifact includes Unicorn, a performance analysis, debugging, and optimization tool designed for highly configurable systems with causal reasoning and inference. It can be used to solve performance problem, guide to optimize the performance, give better understanding of performance issue, and motivate future research.
Thank you for your submission to Artifact Evaluation at EuroSys'22. Congratulations on your paper! The paper is very impressive and insightful.
Your work is qualified for "Artifact Available", "Artifact Functional" and "Results Reproduced" badges.
First, you have made the key source code, benchmarks, and instructions in this paper publically through GitHub. Benchmarks used in the paper are open-sourced. So, "Artifact Available" is qualified.
Second, the main components in your artifacts are source code of Unicorn, benchmarks, some scripts and instructions. They are good enough for an experienced researcher or developer to follow. I can run the scripts to test the benchmarks in this paper successfully. Therefore, "Artifact Functional" should be awarded.
Third, I can reproduce most results (the tables and figures) in this paper. The results are matched with the ones in the paper. Authors even provide a video to show how they reproduce the results, which is very helpful. Since the artifact meets with the requirement of "Results Reproduced" badge, I would like to award "Results Reproduced" badge to it.
- Yes
- Yes
- Yes
This artifact along with the instructions give clear instructions on how Unicorn is able to debug faults in configurations on different systems. Key results included in the artifact are energy consumption faults used in the Xception machine learning model, as well as how Unicorn can be used to tune and optimize performance.
Thanks so much for submitting this artifact! I genuinely enjoyed reading the paper and learning about this artifact. Thanks for taking the additional steps for making the online version work for us, too!
Artifacts Available: The artifact is publically available on GitHub, with a clearly-written README, making it extremely convenient for external users. Therefore, the AA badge should be awarded.
Artifacts Functional: The README contains clear instructions on how to get the artifact up and running, and provides the exact commands to run various benchmarks/experiments. Therefore, the AF badge should be awarded.
Results Reproduced: I was able to reproduce the results on Table 2, and in addition given the video that the authors provided, I think combined this meets the RR badge criterion.