-
Notifications
You must be signed in to change notification settings - Fork 203
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Documentation: Part I #1031
Documentation: Part I #1031
Conversation
Thanks! Now I can see much better what the changes are. Some thoughts that come to my mind when reading this:
What do you think about this? I would highly appreciate hearing your thoughts about these questions, because it is not easy to consider what is the best reading flow for new people. |
I'd rather remove it from the other places. Without runexec being mentioned there, I wouldn't look further. From personal experience: I did briefly look at BenchExec several years ago during my PhD and thought BenchExec is just
I would argue that (initially) In my perception,
Yes, IMO the README should briefly summarize what BenchExec offers and what it needs so that one can make a judgement. If you have to click through several files just to see what all the main features and requirements are, I am rather certain that people will simply not care enough. After all, there is no "pressure" to use this, |
I understand, and we will follow this advice. Thank you! What do you think about the following as an extension of the current feature list? BenchExec provides three major features:
For the "on top of" part I am unsure what the best wording is. But overall I would hope that this makes it pretty clear that if you want the first feature that you can just have this and how, or doesn't it?
I am not so sure. If you start with benchmarking (e.g., as a new researcher), and you do not have all the scripts already that allow you to run sets of commands, collect the results, compare the results across different configurations etc.,
For me it is important that we do not imply this in the documentation. We do not want to scare people away from
I meant to say that in this PR the system requirements appear twice inside |
That sounds good!
oops that was a mistake :-)
I think the target audience you have in mind is different from what I think of :-) A notable fraction of people in my area barely know one programming language and "writing an XML file" or "write a python class implementing an interface" is a significant task. The implementations often aren't stable tools but prototypes. And yes, if these are evaluated on their own, there isn't much reason for precise measurements, but when comparing to existing tools, I think this is relevant. I would argue that using Just for comparison: Using My main point is that currently there is no real reward for using BenchExec except "doing things right", which I believe is not enough to justify spending more than an hour on this for most PhD students, or, in any case, something quite a bit of supervisors won't appreciate. (Bluntly: Using benchexec over It is a question of whom you want to reach - I think it is totally fine if you want to cater towards an "experienced" crowd but for a beginner I am rather certain that benchexec inherently is "scary" (even though it is easy to use once you know how!) |
We want to make it clear that BenchExec is not only benchexec and one can use runexec on its own as a replacement for time. This is not mentioned in the README and also not prominently in the documentation, and has led to misunderstandings about BenchExec in the past. In particular, we want to advertise runexec for users who already have their own benchmarking scripts, while still advertising benchexec as the main and recommended solution in general for most users. For this it seems useful to make the connections between the tools more explicit in the documentation. This commit is an outcome of the discussion in #1031 and implements a part of what is suggested in #999 and #1031. Co-authored-by: Tobias Meggendorfer <tobias@meggendorfer.de>
Sorry for the long delay. It was a busy time.
Thanks, I will commit this. Step-wise improvements :-) Would you like to be accredited as a co-author of the commit?
Not at all! Beginners have most to gain from using BenchExec and beginners are the most important people who should use BenchExec (instead of taking care of all the tricky details of benchmarking themselves).
I know, of course, that there is some initial barrier for using BenchExec because you need to learn several things. And I am always glad about feedback and hints about how we could improve this by making it easier or having more documentation. But I think that pushing people and especially beginners who know little about benchmarking into the direction of using less of BenchExec and more of their own hand-written scripts or manual steps is going in the wrong direction.
But this group of people would also struggle with having to write their own benchmarking scripts. So precisely for this group of people I would argue that using
Even if they compare it against the effort of writing an own script that takes care of collecting all the benchmark runs and storing the results in a machine-readable way?
If you just replace So no, I do not want to force people to use
I see the point for those users who already have such scripts and I agree that for those users your suggestion is a good selling strategy. Thank you! I think we can incorporate this without pushing everyone to In order to lock in place what we have already and allow easier iteration, I have committed an attempt at this together with what was discussed before in 48cfd9b. I would be nice if you could have a look whether you like it or have suggestions and whether you agree with being a co-author. Then I would push this to the main branch. Afterwards, I would be glad to hear what you think is still missing between that commit and this PR and what the goals of the remaining changes are. |
Can relate, no problem :)
No need, maybe when I contribute a larger chunk. But I also won't object :)
But they often are not aware that their scripts are a problem ;) So I would see it as a compromise - make it easy to use the basic bits so at least that is done right. I agree that much more people should be using
If only that were the case. Example from a recent artifact evaluation: 1) needed to manually modify the source code to change the model being run 2) was an IDE packaged inside a VM where you needed to change the run parameters inside the VM (i.e. no usable command line binary). Yes, these are extreme examples, but this is the sort of context that I want to make proper benchmarking appealing to. Most artifacts do not store results in a machine readable way.
But I care about this! :) I see where we are going. My point here is: I want to be able to establish using Again, I agree that
Great! I like the changes in the commit. I think the only addition I would make is that |
Thanks! Added and pushed to main.
Thank you very much! We highly appreciate this (and not because pushing our tool in particular, but because of the general improvement to the state of how research is done).
Oh, yes! This is a great suggestion and I definitively want to provide this now! Do you think extending https://github.com/sosy-lab/benchexec/blob/main/doc/runexec.md would be a good place? |
Same here :)
I am unsure. There are two separate points; 1. "this is what the tool can do" 2. "this is how to do basic proper benchmarks". I think runexec.md should be more a "documentation" of the tool, while the other should be a self-contained set of "instructions". I imagine something like this:
Basically, I would like to be able to write on the webpage of an artifact evaluation "We strongly encourage authors to follow established best practices for benchmarking [cite to your paper]. See here [link to the above page] for a guide on how to do it." (or something like this) and similarly just paste this in a review. (I think AE is probably the best way to raise awareness as the reviewers can actually inspect the methodology.) Point being, I think it would be nice if this is a "landing page" that can be directly linked to, if you see what I mean? |
I see, and I fully agree that this would be great to have. We could call it "Quick start guide" or "Runexec Tutorial" or so? It should then probably be its own page in Btw., also have a look at https://github.com/sosy-lab/benchexec/blob/main/doc/benchmarking.md. It also covers "stuff that you should do when benchmarking" and should be tightly coupled via links to what you are proposing, but probably keeping this checklist and the technical quick-start guide on separate pages is better. |
I like "quickstart.md" a lot (its not exclusive about Yes, I saw i could sketch a quickstart if you want (but it will need some polishing / iteration I think) |
I would be glad about this! Thank you a lot for your invaluable help! |
Will do soon (tm); closing this MR then |
Improve README with usage notes