-
Notifications
You must be signed in to change notification settings - Fork 203
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Task filters in benchmark definition #529
Comments
That makes sense, and we might want to extend the possible task filters to cover more than the expected result. I am not sure how to express this in the XML, though. One possible way would be to use the expression language that tools like yq and jq use. However, we would not want to implement this ourselves and I do not know whether an appropriate library exists. For now, it is quite easy to do this manually by writing a small script that extracts information from task-definition files using |
It might be nicer for a developer and for documentation to directly have something like <tasks name="ECA"> <includesfile>../programs/benchmarks/ReachSafety-ECA.set</includesfile> <propertyfile>../programs/benchmarks/properties/unreach-call.prp</propertyfile> <propertyfilter>true</propertyfilter> </tasks> or <tasks name="ECA"> <includesfile>../programs/benchmarks/ReachSafety-ECA.set</includesfile> <propertyfile filter="true">../programs/benchmarks/properties/unreach-call.prp</propertyfile> </tasks> than filtering with a separate script and having an XML file directly containing hundreds of tasks. |
I had in mind the following:
This defines a set of tasks N x (P x V), where
|
@kfriedberger In the meanwhile I like your original proposal (expected verdict as attribute for property, which I wrote above) much better than my original proposal (expected verdict as tag, which you wrote above). |
@kfriedberger Of course it is nicer, I already agreed to that. And of course you never need to use XML files that directly contain hundreds of tasks, I already suggested to generate a set file instead. @dbeyer Interesting idea. This would certainly be implementable, and could also be extended to allow filtering for specific subproperties. So it would cover the current use case. It seems difficult to extend in the future, though. If we add more information to the task-definition files (obvious ideas here are the language of a task, or things like the machine model which SV-Benchmarks currently needs to track separately), then people naturally would want to filter on these fields as well. And then we would have two different ways on how to specify task filters, although both ways would filter based on the content of the task-definition files. So the question would be that given the existing workaround, is it still important enough to have this soon rather than as part of a more general solution in the future? |
Temporarily, just for the BenchExec run, or maintain an extra set of .set files in the repository? I would also prefer the way it is now: the task definitions (.yml) define the "abilities" or "use cases" for the program, and the benchmark definition (.xml) define the selection for a particular benchmark.
You mean asking the validator developers to implement an option? I agree that a query language would be nice, but wouldn't this replace the property-based selection anyway? And the expectedverdict-based selection is just part of the property-based selection? I like the idea of using a query language, but we would need to experiment with it. It could be that people find this too difficult to understand. |
I would suggest just temporarily, but this is a matter of taste.
No, I mean the generated set files.
No, the property file is used for more than just filtering, and would not be replaced by some general filter. For example, the property file is passed on to the tool, and it is necessary for
Could be the case. Your proposal would of course be easy to understand, I like that aspect. |
This makes the choice between 'tag' an 'attribute' easy: |
It makes sense to extend this functionality to matching for subproperties while we are at it. Would you prefer
or
|
Implementation of the most important part of #529. The syntax is <propertyfile expectedverdict="...">, where the value can be either "true", "false", or "unknown". Filtering on anything else is not implement. The filters also work only for tasks that are defined in task-definition files, not for the deprecated kind of tasks where the expected verdict is encoded in the file name. No tests exist yet.
I would slightly prefer the former because this coincides with what appears in the tables. |
This is implemented now, and I think this covers all kinds of filters that would make sense for the current state of task-definition files. We can reconsider this after task-definition files have been extended, so I am closing this for now. Boolean combinations of filters are not implemented, but only disjunctions make sense at all (e.g., all tasks with expected verdict |
Currently, I can select tasks by two components:
But, for example in SV-COMP, it is often crucial to also select
by expected verdict.
For example, I would like to run a result validation based on violation witnesses
on files that contain a bug, in order to validate that the bug is described by the witness.
The text was updated successfully, but these errors were encountered: