Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uses PkgBenchmark to run benchmarks on each PR, using some nice code from @tkf's Transducers.jl.
The benchmarks themselves are written with an
@add_benchmark
macro taken from MOI. What this does is allow defining benchmarks which take a callback, and then using them to generate different benchmark suites easily:Here, you pass a callback for what to do with the problem (solve it with ECOS, with SCS, or just generate the MBP standard form with
Convex.conic_problem
). This runs all the benchmarks except those excluded by a regex search. This could also be used for comparing the speed of solvers on Convex.jl problems; maybe at some point we could even incorporate benchmark data into the docs, all via travis.The actual benchmark problems themselves are mostly taken from Convex.jl's tests, and are certainly incomplete: they cover some of the affine and sdp tests only. I think they might be enough however to see the effect of large changes which affect all atoms more-or-less the same. Ideally at some point we could have benchmarks for every atom, like we have tests for each atom.
Hopefully I've got the CI bits done correctly. I suspect the tests will fail on this PR, because it will try to compare benchmarks to master, which doesn't have any, but that doesn't mean there's something wrong. The real test will be the next PR after this one is merged.
Edit: no, this PR passed because Travis doesn't use the PR's .travis.yml. That makes sense :).
This is priority 1 on #320.