Fix benchmarking, add test, update docs

benchmark/benchmarks.jl

@@ -11,11 +11,9 @@ using ECOS: ECOSSolver
 using BenchmarkTools
 
 
-include("benchmarks/benchmarks.jl") # defines module Benchmarks
-
 const SUITE = BenchmarkGroup()
 
-# SUITE["SCS"] = ProblemDepot.suite(p -> Convex.solve!(p, SCSSolver(verbose=0)))
-# SUITE["ECOS"] = ProblemDepot.suite(p -> Convex.solve!(p, ECOSSolver(verbose=0));
+# SUITE["SCS"] = ProblemDepot.benchmark_suite(p -> Convex.solve!(p, SCSSolver(verbose=0)))
+# SUITE["ECOS"] = ProblemDepot.benchmark_suite(p -> Convex.solve!(p, ECOSSolver(verbose=0));
     # exclude = [r"sdp", r"exp"])
-SUITE["formulation"] = ProblemDepot.suite(Convex.conic_problem; exclude=[r"fix"])
+SUITE["formulation"] = ProblemDepot.benchmark_suite(Convex.conic_problem; exclude=[r"fix"])

docs/src/contributing.md

@@ -5,57 +5,58 @@ We'd welcome contributions to the Convex.jl package. Here are some
 short instructions on how to get started. If you don't know what you'd
 like to contribute, you could
 
-> -   take a look at the current
->     [issues](https://github.com/JuliaOpt/Convex.jl/issues) and pick
->     one. (Feature requests are probably the easiest to tackle.)
-> -   add a [usage
->     example](https://github.com/JuliaOpt/Convex.jl/tree/master/examples).
+ -   take a look at the current
+     [issues](https://github.com/JuliaOpt/Convex.jl/issues) and pick
+     one. (Feature requests are probably the easiest to tackle.)
+ -   add a [usage
+     example](https://github.com/JuliaOpt/Convex.jl/tree/master/examples).
 
 Then submit a pull request (PR). (Let us know if it's a work in
 progress by putting \[WIP\] in the name of the PR.)
 
 Adding examples
 ---------------
 
-> -   Take a look at our exising [usage
->     examples](https://github.com/JuliaOpt/Convex.jl/tree/master/examples)
->     and add another in similar style.
-> -   Submit a PR. (Let us know if it's a work in progress by putting
->     \[WIP\] in the name of the PR.)
-> -   We'll look it over, fix up anything that doesn't work, and merge
->     it!
+ -   Take a look at our exising [usage
+     examples](https://github.com/JuliaOpt/Convex.jl/tree/master/examples)
+     and add another in similar style.
+ -   Submit a PR. (Let us know if it's a work in progress by putting
+     \[WIP\] in the name of the PR.)
+ -   We'll look it over, fix up anything that doesn't work, and merge
+     it!
 
 Adding atoms
 ------------
 
 Here are the steps to add a new function or operation (atom) to
 Convex.jl. Let's say you're adding the new function $f$.
 
-> -   Take a look at the [nuclear norm
->     atom](https://github.com/JuliaOpt/Convex.jl/blob/master/src/atoms/sdp_cone/nuclearnorm.jl)
->     for an example of how to construct atoms, and see the [norm
->     atom](https://github.com/JuliaOpt/Convex.jl/blob/master/src/atoms/second_order_cone/norm.jl)
->     for an example of an atom that depends on a parameter.
-> -   Copy paste (eg) the nuclear norm file, replace anything saying
->     nuclear norm with the name of the atom $f$, fill in monotonicity,
->     curvature, etc. Save it in the appropriate subfolder of
->     `src/atoms/`.
-> -   Add as a comment a description of what the atom does and its
->     parameters.
-> -   The most mathematically interesting part is the `conic_form!`
->     function. Following the example in the nuclear norm atom, you'll
->     see that you can just construct the problem whose optimal value is
->     $f(x)$, introducing any auxiliary variables you need, exactly as
->     you would normally in Convex.jl, and then call `cache_conic_form!`
->     on that problem.
-> -   Add a test for the atom so we can verify it works in
->     `test/test_<cone>`, where `<cone>` matches the subfolder of
->     `src/atoms`.
-> -   Submit a PR, including a description of what the atom does and its
->     parameters. (Let us know if it's a work in progress by putting
->     \[WIP\] in the name of the PR.)
-> -   We'll look it over, fix up anything that doesn't work, and merge
->     it!
+ -   Take a look at the [nuclear norm
+     atom](https://github.com/JuliaOpt/Convex.jl/blob/master/src/atoms/sdp_cone/nuclearnorm.jl)
+     for an example of how to construct atoms, and see the [norm
+     atom](https://github.com/JuliaOpt/Convex.jl/blob/master/src/atoms/second_order_cone/norm.jl)
+     for an example of an atom that depends on a parameter.
+ -   Copy paste (eg) the nuclear norm file, replace anything saying
+     nuclear norm with the name of the atom $f$, fill in monotonicity,
+     curvature, etc. Save it in the appropriate subfolder of
+     `src/atoms/`.
+ -   Add as a comment a description of what the atom does and its
+     parameters.
+ -   The most mathematically interesting part is the `conic_form!`
+     function. Following the example in the nuclear norm atom, you'll
+     see that you can just construct the problem whose optimal value is
+     $f(x)$, introducing any auxiliary variables you need, exactly as
+     you would normally in Convex.jl, and then call `cache_conic_form!`
+     on that problem.
+ -   Add a test for the atom so we can verify it works in
+     `src/problem_depot/problem/<cone>`, where `<cone>` matches the subfolder of
+     `src/atoms`. See [How to write a ProblemDepot problem](@ref) for details
+     on how to write the tests.
+ -   Submit a PR, including a description of what the atom does and its
+     parameters. (Let us know if it's a work in progress by putting
+     \[WIP\] in the name of the PR.)
+ -   We'll look it over, fix up anything that doesn't work, and merge
+     it!
 
 Fixing the guts
 ---------------
@@ -69,18 +70,18 @@ helpful as well; you can find the ipython notebook presented in the talk
 
 Then read the conic form code:
 
-> -   We define data structures for conic objectives and conic
->     constraints, and simple ways of combining them, in
->     [conic\_form.jl](https://github.com/JuliaOpt/Convex.jl/blob/master/src/conic_form.jl)
-> -   We convert the internal conic form representation into the
->     [standard form for conic
->     solvers](http://mathprogbasejl.readthedocs.io/en/latest/conic.html)
->     in the function
->     [conic\_problem](https://github.com/JuliaOpt/Convex.jl/blob/master/src/problems.jl#L97).
-> -   We solve problems (that is, pass the standard form of the problem
->     to a solver, and put the solution back into the values of the
->     appropriate variables) in
->     [solution.jl](https://github.com/JuliaOpt/Convex.jl/blob/master/src/solution.jl#L8).
+ -   We define data structures for conic objectives and conic
+     constraints, and simple ways of combining them, in
+     [conic\_form.jl](https://github.com/JuliaOpt/Convex.jl/blob/master/src/conic_form.jl)
+ -   We convert the internal conic form representation into the
+     [standard form for conic
+     solvers](http://mathprogbasejl.readthedocs.io/en/latest/conic.html)
+     in the function
+     [conic\_problem](https://github.com/JuliaOpt/Convex.jl/blob/master/src/problems.jl#L97).
+ -   We solve problems (that is, pass the standard form of the problem
+     to a solver, and put the solution back into the values of the
+     appropriate variables) in
+     [solution.jl](https://github.com/JuliaOpt/Convex.jl/blob/master/src/solution.jl#L8).
 
 You're now armed and dangerous. Go ahead and open an issue (or comment
 on a previous one) if you can't figure something out, or submit a PR if

docs/src/problem_depot.md

@@ -3,9 +3,10 @@ Problem Depot
 
 Convex.jl has a submodule, `ProblemDepot` which holds a collection of convex optimization problems. The problems are used by Convex itself to test and benchmark its code, but can also be used by solvers to test and benchmark their code.
 
-ProblemDepot has two main methods for accessing these problems: `Convex.ProblemDepot.run_tests` and `Convex.ProblemDepot.suite`.
+ProblemDepot has two main methods for accessing these problems: `Convex.ProblemDepot.run_tests` and `Convex.ProblemDepot.benchmark_suite`.
 
 For example, to test the solver SCS on all the problems of the depot except the mixed-integer problems (which it cannot handle), run
+
 ```julia
 using Convex, SCS, Test
 @testset "SCS" begin
@@ -15,10 +16,82 @@ using Convex, SCS, Test
 end
 ```
 
-## Reference
+How to write a ProblemDepot problem
+-----------------------------------
+
+The problems are organized into folders in `src/problem_depot/problems`. Each is written as a function, annotated by `@add_problem`, and a name, which is used to group the problems. For example, here is a simple problem:
+
+```julia
+@add_problem affine function affine_negate_atom(handle_problem!, ::Val{test}, atol, rtol, ::Type{T}) where {T, test}
+    x = Variable()
+    p = minimize(-x, [x <= 0])
+    if test
+        @test vexity(p) == AffineVexity()
+    end
+    handle_problem!(p)
+    if test
+        @test p.optval ≈ 0 atol=atol rtol=rtol
+        @test evaluate(-x) ≈ 0 atol=atol rtol=rtol
+    end
+end
+```
+
+The `@add_problem` call adds the problem to the registry of problems in [`Convex.ProblemDepot.PROBLEMS`](@ref), which in turn is used by [`run_tests`](@ref) and [`benchmark_suite`](@ref). Next, `affine` is the grouping of the problem; this problem came from one of the affine tests, and in particular is testing the negation atom. Next is the function signature:
+
+```julia
+function affine_negate_atom(handle_problem!, ::Val{test}, atol, rtol, ::Type{T}) where {T, test}
+```
+
+this should be the same for every problem, except for the name, which is a description of the problem. It should include what kind of atoms it uses (`affine` in this case), so that certain kinds of atoms can be ruled out by the `exclude` keyword to [`run_tests`](@ref) and [`benchmark_suite`](@ref); for example, many solvers cannot solve mixed-integer problems, so `mip` is included in the name of such problems.
+
+Then begins the body of the problem. It is setup like any other Convex.jl problem, only `handle_problem!` is called instead of `solve!`. This allows particular solvers to be used (via e.g. choosing `handle_problem! = p -> solve!(p, solver)`), or for any other function of the problem (e.g. `handle_problem! = p -> Convex.conic_problem(p)` which is used for benchmarking problem formulation speed.) Tests should be included and gated behind `if test` blocks, so that tests can be skipped for benchmarking, or in the case that the problem is not in fact solved during `handle_problem!`.
+
+The fact that the problems may not be solved during `handle_problem!` brings with it a small complication: any command that assumes the problem has been solved should be behind an `if test` check. For example, in some of the problems, `real(x.value)` is used, for a variable `x`; perhaps as
+
+```julia
+x_re = real(x.value)
+if test
+    @test x_re = ...
+end
+```
+
+However, if the problem `x` is used in has not been solved, then `x.value === nothing`, and `real(nothing)` throws an error. So instead, this should be rewritten as
+
+```julia
+if test
+    x_re = real(x.value)
+    @test x_re = ...
+end
+```
+
+Benchmark-only problems
+-----------------------
+
+To add problems for benchmarking without tests, place problems in `src/problem_depot/problems/benchmark`, and include `benchmark` in the name. These problems will be automatically skipped during `run_tests` calls. For example, to benchmark the time it takes to add an SDP constraint, we have the problem
+
+```julia
+@add_problem constraints_benchmark function sdp_constraint(handle_problem!, args...)
+    p = satisfy()
+    x = Variable(44, 44) # 990 vectorized entries
+    push!(p.constraints, x ⪰ 0)
+    handle_problem!(p)
+    nothing
+end
+```
+
+However, this "problem" has no tests or interesting content for testing, so we skip it during testing.
+Note, we use `args...` in the function signature so that it may be called with the standard function signature
+
+```julia
+f(handle_problem!, ::Val{test}, atol, rtol, ::Type{T}) where {T, test}
+```
+
+Reference
+---------
 
 ```@docs
 Convex.ProblemDepot.run_tests
-Convex.ProblemDepot.suite
+Convex.ProblemDepot.benchmark_suite
+Convex.ProblemDepot.foreach_problem
 Convex.ProblemDepot.PROBLEMS
 ```