Dev plan

[ericphanson]

I'm planning on making some changes and improvements, and I thought I'd write out my plan, in case there are objections and just so these changes aren't coming out of nowhere. It's hard to say how much of this I'll actually get done, but I've consistently been feeling pretty motivated on weekends to work on this, so hopefully it's not too wildly ambitious.

Ordered priorities:

• (Add benchmarks #321) Add some benchmarking. This is the first priority, because I want to make sure subsequent changes don't hurt performance. I'll probably put up a PR today or tomorrow and merge it fairly quickly, so that subsequent PRs can get benchmarked against master. This won't involve code changes to the Convex.jl src folder so there should be little chance of breaking anything.
• (Remove global variables #322) Get rid of global dictionary of variables (id_to_variables). This is half the cause of Memory leak #83, and I think there's a relatively easy fix. The hard bit is to make sure to not hurt perf too much, since we will have to do the work locally in the problem instead of once at variable construction time, so benchmarking is key.
• (Remove global variables #322) Get rid of the other global variable? I haven't looked into this, but it's used for dual values so it might be nice to do this before the next point. This would then close Memory leak #83.
• Switch to MOI. This is maybe 80% done in my branch https://github.com/ericphanson/Convex.jl. I still need to populate dual variables, decide what to do with the various can_solve_sdp type methods, and clean up the code a bit (either get rid of MBP or make Convex usable with both MBP and MOI). Benchmarking is important here too, to make sure we haven't hurt model formulation speed with the new internals generating MOI functions and constraints. Ideally, we might get improvements.
• Tag an MOI release! (https://github.com/JuliaOpt/Convex.jl/milestone/2)
• Formulate a coherent public API (Better public API (deprecate field access) #346, AbstractVariable interface; allow variables to carry constraints #358) with an aim towards 1.0

Optional:

• (Improve printing of problems, constraints, and variables #325) Add AbstractTrees dependency + methods, so a Convex.jl problem can be seen as an AbstractTree. I thought I might need this for getting rid of id_to_variables, but I think I won't now, so it's kind of optional. It's nice because you can use AbstractTrees.print_tree(p::Problem) to generate a nice representation of the problem. This might actually give a nicer way to show the problem in the REPL which we could use for show. You can also use GraphPlots master and do plot(TreePlot(p)) where p::Problem and get a visual representation of the problem (which was my motivation for Add AbstractTrees plotting JuliaPlots/GraphRecipes.jl#80). The only downside is the dependency, which is 700 lines of Julia with no dependencies itself, which doesn't seem too bad. I might PR this one soon because it's already done locally.
• Finish + merge (WIP) Allow variables to carry constraints, custom variable types #313 AbstractVariable interface; allow variables to carry constraints #358. This PR touches almost every use of Variable (if only trivially replacing it by AbstractVariable) so it will need rebasing with each of the above changes. But it's really an enhancement / feature, not blocking for MOI, and MOI is probably more important. It shouldn't have any perf consequences, but benchmarking would be nice to confirm. I see this as opening the door to DSLs based on top of Convex, and it may be better to wait for some of the big internals changes (e.g. switch to MOI) first, so that there's a stable base for those DSLs.
• Partially specified problems #310: allows implementing more atoms, and make it easier to write DSLs on top of Convex. Might be able to remove some current atoms too (without feature loss).
• Cache conic forms between solve!s? #318: seems important for making fix! and free! have meaningful perf advantages for model formulation.
• Various doc changes and small fixes. E.g. I have some open PRs that I should rebase and double-check.

Speculative:

• Change how conic_form! works to be closer to MOI. Theoretically, each conic_form! could be directly generating MOI functions and constraints, instead of using ConicObj's. This would give nicely typed objects which would hopefully address performance issues like Massive RAM for moderate problem #254, and maybe others (which I haven't looked into too much). The end-state of this would be Convex being a front-end to MOI like JuMP, but with different syntax, DCP checking, and some pseudo-bridges on top. I think that's a reasonable place to be, and positions Convex well to take advantage of MOI's fast + high quality development. This would likely involve rewriting conic_form! for every atom.

[matbesancon]

I'm ok with at least the high-priority you defined above

[ericphanson]

For those following this plan: there’s a brief detour to #328, which came about partly trying to get the tests working for MPB and MOI simultaneously, with the plan of supporting both— at least until it’s inconvenient; since Convex has a few points of contact with the middle layer, at this point it’s not really a problem to support both because the MPB code is already written. If we get to my speculative point above, I’d probably drop MPB.

Edit: this also solves another minor problem, namely we no longer need the can_solve* methods, since we get regex excludes with the problem depot.

[ericphanson]

We can’t super easily support both MPB and MOI simultaneously because they each represent SDP constraints in a different way, and we’d need a different conic_form! for each. We could propagate down the tree the information of whether or not we’re using an MOI solver. But I’m not sure that’s worth it, since it’s adding complexity to support superseded code.

[ericphanson]

I think updating the examples (and #335) will be my next priority now that #330 is essentially done
edit: done by #339

[ericphanson]

Next up is #358 and #346 to try to get an MOI release out soon.

[ericphanson]

See #361 for new thoughts on v0.13.

[ericphanson]

0.13 is out!

The next steps that I see are #346 and #358 to setup a coherent public API as a minor release, with deprecations for the breaking changes, then a 1.0 release with that API (and the deprecations removed). After that I’d like to focus on a non-breaking internals revamp for performance and ideally #310 as feature versions.

I’m not sure when all that will happen though though, possibly not for months, so if someone else has a different plan they want to pursue sooner, that seems fine as well :).

[ericphanson]

A bit of an update:

• I cleaned up and merged AbstractVariable interface; allow variables to carry constraints #358. Check it out!
• I have not worked on the public API (ref Better public API (deprecate field access) #346). It's hard and not super fun! It's also not completely clear if it's the right time for commit to a public API.
• I started working a bit on the "speculative" item "Change how conic_form! works to be closer to MOI". There is a proof of concept here Proof of concept more-MOIified implementation #393 which can solve one test problem (Too much memory consumption right before trying to solve a problem. #390), and matches JuMP performance (beating Convex.jl's current performance by a lot). I think this may be the right way forward for Convex.jl's internals, and can help with e.g. geomean: Support for vectors #388. To get better type stability, we may want to make some syntax changes like stopping supporting x >= 0 where x is a vector or matrix. This definitely needs a lot more thought though.

One other note: as I mentioned in my JuliaCon talk (https://www.youtube.com/watch?v=iczcGK6G_sE), I may not be working on Convex.jl as often from now on. (But this MOI updating thing has gotten me excited again...)

[ericphanson]

I haven't been working on this since July 2020, so this isn't really an active plan, and I'll close it at this point.