AISTATS camera-ready plan

[xukai92]

• More benchmark (Improving Wiki #335, Add more models to benchmarking #309)

  • LDA (=clustering)
  • Naive Bayes (=classification)
  • Cross cat? @yebai I didn't catch the name in the meeting
  • Stochastic volatility (time series)
  • HMM (time series)
  • High-dim Gaussian

• Improve documentation (Doc Turing.assume, Turing.observe and Turing.VarName #344, Better documentations, especially get-started #280, Turing Interface Design #104)

• Finishing ReverseDiff.jl integration (AutoDiff #331, ForwardDiff.gradient is slow #234)

• Only return log posterior and change vi as side effect

• Better typing in the code

  • Try use @code_warntype (suggested by Maxime)

• Replace vi.vals by vi.rvs

  • This will also solve VarInfo: dist ==> typeof{dist} #191

• Try to resolve adaptation problem (Robust adaption for NUTS #324 (comment))

[xukai92]

Note: for testing we need to test both with ReverseDiff.jl and ForwardDiff.jl as backend. Also I have set a flag to use ReverseDiff.jl either "safely" or `unsafely". "safely" means Turing.jl force to re-compile the tape each time it get gradient - it is much slower but can handle dynamic model.

• a68e9af passes all tests except gibbs.jl/gibbs2jl for unsafe reverse diff. I didn't figure out why.

[yebai]

Here is the CrossCat model I mentioned - http://probcomp.csail.mit.edu/crosscat/. This model is not super urgent, let's focus on existing models for now. When we are done, we can consider the following two models

• Prophet: forecasting at scale - https://research.fb.com/prophet-forecasting-at-scale/
  This is a model used by Facebook and Uber in production for forecasting time series.

• Bayesian Logistic Regression from the NUTS paper - https://arxiv.org/pdf/1111.4246.pdf

[ChrisRackauckas]

If you need more realish benchmarks you can use ours from DiffEqBayes.jl. Some of them we have Turing.jl omitted because of SciML/DiffEqBayes.jl#30 , but for what works it seems to work well according to the benchmarks in https://github.com/JuliaDiffEq/DiffEqBenchmarks.jl/

There's a question of how to address time and accuracy at the same time that is being worked on by @Vaibhavdixit02. If you have any pointers that would be helpful.

[xukai92]

Thanks! I checked the benchmark results. Just FYI the benchmark results actually indicates Turing.jl is slower than Stan as the sampling algorithm used is different. E.g. https://github.com/JuliaDiffEq/DiffEqBenchmarks.jl/blob/master/ParameterEstimation/DiffEqBayesFitzHughNagumo.ipynb shows Stan uses 93.0s and Turing.jl uses 30.9s, however, Stan takes 79*600 evaluations of the model but Turing.jl only takes 4*500, which together says Stan is 7.65 times faster.

I managed to make Turing.jl faster on high dimensional problem by adapting ReverseDiff.jl (branch not merged yet) but it seems that another main slowness of Turing.jl comes from things mentioned in #400, which I don't have yet a very clear approach about how to improve.

[xukai92]

Benchmark models are added in PR #424