Is there a LOO R2?

[hadjipantelis]

I am just playing with ArviZ and it is great. There is a r2_score to get Bayesian R²; is there a LOO equivalent akin to the one discussed in https://avehtari.github.io/bayes_R2/bayes_R2.html or in loo_R2 for brms?

[OriolAbril]

Missed that message somehow. We don't have that yet, and I'm not too familiar with the method so I'm not sure if it should be here or at bambi cc @aloctavodia @tomicapretto what do you think?

Also, would you be interested in contributing it to either ArviZ or bambi @hadjipantelis ?

[tomicapretto]

It sounds interesting, and it looks like something we could have in Bambi. However, I'm not very familiar with the method and it would require me to study it a little.

I can add it to our TODO list, but I don't think we're going to tackle it very soon.

[hadjipantelis]

Hmm... OK, @OriolAbril.

Would you guys prefer this in bambi or arviz? (Just so I know from which repo to clone.)
Downstream I would need a bit more feedback regarding where this functionality needs to reside module-wise, testing, etc.

[tomicapretto]

I think ArviZ has a much wider adoption than Bambi. However, as far as I've read so far, you need more information about the model for which you're computing R^2. And in Bambi you have all that information so it seems like a better place.

[ahartikainen]

What are the needed parts for loo_r2?

[tomicapretto]

I'm not sure. But the code in brms seems to do different things for at least two different families of regression models.
https://github.com/paul-buerkner/brms/blob/master/R/loo_predict.R

[hadjipantelis]

@ahartikainen: Reading through brms::loo_predict the immediate thing missing in arviz is loo::E_loo functionality (that is assuming that arviz.psislw can replicate the output of loo::psis).

@tomicapretto: It seems to me that the implementation errs for unordered categorical models and/or gives some rather strong-worded warnings if used with ordinal families. Family type doesn't look like to be a show-stopper here. (Most people would probably look twice at an R^2 for a non-continuous variable anyway...)

[OriolAbril]

It looks like E_loo or equivalent should go in ArviZ and use the data in posterior predictive group. loo_predict (and similars) would go in bambi (if the posterior predictive group is not present, ArviZ can't populate it, bambi can).