clarify implementation of priors from Vanilla BO works great in high dimensions

[R-M-Lee]

Hey @jduerholt ,

I went through the paper and code of https://arxiv.org/abs/2402.02229 (code: https://github.com/hvarfner/vanilla_bo_in_highdim/tree/62cc3846108ee9ba6fe865a0da65e51f443c768b)

and saw your additions in 22c83fa

Just wanted to check a couple of things. In the paper they do not change the scale part of the lengthscale prior, but they do in their code (

bofire/bofire/priors/mapper.py

Line 42 in b730c29

scale=(data_model.scale**2 + math.log(d) * data_model.scale_scaling) ** 0.5,

). We in bofire follow the code rather than the paper. But that nice blogpost you shared https://www.miguelgondu.com/blogposts/2024-03-16/when-does-vanilla-gpr-fail/ the author notes that the results in the paper were generated without scaling sigma_0 and that the scaling we have here in bofire is apparently less stable. So should we consider removing it?

Second question: where is the noise prior we have implemented (

bofire/bofire/data_models/priors/api.py

Line 47 in b730c29

HVARFNER_NOISE_PRIOR = partial(LogNormalPrior, loc=-4, scale=1)

) from?

Third question: In section 5.2 of the paper they say that they fix the signal variance parameter to 1 because, among other reasons, the observations are standardized. I can't see straight away where this parameter comes into our setup... an RBF kernel does not out-of-the-box specify this parameter at all, so it's already fixed to 1, is that right?

I wanted to check these points because the paper has convinced me that these things are important.

[jduerholt]

Hi @R-M-Lee,

by default we are also not scaling the scale part, we just provide the option. In the accompanying data model the scale_scaling parameter is set to zero:

bofire/bofire/data_models/priors/normal.py

Line 47 in b730c29

scale_scaling: float = 0.0

This also holds true for their implementation. Have a look here: https://github.com/hvarfner/vanilla_bo_in_highdim/blob/62cc3846108ee9ba6fe865a0da65e51f443c768b/benchmarking/gp_priors.py#L34

We are just giving the possibility to also scale the scale parameter.

The noise prior is also from their code, have a look here: https://github.com/hvarfner/vanilla_bo_in_highdim/blob/62cc3846108ee9ba6fe865a0da65e51f443c768b/configs/model/default.yaml

Concerning signal variance, we are doing it as in gpytorch/botorch by having a ScaleKernel around the RBF/Matern Kernel. You can see this here:

bofire/bofire/data_models/surrogates/single_task_gp.py

Line 102 in b730c29

default_factory=lambda: ScaleKernel(

Note that botorch has now switsched to the priors from the paper as new default: pytorch/botorch#2451

We should do the same, if you want you can file a PR ;)

Best,

Johannes