Constrained Bayesian optimization

[ValerioPerrone]

Description of changes:

Extends the acquisition function to handle multiple outputs and implements constrained Bayesian optimization.

This is useful to tackle problems such as optimizing accuracy with an upper bound on memory requirements or on unfairness.

The main change is meanstd_acqfunc.py. The acquisition function now internally operates with a dictionary Dict[str, SurrogateModel], allowing for a model for each output metric. The standard case is a dictionary with a single model but the change is backward compatible: if a plain SurrogateModel is passed, the acquisition function automatically turns it into a one-key dictionary.

Other key changes:

• Implemented CEIAcquisitionFunction and EIpuAcquisitionFunction, two multi-output acquisition functions. These require the predictions from both the main model and, respectively, the constraint or cost model, and return the head gradients wrt to the mean and standard deviation of each model.

• Introduced 'constrained_bayesopt', a new searcher to run end-to-end experiments with constrained BO.

• Extended GPModelPendingCandidateStateTransformer to support multiple models with a shared state.

Testing

• All previous tests pass. The behavior of existing code is unaffected by the acquisition function refactoring.

• test_bayesopt_eipu.py: tests cost-aware EI, an example of multi-output acquisition function.

• test_bayesopt_cei.py: tests the constrained EI. This includes tests with/without MCMC, with/without fantasizing, and with/without feasible candidates.

• test_multi_output_searcher.py: tests the constrained BO searcher.

[szha]

Job PR-1034-1 is done.
Docs are uploaded to http://autogluon-staging.s3-website-us-west-2.amazonaws.com/PR-1034/1/index.html

[Innixma]

A general thought: This is a lot of changes, do we think it makes sense to have an entire submodule autogluon.search dedicated to this code / searcher code in general? This could be an optional dependency for other modules which would allow to plug-in searcher backends. This would lighten the autogluon.core code and ensure that the blast radius of searcher changes are contained within the searcher module.

[Innixma]

This would also make migration easier if we eventually move the searcher logic to ray or some similar package.

[mseeger]

In fact, we will move all this code out of the AutoGluon codebase and bring it back as a dependence. We are currently working on this full steam, our code would then work on AutoGluon, but also on Ray Tune. In the end, this is the cleanest solution.

This will happen likely in April, latest in May.

Of course, we'll make sure everything will work just as before.

[Innixma]

That's great, I think that is the cleanest approach as well.

[szha]

Job PR-1034-2 is done.
Docs are uploaded to http://autogluon-staging.s3-website-us-west-2.amazonaws.com/PR-1034/2/index.html

[szha]

Job PR-1034-3 is done.
Docs are uploaded to http://autogluon-staging.s3-website-us-west-2.amazonaws.com/PR-1034/3/index.html

[mseeger]

Use keys for the arguments

[ValerioPerrone]

Done.

[mseeger]

Use key: active_output=active_output

[ValerioPerrone]

Done.

[mseeger]

Add comment: We need two GP models here, one for active metric, the other for constraint metric

[mseeger]

BTW: You could make this really clean by making a temporary function which creates gpmodel, model_args, and then call that both in _create_common_objects and here. This way, we make sure this is done in the same way.

[mseeger]

Please do this, let us avoid copy&paste

[ValerioPerrone]

I've factored it out as a common function.

[mseeger]

These two seem to be parameters the user can pass elsewhere, e.g. in searcher_factory.py.

I think either way is OK (user can choose, or we fix them). Since these are internal names, it is fine to fix them. But please only do one of them.

[ValerioPerrone]

I've now fixed them as they are internal only.

[mseeger]

Since active_metric is just an internal name (right?), we can just fix it to DEFAULT_METRIC, no?

[ValerioPerrone]

Done.

[mseeger]

These are internal names, no?

Of course, the user-facing names (appearing in train_fn) must be configurable.

[ValerioPerrone]

Done.

[mseeger]

See above. If these are internal, there is no value having the user set them. Correct me if I am wrong.

[ValerioPerrone]

Done. They are internal and fixed.

[szha]

Job PR-1034-5 is done.
Docs are uploaded to http://autogluon-staging.s3-website-us-west-2.amazonaws.com/PR-1034/5/index.html

[mseeger]

nitpick:
This could be done without copy&paste, something like this:

if self._use_single_model:
gpmodels = dictionarize_objective(self._gpmodel)
else:
gpmodels = self._gpmodel
output_models = {
GaussProcSurrogateModel(...)
for metric_name, gpmodel in gpmodels
}
if self._use_single_model:
self._model = output_models[self.active_metric]
else:
self._model = output_models

You are using dictionarize_objective elsewhere for this. That's elegant, but maybe the name is misleading then.

[mseeger]

Actually, please do this. We don't want c&p.

[mseeger]

OK, is there not a potential problem here?

• In the code, you allow the user to define the active metric name, and only if not is DEFAULT_METRIC used
• But in dictionarize_objective, DEFAULT_METRIC is hardcoded.

I'd prefer if the hardcoded names are used everywhere, no? There seems no advantage of the user choosing these names, because they are internal anyway.

[mseeger]

Can you expand this docstring? This type def is used all over the place. Please do explain why this is sometimes a dict.

[mseeger]

Is there a good reason for moving this to base_classes, and not leave it here?

I've never been a big fan of this defaults.py, but the intention was to collect constants here, and why not? Can't you import from base_classes?

[ValerioPerrone]

The reason is that EIAcquisitionfunction requires it, resulting in a circular import.

[mseeger]

Why not just:
self.initial_scoring = 'acq_func'
assert (...)

[mseeger]

Again, DEFAULT_CONSTRAINT_METRIC hardcoded here. Please make it hardcoded everywhere then, do not let the user choose.

[mseeger]

Oh wait, no! This name is user facing. You cannot use DEFAULT_CONSTRAINT_METRIC here, you need to use something like self._constraint_attribute !

OK, there are:

• Internal names, in internal searcher code: You can fix them and hardcode
• User facing names, which appear in train_fn. These have to be arguments, of course

[ValerioPerrone]

Thanks for your comments. I've addressed them in the following commit. I've now created a self._constraint_attribute, which is user facing, and hardcoded the rest.

[mseeger]

I am OK once my comments from yesterday are addressed. In particular, the scheduler needs a _constraint_attribute attribute which can be passed in. I'd also strongly prefer the internal names of these metrics not to be user-selectable, this is just creating confusion.

[szha]

Job PR-1034-7 is done.
Docs are uploaded to http://autogluon-staging.s3-website-us-west-2.amazonaws.com/PR-1034/7/index.html

[mseeger]

OK,

[mseeger]

Any cycles for this one? This code is the basis for a tutorial on Fair HPO, which has been published now, and which would also go into a blog post, creating visibility for AG.

[Innixma]

Looks good in terms of not altering much existing functionality that is actively used by submodules. I don't have context in terms of the actual changes themselves, so I'm deferring to @mseeger for that.

I'm ok to merge, @mseeger feel free to merge whenever you are ready.