[BREAKING] Cleanup gpu_id configuration.

[trivialfis]

Recently found a bug that was introduced in updating the prediction cache for gpu_hist. When a custom objective is used, the gradient is on CPU. This PR fixes it by prioritizing the gpu_id parameter.

Other than the bug fix, this PR also tries to cleanup and clarify the configuration of this parameter. Please see the note in gbtree.cc for details.

Here breaking means, one can no longer run into edge cases like having gpu_id = 0 while running CPU exact. Overall it should not change normal usage.

Todos:

• Correct linear updaters.
• Add tests for the state transition.

[trivialfis]

@RAMitchell @hcho3 I will probably close this PR and open a different one to simply fix the bug. Getting it to work without some user-visible breaking changes seems impossible. For example:

Setting "gpu_hist" leads to gpu_id >= 0

"gpu_hist" -> gpu_id = 0

then if a user wants to run prediction on the CPU, he will set

booster.set_param({"predictor": "cpu_predictor"})

then xgboost will throw an error as gpu_id is invalid for CPU.

[trivialfis]

I'm picking up this PR again. Will proceed with breaking changes.