Booster.refit() fails when the booster used a custom objective function fobj

[jrvmalik]

Description

I first train an lgb.Booster with a custom objective function. I then try to update the leaf values using another dataset, but it fails on an assertion that the objective must not be None.

Reproducible example

import lightgbm as lgb
import numpy as np
from dataclasses import dataclass

X = np.random.randn(300, 3)
y = np.random.randn(300)
y_new = np.random.randn(300)

data = lgb.Dataset(X, label=y)
params = {"max_depth": 3, 'learning_rate': 0.1}


def fobj(predictions, dataset: lgb.Dataset) -> tuple[np.ndarray, np.ndarray]:
    targets = dataset.get_label()
    return (predictions - targets), np.ones(predictions.shape)
    

model = lgb.train(params, data, num_boost_round=10, keep_training_booster=True, fobj=fobj)
model = model.refit(data=X, label=y_new, decay_rate=0.0)

This yields the following error.

---------------------------------------------------------------------------
LightGBMError                             Traceback (most recent call last)
Input In [3], in <cell line: 19>()
     15     return (predictions - targets), np.ones(predictions.shape)
     18 model = lgb.train(params, data, num_boost_round=10, keep_training_booster=True, fobj=fobj)
---> 19 model = model.refit(data=X, label=y_new, decay_rate=0.0)

File /data/johnmalik/miniconda3/envs/rig_env-1.25.0/lib/python3.9/site-packages/lightgbm/basic.py:3565, in Booster.refit(self, data, label, decay_rate, **kwargs)
   3543 """Refit the existing Booster by new data.
   3544 
   3545 Parameters
   (...)
   3562     Refitted Booster.
   3563 """
   3564 if self.__set_objective_to_none:
-> 3565     raise LightGBMError('Cannot refit due to null objective function.')
   3566 predictor = self._to_predictor(deepcopy(kwargs))
   3567 leaf_preds = predictor.predict(data, -1, pred_leaf=True)

LightGBMError: Cannot refit due to null objective function.

The expected behavior should be getting the same result as this, where the refitting succeeds.

import lightgbm as lgb
import numpy as np
from dataclasses import dataclass

X = np.random.randn(300, 3)
y = np.random.randn(300)
y_new = np.random.randn(300)

data = lgb.Dataset(X, label=y)
params = {"max_depth": 3, 'learning_rate': 0.1}


def fobj(predictions, dataset: lgb.Dataset) -> tuple[np.ndarray, np.ndarray]:
    targets = dataset.get_label()
    return (predictions - targets), np.ones(predictions.shape)
    

model = lgb.train(params, data, num_boost_round=10, keep_training_booster=True)
model = model.refit(data=X, label=y_new, decay_rate=0.0)

[jmoralez]

Hi @jrvmalik, thanks for using LightGBM. This is deliberate because refit doesn't support using a custom objective #1699 (comment). The reason is that when training we have the option to run one boosting iteration passing the gradients and hessians, but the refit method updates all the trees in one go.

LightGBM/src/boosting/gbdt.cpp

Lines 353 to 389 in 6fa4673

	void GBDT::RefitTree(const std::vector<std::vector<int>>& tree_leaf_prediction) {
	CHECK_GT(tree_leaf_prediction.size(), 0);
	CHECK_EQ(static_cast<size_t>(num_data_), tree_leaf_prediction.size());
	CHECK_EQ(static_cast<size_t>(models_.size()), tree_leaf_prediction[0].size());
	int num_iterations = static_cast<int>(models_.size() / num_tree_per_iteration_);
	std::vector<int> leaf_pred(num_data_);
	if (linear_tree_) {
	std::vector<int> max_leaves_by_thread = std::vector<int>(OMP_NUM_THREADS(), 0);
	#pragma omp parallel for schedule(static)
	for (int i = 0; i < static_cast<int>(tree_leaf_prediction.size()); ++i) {
	int tid = omp_get_thread_num();
	for (size_t j = 0; j < tree_leaf_prediction[i].size(); ++j) {
	max_leaves_by_thread[tid] = std::max(max_leaves_by_thread[tid], tree_leaf_prediction[i][j]);
	}
	}
	int max_leaves = *std::max_element(max_leaves_by_thread.begin(), max_leaves_by_thread.end());
	max_leaves += 1;
	tree_learner_->InitLinear(train_data_, max_leaves);
	}
	for (int iter = 0; iter < num_iterations; ++iter) {
	Boosting();
	for (int tree_id = 0; tree_id < num_tree_per_iteration_; ++tree_id) {
	int model_index = iter * num_tree_per_iteration_ + tree_id;
	#pragma omp parallel for schedule(static)
	for (int i = 0; i < num_data_; ++i) {
	leaf_pred[i] = tree_leaf_prediction[i][model_index];
	CHECK_LT(leaf_pred[i], models_[model_index]->num_leaves());
	}
	size_t offset = static_cast<size_t>(tree_id) * num_data_;
	auto grad = gradients_pointer_ + offset;
	auto hess = hessians_pointer_ + offset;
	auto new_tree = tree_learner_->FitByExistingTree(models_[model_index].get(), leaf_pred, grad, hess);
	train_score_updater_->AddScore(tree_learner_.get(), new_tree, tree_id);
	models_[model_index].reset(new_tree);
	}
	}
	}

Supporting this would involve changing the refit method, I'm not sure how extensive these changes would have to be, maybe @guolinke can comment.

[jrvmalik]

I see. Yeah it would be great if this was supported!

[jmoralez]

Closed in favor of being in #2302, where we keep all feature requests.

You're welcome to contribute this feature! Please re-open this issue (or post a comment if you are not a topic starter) if you are actively working on implementing this feature.

[github-actions]

This issue has been automatically locked since there has not been any recent activity since it was closed.
To start a new related discussion, open a new issue at https://github.com/microsoft/LightGBM/issues
including a reference to this.