Tutorial for adding custom loss function in C++

[JoshuaC3]

Hi, I am implementing the fair loss function to approximate MAE. 1 2 3 However, it runs incredibly slowly. Is this a known issue or is my implementation of the function below just a bad implementation?

import math
from catboost import Pool, CatBoostClassifier

class fair_obj(object):
    def calc_ders_range(self, approxes, targets, weights):
        # approxes, targets, weights are indexed containers of floats
        # (containers with only __len__ and __getitem__ defined).
        # weights parameter can be None.
        # Returns list of pairs (der1, der2)
        assert len(approxes) == len(targets)
        if weights is not None:
            assert len(weights) == len(approxes)

        x = []
        for index in range(len(approxes)):
            x.append(targets[index] - approxes[index])
        
        result = []
        c=1
        for index in range(len(targets)):
            p = abs(x[index]) + c #x[index] / (1 + x[index])
            der1 = c*x[index] / p
            der2 = c*c / p ** 2

            if weights is not None:
                der1 *= weights[index]
                der2 *= weights[index]

            result.append((-der1, -der2)) # negatives as it says to maximise loss functions.

        return result

The training time difference for 10, 100 and 1000 samples was around 5x - 10x as long. For my entire dataset 200,000, it took too long and threw a python/jupyter notebook warning.

IOPub data rate exceeded.
The notebook server will temporarily stop sending output
to the client in order to avoid crashing it.
To change this limit, set the config variable
--NotebookApp.iopub_data_rate_limit.

FYI: Using the built in RMSE gives a decent MAE score of 183 MAE on my application. Using the MAE objective function gave around 360 MAE (hence trying my own MAE approx.). Using the fair loss custom on a 1000 samples (0.5% of the dataset) gave a score of 280 MAE (so good improvements expected if can run it for all 200,000 sample points).

Benchmarking: using XGBoost with custom fair loss gave an MAE of 179 with little speed change (190 MAE when using RMSE as objective function).
Using lightgbm with built-in fair loss object gave 187 MAE and 192 MAE when using RMSE objective function.

[annaveronika]

Hi, we know about the issue and this is because we calculate derivatives not once per iteration, but many more times.
If you want to do it faster, you could write your function in c++, we'll add a tutorial for this also.
MATRIXNET-974

For now you can try to do in on your own. You need to add a function to catboost/libs/algo/error_functions.h add this function name to enumeration with functions and look carefully that the correct enum value is passed everywhere.

[JoshuaC3]

@annaveronika thank you for your reply. That is a shame. Hopefully it can be resolved soon with a speed up for the python Classes and functions.

I have no c++ knowledge so will not be able to rewrite this in c++ from scratch. I will look forward to the c++ tutorial as I might be able to edit this for my needs.

Is it worth rewriting the above in Cpython? Would this help with a speed up? What about in numba?

Finally, it seems from what you are saying the model should run, just very slowly. I will run this on my full dataset and let it go. Hopefully it completes. I will update with a time for the 200,000 ~ points.

[Ubikas]

Hi @annaveronika,
I would like to join @JoshuaC3 to ask to add the fair loss function and even huber at the same time. I suppose it shouldn't take a lot of time :) I would really appreciate. Probably we could do it on our own, but these loss functions have extra parameter what probably brings some more complexity to make them on my own.
Thank you, catboot is really nice tool especially dealing with categories.

[annaveronika]

@Ubikas You could add it to C++ code and send it as a pull request.
We already have functions with parameters - Quantile and LogLinQuantile.
To add a new one you need to write a class in error_functions.h and metric.h and pass the parameter through train_model.cpp so that this two classes are used.

[rupak-118]

Can someone please explain to me what do the der1 and der2 variables signify in the above code as well as in the usage example in the official docs?

first and second derivatives? with respect to what?

[JoshuaC3]

@rupak-118 w.r.t the loss function. If the loss function is MSE, f(x) = x^2 where x is y-yhat, y is the target and yhat is the prediction.

der1 = f'(x) = 2x
der2 = f''(x) = 2

The MSE is a nice example. My issue, and the main topic in this post is with calculating MAE, and more generally, calculating custom loss function quickly enough to be useful.

[rupak-118]

@JoshuaC3 Thanks. I was hoping to build one for Recall, but it seems it will be too slow. I'll try and explore the C++ way as mentioned by @annaveronika. Not sure if I'll be able to bring it to fruition, though.

[JoshuaC3]

@rupak-118 do let us know if you get this to work.

[nikitxskv]

We added c++ user error function tutorial.
You can find it here.