Question about GBT-OMLT model Structure

[Yunshan-Liu67]

Hi,
As you suggested, I switched from RF model to GBT to utilize all the useful functions in OMLT for optimization problem construction. And therefore had one question about the capability of OMLT GBT functions. As noticed in the GBT example, the GBT model has the shape of multi-input single output. I'm wondering will OMLT accept a multi-input multi-output GBT model? Also, since lightgbm pkg only supports multi-input single output GBT model, does OMLT accept onnx model converted from pkg other than lightgbm, let's say xgboost?

[rmisener]

At this point in time, OMLT only accepts multi-input single-output GBT models.

OMLT will accept an onnx model that's converted from xgboost, but we haven't tested this connection, so we don't officially support it. As you can see in #80 and #81, we have tested the connections from lightgbm and scikit-learn to onnx. There are subtle differences in the way that onnx stores GBTs from each of the different packages, so it's possible there will be something weird in the xgboost connection.

[Yunshan-Liu67]

Thank you for your insightful reply! I switched from xgboost to sklearn and the single-output GBT model has been successfully imported into omlt block.
And with pkg sklearn, I tried to add MultiOutputRegressor out of my single-output GBT model which turns the single-output into multi-output. The updated model also being successfully taken into omlt block. I pprint outputs of the new omlt block and the index of it is indeed more than one. Since you mentioned OMLT only accepts multi-input single-output GBT models, could you explain this rather odd behavior of omlt?

[rmisener]

Check the GBT formulation in https://github.com/cog-imperial/OMLT/blob/main/src/omlt/gbt/gbt_formulation.py

You'll see that it's written, e.g. on Lines 40-42, to allow multiple outputs. But you'll also notice on Line 325 that only one of these outputs will have a meaningful value. So you can indeed have many output indexes, but only one of these outputs will have a value relevant to your trained machine learning model.

If you're doing any post-processing on the OMLT block, then I'm sure you could extend OMLT to multi-output GBT (since we already have the indexing ready). It's just that the constraints will not be correct.

[Yunshan-Liu67]

Thanks again for your explanation! As I sticking with single-output GBT model with sklearn package, another issue occurred. I could not getting the same result from sklearn predict method (link here) and OMLT XGB block with same input value.
Shouldn't XGB OMLT block (GBTBigMFormulation) ,like FullSpaceNNFormulation and ReducedSpaceNNFormulation, giving the same value as predict method in sklearn? Why I'm getting the opposite behavior?