Could not implement the ensemble method

[wenxichen2746]

Dear authors,

Amazing works on the benchmark testing of OOD detections. Please kindly suggest on how to test ensemble method using openood. I can see ensemble method is defined among postprocessors, but the string 'ensemble' can not be passed to Evaluator directly as its config is not loaded somehow.

The specific lines of code I am running is as follow:

evaluator = Evaluator(
net,
id_name='cifar10', # the target ID dataset
data_root='./data', # change if necessary
config_root=None, # see notes above
preprocessor=None, # default preprocessing for the target ID dataset
postprocessor_name="ensemble", # the postprocessor to use
postprocessor=None, # if you want to use your own postprocessor
batch_size=200, # for certain methods the results can be slightly affected by batch size
shuffle=False,
num_workers=2)

And the key error is as follow:

site-packages/openood/postprocessors/ensemble_postprocessor.py:18, in EnsemblePostprocessor.__init__(self, config)
 15 self.postprocess_config = config.postprocessor
 16 self.postprocessor_args = self.postprocess_config.postprocessor_args
 17 assert self.postprocessor_args.network_name == \

---> 18 self.config.network.name,
19 'checkpoint network type and model type do not align!'
20 # get ensemble args
21 self.checkpoint_root = self.postprocessor_args.checkpoint_root
AttributeError: 'NoneType' object has no attribute 'name'

Your help would be greatly appreciated, thanks!
Best.

[zjysteven]

This is an edge case of the new evaluator where the config passed to the postprocessor doesn't really have config.network (this is why you see the error 'NoneType' object has no attribute 'name'). A quick fix would be to just comment out the assertion in your site-packages/openood/postprocessors/ensemble_postprocessor.py

#        assert self.postprocessor_args.network_name == \
#            self.config.network.name,\
#            'checkpoint network type and model type do not align!'

Also note that to get the ensemble postprocessor working correctly you will need to modify the configs/postprocessors/ensemble.yml according to your case.

[wenxichen2746]

Thank you for getting back to me! That feedback is really helpful.

Also, I wonder if the trained ensemble models should be downloaded through:

sh ./scripts/download/dowanload.sh

It seems that there is a permission issue with the shared document:

/.local/lib/python3.8/site-packages/gdown/download.py", line 267, in download
url = get_url_from_gdrive_confirmation(res.text)
File "/home/aims-exxact/.local/lib/python3.8/site-packages/gdown/download.py", line 55, in get_url_from_gdrive_confirmation
raise FileURLRetrievalError(
gdown.exceptions.FileURLRetrievalError: Cannot retrieve the public link of the file. You may need to change the permission to 'Anyone with the link', or have had many accesses. Check FAQ in https://github.com/wkentaro/gdown?tab=readme-ov-file#faq.

I wonder if you could kindly look into this. Again, I would greatly appreciate your help. I am trying to test my OOD detection algorithm under this framework, and the ensemble really serves as a key baseline that I want to reproduce.

Best regards,

[zjysteven]

@wenxichen2746 Try upgrade gdown by python -m pip install --upgrade gdown. I just tested locally and saw the same error, which got fixed after upgrading gdown.