Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

BOHB loop does not abort when there is an error #91

Open
poojitharamachandra opened this issue Jul 29, 2020 · 1 comment
Open

BOHB loop does not abort when there is an error #91

poojitharamachandra opened this issue Jul 29, 2020 · 1 comment

Comments

@poojitharamachandra
Copy link

poojitharamachandra commented Jul 29, 2020

Hi,
Returning just the loss from 'compute' function of the worker causes error, But the execution does not abort, instead goes into an infinite loop.

I wrote 'return loss' in my code , instead of
return ({'loss': s, # remember: HpBandSter always minimizes!
'info': {'r2_score': r2,'RMSE': accuracy}
})#something like this

the debug file had error :

File "/home/....../lib/python3.8/site-packages/hpbandster/core/base_iteration.py", line 133, in register_result
if (not job.result is None) and np.isfinite(result['loss']):
TypeError: 'float' object is not subscriptable

I saw this error after 12 hours.
When this was corrected, the complete setup ran for just 5 min and gave me results (of course I was using a very small dataset and model)

Thanks

@alexgawrilow
Copy link

I find this behavior not optimal, too. Any error in the compute method from the worker is caught at some place and causes the optimization to continue running. I'm wondering if that behavior is desired and implemented on purpose.
In my opinion a better approach would be not to catch exceptions in the compute method.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants