-
Notifications
You must be signed in to change notification settings - Fork 173
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
log warning for batch_size 1 #1503
base: main
Are you sure you want to change the base?
Conversation
Could this be made to be more in-line with the already existing validation check, or perhaps, just use that check instead? See https://github.com/neuralmagic/deepsparse/blob/main/src/deepsparse/engine.py#L108 Separately, while not a huge deal, I would be curious to know why the existing check catches, e.g. Pipeline.create(task="text_classification", batch_size=batch_size) In that test, the existing warning is thrown if |
@dbarbuzzi , good catch. @horheynm, it may be good to understand why pipeline.py overrides batch_size when engine.py is already doing it differently? and if the statement on the old lines 195 and 209 is going to stay around, they should probably be refactored to be a single line in whatever is the appropriate place in the init function. |
Yeah I actually wanted the change there for overriding the values, but then i dont understand
I do prefer changing the values in the locs you sent bc I also do have to initialize a new logger. If there are no conflicts changing the code snippet you mentioned, then ofc its better to make edits to that line. @bfineran Any issues changing the snippet of code in engine.py? |
@horheynm @dbarbuzzi @derekk-nm batch_size=0 was added as a special case to disable the engine from setting the batch size of the model. This is important because some models do not have a batch size at all! So we do need a way to express that case with the engine. Please see this PR for the original creation #1169 |
Thanks for that info @mgoin ! So it sounds like the changes we need are not to "fix" any logic, but just to ensure that this special case is properly displayed in a log message? |
Yes, I misunderstood that it is not exactly 0 that is a special case but anything under 1. That brings us back to the original problem: when using 0 specifically, that log message is not being triggered/displayed like it is if you use -3. That inconsistency is what this PR is supposed to address. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
For context, I believe this setting was added so that users who would set the pipeline kwarg batch_size
to 0 get the "fake" dynamic batch we had supported - in practice this needs to be set to 1.
There's two options here
- we can get rid of this convienence and require users of the legacy pipeline to manually set to 1 for dynamic batch (to get the engine disable behavior, BS would need to be set to <0 instead of <1)
- we can get rid of this override, and allow the engine to raise its usual warning even when bs is set to 0
Show warning when overriding batch_size 0 to 1
https://app.asana.com/0/1201735099598270/1206262288703592