-
Notifications
You must be signed in to change notification settings - Fork 2.2k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Batching parameters #344
Comments
Check out batching/README.md. |
Thanks @chrisolston . I went thru the README.md for BatchingSession and BasicBatchScedhuler. Most of the parameters needed are in these 2 files. However, based on query client (inception_client.py), it doesn't look like it calls basic_batch_scheduler and/or BasicBatchScheduler. So, the question is how are these session and scheduling parameters passed to the query client? Looks like I am missing something... |
With the ModelServer binary the batching takes place in the server not the client. I just checked and unfortunately the batching parameters are currently hard-coded to the defaults. It would be easy to extend model_servers/main.cc to be able to accept the parameters via a textual proto file (analogous to model_config_file) containing a BatchingParameters proto, if you're up for making a (simple) code contribution :) As a work-around, you can hard-code some specific values into the BatchingParameters proto in main.cc. |
On second thought, since it's only a few lines of code I will add the flag. I'm working on the change internally at Google, and if things go smoothly it will propagate to open-source in a week or so. |
@chrisolston Thanks for the clarification. Will try the workaround for now. |
As of now, tensorflow_model_server seems to support batching_parameter_file as an argument. Can someone point me to a template or specification for this file? I searched around and could not come across anything in a timely manner |
As the flag documentation states, it's an ascii protobuf for the BatchingParameters proto (defined in session_bundle_config.proto). You can find information about the ascii protobuf format elsewhere. It basically looks like: |
What is the default setting for batching parameters when enable_batching=true? I looked at the following config file, played with the parameter settings and ran the model server against each of the settings, but the results do not match up with what I get when I do not supply any file name to batching_parameters_file (but have enable_batching=true).. |
I am running TFS (custom build for GPU) with standard InceptionV3 model. Amazon EC2 P2.xlarge instance:
batching.conf file contents
I have set num_batch_threads equal to number of CPU cores i.e. 4 Tensorflow Serving initialisation logs shows that GPU is visible and utilised.
will show how GPU is being utilised when performance script is run. Test 1: Single process with 100 thread(s) per process and 1 request per thread
With batching using above mentioned config (average response time for multiple execution is 1.1 sec):
Test 2: Single process with 100 thread(s) per process and 10 requests per thread
With batching using above mentioned config (average response time for multiple execution is 1.4 sec):
Numbers looks better with batching. We have TF benchmark for training: https://www.tensorflow.org/performance/benchmarks |
Hi @sreddybr3, |
This is a great discussion and points to a need for documentation of batching for modelserver binary. I have opened #1379 to add docs - if anyone has any thoughts or suggestions please comment on that issue :) |
I am trying to use a batching config file using --batching_parameters_file. Where this file has to be placed in order to be used? |
just give the absolute path of the file. In the case of docker, you will have to mount your local folder or have the file in the container itself. |
Can we reload the batch config after the server starts ? I see model config has such function. |
Same problem . I run tf-serving like this :
And my batch_size.config is :
It works as slow as I don't use |
What is the best way to use batching parameters like max_batch_size, batch_timeout_micros, num_batch_threads and other parameters? Tried using them while running the query client.
In the below example I have 100 images and I want to batch in size of 10. The query runs for all images instead of 10.
bazel-bin/tensorflow_serving/example/demo_batch --server=localhost:9000 --max_batch_size = 10
Also, for batch scheduling how to make it run every 10 secs after the first batch is done? Appreciate some ideas..
The text was updated successfully, but these errors were encountered: