torch.compile() support

[msaroufim]

Caveats

1. Models will take longer to initialize
2. Changes in batch sizes will trigger a recompilation, so batch size choice needs to be sufficiently small otherwise recompilations will outweigh any benefits
3. TensorRT still has better performance than Inductor for inference, it's not clear to me yet whether users should do the tensorRT conversion via pytorch/tensorrt or via dynamo

scope of this PR

Just went ahead and added torch.compile support - to make it happen I made a few changes

1. Added a tutorial
2. In the base handler check if _dynamo is present and if yes enable pt 2.0
3. If pt 2.0 is enabled and a user passed in a compile.json file to specify the backend then model compilation is enabled
4. Enabled some reasonable defaults for mode that reduces overhead - aka cuda graphs
5. For convenience I also changed install_dependencies.py to include a mode that installs nightly torch

Logs

2022-12-05T23:56:53,168 [INFO ] W-9000-densenet161_1.0-stdout MODEL_LOG - Listening on port: /tmp/.ts.sock.9000
2022-12-05T23:56:53,172 [INFO ] W-9000-densenet161_1.0-stdout MODEL_LOG - Successfully loaded /home/ubuntu/serve/ts/configs/metrics.yaml.
2022-12-05T23:56:53,172 [INFO ] W-9000-densenet161_1.0-stdout MODEL_LOG - [PID]15049
2022-12-05T23:56:53,172 [INFO ] W-9000-densenet161_1.0-stdout MODEL_LOG - Torch worker started.
2022-12-05T23:56:53,173 [INFO ] W-9000-densenet161_1.0-stdout MODEL_LOG - Python runtime: 3.8.13
2022-12-05T23:56:53,187 [INFO ] W-9000-densenet161_1.0-stdout MODEL_LOG - Connection accepted: /tmp/.ts.sock.9000.
2022-12-05T23:56:53,228 [INFO ] W-9000-densenet161_1.0-stdout MODEL_LOG - model_name: densenet161, batchSize: 1
2022-12-05T23:56:57,592 [INFO ] W-9000-densenet161_1.0-stdout MODEL_LOG - Compiled model with backend inductor

[codecov]

Codecov Report

Merging #1960 (2e5c215) into master (c59c6ac) will decrease coverage by 0.44%.
The diff coverage is 20.96%.

@@            Coverage Diff             @@
##           master    #1960      +/-   ##
==========================================
- Coverage   53.80%   53.35%   -0.45%     
==========================================
  Files          70       70              
  Lines        3169     3220      +51     
  Branches       56       56              
==========================================
+ Hits         1705     1718      +13     
- Misses       1464     1502      +38     

Impacted Files	Coverage Δ
ts/torch_handler/base_handler.py	0.00% <0.00%> (ø)
ts/utils/util.py	43.66% <56.52%> (+6.16%)	⬆️

📣 We’re building smart automated test selection to slash your CI/CD build times. Learn more