Regarding replacing the lidar centerpoint net trt engine

[VeeranjaneyuluToka]

Hi,
Replacing the autoware->perception->lidar centerpoint onnx/trt engine with custom trt engine.

As mentioned above, i have a custom trt engine which is generated using mmdeploy and it is a single file. But when i analyse autoware-> Perception-> lidar centerpoint node, it loads two onnx's or engine's (pts_voxel_encoder_centerpoint_tiny.onnx, pts_voxel_encoder_centerpoint_tiny.engine and pts_backbone_neck_head_centerpoint_tiny.onnx, pts_backbone_neck_head_centerpoint_tiny.engine).

I am not sure how to generate these two separate files. I have used mmdeploy (https://github.com/open-mmlab/mmdeploy) to generate in-house onnx/trt engine. Would be great help if somebody can point out on how to generate in the required format (as two egines)? I could not find any references on the same (maybe i might have missed something in case if there is already).Thanks!

[kminoda]

Hi, thank you for your interest. I am an perception engineer from TIER IV.

TIER IV has an internal machine learning library that generates above ONNX files. Although we are working on making it public as well, but still taking a bit of time.

As a temporary solution, you may also find this work helpful: autowarefoundation/mmdetection3d#1

[VeeranjaneyuluToka]

Hi, Thanks for your quick reply and it is really helpful.

I think autoware expects two .onnx and .trt engines as well, is not it? And the pull request that you have shared generates only two .onnx files it seems to be.

Thanks!

[VeeranjaneyuluToka]

Hi @kminoda , thanks for quick reply. I tried to convert my onnx and ended up with the below errors

Error with the below config:
/**:
ros__parameters:
class_names: ["human", "tree"]
point_feature_size: 4
max_voxel_size: 40000
point_cloud_range: [-115.0, -115.0, -8.0, 115.0, 115.0, 32.0]
voxel_size: [0.4, 0.4, 8.0]
downsample_factor: 1
encoder_in_feature_size: 10
# post-process params
circle_nms_dist_threshold: 0.5
iou_nms_target_class_names: ["human"]
iou_nms_search_distance_2d: 10.0
iou_nms_threshold: 0.1
yaw_norm_thresholds: [0.3, 0.0, 0.3]

[lidar_centerpoint_node-1] [ERROR] [1718110422.654127020] [lidar_centerpoint]: Expected and actual number of classes do not match
[lidar_centerpoint_node-1] [ERROR] [1718110422.654162701] [lidar_centerpoint]: Failed to set profile
[lidar_centerpoint_node-1] [ERROR] [1718110422.698257763] [lidar_centerpoint]: Failed to create context: Engine was not created

The above error disappear and gives the below error when i change config as follows: Change is basically getting rid off one class name.

/**:
ros__parameters:
class_names: ["human"]
point_feature_size: 4
max_voxel_size: 40000
point_cloud_range: [-115.0, -115.0, -8.0, 115.0, 115.0, 32.0]
voxel_size: [0.4, 0.4, 8.0]
downsample_factor: 1
encoder_in_feature_size: 10
# post-process params
circle_nms_dist_threshold: 0.5
iou_nms_target_class_names: ["human"]
iou_nms_search_distance_2d: 10.0
iou_nms_threshold: 0.1
yaw_norm_thresholds: [0.3, 0.0, 0.3]

[lidar_centerpoint_node-1] [INFO] [1718109803.745353333] [lidar_centerpoint]: Applying optimizations and building TRT CUDA engine (/home/veeru/autoware/pre-trained/marine/pts_backbone_neck_head_centerpoint_marine.onnx) ...
[lidar_centerpoint_node-1] [E] [TRT] 4: [network.cpp::validate::3008] Error Code 4: Internal Error (spatial_features: for dimension number 1 in profile 0 does not match network definition (got min=32, opt=32, max=32), expected min=opt=max=64).)
[lidar_centerpoint_node-1] [E] [TRT] 2: [builder.cpp::buildSerializedNetwork::636] Error Code 2: Internal Error (Assertion engine != nullptr failed. )
[lidar_centerpoint_node-1] [ERROR] [1718109803.752861196] [lidar_centerpoint]: Failed to create serialized network
[lidar_centerpoint_node-1] [ERROR] [1718109803.796516124] [lidar_centerpoint]: Failed to create context: Engine was not created

Would be great help if you have any suggestions, thanks!

[kminoda]

Please check if the onnx structure (especially input and output) is aligned with the sample onnx provided in awf/autoware repository

[VeeranjaneyuluToka]

@kminoda , Thanks again for quick reply. Yes, i started investigating that way and found smaller differences in the voxel encoder graphs
The sample onnx voxel encoder graph is as follows

And custom graph that i generated is as follows

I could see there is some difference in input and ouputs from voxel ecoder, but i think i need to change .pth model config to change this, is not it? or is there any other way i can change?

And also here is sample backbone_neck_head graph

and custom backbone_neck_head graph

If i am correct, correcting input_features in voxel encoder should fix the issue. However my original pytroch model is generated for 2 classes but it is not accepting the same two classes here rather it works when i give 1 class.

Also, what is the significance of encoder_in_feature_size? when i pass 9, it is failing while creating voxel encoder engine itself.

And also, could you plz tell me the config that you have used to train pytorch model?

Thanks!

[VeeranjaneyuluToka]

@kminoda , I have changed my pytorch model esp pillarfeaturenet feat-channels, then i am able to generate trt engine successfully. However i am yet to visualize the inference results, will let you know how it goes as soon as i get some results. Thanks for your help.

[VeeranjaneyuluToka]

@kminoda , my trt engine predictions are so bad that they are not matching with pytorch model at all. Do you have any suggestions for me on this behavior? Thanks!

[VeeranjaneyuluToka]

@kminoda , can you please share the config of the pytorch model that you have trained? I could notice that there is some difference in voxel encoder esp, so sharing your pytorch model config would help me to analyse in right direction. Thanks!

[VeeranjaneyuluToka]

@kminoda , i think i got an answer for my question after i had a look into recent submission of your changes in autowarefoundation mmdetection3d esp by looking at lidar_centerpoint readme.md. I started following the same but PillarFeatureNetAutoware forward is not getting called. I am getting the below error when i try with custom config file.

"
File "/mmdetection3d/mmdet3d/models/detectors/mvx_two_stage.py", line 213, in extract_pts_feat
batch_input_metas)
File "/opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py", line 1051, in _call_impl
return forward_call(*input, **kwargs)
File "/opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py", line 201, in _forward_unimplemented
raise NotImplementedError
NotImplementedError
"

I am wondering if i missed anything here?

[VeeranjaneyuluToka]

Again, there was small typo, i could solve the above issue as well! Ignore the above msg.

[VeeranjaneyuluToka]

@kminoda , I have re-trained a model with same config that you are using, converted to onnx and try to run ros2 node but it failed to create backbone-neck-head network with the below error

"
[lidar_centerpoint_node-1] [INFO] [1719320091.351255779] [lidar_centerpoint]: Applying optimizations and building TRT CUDA engine (/home/veeru/autoware/pre-trained/marine_250624/pts_backbone_neck_head_centerpoint_marine.onnx) ...
[lidar_centerpoint_node-1] [E] [TRT] 4: [graphShapeAnalyzer.cpp::processCheck::587] Error Code 4: Internal Error (Concat_40: dimensions not compatible for concatenation)
[lidar_centerpoint_node-1] [E] [TRT] 2: [builder.cpp::buildSerializedNetwork::636] Error Code 2: Internal Error (Assertion engine != nullptr failed. )
[lidar_centerpoint_node-1] [ERROR] [1719320091.367612318] [lidar_centerpoint]: Failed to create serialized network
[lidar_centerpoint_node-1] [ERROR] [1719320091.412796339] [lidar_centerpoint]: Failed to create context: Engine was not created
"

When i analysed the onnx graph, the graph that you created and my graphs are exactly same and attached the concat node. my graph is as below

and you grapsh is as below

Do you have any suggestions why am i getting still the above error? Thanks!

[VeeranjaneyuluToka]

@kminoda , found the reason for this again. Thanks!

[kminoda]

Sorry for not being able to catching up lately. What is your current issue here?

[VeeranjaneyuluToka]

There was some mistmatch in grid_size between what it is calculated (using pcl range and voxel size) and the one that i feed from config. I corrected it, then the recent issue solved and am able to create trt engine. Am good so far, will have to compare the pytorch model results and trt engine results. If they are same, then i close this request. Thanks for your reply.

[VeeranjaneyuluToka]

It works as expected. Thanks for all your help!