Questions about autoware.universe end-to-end performance issues.

[nainaigetuide]

Hello.

• I'm doing work on profiling autoware.universe. At present, I have encountered the problem of high end-to-end delay, and I would like to ask for advice.

  The machine configuration is as follows:
  Scenario: https://autowarefoundation.github.io/autoware-documentation/main/tutorials/ad-hoc-simulation/rosbag-replay-simulation/
  Environment
  OS: Ubuntu 20.04, 5.15.0-52-generic
  ROS: ROS2 Galactic
  CPU: Intel(R) Xeon(R) Silver 4114 CPU @ 2.20GHz 40 Core
  GPU: Tesla T4
  Memory: 64GByte
  Storage: 180GByte

• When using the Caret tool to analyze the delay, it is found that the delay is much higher than 100ms, and the bottleneck is not in the calculation of a certain ROS node. Every part seems to have a relatively high delay, such as attachments.

test.pdf

• When running autoware, the average CPU utilization rate is 40%, the highest load is 20, and the highest IO TPS is 190. These indicators do not show the bottleneck of the program.

• I have the following questions:
  Do you know the bottleneck that restricts the high latency of the system?

[takeshi-iwanari]

Please refer to the following comment.
tier4/caret_analyze#203 (comment)