Understanding and improving IK speed

[peter-mitrano-bg]

I'm trying to get a deeper under understanding of the IK solver and how to make it as fast as possible for my use cases, namely workspace analysis and iteratively solving collision-free IK to nearby poses, all with a dual arm robot with many spheres.
I'm still in the exploration phase, but I have some specific questions:

1. When I solve Ik it seems to run MPPI and L-BFGS, is that necessary? Could it be faster just to use one or the other? How much performance can I squeeze out by tuning the many many parameters involves? Specifically, running MPPI just to solve IK is a bit confusing to me, since usually that is used for whole trajectories.
2. What does "synchronize" in cuda do, and when should it be done? I can see the sync_cuda_time flag exists and seems to be true by default in several places, which feels wrong but honestly I'm not sure. I did a profile run and it seems to be dominating IK solve time?
3. I've noticed that my robot is ~8x to solve than even the quad ur5e. It has many spheres (~150), which explains part of the slow down, but I tried just repeating spheres on the ur5e to match and it was still way faster. I can't share all my code, but are there any likely culprits? Is it number of links, joints, spheres, .... ? Does the self-collision configuration effect performance?
4. I've also noticed that solve time scales ~linearly with batch after batch size exceeds only 10 or 20. This surprised me, since I thought all the solving was happening in parallel, and so doing 1000 problems shouldn't take that much longer than doing 100 problems, and it certainly shouldn't take 10x longer. The paper also shows this, but I'm just confused as to why it doesn't scale better.

Thanks in advance, sorry this is long and lacking some details.

[balakumar-s]

MPPI + L-BFGS

MPPI takes around 10% of the IK time as it's a highly parallel solver and is only run for 4 iterations. MPPI is similar to CMA-ES and helps in quickly moving the initial seeds to regions of local minima. L-BFGS then converges to sub-mm accuracy in these regions of local-minima.

CUDA Synchronization

Measuring compute time without synchronizing cuda ops is wrong as the measured time will not wait for cuda ops to complete. This is similar to joining threads in multi-threaded programs. Profile using pytorch's profiler, an example is here:

curobo/benchmark/curobo_profile.py

Line 146 in 1e0b5a8

with profile(activities=[ProfilerActivity.CPU, ProfilerActivity.CUDA]) as prof:

Solve Time

What's your system specs? We benchmark on RTX 6000 Ada and also RTX 4090.

In addition when cuda_graph is enabled, cuRobo's ik_solver does not support changing the batch size. Do you have cuda_graph enabled?

We have provided a script that will sample random joint configurations and do IK at different batch sizes. This could be used to tune cost terms as well.

Add your robot configuration file path to benchmark for your robot:

curobo/benchmark/ik_benchmark.py

Line 128 in 1e0b5a8

robot_list = get_motion_gen_robot_list() + get_multi_arm_robot_list()[:2]

[peter-mitrano-bg]

Thanks! I've already done a bit of benchmarking, currently on a RTX 2000 Ada, but I plan to do more on a 4090. I do have cuda_graph enabled, and yes turning it off makes things way slower, as expected.

What about these lines at the top of some scripts? From my testing on 2000 Ada they have no real effect on performance. Are there cases where I should/shouldn't use them? The first one I'd probably not use, but the second and third seem interesting.

torch.backends.cudnn.benchmark = True
torch.backends.cuda.matmul.allow_tf32 = True
torch.backends.cudnn.allow_tf32 = True

[balakumar-s]

I also forgot to mention, before measuring compute time run a few ik solve calls so that all tensors have been initialized and cuda graphs have been captured.

An example snippet

for _ in range(5):
    result = ik_solver.solve_batch()

sample_poses  = // your poses
torch.cuda.synchronize()

st_time = time.time()
result = ik_solver.solve_batch(sample_poses)
torch.cuda.synchronize()
solve_time = time.time() - st_time

In regards to torch flags, having these True will save 1%-5% on solve time. We have not found any use cases where we need to set them to False.

[balakumar-s]

@peter-mitrano-bg Checking again if your IK time is better or still 8x slower.

[peter-mitrano-bg]

I've moved from a laptop GPU to a desktop GPU, and it's fast enough for me at the moment. But I never did resolve why it was so much slower than a ur5e with comparable number of spheres. I may revisit this later but for now it's not an issue.

[tahsinkose]

Hey @balakumar-s, I'm also trying to reproduce the cuRobo-Cfree results on my local computer that has a RTX 3050. I specify num_seeds of IKSolver to be 30. The compute times I get (with franka.yml) using benchmark/ik_benchmark.py are as follow:

Batch-Size:
- 1
- 10
- 100
- 500
Collision-Free-IK-time(ms):
- 1994.2009449005127
- 251.005744934082
- 102.86066055297852
- 87.71210765838623

Collision-Free-IK-time(ms) shows the per pose compute time. In cuRobo report, it is stated that 100 poses are solved in 14.7ms (Figure 17-Plot 1), which means a compute time of 0.147ms per pose. I don't quite get almost ~700x (102.8ms / 0.147ms) slowness that I observe locally 🤔 Could you please elaborate?

[balakumar-s]

Can you confirm that you didn't modify anything in the script? E.g., did you change the world model?

Here is what I get on a 4090:

Batch-Size:
- 1
- 10
- 100
- 500
- 2000
Collision-Free-IK-time(ms):
- 5.855321884155273
- 0.6350517272949219
- 0.12480735778808594
- 0.10863113403320312
- 0.10627758502960205
IK-time(ms):
- 4.034996032714844
- 0.43327808380126953
- 0.056955814361572266
- 0.025595664978027344
- 0.02368748188018799

[tahsinkose]

Nope, I didn't change the world model, it is collision_test.yml

[tahsinkose]

OK, I found the problem. I had enabled -g -G flags to debug some of the CUDA kernels and left them as such. That's why I observed this slowdown. I now tested on a separate workstation with RTX 4090 and the timing I got with num_seeds=16 is:

Batch-Size:
- 1
- 10
- 100
- 500
- 2000
Collision-Free-IK-time(ms):
- 5.861520767211914
- 0.6744861602783203
- 0.12660503387451172
- 0.10953521728515625
- 0.10719847679138184

Thank you very much for the help @balakumar-s 🙏🏼

[balakumar-s]

To help future users, can you also share the IK numbers you get on RTX 3050 without the debug flags.

Glad that you could figure it out.

[tahsinkose]

Since I only have 4GB of GPU VRAM, I could provide up to a batch size of 500. Following are the results from RTX3050 without debug flags. When num_seeds=12:

Batch-Size:
- 1
- 10
- 100
- 500
Collision-Free-IK-time(ms):
- 7.568359375
- 1.3899803161621094
- 0.6724166870117188
- 0.6260318756103516

When num_seeds=30:

Batch-Size:
- 1
- 10
- 100
- 500
Collision-Free-IK-time(ms):
- 8.116483688354492
- 2.222919464111328
- 1.6072845458984375
- 1.550224781036377

[frederiknolte]

I am also seeing that solving time scales roughly linearly and I'm sorry if I misunderstand, but the numbers you are getting for the benchmark also seem to show that after a certain threshold (between 100 and 500 batch size), the total time to solve scales linearly with batch size?

[peter-mitrano-bg]

Yes, I think that's what everyone's testing shows. Linear after 50-100 batch size. Feels like it should scale much better than that for much higher batch sizes...

[frederiknolte]

It is odd though that the GIF here suggests that IK could be solved in less than a second even with a batch of 500 or so poses. @balakumar-s may I ask if this GIF shows the real-time performance?