Custom AI Agent LLM

[aaron4522]

NASA released ROSA, which allows LLMs to interact with ROS2 by executing commands, modifying code, read logs and help with code creation or debugging. Also we can get personalized suggestions by giving the LLM the context of our project and the code we've already written.

Benefits are that it can help us be more productive by resolving roadblocks faster.

Tasks

• Follow ROSA's README instructions
• Create Azure resource to take LLM queries (we get $100 of free credits)
• Test demo code
• Integrate with our sailbot repo

Its new, so documentation isn't great. Currently stuck at trying to test demo code:

• X11 forwarding can be done with sudo apt install x11-xserver-utils.
• Environment variables can be added with export.
• Currently stuck due to exceeding Azure's 1k token/min limit on a single prompt due to ROSA's multi-hop API calls. Reducing the amount of text in prompts.py is insufficient. Request to increase Azure rate limit was denied due to a lack of availability on eastaustralian servers. Switching from 4o to 3.5 turbo may be allowed to increase the rate limit. Custom responses outside of ROSA work otherwise.

There's also pre-made custom agents already targeting ROS. Difference is that responses won't be personalized and it can't interact directly with ROS.