Simulation time for message headers

[samiamlabs]

I have a feature request!

I want to do this:
Have a standard way of updating message headers with correct timestamps.

For being able to do it, I wish ROS# had the following feature:
Possibly a RosBridgeClient.time.now() or an improvement of StandardHeaderExtensions that uses a clock synchronized with the /clock topic (published by a ROS node).

Alternatively, ros-sharp should publish its own clock on /clock. I have some concerns about the stability of this though...

I implemented a ClockSubscriber in order to get a laser scan from Unity to work with Cartographer:

It seems to me that it would be good to have this as standard functionality in ros-sharp.
I currently set the roundtrip delay manually, but it would be better to measure it...

Here is a bonus video of the ROS navigation stack and Cartograper working with ros-sharp.

[MartinBischoff]

Hi @samiamlabs !

Very nice videos! Would you like to drop some lines on what you are working on here #20 , togehter with your video links?

I'd like to learn more about what improvement you suggest / what you want to achieve:

• Why is the timing set in StandardHeaderExtensions not correct for your application? From the title of your issue I read that you want to publish the Unity simultion time. To my understanding, the advantage of UnityEngine.Time.realtimeSinceStartup is that the time difference to the clock time published by /clock on the ROS system will remain constant. Is (or is there going to be) real hardware involved in your setup? If so, using Unity simulation time at any point will likely cause problems.
• Would it make sense for your application to remove the header completely from the message and let rosbridge_suite sort things out? See the last paragraph in Section 3.4.3 here.
• Which concerns about stability do you have in terms of publishing a /clock topic from within Unity?
• In the previous Release 1.2c we had a ClockTimePublisher.cs (you find it in \Unity3D\Assets\RosSharp\Scripts\RosCommuncation) that publishes topics at "constant" time intervals, rather than triggered by Unity event functions. Would something like this solve your problem?
• How exactly would you like the synchronization with a ROS /clock topic to take place?

[samiamlabs]

I will probably have some videos that make easier to understand what I'm trying to at the end of this week. I can post them at that time.

I'm referring to ROS simulation time in the title, not Unity simulation time. The concepts are very similar though. You can read about ROS simulation time here: http://wiki.ros.org/Clock. The basic idea is to enable you to speed up or slow down your entire ROS system.

In the case of using a source for /clock in ROS (not Unity Simulation Time) you would need to keep track of the slow-down/speed-up factor and update https://docs.unity3d.com/ScriptReference/Time-timeScale.html accordingly. ROS nodes will use simulation time if the rosparam /use_sim_time parameter is set.

For real robots, wall time should be used so you should synchronize all computers the same time server if possible. I will be putting vive trackers on real robots for localization and visualization at some point. Will probably need wall-time header stamps in message headers then.

The reason I need accurate headers for sensor data is that I make pretty heavy use of tf (http://wiki.ros.org/tf) in order to do things like track objects in the world/map frame while the robot is moving. For this reason, i need accurate timestamps and appropriate frame_id:s. Rosbridge_suit just sets the fame_id filed to "" if I understand correctly.

I don't know how often rosbrigde drops messages when under heavy load (several simulated 3d cameras for example). All the ROS nodes will freeze up (get stuck in sleep functions) if they don't receive times on the /clock topic (and /use_sim_time is set). I also have some concerns about shifting ping between simulation computer and ROS computer causing problems with tf. I would be less concerned about publishing /clock from Unity if ros-sharp was using DDS (https://en.wikipedia.org/wiki/Data_Distribution_Service) instead of websockets. Possibly https://github.com/ros2/ros1_bridge and https://github.com/firesurfer/rclcs could be an option, but I think they are still fairly experimental.

I guess it might be smart to try to publish /clock from Unity in FixedUpdate and determine if it works for my applications before implementing more complicated time synchronization.

As for implementation, here is how they solved it for SIGVerse: https://github.com/PartnerRobotChallengeVirtual/common-unity/blob/master/Assets/SIGVerse/Common/ROSBridge/SIGVerseRosBridgeTimeSynchronizer.cs
But I'm not so sure that is the right way to do it.

I think the best solution I thought of so far is make something like a get_current_unity_time service provider/server in Unity and have a ros node request time at set intervals (measuring roundtrip). The ROS node can then publish the synchronized time on the ROS computer. In that case the StandardHeaderExtension should work as is for simulation time and only need to be updated to support wall time for real robots. It would also make it easy to implement a slider for speeding up and slowing down the simulation in Unity and ROS simultaneously.

[samiamlabs]

Upon further reflection, I think I will implement this on my own fork and make a pull request if I want it on the main master branch. If implemented as I suggested at the end of the last comment, there should not be any breaking changes.

Should I close the issue?

[MartinBischoff]

That's a good @samiamlabs .
You know your requirements best and how to fulfill them.
When done, please do think about whether your new features are useful for ROS# in general.
If so, we all are happy to receive your pull request.

We should refer to this discussion in your pull reqest. Fell free to close it or to keep it open for further discussions in the meantime.

[samiamlabs]

Some updates on this issue:

I wanted a replacement for gazebos planar move plugin that works with the navigation stack and SLAM (TEB and cartographer in my case), so I implemented a general purpose rigidbody based twist subscriber and odom publisher in Unity.

As I expected, both cartographer and the local planner were very sensitive to correct timestamps. Was not able to get it working by publishing /clock directly from unity. The approach I used for the forklift, where i subscribed to /clock from ROS for timestamps did not work either because of complications related to publishing odom from Unity.

I implemented a sync node in ROS and finally got it working:(https://github.com/samiamlabs/dyno/blob/master/dyno_unity/src/clock_sync.cpp)

I used i simplified version the approach I mentioned at the end of my last comment with unity time from UnityEngine.Time.realtimeSinceStartup.
(I aslo noticed that StandardHeaderExtensions is currently putting milliseconds in the nanosecond field btw).

Unfortunately, ROS simulation time can't move backwards without all ROS nodes that use ROS-based sleep() methods throwing exceptions. This means that if you restart Unity you have to wait for simulation time to reach its previous largest value before ROS starts working again. I was enjoying the ability to restart Unity without restarting ROS so I think I will try to find a way to properly sync Unity time with /clock published by ROS instead after all... at some point...

[MartinBischoff]

Thank you again for the impressive videos about your ROS# application and also for your initiative to improve the timing between ROS and Unity!

It is possible to implement an own timer in Unity e.g. by using Physics.Simulate() and Physics.autoSimulation(). This will solve the problem you reported above.
Though this might be a help for some applications, I would not like to implement any own timing concept in ROS#. It would make the framework less flexible and e.g. restrict the use of other assets in combination with ROS#.

How shall we proceed with this issue? Are you planning to work on this topic in the future?
Shall we keep it open such that others find your fork can support you there?

In case you consider your work done (for the time being), I'd suggest to close the issue.

[samiamlabs]

The main problem is that nodes that use TF can be very sensitive when it comes to timestamps. I found that it is pretty difficult to keep ROS and Unity properly synchronized so that all the navigation and localization systems work as they should

I made enough progress so that it is usable for my purposes with my on clock-sync-system, but it is far from perfect.

I have some more advanced simulation scenarios that I am planning to implement, but I think I will attempt to use ROS2 and https://github.com/firesurfer/rclcs for most of that.

My attempt at clock-sync can be found in the links below in case anyone is interested:
https://github.com/samiamlabs/dyno/tree/master/dyno_unity
https://github.com/samiamlabs/ros-sharp/tree/new-warehouse

Closing the issue is fine with me :)

[MartinBischoff]

Thank you for the update and for sharing these infos and your work.
ROS2 is something we are planning to adress too.
The nice thing is that all Websocket communication between Windows and Linux will become redundant, whereas Unity Eidtor capabilities will stay equally useful.