Why do we prefer CLOCK_MONOTONIC over CLOCK_REALTIME

[erlingrj]

According to the manpages (man clock_gettime):

CLOCK_MONOTONIC: Counts nanoseconds since boot and is not affected by discontinuous jumps in system time. But it does not count time when the system is suspended (what does that mean?)

CLOCK_REALTIME: Gives the system POSIX time since 1970. Affected by discontinuous jumps.

While discontinuous jumps could be disadvantageous, I do believe that the linuxPTP tools use CLOCK_REALTIME. (I will ask Henrik about this). Also, seems dangerous that the LF clock should remain unchanged during system suspension.

[henrikau]

As the name suggests, CLOCK_MONOTONIC is a monotonically increasing clock source. It will never make large jumps but instead adjust the phase to adjust the time as the local clock drifts. CLOCK_MONOTONIC_RAW is the more extreme example of this, which will not even adjust the phase but instead provide a semblance of a clock by scaling the value of a CPU counter.

CLOCK_REALTIME is kept as close to the real time in your clock domain as possible. This means that if you have 2 systems, the only clock that makes sense for the two is CLOCK_REALTIME, you should not compare CLOCK_MONOTONIC between two machines!

However, when you start setting timeouts or timers that need to fire very closely to your target, you must be a bit careful with CLOCK_REALTIME as this clock is free to move backward in time. In netchan we used CLOCK_MONOTONIC for this reason and read the PTP Time directly from the NIC. However, this opens you up to a lot of extra complexity. At the same time, if you have microsecond accuracy as a target, you should disable frequency scaling and avoid moving between c-states (see https://github.com/henrikau/net_chan/blob/main/src/netchan.c#L1012 for an example)

If you have a PTP synchronized domain and the clocks are reasonably stable, using CLOCK_REALTIME will save you a lot of complexity (which translates into fewer sources of errors) at the cost of suddenly overshooting a timer when the base is yanked backward from under you.

[erlingrj]

Thanks for this input Henrik. Very insightful.

• netchan uses CLOCK_MONOTONIC to configure very accurate timers to wakeup a thread after some duration, but it reads out the PTP clock to calculate the duration. You don't use phc2sys at all?
• Currently, in LF, at startup CLOCK_REALTIME and CLOCK_MONOTONIC is both read and the offset is computed and applied to all readouts of CLOCK_MONOTONIC. In this was we can compare timestamps across different systems.
• In the C-runtime of LF we are using the pthread_cond_timedwait to sleep our system until the next event (or until an asynchronous event is scheduled). This uses CLOCK_REALTIME, as far as I understand, so I am not sure whether using CLOCK_MONOTONIC for deciding what time it is helps us that much, worse, if there are discontinous jumps, then our clock used for sleeping is out-of-sync with our clock used to get current time.
• However, in phc2sys, do I understand it correctly that it works on CLOCK_REALTIME only? Or is it actually configurable and CLOCK_REALTIME is default? Also how do we disable frequency scaling (is this a good approach). ALso how do we avoid moving between C-states? I didn't totally understand from your code example.

[erlingrj]

I am curios to hear what @edwardalee has to say here since the choice of CLOCK_MONOTONIC seems deliberate

[edwardalee]

The choice of clock in LF is determined by the _LF_CLOCK macro, which is defined in the header files for Linux, Mac, and nrf52 to use CLOCK_MONOTONIC. Oddly, the Windows support header file says this:

// FIXME: Windows does not #define _LF_CLOCK

Regardless of what clock you choose, LF uses two safeguards:

1. The initialization code sets _lf_time_epoch_offset to the difference between what CLOCK_REALTIME and whatever clock you choose report. This offset is then added to all reported times. Hence, you should always get Epoch time if CLOCK_REALTIME reports epoch time.
2. The functions that report physical time defined in tag.h and tag.c always report monotonically increasing times. If the underlying clock is not monotonic, then these functions add 1 ns to the last reported time and report that.

If indeed pthread_cond_timedwait uses CLOCK_REALTIME, then have an inconsistency. Is there a way to change this in the POSIX API?

[erlingrj]

It should be possible to set which underlying clock should be used by pthread: https://linux.die.net/man/3/pthread_condattr_setclock

[henrikau]

* netchan uses CLOCK_MONOTONIC to configure very accurate timers to wakeup a thread after some duration, but it reads out the PTP clock to calculate the duration. You don't use `phc2sys` at all?

Correct. The reason was that I had some plans of using netchan to bridge multiple networks where each network did not necessarily live in the same clock domain. This was a mistake and I am currently moving net_chan away from this approach and towards expecting phc2sys to sync the clocks.

* Currently, in LF, at startup CLOCK_REALTIME and CLOCK_MONOTONIC is both read and the offset is computed and applied to all readouts of CLOCK_MONOTONIC. In this was we can compare timestamps across different systems.

Be aware that if you do this at startup, the offset is not necessarily constant. CLOCK_MONOTONIC will have the phase adjusted so if the clocks become better synchronized after a while, then the original offset may not be accurate any more.

* In the C-runtime of LF we are using the `pthread_cond_timedwait` to sleep our system until the next event (or until an asynchronous event is scheduled). This uses CLOCK_REALTIME, as far as I understand, so I am not sure whether using CLOCK_MONOTONIC for deciding what time it is helps us that much, worse, if there are discontinous jumps, then our clock used for sleeping is out-of-sync with our clock used to get current time.

* However, in `phc2sys`, do I understand it correctly that it works on CLOCK_REALTIME only? Or is it actually configurable and CLOCK_REALTIME is default? Also how do we disable frequency scaling (is [this](https://askubuntu.com/questions/523640/how-i-can-disable-cpu-frequency-scaling-and-set-the-system-to-performance) a good approach). ALso how do we avoid moving between C-states? I didn't totally understand from your code example.

Yes, It does not make sense for phc2sys to adjust CLOCK_MONOTONIC.

By writing 1 to /dev/cpu_dma_latency and keeping the file open (i.e. do NOT close it), cstate transitions are disabled. You can pass a flag to net_chan to disable this, wakeup accuracy then goes from 18us to above 500us.

[erlingrj]

After a discussion with Henrik yesterday I now understand the clock and timer subsystem of Linux a lot better. I have made the following observation:

1. On a non-federated LF program, using CLOCK_MONOTONIC can make sense. We are mainly interested in the local relative time. The frequency of CLOCK_MONOTONIC is affected by NTP/PTP but it doesnt receive any hard resets. This means that CLOCK_MONOTONIC (and thus our LF time) can get out of sync with CLOCK_REALTIME (and probably UTC). But in a non-federated program, is that a big deal, what is important is that the frequency of our underlying clock is correct, not the phase. CLOCK_MONOTONIC+NTP/PTP makes sure we have an underlying clock with correct frequency (no guarantee on phase).

2. On a fedearated LF program, CLOCK_REALTIME or CLOCK_MONOTONIC_RAW is the better option. Because
   2a) If we use external (NTP/PTP) clock sync, this will only work properly on CLOCK_REALTIME since actually hard jumps on the clock are only applied on that one.
   2b) If we are running our own clock sync, then CLOCK_MONOTONIC_RAW seems better since this is totally unaffected by any NTP/PTP service. It is hard to run a clock sync algorithm ontop of CLOCK_MONTONIC (or CLOCK_REALTIME) when the frequency of that clock is being adjusted by NTP/PTP.

[edwardalee]

This makes sense. For all of these clocks, our physical time will still be monotonic because this is enforced in lf_time_physical().

[lhstrh]

Should we then update our code generator to pick CLOCK_MONOTONIC, CLOCK_MONOTONIC_RAW, or CLOCK_REALTIME accordingly?

[henrikau]

If you have safeguards in place for detecting if the clock jumps backward, I would consider CLOCK_REALTIME purely because it tracks global time.
Note that CLOCK_MONOTONIC_RAW, although it is named clock, is not really a clock, but rather a scaled counter such that it looks like something that could be a clock.

If all you need is a monotonically increasing value, wouldn't the CPU counter be just as effective? And would be transferrable to other platforms than x86 as well

[erlingrj]

I think we should have a testbed for distributed LF programs. This way we can actually experiment with different underlying CLOCKS and how it interacts with NTP and PTP or hard clock resets from the user.

I think we should add a PR with the following changes:

• _LF_CLOCK is adjustable for the user to change with a preprocessor command (reactor-c)
• In unfederated programs. CLOCK_MONOTONIC or CLOCK_REALTIME is default. Need to think a bit more to make up my mind here. (lingua-franca)
• In federated programs with clock-sync. CLOCK_MONOTONIC can still be default (lingua-franca)
• In federated programs without clock-sync. CLOCK_REALTIME is default (lingua-franca)
• Always configure underlying platform to use _LF_CLOCK for lf_cond_timedwait. Currently it always uses CLOCK_REALTIME. (reactor-c)

[petervdonovan]

Is the mechanism by which we enforce monotonicity thread-safe? To me it looks like it isn't because it uses the global variable _lf_last_reported_unadjusted_physical_time_ns, but I could be missing something. I'm sure we could add (yet another) global mutex and some preprocessor hacking to get _lf_physical_time to be thread-safe, but if we have the option of just sticking with a monotonic underlying clock, then maybe we wouldn't have to worry about that.

[erlingrj]

That is a good point, Peter. I think you are right. By the way, what would be the consequence of a non-monotonic underlying clock? Might our runtime crash because of some assertions on the current physical time and time of the last handled event? I am asking to understand what could happen if we remove this monotonic-guaranteeing logic.

I think the best for us is to let NTP and PTP keep our physical clocks in sync. But they might step the clock forward or backward. It is possible to configure NTP/PTP so that they don't step the clock, or only step the clock when booting the clock sync client. However, it will take these clients a long time to correct even small clock offsets, so stepping the clock might be what you want in many cases.

In other words. We shouldn't rely on the underlying physical clock always being monotonic unless we can guarantee that NTP/PTP are configured in a certain way.

[erlingrj]

I have been thinking more about this problem and my conclusion now is that we should just stick with CLOCK_REALTIME and forget about CLOCK_MONOTONIC. This means that we need to ensure monotonicity on top of the clock. Also, we must make it thread-safe as Peter observes above.

My logic is as follows: Since we want to support using PTP and NTP to synchronize federates, we need to support CLOCK_REALTIME, so we will need the monotonicity enforcing anyway.

The main argument for CLOCK_MONOTONIC is if we have federates with clock-sync enabled then we don't want NTP/PTP to mess up our clock sync. But is it a realistic scenario that we have NTP/PTP enabled and still want LF clock sync? I think it is reasonable to require either:
A) The system provides sufficient clock sync through NTP/PTP
B) The system does not adjust the clock.

If this is an unreasonable requirement for the system on which our federates are running. Then we must consider CLOCK_MONOTONIC.

The advantages of removing CLOCK_MONOTONIC are the following:

• We can read time directly from _lf_clock_now without having to adjust for the epoch offset between LF_CLOCK and CLOCK_REALTIME
• pthread_cond_timedwait on macOS only works relative CLOCK_REALTIME. This means that if LF is using CLOCK_MONOTONIC then we cannot use lf_cond_timedwait without translating the timeout into a time_point which is relative CLOCK_REALTIME. If that clock is being adjusted by NTP/PTP then we lose wakeup accuracy relative CLOCK_MONOTONIC. (On Linux it is possible to du pthread_cond_timedwait relative CLOCK_MONOTONIC).

I think that moving to just using CLOCK_REALTIME removes some complexity and puts a little more responsibility at the user. Which I think is OK.

What do you think?

[edwardalee]

I agree. Let's go with CLOCK_REALTIME. I think we already ensure monotonicity...