NTP / dnscrypt chicken & egg problem

[rootTHC]

Platform: EdgeOS (ubnt, USG).

Following instructions on https://github.com/DNSCrypt/dnscrypt-proxy/wiki/Installation-on-EdgeOS.

System running in perfect condition. Then this happens:

1. Factory reset is triggers by admin (on purpose).

2. USG goes through adoption after factory reset. The dnscrypt-proxy json file from the cloud key is automatically installed to the USG. This sets the local resolver.conf to 127.0.0.1. This is an automated process.

3. Now following above instructions to install the dnscrypt-proxy again will fail because the ntpd could not resolve any hostname to set the correct time. Incorrect time means that dnscrypt-proxy will not resolve any hostname and thus ntpd will never be able to set the correct date - which dnscrypt-proxy needs to resolve the domain for the ntp servers.

quick workaround after adoption but before dnscrypt-proxy installation:
/etc/init.d/ntp stop
ntpdate -b 143.210.16.201
/etc/inid.d/ntp start

real workaround might be that local resolver is not set to 127.0.0.1 so that USG can update its own date/time via ntpdate.

[jedisct1]

Maybe you can simply include some IP addresses in the NTP server configuration?

If this is really hopeless, dnscrypt-proxy has an option to ignore timestamps in certificate checks.

[welwood08]

For what it's worth, I run dnscrypt-proxy on an EdgeRouter device and am therefore personally invested in making the experience on EdgeOS smooth. Since there is no hardware clock, it must rely almost entirely on NTP to set the correct time during every boot. I'm not familiar with anything on the UniFi side of things, but if their hardware is similar then it's not just factory reset to worry about.

In step 2 you refer to "The dnscrypt-proxy json file" but I'm not sure what that is other than the full EdgeOS config.boot file? It may be possible to have the controller's adoption process push more than just config.boot, in which case you could have a script that makes sure NTP is working and then automates the installation of dnscrypt-proxy but this is really a question for the UBNT forums rather than here.

@jedisct1's suggestion to ignore timestamps in certificate checks is something I hadn't considered, but in this case it would have no effect since dnscrypt-proxy is not yet even installed. Using IP addresses for NTP would surely work, but be aware that the default configuration on EdgeOS is to use a subdomain of pool.ntp.org so using those specific IP addresses is brittle as servers can rotate out of the pool.

Unfortunately I don't have much more advice to share since I haven't properly tested my own solution to this chicken/egg problem. Suffice to say I was uncomfortable with the relaxed security and poor integration of the built-in service installation mechanism and instead make my own .deb package so I don't have to reinstall manually after every firmware update. This method could potentially survive factory reset too. If testing works out and there's any interest, I may tidy it up and start releasing these EdgeOS-specific packages on my fork or something.

[rootTHC]

Maybe you can simply include some IP addresses in the NTP server configuration?

If this is really hopeless, dnscrypt-proxy has an option to ignore timestamps in certificate checks.

I'm using the USG from Ubiquiti. On factory reset and adoption the USG pulls a static configuration file from the Cloud Key. That configuration file forces /etc/resolv.conf to 127.0.0.1.

Any NTP server config wont survive 'factory reset'. Factory reset flushes the entire HD and pulls the resolv.conf from the central configuration server. Dnscrypt-proxy has then to be installed manually again. That installation fails because resolv.conf is set to 127.0.0.1 and ntpd fails forever because dnsproxy is not working (bad timestamp).

Surprisingly my time was only behind (slow) by 11h and it did not work.

It should be mentioned EdgeOS instructions that timestamp must be set correctly manually before installation even if ntpd is running (because ntpd will fail without resolver - which only happens after factory reset).

[jedisct1]

Looks like this is not going to help if the NTP configuration is reset, but for what it's worth, there are some pretty stable IP addresses that can be used as NTP servers. Notably 216.239.35.0, 216.239.35.4, 216.239.35.8 and 216.239.35.12 that are Google's NTP servers, and these addresses haven't changed ever.

[jdrch]

timestamp must be set correctly manually before installation

Can confirm from previous experience with dnscrypt-proxy on a Pi 3B+. BTW, this isn't unique to EdgeOS or Ubiquiti: basically any device lacking an RTC is susceptible to this issue.

I got an RTC for the Pi after that episode, but I wound up moving dnscrypt-proxy to a proper x86-64 Debian Stable machine and it's been smooth sailing since.

[pradorocchi]

About running on such device: Better is to install a simple NTP script to run 100% of the time, in background on the device, checking every X minutes.

Because internet link may still be connecting when the device starts up, thus the ntpdate initial command will fail, but using a script to check every X minutes deals well with the problem, and as soon as the internet is available the device updates the time.

Also, the domain name of the timeservers (if using domain name) have to bypass the localserver and use any common external DNS server, this can easily be done and you can have a bypass name group like time.nist.gov or time.windows.com that are resolvable via external DNS (eg: 1.1.1.1), (or you can use their IP address directly as mentioned above to adjust NTP).

[rootTHC]

dnsmasq had the same problem and they found an 'elegant' solution (perhaps worth adopting). From the man page:

--dnssec-no-timecheck
DNSSEC signatures are only valid for specified time windows, and should be rejected outside those windows. This generates an interesting chicken-and-egg problem for machines which don't have a hardware real time clock. For these machines to determine the correct time typically requires use of NTP and therefore DNS, but validating DNS requires that the correct time is already known. Setting this flag removes the time-window checks (but not other DNSSEC validation.) only until the dnsmasq process receives SIGINT. The intention is that dnsmasq should be started with this flag when the platform determines that reliable time is not currently available. As soon as reliable time is established, a SIGINT should be sent to dnsmasq, which enables time checking, and purges the cache of DNS records which have not been thoroughly checked.
Earlier versions of dnsmasq overloaded SIGHUP (which re-reads much configuration) to also enable time validation.

If dnsmasq is run in debug mode (--no-daemon flag) then SIGINT retains its usual meaning of terminating the dnsmasq process.

[jedisct1]

dnscrypt-proxy has a cert_ignore_timestamp option to ignore DNSCrypt certificate timestamps until at least one resolver successfully returned a response, but having working NTP is way better and safer. If only to get log files with correct time stamps.