Switch RTC clock to external 32kz xtal

[errolt]

Hardware:

Board: ESP-WROVER-KIT
Core Installation/update date: 2018-03-10
IDE name: Arduino IDE
Flash Frequency: 40Mhz
Upload Speed: 921600

Description:

Is there any way, or plans for a way, to enable the slow xtal osc for the RTC? The current way the RTC is set up seems to keep time during deep sleep, but drift seems extreme at times.

I have tried to change the RTC clock using:
rtc_clk_32k_bootstrap();
rtc_clk_32k_enable(true);
rtc_clk_slow_freq_set(RTC_SLOW_FREQ_32K_XTAL);
But that did not enable the osc, and it prevents deepsleep from ever waking up.

Sketch:

#include <WiFi.h>
#include "time.h"
#include <rom/rtc.h>

const char* ssid       = "abc";
const char* password   = "xyz";

const char* ntpServer = "pool.ntp.org";
const long  gmtOffset_sec = 3600;
const int   daylightOffset_sec = 3600;

void printLocalTime()
{
  struct tm timeinfo;
  if(!getLocalTime(&timeinfo)){
    Serial.println("Failed to obtain time");
    return;
  }
  Serial.println(&timeinfo, "%A, %B %d %Y %H:%M:%S");
}

void setup()
{
  Serial.begin(115200);
  
  //init and get the time
  if(rtc_get_reset_reason(0)!=DEEPSLEEP_RESET)
  {
      //connect to WiFi
      Serial.printf("Connecting to %s ", ssid);
      WiFi.begin(ssid, password);
      while (WiFi.status() != WL_CONNECTED) {
          delay(500);
          Serial.print(".");
      }
      Serial.println(" CONNECTED");

      Serial.println("Getting Time.");
      configTime(gmtOffset_sec, daylightOffset_sec, ntpServer);

      printLocalTime();

      //disconnect WiFi as it's no longer needed
      WiFi.disconnect(true);
      WiFi.mode(WIFI_OFF);
  }

  configTime(gmtOffset_sec, daylightOffset_sec, ntpServer);
  printLocalTime();

  esp_sleep_pd_config(ESP_PD_DOMAIN_RTC_PERIPH, ESP_PD_OPTION_ON);
  Serial.println("Going to sleep");
  esp_sleep_enable_timer_wakeup(5000000);

  esp_deep_sleep_start();

}

void loop()
{
}

[lbernstone]

I think this discussion is probably more appropriate for esp-idf or the espressif bbs. That said, I think you would need to build your own module to implement the XTAL32K, since those pins are generally not exposed, and you would want the crystal close to the chip or else you will end up with the drift you are trying to get rid of. Maybe you can find a manufacturer that is looking to hit the low power market and convince them to make such a module. An easier/cheaper technique might be to do a short sleep to calibrate the slow_clk and then use a scaling factor from then on.

[errolt]

Those pins are brought out on the ERP-WROVER module, possibly others, as IO32 & IO33.
It would not help to to ask in the esp-idf forum or bbs as this question is specifically arduino core related. This is not an issue, and it is possible, in esp-idf, but is not catered for in the adruino core.

[kriswiner]

I have an external xtal on all of my ESP32 dev boards here and here I would like to be able to use it for the RTC as well. I'd like to see a working example of this please.

Does the above work? if not, I hope we can figure out how to fix it.

Can you add some comments to the above sketch to make it a bit clearer for the novice what is happening?

[errolt]

The code is just a combination of the "Simple Time" esp32 example and the "Deep Sleep" example. The sketch as posted will get the current time from the internet after a hard rest/power on. It will then enter deep sleep for 5 seconds, wake without getting the time from the internet again, print the time as stored and go to sleep. This does not use the external rtc xtal. I have also found that it does not use the internal rtc to keep time.

Instead in the background it knows that the sketch will sleep 5 seconds, so after waking up it adds 5 seconds to the last known time, and assumes that sleep, sleeping, and waking was exactly predictable.

I have been able to modify the sketch with the three lines of code noted in my issue. This will power up the external crystal. It also switches the hardware rtc to the 32KHz xtal. The result is that the adruino sleep function expects the rtc to be clocked at 150KHz, not 32KHz, thus a 5 second sleep becomes at 25 second sleep. And because the timekeeping in the arduino core does not use the rtc hardware and assumes that the sleep took 5 seconds, the reported time advances only 5 seconds during every 25 second sleep.

A lot of the mods required are in the precompiled libs, and I can't figure out how to rebuild the arduino-esp32 project...

[lbernstone]

How to rebuild core libs: #1142

[errolt]

Looks like the only way do enable the external 32KHz crystal is, and will always be, to compile a custom libesp32.a where the RTC options was modified by "make menuconfig". Relevant settings:
Component config->ESP32-specific->"Timers used for gettimeofday" and "RTC clock source"

The clock settings are hardcoded into libesp32.a, and although they can be configured by code after the fact, the changes are lost after a deepsleep wake when the hardcoded values are loadef again. This will cause glitches as the RTC clock is switched back and forth.

I wish the IDF code would leave some things alone, like the RTC, if the chip is waking from deep sleep. It already leaves the RTC ram alone, but not the clock settings...

Oh, by the way, thank you Ibernstone for your assistance in this.

[rin67630]

I have the same issue.
Light sleep is practically unusable if you need to keep time, the drift is awful.
The built-in RTC do not deserve the name without a decent xtal at 32.768 Khz.
So there is no way to run at low power keeping time, without additional hardware RTC.
:-(
I also miss the possiblity to run at 2MHz, which could have been a workaround for many of us.

[Matheus-Garbelini]

Got it working somehow by repeating the bootstrap block. Even compiling the libesp32.a again with rtc support, the sdk does the bootstrap one time during bootloader_clock.c and never again. Also the bootstrap cycle I've used was 512, so 2 calls of 512 did the job. Even modifying the SDK source code to force call twice the bootstrap procedure was not working, so I'm glad that adding the following lines in void setup did the trick. Maybe @igrr could help to explain why this happens, as he did a unity test for the rtc clock calibration.

I have tested the code with the TimerWakeUp example and it works perfectly fine:
https://github.com/espressif/arduino-esp32/blob/master/libraries/ESP32/examples/DeepSleep/TimerWakeUp/TimerWakeUp.ino

Here's the piece of code I've added (With external rtc clock enabled in esp32.a):

        rtc_clk_32k_bootstrap(512);
        rtc_clk_32k_bootstrap(512);
        rtc_clk_32k_enable(true);

        uint32_t cal_32k = CALIBRATE_ONE(RTC_CAL_32K_XTAL);
        rtc_clk_slow_freq_set(RTC_SLOW_FREQ_32K_XTAL);

        if (cal_32k == 0)
        {
            printf("32K XTAL OSC has not started up");
        }
        else
        {
            printf("done\n");
        }

        if (rtc_clk_32k_enabled())
        {
            Serial.println("OSC Enabled");
        }

I've used the exposed calibration function that @igrr did:

#include "soc/rtc.h"

#define CALIBRATE_ONE(cali_clk) calibrate_one(cali_clk, #cali_clk)

static uint32_t calibrate_one(rtc_cal_sel_t cal_clk, const char *name)
{

    const uint32_t cal_count = 1000;
    const float factor = (1 << 19) * 1000.0f;
    uint32_t cali_val;
    printf("%s:\n", name);
    for (int i = 0; i < 5; ++i)
    {
        printf("calibrate (%d): ", i);
        cali_val = rtc_clk_cal(cal_clk, cal_count);
        printf("%.3f kHz\n", factor / (float)cali_val);
    }
    return cali_val;
}

Please someone fix this, we're not buying external RTC. Maybe someone can patch the SDK with this same calibration functino that igrr did, because the one in cpu_clock.c is not working the way it's supposed to, or the crytal is bootstrapping too slow.

Notes:

• This was tested in my own board that I've developed, it contains the Wrover kit with an external 32.768Khz crystals on pins 32 and 33.
• rtc_clk_32k_enabled return false even tough the clock was enabled. During my sdk modifications I may have got it to return true but in the end it didn't matter as the deepsleep was working even if this function returned false.
• If I call only once and with double the cycle I've used like: rtc_clk_32k_bootstrap(1024) or more, it doens't work ether and my output becames:

Here's the result of my ouput by using this calibration function after adding two bootstrap calls:

As you can see the reset reason worked this way. The first output the esp32 is never waken up, so clearly after the calibration, the clock was bootstraped.

• Please if someone need my idf-sdk and toolchain to test this, just ask-me and I'll zip everything and provide you with the code for testing. I really don't like to make people compile everything again and without knowing exactly which sdkoptions I've used. I'm saying that because the sdkoptions that arduino provided is not enough, If you try to replicate and just copy the libesp32.a to your arduino installation it works (sometimes you need to copy the libfreertos also), but if you copy all the generated libs as it is supposed to be, it gives linker errors, this tells that arduino the repository is not being totally transparent how thet binaries are generated in order to help developers. Every change in the core should NOT obligate people to know everything about computer enginnering to get it working, this is just making things harder. That's is just my opinion, so don't take it seriously :-)

[Matheus-Garbelini]

https://github.com/Matheus-Garbelini/Esp32-Pre-Compiled-SDK-Toolchain

[nzagorec]

I compiled a custom libesp32.a with 32.768 kHz XTAL enabled in menuconfig but the ESP still reports "clk: RTC: Not found External 32 kHz XTAL. Switching to Internal 150 kHz RC chain" after power on. If I flash it with IDF it works fine (I don't get that message). What else do I have to do to get it working? I'm using ESP-WROOM-32.

[Matheus-Garbelini]

@nzagorec It's common to happen right now with the current IDF SDK. There's some sort of undocumented bug in the underling external crystal bootstrap code that maybe only @igrr would know what's the reason (he did the code for crystal calibration). Please, see my answer from before on this issue (#1225 (comment)). The boot will always print this error for now so you should add this code manually.

I've seen on my email that you already tested this. Yes, the module should detect the RTC but that's a SDK bug right now. Maybe you could do some fallback function to call CALIBRATE_ONE(RTC_CAL_32K_XTAL);. If the external crystal fails you can try again with other argument other than the RTC_CAL_32K_XTAL so you can use the internal crystal.

Until now no one seemed to care about this hehe, but it's a big deal if we're talking about using deep sleep with precise sleeping times (low drift).

Cheers

[nzagorec]

@Matheus-Garbelini Thanks! So this manual calibration in setup loop should be done every time ESP wakes up from deep sleep? Or only the first time setup is executed?

@errolt mentioned earlier that executing it on every wakeup will cause glitches because RTC clock is switched back and forth. Is this the case only if XTAL is not enabled in libesp32.a or here as well?

The clock settings are hardcoded into libesp32.a, and although they can be configured by code after the fact, the changes are lost after a deepsleep wake when the hardcoded values are loadef again. This will cause glitches as the RTC clock is switched back and forth.

I agree, if you're making a very low power consumption device that has to send messages every 24h or so it's extremely important to have precise clock that doesn't drift with time.

[Matheus-Garbelini]

@nzagorec I've always got it working by re-calibrating on setup. I didn't tested the code not recalibrating after every wake up, but, for my understanding if you wakeup from deepsleep and do not bootstrap or re-calibrate the RTC you may experience some instabilities with it.

I'm not sure if @errolt recalibrated every wake-up then. The calibration is supposed to be done by the core in cpu_clock.c if the RTC was detected earlier on any startup. As this is not done for now, you need to repeat the calibration and bootstrapping procedure on arduino setup.

Some settings such as the boostrap time are hardcoded as a value defined in the menuconfig that are used as constant arguments during libesp32.a (you can see them in cpu_clock.c for instance) compilation. But by passing your own argument in rtc_clk_32k_bootstrap(512); you then need to create your own logic to restore this value (it's best to just leave it as a constant value if calibration works every time).

[nevercast]

Trying to follow those instructions I'm getting

RTC_CAL_32K_XTAL:
calibrate (0): inf kHz
calibrate (1): 16.868 kHz
calibrate (2): 22.210 kHz
calibrate (3): 33.604 kHz
calibrate (4): 52.245 kHz
done
OSC Enabled

Does someone have a complete Sketch I can just copy paste, that is known to work? That'll help me at least eliminate the code being a problem and I can focus on IDF and hardware.

Edit/Update:
After a bit of playing I've managed to get the crystal to oscillate consistently. A lot of it was hardware layout and the capacitors I was using. (You can see Github comment history for my edits over time)

Quick question, does the ESP32 require a 32.768 or a 32.000 crystal? Can it use either? I currently do not see the CPU always start the external oscillator, I will continue working on that.

[nzagorec]

@Matheus-Garbelini I tried to add your code to my board and, although it does show 32.768 kHz for "calibrate", it's no longer possible to wake the MCU from deep sleep. As soon as I remove these lines of code it works fine.

I also tried to modify esp32/clk.c by adding rtc_clk_32k_bootstrap(512) before rtc_clk_32k_enable(true) and it also seems to work (cal_val is >= 15000000L) but I have the same problem - I can't wake my board from deep sleep using ext.

Any suggestions?
@igrr any comments?

[Matheus-Garbelini]

@narongrat I'm not sure, rtc_clk_slow_freq_set(RTC_SLOW_FREQ_32K_XTAL); should have done it. Maybe it's the IDF version you are using? Did you enabled RTC in the settings? If using the deep sleep with internal osc it works right?
I don't know if igrr can help now, he's focusing in other fields of ESP32 such as the wifi stack. Unfortunatelly esp32 arduino core generation is still not documented, so we don't now what's the .config they are using after each idf update. It doesn't seem they intend to disclose it also :p

[nzagorec]

@Matheus-Garbelini I've enabled external crystal in menuconfig and copied libesp32.a to my arduino core, yes. If XTAL is not started and it falls back to internal, it boots normally.

Op also mentioned this problem in his original post:

I have tried to change the RTC clock using:
rtc_clk_32k_bootstrap();
rtc_clk_32k_enable(true);
rtc_clk_slow_freq_set(RTC_SLOW_FREQ_32K_XTAL);
But that did not enable the osc, and it prevents deepsleep from ever waking up.

@errolt did you ever get external XTAL running in arduino-esp32?

[Matheus-Garbelini]

@nzagorec yes, as I depicted in the pictures of my first post. Also check my repo which I've used RTC:
https://github.com/Matheus-Garbelini/TransformerMonitoringSystem/blob/master/GPSManager.hpp
Altough I've not used deepsleep in the project, I tested this to ensure my core was working. You can check the libs I was using in this repo (Already compiled): https://github.com/Matheus-Garbelini/Esp32-Pre-Compiled-SDK-Toolchain

Check if it works with the older libs provided (Altough I've used psram). I don't remember exactly the version of the arduino core I was using, but you can check the last commit since my first comment to get the corresponding board package. The defconfig file I've used is in esp-idf/examples/get-started/hello_world/sdkconfig

Use the script ./dumplib.sh to automatically find the lib files and dump to lib folder.

PS: At the time it was important to me to repeat the rtc_clk_32k_bootstrap(512); code twice before enabling the RTC, this may not be true with the current SDK.

[joaocarlosss84]

@nevercast, please take a look at the ESP32 datasheet. It says to use a 32kHz crystal, but it will work if you use a 32.768kHz too. Please keep in mind that you need to calibrate the RTC.

Another thing that I noticed is the PIN 19 VDD3P3_RTC - it should be connected in your module/breakboard to make the internal RTC hardware work. This is true inside the ESP32-WROOM-32 and ESP32-WROVER.

About RTC precision: sometimes you may need to add external capacitors close to the XTAL as well. You can see this inside the ESP-WROVER-KIT schematic, they are using a 12pF capacitors.

https://www.espressif.com/sites/default/files/documentation/esp32_datasheet_en.pdf
https://www.espressif.com/sites/default/files/documentation/esp32-wroom-32_datasheet_en.pdf
https://www.espressif.com/sites/default/files/documentation/esp32-wrover_datasheet_en.pdf
https://dl.espressif.com/dl/schematics/ESP-WROVER-KIT_V4_1.pdf

[nevercast]

Unfortunately I cannot find anywhere that tells of the crystal used on the ESP-WROVER-KIT V4.1, without a datasheet for the crystal, the schematic isn't much use. For a 12.5pF CL crystal I've found 12pF capacitors to be too small. I am getting some success using 18pF capacitors, at least the crystal starts oscillating, however when I attempt to deepsleep it never wakes.

I have also received the 32768Hz crystal, one that was recommended on the forums. I am using this.

I am using a WROVER module currently, so I expect the power system to work correctly.

[nevercast]

Best results so far is a 12.5 CL Crystal, with 18pF loading caps, and 7Mohm bias resistance. Crystal oscillates, deepsleep works, the ESP32 wakes up but it wakes up about 8-10 times late, and when it wakes from deepsleep it reports that the 32 kHz XTAL is not found.

[doanerock]

I am not looking to put the esp32 into sleep mode. I simply want to highly increase the accuracy of the clock. I know you can set the eap32 via NTP which is what I plan on doing. I want the esp32 clock to be able to be able to keep accuracy to a few seconds for months at a time. I want to attach a VCTCXO Oscillator that offers ±100ppb in order to keep the accuracy. So it looks like I need to attach the output of my Oscillator to IO32 or IO33. Can I use other frequencies like 26MHz, or 40MHz or would only one frequency work? Do I have to use a certain type of Oscillators? Would a Clipped Sine Wave work, would others work like sign wave, plus a lot of others to choose from on digikey. Thank for the help I have been searching in a lot of difrent places for a while now.

[ryan4volts]

Interesting to note that when I load IO32 the XN of crystal my osilloscope probe @x10, so ~5pF, the RTC stops reporting the issue. It still occurs at power up/rst regardless. Though after a deep sleep the warning disappears. Maybe the internal circuit doesn't work well by default to effectly load one side to jump start the xtal?

[nevercast]

Interesting to note that when I load IO32 the XN of crystal my osilloscope probe @x10, so ~5pF, the RTC stops reporting the issue.

Try increasing the loading capacitors ?

[ryan4volts]

Yet to do actual time test, but all my boards are now working. Turns out I selected 32khz clock instead of 32khz crystal in the menuconfig. Below is my code. This is working on the sparkfun thingy, my custom pcb and the esp32-lcd-kit. Specifically I mean I don't get the error anymore and deep sleep works fine.

`void app_main() {
struct timespec tp;
memset(&tp, 0, sizeof(tp));
clock_gettime(CLOCK_REALTIME, &tp);

printf("Got time %lus\n", tp.tv_sec);
if (tp.tv_sec < 1573532) {
	tp.tv_sec = 1573532;
	tp.tv_nsec = 198;
	clock_settime(CLOCK_REALTIME, &tp);
	puts("Time set");
}

const int deep_sleep_sec = 10;
printf("Entering deep sleep for %d seconds", deep_sleep_sec);
esp_deep_sleep(1000000LL * deep_sleep_sec);

}`

[dajtxx]

FWIW, I copied and modified the select_rtc_slow_clk function from clk.c. It cannot be called directly because it is static.

My mods are to make it always try to set the crystal as the RTC source, but still has the fallback to the 150 kHz oscillator. I call this early in setup figuring that it's only setting the source for the RTC related circuits so is probably ok to call at that point.

On my first board the RTC is still running fast by 1s in 150 minutes, or about 1m every 6 days with a deep sleep interval of as close to 15m as I can make it. I have not tried it on other boards yet.

#define SLOW_CLK_CAL_CYCLES 1024

void select_rtc_slow_clk(void)
{
    rtc_slow_freq_t rtc_slow_freq = RTC_SLOW_FREQ_32K_XTAL;
    uint32_t cal_val = 0;
    /* number of times to repeat 32k XTAL calibration
     * before giving up and switching to the internal RC
     */
    int retry_32k_xtal = 5;

    do {
        /* 32k XTAL oscillator needs to be enabled and running before it can
         * be used. Hardware doesn't have a direct way of checking if the
         * oscillator is running. Here we use rtc_clk_cal function to count
         * the number of main XTAL cycles in the given number of 32k XTAL
         * oscillator cycles. If the 32k XTAL has not started up, calibration
         * will time out, returning 0.
         */
        ESP_LOGI(TAG, "waiting for 32k oscillator to start up");
        rtc_clk_32k_enable(true);

        // When SLOW_CLK_CAL_CYCLES is set to 0, clock calibration will not be performed at startup.
        if (SLOW_CLK_CAL_CYCLES > 0) {
            cal_val = rtc_clk_cal(RTC_CAL_32K_XTAL, SLOW_CLK_CAL_CYCLES);
            if (cal_val == 0) {
                if (retry_32k_xtal-- > 0) {
                    continue;
                }
                ESP_LOGW(TAG, "32 kHz XTAL not found, switching to internal 150 kHz oscillator");
                rtc_slow_freq = RTC_SLOW_FREQ_RTC;
            }
        }

        rtc_clk_slow_freq_set(rtc_slow_freq);

        if (SLOW_CLK_CAL_CYCLES > 0) {
            /* TODO: 32k XTAL oscillator has some frequency drift at startup.
             * Improve calibration routine to wait until the frequency is stable.
             */
            cal_val = rtc_clk_cal(RTC_CAL_RTC_MUX, SLOW_CLK_CAL_CYCLES);
        } else {
            const uint64_t cal_dividend = (1ULL << RTC_CLK_CAL_FRACT) * 1000000ULL;
            cal_val = (uint32_t) (cal_dividend / rtc_clk_slow_freq_get_hz());
        }
    } while (cal_val == 0);

    ESP_LOGI(TAG, "RTC_SLOW_CLK calibration value: %d", cal_val);
    esp_clk_slowclk_cal_set(cal_val);
}