XOSC32K running slow on avr64db32?

[ObviousInRetrospect]

[edit: solved, was losing about 10/8192 pit interrupts due to a print in the pit.]

So on several samples I've been seeing XOSC32K losing hugely out of spec amounts of time (2 minutes per day). I figured I was just being sloppy with the loading capacitors but I finally sat down and did some more rigorous testing across 2 samples with 3 different crystal types and several permutations of loading caps per crystal. Surprisingly all were slow by similar amounts.

Any ideas what I am doing wrong (besides using crystals with similar specs to ones on the approved list instead of ones on the approved list - waiting for one of those to arrive)?

I've used a couple methods. Mostly via a GPS (both the 1 pulse per second signal and the time output). But my oscilloscope matches (on a pin toggled from the PIT ISR running at 8192hz) within its more limited precision.

• 8192hz pit increments a ticks counter and prints the difference between rising edges (averages between 8180 and 8182, should be 8192
• also printing wall-clock time from the same gps over long periods of time to verify it isn't just the pps signal being off
• also printing millis() to distinguish variation in the RTC from variation in the GPS output. millis/pps is stable, rtc/pps takes time to settle.
• ran the test long enough to compare NTP time to GPS time, the GPS time is accurate (over a time period with a 37 second discrepancy in the RTC).

All rigorous testing was done at 2.8v. But I originally noticed the problem of losing about 2m per day on a third board running 3.3v.

(yes, in theory I could read the rtc counter value instead of using the pit for 2 more bits of precision. I wish I had it accurate enough for those 2 bits to be useful. I figured the synchronization time might make that a bit harder to reason about.)

In short tests, a 6pf crystal https://www.digikey.com/en/products/detail/micro-crystal-ag/CM8V-T1A-32-768KHZ-6PF-20PPM-TA-QC/10499094 with the minimum caps I could get to start (2.7pf = 2.7/2+4/2+1/2 = 2.85pf load) was slightly better than either correct-ish caps (6pf/2=3+4/2+1/2 5.5 load ) or too-large caps (12pf caps = 12/2+4/2+1/2 = 8.5pf load)

Additionally, the crystals all seem to take about 90-120 seconds to stabilize during which they get worse and go from ~8184 to 8181ish and then seem to further slow over a couple hours to 8180.3ish.

The best results were with the undersized caps (which would pull it faster) so I ran that overnight. In 29082 seconds (wall clock measured by GPS) I had 237927797 ticks, for a rate of 8180.9922291373Hz and a loss of 39 seconds. If I discard the first 10k seconds it gets worse: 8180.62395090222hz or a loss of 26.47/19064 seconds.

These are all 2-layer PCBs, ground plane under the crystal which is right next to the MCU, 0402 loading caps.

In short tests (minutes), a 6pf crystal https://www.digikey.com/en/products/detail/micro-crystal-ag/CM8V-T1A-32-768KHZ-6PF-20PPM-TA-QC/10499094 with the minimum caps I could get to start (2.7pf = 2.7/2+4/2+1/2 = 2.85pf load) was slightly better than either correct-ish caps (6pf/2=3+4/2+1/2 5.5 load ) or too-large caps (12pf caps = 12/2+4/2+1/2 = 8.5pf load). (4pf, 9pf) were not meaningfully different.

This was repeated against a couple capacitor variations with a second copy of the same crystal type on a second board with functionally identical results.

I also tried both:
Q13FC1350000200 (needed 6pf loading caps to start)
and
Q13FC1350000400 (similar between 6, 12, and 18pf loading caps)

Here is the test code (omitted configuration of gps + pin setup)

I initialize the crystal:

void setup_rtc(){
uint32_t ims = millis();
  //setup watch crystal XOSC32K
  _PROTECTED_WRITE(CLKCTRL.XOSC32KCTRLA,
	             CLKCTRL_CSUT_64K_gc            /* 64k cycles */
	                 | 1 << CLKCTRL_ENABLE_bp   /* Enable: enabled */
	                 | 1 << CLKCTRL_RUNSTDBY_bp /* Run standby: enabled */
	                 | 0 << CLKCTRL_SEL_bp      /* Select: disabled */
	                 | 0 << CLKCTRL_LPMODE_bp /* Low-Power Mode: disabled */);

  while(0==(CLKCTRL.MCLKSTATUS & CLKCTRL_XOSC32KS_bm))
  {
    if((millis()-ims)>2000){ //taking too long, make noise
      Serial.println(F("Waiting for XOSC32K to start...")); delay(1000);
      Serial.print(F("XOSC32KCTRLA: ")); Serial.println(CLKCTRL.XOSC32KCTRLA,2);
      Serial.print(F("MCLKSTATUS: ")); Serial.println(CLKCTRL.MCLKSTATUS,2);
      Serial.print(F("CLKCTRL_XOSC32KS_bm: ")); Serial.println(CLKCTRL_XOSC32KS_bm,2);
    }
    if((millis()-ims)>4000){ //give up and use OSCULP32K instead
        Serial.println(F("XOSC32K startup FAILED"));
        //remove fallback to ULP for brevity, not happening in tests
    }
  }
  errors &= ~ERR_CRYSTAL;
  Serial.print(millis()-ims);
  Serial.println(F(" XOSC32K started. Initializing RTC."));
Serial.println(F(" XOSC32K started. Initializing RTC."));

  /* Initialize RTC: */
  while (RTC.STATUS > 0) {
    ; /* Wait for all registers to be synchronized */
  }
  RTC.CLKSEL = CLKCTRL_CLKSEL_XOSC32K_gc; /* 32.768kHz crystal */
  RTC.PITINTCTRL = RTC_PI_bm; /* PIT Interrupt: enabled */
 RTC.PITCTRLA = RTC_PERIOD_CYC4_gc     /* RTC Clock Cycles 3, resulting
                                    in 32.768kHz/4 = 8.192kHz */
                 | RTC_PITEN_bm; /* Enable PIT counter: enabled */
  Serial.println(F("RTC initialized using XOSC32K"));
}

volatile uint8_t lpps; //last pps read
uint32_t lr; //last rising edge, ticks
uint32_t lrm; //last rising edge, millis
volatile uint8_t pf;
ISR(RTC_PIT_vect) {
  pit_fire=1;
  RTC.PITINTFLAGS =
      RTC_PI_bm; /* Clear interrupt flag by writing '1' (required) */
  ticks++;
  digitalWriteFast(P_LED,ticks&1); //signal for oscilloscope
  uint8_t pps=digitalReadFast(P_PPS);
  if(lpps != pps){
    lpps=pps;
    if(pps){ //rising edge of PPS
      Serial.print("R "); //dirty practice, printing in ISR, however timing stuff is done and this is simpler
      uint32_t n = millis();
      Serial.print(millis());
      Serial.print(" ");
      Serial.print(ticks);
      Serial.print(" ");
      Serial.print(n-lrm);
      Serial.print(" ");
      Serial.print(ticks-lr);
      lr=ticks;
      lrm=n;
    }
    else{ //falling edge happens long enough after (100ms) for it to be safe for main loop to print the rest of the line
      pf=1;
      /*
      Serial.print("F ");
      Serial.print(millis());
      Serial.print(" ");
      Serial.println(ticks);*/
    }
  }
}



uint32_t last;
void loop(){
      while(Serial2.available()){
      char c=Serial2.read();
      nmea.process(c);
      //Serial.print(c);
    }
  if(pf){
    if(1 || nmea.isValid()){
      cli();
      uint32_t lticks=ticks;
      sei();
      uint8_t h = nmea.getHour();
      uint8_t m = nmea.getMinute();
      uint8_t s = nmea.getSecond();
      uint16_t y= nmea.getYear();
      uint8_t mo = nmea.getMonth();
      uint8_t d = nmea.getDay();
      if(nmea.isValid()){
            setTime(h, m, s, d, mo, y);  
      }
      //Serial.print(d);
      
      int8_t hl =h-7;
      if(hl<0) hl+=24;
      Serial.print(" ");
      Serial.print(nmea.isValid());
      Serial.print(" ");
      Print2(hl);
      Serial.print(':');
      Print2(m);
      Serial.print(':');
      Print2(s);
      Serial.print(" ");
      Serial.print(now());
      Serial.print(" ");
      Serial.print(millis());
      Serial.print(" ");
      Serial.println(lticks);
      nmea.clear();
    }
  pf=0;
  }
}

Here is a short snippet of the output:

rise millis ticks ms/pps ticks/pps gps_valid time  time_utc_s   millis_for_time_utc_s  ticks_for_time_utc_s
R 3184855 26092131 998 8180 1 07:56:19 1658415379 3184957 26092939
R 3185853 26100311 998 8180 1 07:56:20 1658415380 3185955 26101119
R 3186851 26108492 998 8181 1 07:56:21 1658415381 3186952 26109299
R 3187848 26116672 997 8180 1 07:56:22 1658415382 3187949 26117479
R 3188845 26124852 997 8180 1 07:56:23 1658415383 3188946 26125659
R 3189842 26133032 997 8180 1 07:56:24 1658415384 3189943 26133839
R 3190839 26141212 997 8180 1 07:56:25 1658415385 3190940 26142020
R 3191837 26149393 998 8181 1 07:56:26 1658415386 3191939 26150200
R 3192835 26157573 998 8180 1 07:56:27 1658415387 3192936 26158380
R 3193832 26165753 997 8180 1 07:56:28 1658415388 3193933 26166560
R 3194829 26173933 997 8180 1 07:56:29 1658415389 3194931 26174740
R 3195826 26182113 997 8180 1 07:56:30 1658415390 3195927 26182921
R 3196823 26190294 997 8181 1 07:56:31 1658415391 3196924 26191101
R 3197820 26198474 997 8180 1 07:56:32 1658415392 3197921 26199281
R 3198817 26206654 997 8180 1 07:56:33 1658415393 3198918 26207461
R 3199814 26214834 997 8180 1 07:56:34 1658415394 3199915 26215641
R 3200811 26223014 997 8180 1 07:56:35 1658415395 3200912 26223822
R 3201808 26231195 997 8181 1 07:56:36 1658415396 3201909 26232002
R 3202805 26239375 997 8180 1 07:56:37 1658415397 3202906 26240182
R 3203802 26247555 997 8180 1 07:56:38 1658415398 3203903 26248362
R 3204799 26255735 997 8180 1 07:56:39 1658415399 3204900 26256542
R 3205796 26263915 997 8180 1 07:56:40 1658415400 3205897 26264723
R 3206793 26272096 997 8181 1 07:56:41 1658415401 3206894 26272903
R 3207790 26280276 997 8180 1 07:56:42 1658415402 3207892 26281083
R 3208788 26288456 998 8180 1 07:56:43 1658415403 3208889 26289263
R 3209786 26296636 998 8180 1 07:56:44 1658415404 3209887 26297443
R 3210783 26304816 997 8180 1 07:56:45 1658415405 3210884 26305624
R 3211781 26312997 998 8181 1 07:56:46 1658415406 3211882 26313804
R 3212779 26321177 998 8180 1 07:56:47 1658415407 3212881 26321984
R 3213777 26329357 998 8180 1 07:56:48 1658415408 3213878 26330164

[ObviousInRetrospect]

And in further testing I added a tune of +1 to the internal oscillator. After 4h54m16s I have reached the somewhat infuriating conclusion that the internal oscillator which is specified to be +/- 1% has 22.71ppm error and the crystal which is specified as +/-20ppm has -1585.88 ppm of error. I suppose a more fair comparison would be to apply the tune to the rtc as well which would subtract 127 ppm for 1458.87 but its not a big enough improvement to warrant the prescaler. Mental note not to use the "auto-tune" setting. facepalm

[SpenceKonde]

I have been less than impressed with the oscillator circuits on these new parts, in general. I haven't noticed accuracy issues, though I haven't measured that with all that much care - I've used it as a timebase to cal th internal oscillator on tinyAVRs from, and it was definitely pretty damned close - I couldn't see a difference on the 'scope's frequency counter with an attiny between theoretical and observed. but that was on a 2-series. Have you tried tricks like not using exactly the same load capacitance on the crystal? unbalanced loading caps have more of a frequency pulling effect.

Also, uh, what exactly is the formula for the loading caps you are "supposed" to use? I have seen 2 formulas, neither of which gives a value that causes the crystal to oscillate... and sometimes the stray capacitance is enough to made the bloody things work. Though note that i've mostly played with HF xtals for system clock (mainly for overclocking - DB will run at 40 MHz from a crystal. At least one non-extended temp range one would almost work at 48 from an external clock. but not quite, but when I moved the external clock to a board with an extended temp range DB, I could run a sketch that printed out millis for at least a half an hour without millis suddenly changing. (the usual failure mode of modern AVRs when overclocked is that they get the wrong answers to arithmetic and logical operations, most often 1's coming out as zeros,.

[ObviousInRetrospect]

So in a tinyAVRs calibration world (warning: the modern tinyAVRs use the X32K_1v8_5v5_ULP oscillator module, the DX use OSC_LP_V10, DS00002648D-page 15) the 1600ppm error is still hugely better than the internal oscillator and is a great reference to tune the internal (1600 ppm = 0.16%). But I noticed I was getting large discrepancies between the RTC's timekeeping over the course of days and the time coming out of my GPS so I started investigating what I assumed was just wrong loading caps when I started down this rabbit-hole.

I initially used "divide the sum of the pin capacitance, the external caps, and the parasitic by 2" which seems to produce identical results to the more complex equation 3-5 in:
https://ww1.microchip.com/downloads/aemDocuments/documents/MCU08/ApplicationNotes/ApplicationNotes/AN2648-Selecting_Testing-32KHz-Crystal-Osc-for-AVR-MCUs-00002648.pdf [edit: and they are mathematically the same]

But following the advice in that app note I was trying to accurately measure the frequency to figure out if I got the caps right since I distrust my measurement of the parasitic capacitance (my LCR meter with kelvin-connected tweezer probe reads 1.5pf with the tweezers nearly closed and 2.5pf with those nearly closed tweezers touching the cap pads on an unpopulated board, ergo my best guess for parasitic is 1pf).

So according to that formula, assuming 1pf parasitic (per pad), 4pf pin (per pin) you need 9pf external to have a load capacitance of 7pf and the 6pf external I started with should have already been pulling it faster but it was running slow. Assuming I have no idea what the actual parasitic is I kept dropping the capacitance until it wouldn't start (2pf) then backed it off to 2.7 which seemed to run and was still mega slow. But since it wouldn't start with any less I definitely didn't have too much.

I have not tried unbalanced caps for greater pulling, I was attempting to measure frequency to verify I wasn't inadvertently pulling it off and discovered it was way off regardless of how I loaded it. In theory the crystal when loaded correctly should be in spec, I want to leverage the 20ppm crystal spec not abuse it to try to drag it around by 1000+ppm by violating the loading spec.

I am pretty sure my board that uses a crystal on an attiny1614 has the same problem as I also noticed it losing what seemed like way more than 20ppm on its timekeeping. But since it doesn't have a GPS with PPS connected natively I started doing the investigation on the DB board.

[ObviousInRetrospect]

and yes equation 3-5 does in fact simplify to "divide the per-pin total by 2" when the load capacitance is the same on both pins:
Cper_pin = (Cparasitic+Cexternal_load+Cpin_load)
loading = (Cper_pin x Cper_pin)/(2 x Cper_pin) [eq.3-5]
cancel a Cper_pin term and you get Cper_pin/2

[ObviousInRetrospect]

I think I found the problem. After repeating the testing with a crystal on the manufacturer recommended list and getting the same result and assembling a different design to make sure it wasn't a layout issue I became convinced this was not possible.

I went ahead and moved the serial.print out of the ISR and am now getting 8192-8193/second (running slightly fast, as expected with intentionally low load capacitance). Sigh. I knew better when I wrote it but yes an overly slow PIT ISR was causing 1500PPM of missing interrupts.

For added hilarity, the reason it was taking a few minutes to "settle" in which it slowed from 8183 to 8180 is the numbers increase in size at the beginning which means an increasing amount is getting printed.

[SpenceKonde]

Wow. How often was the pit triggering? It would have to be somehow spending enough time be busy for a whole interrupt cycle.either thst or the transmission is was playing a role

…

____________ Spence Konde Azzy’S Electronics New products! Check them out at tindie.com/stores/DrAzzy GitHub: github.com/SpenceKonde ATTinyCore: Arduino support for almost every ATTiny microcontroller Contact: ***@***.***

On Mon, Jul 25, 2022, 22:47 ObviousInRetrospect ***@***.***> wrote: I think I found the problem. After repeating the testing with a crystal on the manufacturer recommended list and getting the same result and assembling a different design to make sure it wasn't a layout issue I became convinced this was not possible. I went ahead and moved the serial.print out of the ISR and am now getting 8192-8193/second (running slightly fast, as expected with intentionally low load capacitance). Sigh. I knew better when I wrote it but yes an overly slow PIT ISR was causing 1500PPM of missing interrupts. — Reply to this email directly, view it on GitHub <#302 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABTXEWY5KX2LUHRA6L2XE5DVV5GTJANCNFSM54H74RLQ> . You are receiving this because you commented.Message ID: ***@***.***>

[ObviousInRetrospect]

its not the transmission alone: the revised sketch prints exactly the same stuff it just sticks the calculated values in globals in the PIT and then prints in the main loop like it should have all along.

[ObviousInRetrospect]

The PIT was supposed to be triggering at 8192hz. It was triggering at 8180hz. I was running the PIT as fast as it will go because you can't CLKO the 32K clock (afaik) unless you use it as the main clock. The oscilloscope measurements were a pin toggled from the PIT so they were of no help.

Did a bit more testing.

The initial numbers with the 6pf caps (a 7pf crystal with 5.5pf applied low) were +116ppm.
Installing the calculated correct 9pf caps produced -3.78 ppm (+/-20ppm crystal).

Just to get a sense of how much it would change I installed 8.2pf caps (6.6pf load) and got +32 ppm. That means a change of 0.4pf of load (for this crystal) produced a swing of +35.8ppm so with +/-0.25pf tolerance caps 9pf is essentially dead on.

after replacing the 9pf caps (with new ones, 0402 caps are not worth saving) I get -1.03ppm error.

repeating the experiment on one of the other boards (different layout, same crystal model & caps) I get -0.7ppm. And the third board (with CM8V-T1A-32.768KHZ-6PF-20PPM-TA-QC) -30ppm.

Things I and others can learn from this:

1. Don't ever print in an ISR. For real. Even when you think you can.2.
2. For JLCPCB's 0.8mm boards and a 2x1.6mm crystal with 0402 caps, 1pf is a reasonable approximation of the per-pin parasitic capacatance3.
3. the formula in this thread (divide per-pin parasitic + pin + external by 2) is the correct one. a 7pf crystal uses 9pf caps (board dependent, sort of).
4. I'm throwing out my +/- 0.5pf caps in that range and buying +/- 0.1pf caps. They are still dirt cheap.
5. I'll repeat some testing with the other crystals
6. A cheap Chinese GPS module is a perfectly reasonable way to check your crystal loading caps. The numbers are clear in seconds and converge in a couple minutes. I'll clean up and publish the tuning sketch.
7. This test victim appears to be more accurate with CLKCTRL_LPMODE_bp (-1.5ppm vs -24ppm). It seems to still start reliably and quickly with CLKCTRL_CSUT_1K_gc (168ms).

[ObviousInRetrospect]

And overnight testing (post-washing) confirms that the results seen in a couple minutes hold (and provides much higher precision, beyond what is useful).

The better of the two tested was -0.3062ppm. the "worse" was -1.4697ppm (untuned).

When I went to extend my prior scheme of USERSIG.write(0,CLKCTRL_OSC20MCALIBA) (from megatinycore) to also store RTC.CALIB I seem to be getting strange results from the USERSIG library on DxCore:

  Serial.print("write(0,0xFF):");
  Serial.println(USERSIG.write(0,0xFF));
  Serial.print("write(1,0x1):");
  Serial.println(USERSIG.write(1,0x1));
  Serial.print("flush():");
  Serial.println(USERSIG.flush());
  Serial.print("read(1):");
  Serial.println(USERSIG.read(1));

produces:
write(0,0xFF):1
write(1,0x1):1
flush():0
read(1):255

My understanding of the docs say that if write() returns 1 there is no need to call flush (and I wasn't originally). Did I miss something or Is this a USERSIG bug on DB?

the simpler test (where I started):

  Serial.println(USERSIG.write(1,0x1));
  Serial.println(USERSIG.read(1));

always prints 255