Sayma v2.0 clocking/synchronisation write up

[hartytp]

Thanks to @jordens for an extremely useful discussion about this! https://freenode.irclog.whitequark.org/m-labs/2019-01-22 https://freenode.irclog.whitequark.org/m-labs/2019-01-23

Below is my proposed clocking/synchronisation scheme for Sayma. There are plenty of other ways of skinning this cat; the recommendations here are influenced by my competences, biases and the kit I have in my lab right now.

• Reference:
  • f_ref = f_rio_coarse
  • reference clock must be phase locked to the RTIO clock (stable to <<8ns)
  • two reference sources to be supported in hardware using a mux (ADCLK948 or similar): external (SMA/MCX); or internal/WR
  • until WR implemented and tested, external reference will be the only option supported in software
• DAC clock
  • DAC clock (typically between 600MHz and 2.4GHz) generated by on-board PLL
  • fanout buffer used to route DAC clock is a mux (ADCLK948 or similar)
  • second mux input attached to UFL/MMCX connectors that can be used to inject an external clock if required as a fall-back in case of unforeseen issues with the PLL (in which case the user takes responsibility for ensuring phase locking/stability w.r.t. the reference clock)
  • DAC clock output phase must be deterministic w.r.t. the reference clock and have suitably low drift both for synchronisation (drift << 1 DAC clock period) and for the user's particular requirements (usually <=1ps for ion traps)
  • DAC clock PLL operated in integer-N mode so phase is deterministic apart from any unsynchronised output dividers or device imperfections/"analog effects" (VCO leakage, drift etc)
  • test ongoing to characterise the device imperfections. Provisionally it looks like we'll want to manually set the VCO band and use an active (3rd order) loop filter. Also highly desirable to operate the PFD at the max possible frequency. Need to measure phase v temp coefficient for PLL (and also for DAC since that may well limit the overall system performance)
  • Plans for overcoming unsynchronised output dividers (not necessarily in order of preference)
  • Plan A: operate PLL in "fundamental" mode (no output divider) and use DAC interpolation
    • For the HMC830, the VCO range is 1.5GHz to 3GHz, so this only works when the DAC is used in interpolation mode (e.g. >=2GHz clock)
    • For the ADF PLL, the VCO range is 3.4 to 6.8GHz, so this is not possible
    • Risks: a DAC bug that prevents synchronisation at this clock rate
  • Plan B: manually synchronise the PLL output divider
    • DAC clock output phase indeterminism quantized in multiples of the VCO period
    • Desirable to use a PLL with lower VCO frequency as phase quantized in larger steps (makes HMC830 preferable to ADF)
    • measure the PLL start-up phase and reset the PLL (e.g. toggle divider value) until we achieve the desired phase
    • Needs an external phase comparator. Various ways to do this, one possibility is to use the DAC SYSREF. Another is to use the FPGA. Let's configure the hw to allow both.
    • if using the DAC: at start-up, scan sysref delay and find the transitions. If these are not where we expect them to be then reset the divider and try again. Relies on the SYSREF detection being long-term stable to better than 1 period of the VCO, but if this isn't the case then we're never going to be able to sync the DAC at max clock rate and so should rethink the design anyway
    • ideally, should be demonstrated using v1.0 hw (work in progress)
  • Plan C:
    • Use a PLL with synchronised output divider
    • I can see two paths to do this
      • 1. PLL with feedback after the output divider
        • either make our own PLL (e.g. combine PLL with external PFD) or use the ADF4356 or similar, which can do this internally
        • ADF pre scaler requirements mean we'd have to operate the PFD at around 20MHz for 600MHz clock. Not a killer, but annoyingly low (worse for noise and drift)
        • Basic idea demonstrated on ADF4356 evaluation hw, but analog effects/device imperfections need further investigations (but that's true for any approach)
        • rolling our own PLL might give best drift performance but would need careful prototyping. Might be best to press ahead with Sayma v2.0 as is and demonstrate this on an external mezzanine board if required (use the mux to inject it into the DAC clock tree)
      • 2. Use a PLL with a synchronisable output divider (or use an external output divider with SYNC pin)
        • If we're not careful, we end up with another 2GHz sync problem and need more delay lines etc, more complexity, doesn't seem worth it
        • possible exception would be to use the HMC7044, which has a simpler sync mechanism (can sync of ref clock). However, that would need testing and I have separate concerns about that chip (e.g. IIRC, it's only possible to use the low-frequency sync input if we also use a 2-loop PLL, which is nasty for drift etc)
  • Given the above I would (weakly) advocate for: use HMC830 as our PLL. Aim for plan A initially, with plan B as a fall-back. If even that doesn't work then we can build a separate PLL board and hook it up via coax/IDCs to inject the DAC clock
• SYSREF
  • SYSREF generated on RTM FPGA using a SERDES TTL Phy
  • routed to a retiming DFF on the RTM. DFF clocked directly from the ref clk fanout
  • low-jitter retimed SYSREF goes to digital delay lines, then routed to each DAC (DC coupled)
  • two synchronisation issues:
    • A make FPGA SYSREF meet S/H timings at the DFF
      • relatively "easy" problem since we have an ~8ns period
      • I would expect that this is something that can be properly constrained in gateware so that we can use a single calibration (e.g. scope) for all boards across PVT
      • If we need to, we can implement a calibration scan, for example by: produce a 1mu (1ns for 125MHz f_rtio) wide pulse from the FPGA and scan the delay in 1mu increments. Ask the DAC whether it's seen a SYSREF edge or not. This allows us to locate the sampling window. We could also route a copy of the retimed SYSREF signal to a RTM FPGA input to allow us to do the calibration without involving the DAC (but requires more HW on the board)
    • B make the retimed SYSREF meet S/H timings at the DAC
      • this is the "harder" part of the problem, since timing window much narrower (must meet S/H on a 2.4GHz clock)
      • use DAC as a phase comparator (one shot-then-monitor mode) sweep SYSREF delay and find the edges. Put SYSREF delay in the middle of this window. Store a seed value to make this repeatable across boots
      • risk: drifts in the delay lines etc. Preliminary tests suggest that this won't be a problem over a decent range of VT
      • risk: some DAC issue that breaks sync at higher clock rates.
• FPGA
  • RTM FPGA clocked from recovered DRTIO clock. This must be stable w.r.t. the ref clock to <<8ns
  • AMC FPGA fabric + transceivers clocked from DRTIO CDR clock
  • Need to ensure that we have a suitable PLL on the AMC to generate the required transceiver clocks
  • AMC SYSREF signal either routed from the RTM FPGA to the AMC FPGA via the AMC <-> RTM connector or (more likely) generated internally from the rtio clock

[jordens]

Apart from the things discussed on IRC:

• It would be helpful to get an output from the RTM ref clock fanout to the AMC FPGA (and the RTM FPGA). Even if it is only for debugging/bandaid to decouple the JESD/SYNC stuff from DRTIO ref clk reconstruction.
• It is apparently recommended to disable the CP leakage current compensation when changing/setting frequency at high PFD frequencies. I.e. large PFD frequencies and large CP leakage current look problematic. (Hittite PLL operating guide).

[hartytp]

For int N mode one sets the CP offset to zero anyway, right? I.e. that’s only a concern for frac-n operation. Or am I misunderstanding you?

[jordens]

Right. That was fractional.

[gkasprow]

What is the conslusion? I have version of schematics with HMC830, ADF4356BCPZ, discrete delay line and HMC7043. I assume we stay with HMC830 and HMC7043?
i want to freeze RTM schematics ASAP.

[hartytp]

I assume we stay with HMC830 and HMC7043?

Yes.