-
Notifications
You must be signed in to change notification settings - Fork 7
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
DRTIO clock recovery/Si5324 in Sinara #515
Comments
@dtcallcock I remember Jeff saying he'd measured timing stability for WR. Would you mind asking him for any data he has on this, please? |
PS sorry for the essay, wanted to get my thoughts clear while this is fresh in my mind! |
Not necessarily: the FPGA PLL output would only drive the Si5324 input, and you still get the jitter filtered (the clock that is used for the rest of the design is the Si5324 output only). I see three ways of doing this:
I'm surprised we didn't see the Si5324 problems before, when we tested this: https://github.com/m-labs/drtio_transceiver_test |
I wonder if this is normal. Wouldn't that also cause problems in typical situations where the Si5324 is used with a shallow elastic buffer? This is also quite difficult or impossible to fix with the FPGA. |
Unsurprisingly, WR is really nicely designed and very carefully thought through. They have lots of good application notes on this, which contain a lot of the info we need (GTX noise measurements etc). I started pooling some of the data on the Wiki https://github.com/m-labs/sinara/wiki/SinaraClocking The good news is that they're able to recover a really pretty good clock over 1 3km fibre, even after FPGA transceivers. -105dBc/Hz at 10Hz (for reference the SRS rubidium clock is -105dBm/Hz at 1Hz, -135dBc/Hz at 10Hz). |
So, it all comes down to designing a good PLL with a good flywheel oscillator. That's not too hard. |
I suspect that the stability/noise of this approach will not be great. But, could work as a bit of a hack to get the current HW up and running while we design a better long-term solution. |
To get the current hardware running, perhaps dropping in a Si5326 will work. Double-check the datasheet, but it seems: it is pin-compatible, it has mostly the same register interface, it has input-to-output skew control, and its features generally are a superset of the 5324's. |
there is one possible fix - add guard shield around the oscillator. It is not written in Si5324 datasheet, but is recommended for some other Silabs chips with similar oscillator. |
@gkasprow what does that do? Just an EMI shield? These problems are mainly thermal. (A screening can to prevent air currents reaching the crystal would probably help a bit however. |
Here's Jeff Sherman's poster on some measurements he did at NIST on the SevenSols WR-LEN. sherman_WSTS2017_poster.pdf He got 100fs because he's got mad skillz. |
@hartytp Yes, I meant EMI shield. |
Thanks David and Jeff! Mad skills indeed. So, Jeff thought that the SevenSols WR-LEN might be implemented in a slightly different way to the original WR and might be somewhat better. Anyone know any details about its implementation? AFAICT it's not open source, or am I missing something? |
There was a mistake in the measurements reported above. @cjbe retook the data more carefully and found:
Once we receive more Kasli, I'll repeat this measurement using an interferometer (mixer) to give a sub ps measurement noise floor. |
So, it seems that the current hardware can provide <<ns timing stability with correct Si5324 settings. Note that the phase of the Si5324 is still random at startup. However, this should be fixable by using a PLL in the FPGA to add a phase shift to the Si5324 input clock (this may degrade the phase stability a little however). The FPGA can then be used to measure the Si5324 startup phase and tune the Si5324 input phase shift appropriately. So, we should be able to get a decent solution with current hardware and only gateware/firmware changes. |
Pending the result of the interferometer measurement to look at the drift of the Si5324 phase, I'd still like to suggest that we should consider using a solution based on a proper low-noise PLL IC with a better reference oscillator (VCO). My thinking here is that if we can get ps-level timing stability for the recovered clock -- and Jeff's data suggests that we should be able to -- then there are a lot of cool things we can do with it.
If ps timing stability can be achieved with the Si5324 + Xtal then great, otherwise, I'd like to prototype a better solution with the intention of applying it to Kasli/Sayma/Metlino once it's been well tested using current hardware. |
To do the design, the one thing we need to finalize is the choice of RTIO frequencie(s).
The RTIO frequency affects our PLL design:
Any objections to fixing the RTIO frequency to 125MHz if that makes the PLL design easier/better (obviously, other frequencies available as population options, potentially with somewhat worse performance)? @jordens |
The reasons to choose 150 MHz f_RTIO and my take on the arguments you bring up are described elsewhere. I don't think rehashing them is necessary as they haven't changed. There would probably be quite some development needed to go to 1 GHz SAWG now but that is also a chance to revise the parametrization and datapath design and rethink the specs. The current and potential SAWG users would need to think hard and weigh in. |
I opened one up and the important chips seem to be: Artix-7 XC7A35T Jeff claims it's a development of what was at some point an open source project. I will ask him for more details. |
@dtcallcock The WR-LEN is not open source hardware but they keep the gateware open since it is based on original WR development. However the chipset is similar as SPEC WR node I developed for CERN a few years ago. |
Edit: they say that use VCXO and TCXO controlled by DAC, so there should be yet another oscillator. Silabs is probably used for general purposes clocking |
@dtcallcock in what bandwidth Sherman got his 100fs? |
@gkasprow From the poster is seems he was comparing the master input 10MHz with the slave output 10MHz with a 50Hz effective BW and averaging down for 10^4s to get to 100fs. This was in a temperature and humidity controlled chamber. If you need more specifics I can ask him or just give you his email. |
For WR-ZEN design they use the same VCXO as we use for Urukul (CVHD-950) + low jitter divider (some chip from NS) to get 10MHz. Then they apply intensive filtering using 2x MCL to get rid of higher harmonics. So they do probably the same for WR-LEN. |
We're comparing two configurations:
@jordens I'm going to try to summarise the arguments here. Please correct me if I'm wrong:
IIRC, we've agreed to start with the 600MSPs case as the quickest way to get going. However, I'm keen to switch to the 1GSPS operation in the long run, as this maps better to my use-cases (and, AFAICT, to most ion trap use cases). So, question for other potential Sayma users (@jbqubit @dhslichter @dtcallcock @cjbe) how do you plan to use Sayma? Do you want/need the 600MSPs (150MHz RTIO) use-case? Or, like me, would you prefer the full 1GSPs bandwidth of the DACs? |
Should that be '...125MHz (x8 parallelisation...'? |
@gkasprow The Si549 and LMK61e07 are the only two DCXOs of this quality that I'm aware of, are you aware of any others? |
What about limiting the I2C noise by adding capacitors that reduce the edges? |
While there is indeed no stock, the Si549 is $16 at Arrow. The dollar amount isn't such a high price to pay if that avoids some horrible silicon bugs and workarounds like those I2C caps. I'm definitely not looking forward to another HMC7043-style chip. |
@gkasprow I tried to add capacitors and using different power sources on LMK61E07. The effectiveness is hardly noticeable. My feeling was the i2c noise is inner coupled to its output, rather than SI/PI issue. |
@WeiDaZhang I think Greg was talking about adding capacitors to SCL and SDA to increase rise and fall times. Depending where the noise coupling happens inside the chip, that may or may not help. |
@gkasprow I don't think that will help. Firstly, the noise we really care about is the stuff below about 100kHz. However, we can't reduce the edge rise time to >>10us because we need to operate the I2C bus quite fast to get the update rate required for closed-loop operation. My guess is that this noise originates internally to the DCXO, so we're stuck with it. Having said that, it might be worth sending the data we have to Ti and asking them about it. @WeiDaZhang are you happy to post something on the TI engineering forums? |
|
TI suspects that my hand soldering may have damaged the chip. |
I'll be surprised if that's really the case, but using the TI eval board does sound like a good plan -- it would be nice to have a second option besides the Si549. |
Initial phase stability using a fast scope:
|
How good is good enough? Depends on the use-case, but a common one would be producing RF at around 300MHz (maybe a bit higher, but that makes the maths nice!). Generally, the target for QIP would be phase stability better than 1deg, with 0.1deg being ideal, measured over say 24 hours ish. 1deg=10ps. So, we're looking for stability better than 10ps, and ideally around 1ps (0.1deg). For direct microwave production at 3GHz (e.g. calcium) assuming we want to generate our LO from the WR-distributed reference, that goes to 0.1ps-1ps, which is much more challenging. Not clear if we can hope to do that via WR... Still, these results show that the WR PLL noise + drift, measured under realistic conditions, is good enough to clock a RF module like Sayma from. |
There is recent development on low jitter version of WR switch. Some people are also experimenting with better oscillators, but for us it would be an overkill. |
@gkasprow thanks for the cool links.
That's a pretty low bar :) From my end, the idea was to do as well as we can with of order $20 USD of parts and about 1cm^2 of board space, so the Wenzel oscillator is not an option. I'd be interested to see the phase noise plot for that, but my guess is that it's heavily into the regime of decreasing returns in terms of dBc/$$$. Cool project though. If you compare the specifications for the WR low jitter VCXO with the Si549, they are basically identical. Except the Si549 performs better at high frequencies, which is to be expected since it has a higher output frequency. And then you have to take into account the fact that using a VCXO + DAC you have to watch out carefully for any analog noise associated with your DAC/reference/pickup/ground loops/etc, where as the Si549 takes care of that all internally. So, I'll be surprised if their results are actually as good as ours once the PLL is locked. (Not to mention the fact that, IIRC, the PLL used by WR is only second-order so there is quite a lot of noise feed-through from the DDMTD at middling offset frequencies. In contrast we're using a 3rd order filter which gives much nicer performance). Basically, it's nice, but I think our way will work better ;) |
@gkasprow why do they use 25MHz VCXOs instead of a 125MHz oscillator? I never understood that. Is it to do with the frequencies where the best VCXOs are available, or some other constraint? |
I'd be interested to hear what they think of it in case I've missed anything! |
@hartytp This design choice was taken long time ago, around 2008. |
The final comment I'll make on this while I'm nerding out on time distribution is that the Si-Labs got the idea right with the Si549: use an XO + frac-n pll instead of a VCXO. XOs give way better performance for a given cost point than VCXOs. Plus, you scrap all the hard analog bits associated with a low-frequency PLL and get a much cleaner system. |
@gkasprow that makes sense. Well, it'll be interesting to see how well this works out on Kasli. |
AFAIR we tested such fractional PLL in CBM clock synchronisation and there were some issues with Silabs chips. I will ask my coleague about it since he was playing with firmware at the time. |
@gkasprow let me know what he says, because we haven't found any issues so far with the Si549. I wonder if they were using some other chip? |
We were using Si5338 |
For CBM experiment we built clock synchronisation using AFCK and SI570, but AFAIR no detailed jitter measurements were done. Our PhD student was implementing WR node on SI570 in similar way that you want to do it with Si549 |
How far did he get? We looked at the Si570 when we started out. I can't remember why we decided it wasn't an option. The noise is worse that the 549, but I think there was some other reason as well. FWIW, I think the two are pin/footprint compatible, and have very similar programming interfaces, so it's easy to swap them.
Wow! That's a lot more complex than the 549/570. I'd have to read the data sheet quite carefully to figure out whether our scheme would work with that. |
@mguminski Can you write a few words about your results? |
@cjbe @WeiDaZhang and I have been looking at the timing stability of DRTIO on Kasli (all comments apply equally to Sayma of course).
Expectation:
E1. The latency between TTLs on Kasli DRTIO slaves and TTLs on the master should be deterministic and fixed between power cycles. Equivalently, the phase relationship between the master clock, measured at the SMA input), and the recovered RTIO clock, measured at the MMCX on the slave, should be fixed between power cycles.
E2. The latency between TTLs on Kasli DRTIO slaves and TTLs on the master (or, equivalenty, the phase relationship between the two clocks) should be stable to <<1ns over standard operating temperature range (say, something like 10C-20C).
E3. DRTIO is a basic piece of ARTIQ infrastructure, and the better it is (within reason) the more uses it is likely to find. Having a high-quality way of distributing time (e.g. clocks for data converters/TDCs etc) over fibres with good stability is a high-value feature for ARTIQ, so we should try to make the performance "as good as reasonably possible" (without overcomplicating our designs). i.e. let's not shoot for something that just barely meets our most lax specification if there is a way of doing significantly better without too much additional cost/complexity.
Observations:
O1. In @cjbe's measurements, the latency was not constant between power cycles. As @jordens has independently pointed out, this appears to be expected for the Si523, data sheet says
Input to output phase skew after an ICAL is not controlled and can assume any value
.O2. The phase of clock output on the slave is extremely unstable over temperature. Touching the Xtal makes the slave clock move w.r.t. the master by multiple cycles of the 150MHz clock. The phase also wanders around noticeably even when Kasli is left alone in standard lab conditions and running at constant duty cycle etc.
Discussion:
AFAICT, DRTIO uses the Si5324 as follows:
Comments:
C1. Since we are very sensitive to the performance of the Xtal/XO on Kasli, we should be using a high-quality one. Kasli currently has a very cheap Xtal. We replaced this with a high-quality Crystek 125MHz XO (different package, dead-bugged on) and found that the phase stability of the Si5324 clock was significantly improved -- although the phase fluctuations of the 150MHz clocks were still large and easily notable by eye on a scope.
C2. Running the PLL at such a low frequency is unlikely to be a good idea from a noise/stability perspective. Using the 125MHz cyrstal, we were able to run the PFD at a higher frequency (around 2MHz IIRC). This made a significant improvement in the phase stability although, again, not enough to solve the problems. We should aim to run the PFD in our "jitter cleaner" loop at the highest frequency possible to minimize noise/drift.
C3. The Si5324 recommends using a reference which is not related by a rational number to our reference/output clocks. AFAICT, that's to minimize in-band-spurs. AFAICT, that recommendation is not good advice for us, since the problems introduced by running the PFD at a really low frequency are much worse than a few spurs (which are generally rejected by subsequent PLLs anyway).
C4. As @sbourdeauducq has pointed out, it may be possible to fix the phase non-determinism of the Si5324 by wrapping it in an additional PLL using the FPGA (although, that somewhat defeats the purpose of having the Si5324 in the first place).
Recommendations for DRTIO v2.0:
R1. Replace the Si5324 with a proper PLL! This could either be an analog PLL or a digital one like WR. The optimal choice will depend a bit on the measured noise and stability of the CDR clock from the FPGA. We'll aim to measure that soon.
R2. Use a TCVCXO for the flywheel oscillator. The best TC(VC)XOs are generally at around 10MHz-25MHz, so we should probably follow WR and use a decent 25MHz TCVCXO for the flywheel oscillator.
R3. Run the PLL PFD directly at 25MHz rather than some low frequency
R4. Use a reasonably high-order loop filter (we used 3rd order for exactly this reason on the clock mezzanine that Weida and I designed).
R5. If the TCVCXO is well-enough specified, we may be able to do our "hitless switching" by just setting the control voltage to mid-range during link initialisation (someone would need to check the numbers). That simplifies things a lot.
The text was updated successfully, but these errors were encountered: