3U DDS (URUKUL) discussion

[gkasprow]

I did early estimation of available board area.
It seems that we can easily fit 4 channels on 100x160 Eurocard.
There are several questions which need to be answered before we continue with schematics:

• I assume we don't need clock shaping circuit in AD9912 and we will route DAC output (pins 50,51) directly to the circuit copied from Allaki.

• do we need to measure output level as it is done in Allaki design? Should I remove AD8363 or add some I2C/SPI ADC ?

• we need to control several IO lines per channel. Which one can be common and which can be common?
  I assume that DDS Reset, S4...S1, IO_update, CLKMODESEL can be common while PWRDOWN should be individual. For output channel switch, we would need individual RFSW control. Attenuator can have common LE, CLK and RSTn.

• How we distribute SYSCLK? I assume we have SMA input on front panel and some other connector (i.e. mmcx) on the other side of PCB to route via jumpers to Kasli clock distribution. I assume we will use 1:4 clock splitter and input circuit as it is on Sayma RTM.

• I copy loop filter from devkit

• What about SPI control? I can connect 4x HMC542 to have 32bit data and one CS. I can also connect them as 2x 16bit but additional CS will be needed.

• the same I can do with SPI registers (SN74LV595) to have 32it data.

• I will use 8 output address decoder ('138) to connect to all CSn lines. We can route additional CSn or treat state 000 as none SPI selected. SPI address decoder map:
  000 - no slave selected
  001 - DDS1
  010 - DDS2
  011 - DDS3
  100 - DDS4
  101 - attenuators CS
  110 - serial register CS
  111 - reserved

• LVDS assignment:
  LVDS1: SCKI
  LVDS2: SDI
  LVDS3: SDO
  LVDS4: SEL CSN3
  LVDS5: SEL CSN2
  LVDS6: SEL CSN1
  LVDS7: IO_UPDATE ??
  LVDS8: RESET ??

• for supply I want to use similar double output converter modules as for Sayma RTM + LDOs regulating down to +5V and 3.3/1.8V

• do we plan to compensate for clock delays? We can use one of clock outputs and route it back to Kasli phase detector.

• below is initial component placement on PCB. Each channel will have individual shield.

[hartytp]

@gkasprow Looks nice!

@jordens How do you want to manage the review process for Urukul+ Kasli? Do you want feedback from us at this stage? I can guess the answers to most of these questions, but I won't add my tuppence worth unless it's wanted!

[hartytp]

Can you remind me what the plan for clock distribution is? Is the idea that we'll distribute a ~100MHz clock to Kasli + Urukul (via SMAs/backplane/CDR from DRTIO) and then use local PLLs to generate the DAC clock? Will you use the PLL inside the DAC, or add an external IC (HMC830 or similar)?

[gkasprow]

At the moment the clock can be taken either from front panel SMA or Kasli clock distribution. I assume we use the PLL in DDS chip.

[dhslichter]

While we are in the early stages: we might consider using AD9914 instead of AD9912, for two reasons:

• profile mode
• potential for higher clock frequency (thus higher output frequencies).

Particularly for DDS chips which are programmed over serial, profile mode is very handy because it can allow fast frequency/phase/amplitude hopping with deterministic timing at faster timescales than one can reprogram everything over SPI. We use this methodology for our microwave generation in the magtrap. The user does not need to use profile mode if they don't need/want it, but it can be handy e.g. for hopping around the hyperfine manifold.

Also, a 1 GHz clock on the DDS limits us to ~400 MHz (assuming we put reconstruction filters etc on the Urukul board). If you go with the AD9914, you can run at e.g. 2 GHz clock frequency, which lets you go up to ~800 MHz output frequencies, which is handy for some groups that drive 600 MHz AOMs, for example. We run ours at 2.4 GHz.

I would use an external PLL for the reference clock generation (e.g. HMC830, something already in Sayma so the gateware will be written) because I think they tend to be better/more flexible than the internal DDS PLLs.

[sbourdeauducq]

Third reason: we already know the main silicon bugs of the 9914, but not those of the 9912.

[sbourdeauducq]

That being said, if someone from ADI can confirm that the 9912 is bug-free, I would prefer it.

[gkasprow]

Take into account that ad9914 cost is 188$ while AD9912 is 61$. This is the most expensive part on the board. It consumes 3W of power while AD9914 consumes about 1.3W. 12W (4 channels) of power dissipation just for DDS chips is quite a lot. I have devkit with 9912 so can test some functionalities.

[dhslichter]

This cost difference is not zero, but given the additional functionality I'd argue it's definitely worth it. $120 per channel is a very small cost in the types of setups these boards would be used in. The power dissipation is definitely an issue to be considered; however, with forced air this can certainly be handled (and this is in fact what we do with our hardware).

The decision that needs to be made here is basically about how capable we want/need Urukul to be. Is the target market just to do simple CW signal generation for AOMs? Or are people going to use these to generate signals for pulsed setups (with some modulation or switch afterward)? What kind of flexibility would be desired?

From my standpoint, the AD9914 gives the option for substantially broader use cases, at a pretty low cost ($500/board extra, assuming a 4-channel board), and with no additional complexity on the gateware/programming side.

One of the main issues I see is that people may want a solution which can do pulsed operations, or fast frequency/phase switching, but may not be ready to commit to the full Sayma system in the crate (which is very large and expensive compared with the Kasli/EEM ecosystem). By changing the Urukul to the AD9914, one would be able to bridge this functionality gap more readily, and provide a more "mid-grade" performance solution.

[gkasprow]

For me the only difference at the moment is that I already purchased devkit with 9912. To do quick tests with 9914 I would need another devkit for 700$. They have different interfaces so cannot connect my board with 9914 to 9912 devkit.

[hartytp]

@dhslichter Particularly for DDS chips which are programmed over serial, profile mode is very handy because it can allow fast frequency/phase/amplitude hopping with deterministic timing at faster timescales than one can reprogram everything over SPI. We use this methodology for our microwave generation in the magtrap. The user does not need to use profile mode if they don't need/want it, but it can be handy e.g. for hopping around the hyperfine manifold.

How do you want to connect the profile pins to Kasli? Bearing in mind that the IDC only has 8 LVDS pairs to control all 4 DDS channels, we'd need to use multiplexed IO. Depending on how we do that, it may not end up being that much faster than reprogramming the DDS over SPI.

12W (4 channels) of power dissipation just for DDS chips is quite a lot.

Currently, we're budgeting for 6W (0.5A of 12V) per EEM. If we go for the AD9914, we should probably increase that to more like 1.5A per EEM. That would mean increasing the current rating of the power connectors on Kasli/VHDCI carrier (currently 5A IIRC).

[dtcallcock]

How about giving users the option of using two IDCs? In that case the timing-critical profile selects and RF switches would have dedicated lines. It would also mean that you only have to draw 0.75A per IDC.

Even in the AD9912 case, it's not clear how the RF switches are controlled.

[jordens]

@gkasprow
Yes. Output circuitry is allaki directly. No output power sensing. No ad8363 etc. AA filtering (and Nyquist zone selection) to be done with the allaki filters. DDS reset, io update are common, s1-4 and clkmodesel are common and can be resistor jumpers. Pwrdown can be individual resistor jumpers. RF switches should be an individual eem line (maybe an addressable latch with the DDS cs?). Attenuators can be daisy chained with a single cs.
More later...

[hartytp]

@jordens +1 to all of that.

Also FWIW, we'd much prefer the AD9912 to the AD9914. We would also really like this design to run from a single IDC.

[hartytp]

@gkasprow When you produce the prototypes for this board please can we buy 5? We'd like these asap, and don't mind using prototype hardware...

[gkasprow]

@hartytp So we will make at least 6 prototypes of DDS? One stays at WUT, others go abroad. Anybody else wants first prototypes?

[hartytp]

@gkasprow Thanks! That sounds good. Just send me the quote when you're ready.

[dhslichter]

I agree with @dtcallcock; my vision was that you have two IDCs to cover the 4 channels (allowing for profile select, plus fast rf switch open/close).

[dhslichter]

I doubt NIST will be buying any of these boards if the AD9912 is used. That is not intended as an attempt to influence the design, but rather just informational -- it would just be a major step below what we already have in terms of capability. I also think that one could end up making two board designs: an Urukul "basic DDS" board and another board with higher performance (fast switching, higher clock frequency) featuring AD9914s and two IDCs. This of course doubles the design and prototyping work and requires additional funding from someone else. I do think that the market for fast switching and higher output frequencies is substantial, and that it should be an important consideration for developing an ARTIQ hardware ecosystem in general. I recognize that Oxford has their own set of priorities here, and that they are focused on that.

[gkasprow]

@dhslichter We can always make two wersions of DDS board. One - simplified with minimal cost, second - high performance used with two IDCs.

[dhslichter]

@hartytp just curious, why the "much prefer" for AD9912?

[dhslichter]

@gkasprow you read my mind :)

[gkasprow]

@hartytp @dhslichter @jordens Guys, we can make variant design.
We have some spare area and can place both DDS chips on both sides of the board.
When you order board you simply specify the variant
So we have same PCB and 2 versions of pick&place files which result in different board assembly.
So ones who want 9912 will get it mounted with 1 IDC, others who want 9914 will get the same board but with other chip and two IDCs.
This adds some work so additional founding would be very welcome. I'd like to buy 9914 devkit to fully characterise the board.

[sbourdeauducq]

This sounds quite complicated. Can we please keep the Kasli ecosystem simple?

[gkasprow]

@sbourdeauducq This is low cost alternative to having two boards design. We can even call them differently :)

[hartytp]

@dhslichter Maybe "much prefer" is too strong, but prefer nonetheless. To explain our priorities:

We are setting up an experiment which will have two ion traps, each with Ca + Sr. We will use Sayma for qubit manipulations, which require pulse shaping etc. For everything else (cooling AOMs etc) we hope to use Urukul, with Novogorny for noise eating (initially in software, but eventually in gateware).

Given that we need 60+ channels of RF for this, both cost and power consumption are important to us (our labs' AC is only 10kW). As the AD9912 can do everything we would want, and is cheaper and less power hungry, we prefer it.

Beyond that, I think there's a lot of advantage to keeping the Kasli ecosystem as simple and low cost as possible and avoid adding features that aren't actually needed.

IMO the profiles on the AD9914 aren't that useful since one can reprogram the DDS over SPI in ~1us, which is fast enough for most cases. Using up two IDCs for this design is a pain: more ribbon cable wiring (or backplane sockets taken up); and reduces the number of these one can run from a Kasli. This is particularly true for our initial experiments, where we want to run as many of these as possible from a single KC705.

Edit: in a similar vein, I'd be pro using the DDS's internal PLLs rather than an external synth IC unless it's really necessary...

[sbourdeauducq]

If we end up with both 9912 and 9914, I would prefer two board designs, and keep each one as simple as possible.

[hartytp]

Separating two different issues here: (1) whether we multiplex the RF switch TTLs, or just run them directly from the IDC; and (2) how we update the DDS chips.

(1) While I agree that @gkasprow's proposal would work in principle, I also agree with @sbourdeauducq and @dhslichter that it wouldn't fit in well with the way we would use this hardware in the lab. I think the extra complexity it introduces isn't worth saving the IDC. So, I'd prefer to run the switches directly from the IDC (1 LVDS pair per switch).

(2) As I mentioned above, our main use case for this hardware would be running the DDSs in "constant update" mode as part of an intensity servo. This is a bit messy because the user can only change the DDS frequency/phase/amplitude at fixed times, but there isn't a way of getting around that without introducing an extra amplitude modulator of some form after the DDS. In practice (aside from the servo loop), the user would only update the DDSs while the switches are shut, so that they aren't sensitive to the DDS update timing.

If we do put 2 DDSs on the same SPI bus then the delay between updates is ~3us, which is a bit long for some of the experiments we want to do. So, it might be worth considering using DDR to update both DDSs at once. As the DDSs are identical, this doesn't introduce the kinds of corner cases that @dhslichter is worried about.

[gkasprow]

@hartytp @dhslichter So we could use double channel DDR SPI bus to control 4 DDS chips at once. This would consume 7 of 8 LVDS lines. The last one would be used for UPDATE signal.
We still can have a compromise here - we can route switches control both to TTL register and to dedicated LVDS signals from second IDC.
So one who wants to have real time control of the switches can use second IDC but one who can live with slower control but need many DDS channels would use only single IDC and serial register.
The LVDS receivers we use have bus hold capability which means they ensure defined state when not connected to the source.
The outputs from LVDS receivers would be ORed with register outputs to make sure that both sources can control the switches.

[gkasprow]

@dhslichter Another issue is with AD9914. With second IDC connector, we will use 4 signals to control the output switches and remaining 4 to switch DDS profiles. But each DDS has 3 profile inputs. How would you like to connect them? One LVDS line per each DDS chip connected to one profile input or some fast serial register that handles all 4 chips * 3 inputs = 12 signals.

[gkasprow]

@jordens @hartytp @dhslichter
What front panel LEDs should indicate?

• RF switch positions
• 9914 profile
• user defined functions (routed to TTL register)
• Overtermperature / Power Good

[gkasprow]

@jordens @hartytp @dhslichter
I published still early version of schematics and semi-random component placement. So far only SDR SPI but with 2 channels is implemented.
I did complete power budget estimation. It's going to consume 12W in worst case for 9912 and almost twice more for 9914. I implemented hardware over-temperature protection.
I added 9914 option - it seems there is no compromise needed and we can fully implement two variants.
To finish schematics we have to decide about SPI mode (SDR vs DDR) and profile pins routing for 9914.

[hartytp]

@gkasprow Thanks for doing that @gkasprow! The schematics look really nice.

Should we consider using a different output amplifier? AFAICT, there's no need for a special low phase noise amp here (e.g. we're using the DDS's internal PLLs), so we could go for something which dissipates less power and is a bit cheaper.

Not sure what P1dB people want for this design -- although, if it's used with the RF PA, which has 40dB of gain, then even 0dBm is enough -- but, something like a AG203-63, which can run directly from a 3V3 rail might be a good choice. This is a little cheaper and would shave 1.6W off the design, which would make thermal management a bit easier.

Edit: anyway, there are plenty of good options for this. @jordens if you specify a P1dB and lowest frequency of operation then we can make a choice.

[hartytp]

• If there is enough room on the front pane without cramming things, could we have a clock output as well. That way users can daisy-chain the clocks between modules. There are plenty of ADCLK948 outputs spare!

• Also, are we planning to float the SMA grounds by default? Or, should we just ground them, but leave the capacitor pads there so that they can be floated later if needed?

• Can we scrap the LTC6957-1? We've made some phase noise measurements with the ADCLK948 and a +10dBm sinusoid input at 100MHz and the performance is pretty similar. As this board isn't aiming for ultra-low noise, this part seems redundant IMO and removing it takes off another 200mW. I'd also recommend leaving the balun DNF by default and replacing it with a 0R resistor in parallel with it.

[hartytp]

What front panel LEDs should indicate?

How about a single bi-colour LED: green power/temp good; red over-temperature.

I don't see the point having LEDs to indicate switch/profile statuses.

[gkasprow]

@hartytp
OK, I will add 0R resistors that connects SMA to GND. They can be removed when somebody needs isolation.
I have place for SMA clock output, I will add it.
OK, I will scrap LTC chip.
Two LEDs are easier to control so I'll leave as it is.

[hartytp]

@gkasprow Thinking more about thermal management: the LVDS interface takes up quite a lot of power in this design. Is there any way we can simplify it/reduce the power a bit without sacrificing performance?

• What default output filter do we want for this? I assume a 400MHz LP response using the MCL pads?

• Once the design has been finalised, please could you annotate the schematic with the RF output power and bandwidth?

[hartytp]

Another point for consideration: we can potentially shave another Watt from the power budget by replacing the ADCLK948 with an ADCLK846. This doesn't provide a mux, but personally I'd be happy with solder jumpers for that. My guess is that users won't switch clock sources often at all, so the solder jumper is fine.

[gkasprow]

@hartytp we can also use CDCLVD1208RHDT which consumes same amount of power but has input mux. Cost is also the same.

[hartytp]

@gkasprow From a quick look at the data sheets, that looks fine to me. Comments:

• Phase noise is better than the AD9912 with the PLL on
• Output frequency is 800MHz max, so we wouldn't be able to drive the AD9912 with a 1GHz external clock directly (not that we need to IMO).
• Users also need to be aware that although we using 3V3 LVPECL in most places, the clock output SMA from Urukul is LVDS (not that it will make a difference in most cases). Also, the phase noise will be slightly degraded compared with the ADCLK. This isn't a problem IMO, but users need to be told to take care when daisy-chaining an Urukul to a Sayma...

[gkasprow]

@hartytp Generally, PECL devices have much better characteristics than LVDS ones due to much lower output resistance and sharper edges. But power consumption is real nightmare. This is the main reason why LVPECL is used mainly for clock distribution.
Is there any source of 1GHz clock which we would need to route to input of the clock buffer?
The output is not LVDS since it is single ended:) It is simply low amplitude AC coupled signal.

[hartytp]

@gkasprow Do you think we'll get decent performance using only half of the LVDS signal for the clock daisy-chaining (after including a bit of loss in cabling etc)? Or, do you think we'll need baluns on the clock input and output after all? If there is a doubt, then maybe we should add DNF pads for the baluns.

[dhslichter]

Can the clock daisy chaining circuit have a powerdown? It would be nice to have a reasonable quality clock available if people want, but be able to DIP switch/jumper to disable (and thus get rid of the power dissipation too).

Waiting for @jordens to weigh in but to my mind, if you are driving AOMs, you'll need another power amp anyway so you might cut down on power dissipation by doing away with the output amp altogether, just using a reconstruction filter (user solderable) and the digital attenuator and switch. This assumes sufficient gain in the power amps though (probably at least ~40 dB would be required).

Most of the DDS chips will work with a sinusoidal input clock and a balun. How about just using a passive splitter and requiring the user to supply all clocks externally? This seems to me to be not so different in terms of the complexity of external wiring (you are still not sending any signals over a backplane, all on front panel), but it cuts down on power dissipation, will not degrade the clock quality, and can work for clocks up to 1 GHz.

[hartytp]

Waiting for @jordens to weigh in but to my mind, if you are driving AOMs, you'll need another power amp anyway so you might cut down on power dissipation by doing away with the output amp altogether, just using a reconstruction filter (user solderable) and the digital attenuator and switch. This assumes sufficient gain in the power amps though (probably at least ~40 dB would be required).

Agreed, if one has a PA with decent gain then the output amp on Urukul is not required. I'd be happy scrapping it.

[hartytp]

Most of the DDS chips will work with a sinusoidal input clock and a balun. How about just using a passive splitter and requiring the user to supply all clocks externally? This seems to me to be not so different in terms of the complexity of external wiring (you are still not sending any signals over a backplane, all on front panel), but it cuts down on power dissipation, will not degrade the clock quality, and can work for clocks up to 1 GHz.

I'd definitely prefer to keep the clock buffer IC on this design.

1. The CMOS fanout IC will dissipate < 150mW (in contrast, the ADCLK948 would have been closer to 1W IIRC). IMO, that's low enough not to worry about.

2. With passive 4-way splitting of the clock, the input clock has to have a reasonable amplitude to avoid seriously degrading the DDS noise floor. Assuming one has a few of Urukuls then one would need a beefy fan-out IC somewhere anyway to get the clock signal level high enough for passive splitting. In that case, it makes most sense to put the fan-out on Urukul itself rather than having to have it in a separate box. Edit: NB signal-level issues are particularly acute if we distribute a 100MHz clock, rather than a 1GHz clock, as the edges are slower. With a n-way-split 100MHz clock, it's pretty easy to get surprisingly poor DDS performance when the signal level drops.

3. Having a passive splitter also prevents daisy changing of Urukuls as the signal level will quickly get too low. My hope is that we can use the recovered clock from Kasli DRTIO as our 100MHz source for all our AOM DDSs. For that, we need a single clock output SMA on Kasli and the ability to daisy-chain it to the Urukuls.

4. The clock quality from the LVDS can still be very high quality: better than the DDS + PLL noise floor; better than the recovered clock from Kasli; and, plenty good enough for cooling/optical pumping/state preparation/etc that Urukul will be used for. Users just need to know not to try to use it to supply an ultra-low noise clock for Sayma.

5. True, the LVDS IC doesn't go up to 1GHz. But, do we really have a use-case for that? Otherwise let's not worry about it. The DDS + PLL noise floor is still really quite good and will be fine for any application I can imagine using Urukul for. NB 1GHz clocks also weren't supported by Greg's original design as the LTC IC only goes up to 300MHz IIRC.

Can the clock daisy chaining circuit have a powerdown? It would be nice to have a reasonable quality clock available if people want, but be able to DIP switch/jumper to disable (and thus get rid of the power dissipation too).

Assuming one can just leave the SMA un-terimated when not in use, the clock daisy chaining circuit doesn't dissipate any power if it's not used. Thus, there's nothing to do here. Even if one does terminate it, the extra power consumption of a single LVDS line is pretty negligible for this design. I'm not sure that's the right place to try to optimise.

[jordens]

I'll catch up on the remaining pile of comments later. Thanks to all for participating in this!

AFAICT we are heading for two board version here. The DDS requirements for Oxford and optIclock are already different. And the IDC connector usage is as well. I have updated the current idea as far as I could on the wiki:

https://github.com/m-labs/sinara/wiki/Urukul#idc-usage

Let's try to discuss the individual topic areas in individual issues, like @sbourdeauducq did with #193

[jordens]

Let's retire this. We have:

#193
#195
#196
#197
#198
#199
#200
#201

[hartytp]

@gkasprow What's the timeline for finishing the designs for Urukul & Kasli and ordering prototype hardware?

[gkasprow]

So far they are not funded yet, I mean the projects officially don't exist at WUT. And we are focused on other hardware. I've just published updated Kasli schematics which seem to be complete. Once we accept them, can begin with routing.