Kasli specification

[jbqubit]

@jordens is tracking design proposal/specification for Kasli on the wiki. Let's discuss PCB_kasli specification here.

https://github.com/m-labs/sinara/wiki/Kasli

How Kasli might interface to peripherals is discussed in #99.

[jbqubit]

At teleconference today we discussed freezing Kasli electrical interface so that there is compatiablity with Metlino-VHDCI #99. Did I get this right?

• Kasli uses +12V supply to Kasli
• 8 IDC connectors on Kasli
• each IDC connector has
  • 16 pins for general purpose IO (eg 8 LVDS). Some of them will be dedicated to SPI as secondary function
  • 2 pins for I2C
  • +12V supply, 100 mA max. per IDC. 5V or any other will be generated locally on every IO board
• Artix-7 FGPA

@gkasprow Which student is working on Kasli? Please add to #19 and github.
Sebastien already added him (@marmeladapk)

[gkasprow]

I edited specification above.

[dtcallcock]

@jordens Can we have subheadings for each proposed mezzanine on the wiki?

Currently the wiki mentions:

• 16 TTL drivers
• 40 DAC channels
• CameraLink connector

But some other options were mentioned in #99 including SPI interface using LVDS over RJ45 and isolated TTLs.

I would also like to suggest a low cost synthesiser for driving non-critical AOMs/EOMs (ie, where you don't need deterministic phase control or sub-kHz resolution). Would ideally also have pre-amp/switch/attenuators/filters so that only an external power amp is needed. Could be based on the ADF4351 (an $8 synth-on-a-chip) like the SynthNV is.

[jordens]

@dtcallcock sure. Do you want to edit that yourself?

[dtcallcock]

@jordens I have had a go at reorganising the wiki page based on these discussion threads and the work done so far by @gkasprow and @marmeladapk. You guys should fill in any blanks and correct where I've misunderstood what you're up to. I'm also a little unsure when exactly we put things on the wiki. Do we do it right away and then discuss in the issue or suggest in the issue and move to the wiki one we've reached some kind of consensus?

I also have a few questions:

• The /ARTIQ_ALTIUM/PCB_3U_BNC folder contains two .pdf output files, one a few weeks older than the other. Is this a mistake?

• What voltages go over the IDC? Some schematics have +5V and some have +3.3/+12V. Also what are the current specs?

• What is the 5th RJ45 on 3U_RJ45 going to be used for?

• Can we use thread-lock barrel connectors on the boards? These are much more robust in a lab setting and are backwards-compatible with non-locking plugs for those that don't care. Example parts are connector and plug.

• Should the barrel connector input on PCB_3U_VHDCI_Baseboard have some minimal EMI filtering?

[gkasprow]

I deleted wrong pdf file.
We use 12V and 3.3 for EEPROM readout. It is available even when the box is not powered.
The fifth R45 was meant as I2C link for slow devices, but I don't have strong opinion about it.
Yes, we can use such connectors. Just add an issue for Michal.
We can install CMC and EMI filter but such should be already in the power supply.

[dtcallcock]

We can install CMC and EMI filter but such should be already in the power supply.

The power supply was originally required to be "just some random wall wart". In my experience these are often horrifically noisy and have poor regulation. Should we add circuitry to deal with that or just change the spec to "reasonably clean isolated +12V supply".

The fifth R45 was meant as I2C link for slow devices, but I don't have strong opinion about it.

I'd probably rather leave this off due to I2Cs lack of robustness to added bus capacitance and possible voltage level incompatibilities. SPI will be better for off-board devices. If remote I2C devices are really desired though, we could utilise bus buffers like these from Intersil.

There is also a high likelihood of accidentally plugging in one of the adjacent LVDS cables or vice-versa, though this could be avoided by using a different connector.

I deleted wrong pdf file.

The .pdf there now is the one that still has the old 5V power rail.

[gkasprow]

Indeed, there is wrong schematic version still with 5V bus.

[gkasprow]

fixed

[jordens]

@dtcallcock looks good. Thanks! I am all on board with the additions/changes and have added a few details.

• I am okay with saying "reasonably clean 12V". But IME there are plenty of examples where standard wall wart supplies have been without problems. E.g. in the USRP. I think we can aim to replicate that.
• Supply-wise I don't care. Greg should probably choose what's best/easiest.
• The thread lock barrel connectors (or similar Bajonett style things) are a good idea.

[hartytp]

Just to check: the current plan is that each Kasili communicates with/receives time from the root Metlino via a fibre link (one fibre per Kasili), and not via a backplane, right?

[jordens]

Yes. The VCO/DDS synthesizer extensions would receive their reference clock individually.

Another issue is experiments where the set of Kasli extensions is sufficient. In those experiments Metlino (plus Rack, clock RTM) seems pointless and too powerful/big and requiring a separate box/piece of hardware just to be able to talk to Kasli is troublesome.

[hartytp]

Yes

@jordens Good, that seems like the correct solution to me. IIRC, the Metlino currently has 3 SFPs, which is extensible to 7 by adding an FMC-SFP board. However, given the number of different functions currently planned for Kasili, I can imagine using these all up pretty quickly. What's the plan for extending this further?

The VCO/DDS synthesizer extensions would receive their reference clock individually.

Sure, if that's useful, although IMO it's a bit overkill: IIRC, the recovered clock should actually be pretty good noise-wise and, I assume that one be better off use Sayma for any ultra-low noise purposes.

Another issue is experiments where the set of Kasli extensions is sufficient. In those experiments Metlino (plus Rack, clock RTM) seems pointless and too powerful/big and requiring a separate box/piece of hardware just to be able to talk to Kasli is troublesome.

What's your plan for this use case?

[dtcallcock]

The VCO/DDS synthesizer extensions would receive their reference clock individually.

Sure, if that's useful, although IMO it's a bit overkill: IIRC, the recovered clock should actually be pretty good noise-wise and, I assume that one be better off use Sayma for any ultra-low noise purposes.

I can see how you might want this as an option for a DDS. For the synth though, can the Kasli FPGA just provide the clock over one of the LVDS pairs? I'm not entirely sure what the Artix clock output capabilities are though.

[jordens]

• You get a bunch of SFPs with each Sayma. You can also hook up another (slave) Metlino to the (master) Metlino. And we can daisy chain Kaslis ad infinitum. All those require DRTIO switch support and consensus about one preferred option that would be implemented.
• Distributing the clock from Kasli to the extension boards is tricky. There is little space on the frontpanel, doing it inside makes assembly even harder than it is now (now you already have to disassemble the crate when you change/add/remove extensions), and there is little space on the PCB for SMA connectors.
• I would be afraid to run a reference clock over uncontrolled impedance, unshielded, IDC connectors, even for a simple Synth. We would also need a clock fan-out as the Artix outputs are very noisy. I am fine with putting an XO on the extension boards as a low complexity alternative to a reference clock.

[hartytp]

You get a bunch of SFPs with each Sayma. You can also hook up another (slave) Metlino to the (master) Metlino. And we can daisy chain Kaslis ad infinitum. All those require DRTIO switch support and consensus about one preferred option that would be implemented.

Good, that works for me. Is the plan for each Kasli to have 2xSFP to allow daisy chaining then?

[jordens]

Right now I don't see a different use for them.

[hartytp]

I would be afraid to run a reference clock over uncontrolled impedance, unshielded, IDC connectors, even for a simple Synth.

Depends on what the reference clock frequency is. For, say, a 100MHz reference clock frequency, routing board-board via pin header will be fine.

IIRC, the AD9912 has a pretty decent clock multiplier (PLL w/ integrated VCO), so there's no need to supply a GHz clock directly, unless one wants to push the noise performance (in which case, the Sayma might be a better choice anyway). Otherwise, one can always use something cheap like an AD9576 to do the same job.

IMO, it'd be great if these Kasli extension boards could be used without needing an external GHz frequency source...

[hartytp]

@jordens My previous post was ambiguous, I meant "Is the plan for each Kasli to have 2xsSFP (which would allow them to be daisy chained)?" Answer: "yes", so I'm happy with this plan.

[hartytp]

We would also need a clock fan-out as the Artix outputs are very noisy.

Won't there be a spare output from the Si5324 we can use to supply the reference clock?

[dtcallcock]

Also, should we add an SMA front panel input and clock fanout chip to VHDCI carrier so it can drive the backplane clocks?

[gkasprow]

We can place IDC connector routed to 96pin connector that follows Kasli connectivity. In this way we could either use 8 IDCs or route unused ones to 95pin connector (DIN 41612) using short IDC jumper.
To make both boards fully compatible, we would need to addd clock fanout as well.
We can implement it in next revision of VHDCI board.

[hartytp]

Thanks for that guys.

Maybe I'm being a little slow here, but I'm still not entirely clear about what exactly we're trying to achieve with this backplane. I can think of three potential reasons to have a backplane:

a. Reducing wiring complexity (no ribbon cables to modules, no external SMAs needed to distribute clocks)?
b. For high-speed digital communication that can't be sent over the ribbon cable
c. Some kind of multi-point bus link to allow communication between extension modules without going through the carrier

(a) Is potentially quite nice, but IMO probably not worth the effort of designing and maintaining a backplane. Also, as previously discussed, if we use 96 pin connectors then a carrier can only drive 4 extension modules, which isn't convenient: each carrier has 8 expansion headers, so the BP wastes some connectivity; and standard racks will hold far more extension modules than this so we end up with empty slots or multiple backplanes per rack or something.

(b) and (c) are what @gkasprow suggests. But, AFAICT, no one has suggested a use case for them yet. We can't (shouldn't) specify a backplane for undefined purposes. So, if this is our motivation, maybe we should just leave the 96-pin connectors NC until we have a more concrete use case to design the BP around.

Whatever the answer, it would be good to agree on our motivation/goals for this, and to put that on the Wiki...

[gkasprow]

Actually, it's hard to fit more than 32 LVDS pairs (equivalent of 4 IDC connectors). They already would occupy 64pins.
Add 6 or 8 clock outputs and we have 80 pins. Add 4* I2C and we have 10 pins left for grounds and supplies which is absolute minimum.

[dtcallcock]

If a) was the priority but only worth doing if we could drive all 8 extensions then we could switch to a 160 pin connector on the Kasli/VHDCI carrier (would also need the I2C switch and/or clock fanout moving to backplane). I think this only makes sense if the BNC/SMA/RJ45 extensions were also made full length to connect to the backplane. As it is those can only be connected via IDC so in most cases I would anticipate a mix of up to four full length cards (DDS/Synth/ADC/DAC etc) connected via the backplane and the rest of the connectivity taken up by the simpler IDC-only extensions.

On the other hand, if we thought a) wasn't worth the trouble but b) and c) were the real worry, we could drop the backplane and stick a bunch of SATA connectors (or similar) on the back of Kasli instead. The hypothetical extension requiring high speed connections would still use the IDC for power/I2C/slowstuff but could also be connected to SATAs as needed for high speed links. SATA would also be used for c).

We could also just wait until we're got our hands on the prototypes before deciding what to do here. I think some hands-on use will inform how much effort we are willing to put in to achieve a). For reference, in moving from the previous to current generation of NIST hardware we did consider it worth the effort to move from IDCs to a backplane. Also, use cases for b) and c) might arise in the meantime.

[hartytp]

Option a

@dtcallcock: I think some hands-on use will inform how much effort we are willing to put in to achieve a). For reference, in moving from the previous to current generation of NIST hardware we did consider it worth the effort to move from IDCs to a backplane.

ack

I think this only makes sense if the BNC/SMA/RJ45 extensions were also made full length to connect to the backplane.

Agreed: IMO, if we're using a backplane to cut down on wiring then all boards should connect directly to it without ribbon cable. This will require a minor revision to some boards.

@gkasprow: Actually, it's hard to fit more than 32 LVDS pairs (equivalent of 4 IDC connectors).

@dtcallcock: If a) was the priority but only worth doing if we could drive all 8 extensions then we could switch to a 160 pin connector on the Kasli/VHDCI carrier

Agreed: if we're planning to use a backplane for this then we will probably want to switch to a higher density connector.

There's also the question of how many extension modules we can fit in the rack. A standard rack is something like 84HP wide. I assume that we'll have "wide" extensions like the BNC breakout, DACs, etc which will be 12HP, and "narrow" extensions like the RJ45 breakout, at 4HP? So, do we want a single carrier per rack (e.g. either VHDCI or Kasli in a rack), and then up to 8 extension cards? A rack can then fit up to 6 wide extensions + carrier. All 8 BP slots can be used if narrow extensions are used as well. Obviously, this will leave a lot of unusable space if one populates the rack with only narrow extensions like the RJ45 breakout, but IMO we can live with that.

---

@dtcallcock: On the other hand, if we thought a) wasn't worth the trouble but b) and c) were the real worry, we could drop the backplane and stick a bunch of SATA connectors (or similar) on the back of Kasli instead. The hypothetical extension requiring high speed connections would still use the IDC for power/I2C/slowstuff but could also be connected to SATAs as needed for high speed links. SATA would also be used for c).

Exactly, I think this was @jordens motivation for putting some SATA connectors on Kasli (inside the rack, rather than on the front-panel).

---

@dtcallcock: We could also just wait until we're got our hands on the prototypes before deciding what to do here. I think some hands-on use will inform how much effort we are willing to put in to achieve

Agreed, it's probably better to wait until we've had a play with the hardware than to rush into specifying a backplane interface without due thought.

However, obviously the longer we put this decision off, the more hardware we're likely to design in the mean time (ADCs DACs SPI I2C etc). That means that we'll have to alter more boards to fit the backplane when a standard is agreed. IMO, so long as we can agree on what we're trying to achieve now then we should be able to specify it now.

It sounds to me like:

• Out chief motivation for a backplane is (a): cutting down on wiring clutter
• Any BP that can works for (a) will also work for (b) and (c), so we get those for free, even though they're not our main aim with the BP.
• High-speed interfaces (b) will additionally be supported by SATA on Kasli (not VHDCI)

Thoughts?

[gkasprow]

We decided to go for IDCs to enable out of rack operation of Kasli extensions.
If we switch to backplane connectivity we would have to develop dedicated custom backplane for every combination of extension modules.
Keep in mind that these modules may have different size so there will always be wasted some slots on the backplane.
IDCs can be low or high performance. You can easily run 2.5Gbit (PCIe) data over such cable so there should be no issues with DDS modules.

[sbourdeauducq]

Also Kasli was intended as a simple device; any backplane support is more secondary (if not feature creep).

[hartytp]

@gkasprow: We decided to go for IDCs to enable out of rack operation of Kasli extensions.

Agreed, we definitely want to keep the IDCs. The questions was whether we have the backplane as well and, if so, what we are trying to achieve with it.

@gkasprow: IDCs can be low or high performance. You can easily run 2.5Gbit (PCIe) data over such cable so there should be no issues with DDS modules.

Agreed, signal integrity for LVDS over ribbon cable should not be an issue.

If we switch to backplane connectivity we would have to develop dedicated custom backplane for every combination of extension modules. Keep in mind that these modules may have different size so there will always be wasted some slots on the backplane.

Agreed, if we don't constrain the module sizes better then the backplane topology gets to be a mess. This is something that would need to be thought about

@sbourdeauducq: Also Kasli was intended as a simple device; any backplane support is more secondary (if not feature creep).

Agreed, backplane support is secondary and definitely not required. But, as @dtcallcock points out, without it the systems can start to descend into a rats' nest of ribbon cable.

FWIW, I don't feel strongly about whether we have a backplane or not. I'm mainly just trying to clarify the various suggestions that have been put forward and ensure that people aren't talking cross purposes.

I would like to make sure that if we do have a BP, that it's properly thought through and has a well-documented motivation. If we're not going to do that then I don't see any point in putting the 96-pin DIN connector on the Kasli/VHDCI boards in the first place.

[hartytp]

@jbqubit: Please update Kasli specification on wiki to reflect details fleshed out in this Issue.

Done. Although, let's not worry too much about the details of this (e.g. 3 v 4 SPF) until someone actually gets funding for it.

[jbqubit]

@hartytp Thanks for the cleanup of the wiki and update of Kasli specification.

Comments on recent discussion:

• The EEM peripherals are about doing simple things simply. The IDC connector for EEM is simple and should be used for our initial round of 3U peripherals. The "rat's nest" is unlikely to be unruly by physicist standards since EEM consolidates IO, I2C, and power.
• The Kasli Carrier and VHDCI Carrier should be able to interchangeably drive EEM peripherals.
• Extend EEM specification to add an extra pin pair for differential clock. 2x15 -> 2x16
• I don't see a strong use case for a backplane. I advocate that we drop the backplane from the specification entirely. Let future users define this. This also permits the use of a smaller FGPA -- see below.

I've taken a quick look at the number of IO used by the current design. Looks like at least XC7A75T is required if we support 8 IDC and 4 backplane EEM. A XC7A75T may suffice if we forgo the back plane. Google doc is open to edit.

[jordens]

To repeat what @sbourdeauducq said: If we do this (we plan to do so), Kasli will be simple and cheap. Backplane support will be there but probably just secondary. The backplane might -- for example --- end up being just an alternative to using the first four IDC connectors. I.e. you can still only connect 8 extensions. But you can use the backplane to conveniently (mechanically and electrically) wire the first four extensions. We see no problem with VHDCI breakout compatibility but where adequate we may choose to give that a low priority.
@jbqubit, please undo your sprinkling of "EEM" everywhere. Inventing new and wrong acronyms is not helpful. None of the extensions are Eurocards at this moment. And currently I see no reason for more than a 50T. Your pin count is off and not how we prioritize of even approach this. I don't understand what you are trying to achieve.
Instead it would great if you could contribute to publication quality datasheets for Sayma and Metlino (actual numbers, features, configuration options, high quality block diagrams) that detail the specification and convince other potential users of your design decisions and of the hardware quality.

[jordens]

Well. The EEM may have been @hartytp. Anyway, AFAICS most of them are not 100+n*60 mm long...

[gkasprow]

@filipswit started seriously working on Sayma and Metlino documentation. Filip just finished schemtic review of all boards and found several issues. He will use tex templates from AFCK.

[hartytp]

@gkasprow: @filipswit started seriously working on Sayma and Metlino documentation. Filip just finished schemtic review of all boards and found several issues. He will use tex templates from AFCK.

What do people think about the idea of using the Wiki as the main source of documentation? This could have a few advantages:

• Easy for people to edit later on, for example to post measurement results or comment on hardware bugs we find
• Allows us to keep the entire Sinara ecosystem documented in a single place, rather than having a tex file for each component
• It's already a git repo, which makes version control easy

[jbqubit]

@hartytp @gkasprow Please move discussion on documentation to #133.

[hartytp]

@jordens The EEM terminology and pinout were mine. Apologies if I've made mistakes here, or if you feel this was unhelpful.

To be clear about my motivation for the updates I've made to the Wiki: we intend to fund the DAC and ADC boards imminently (hopefully in the next couple of weeks). The plan is to make something that we can run from the VHDCI carrier in the short term, but which will ultimately work with Kasli. From reading the issues, I get the impression that there are various conflicting ideas about what this hardware will do, and how it should fit together. I'd like to get a coherent, well documented plan before we start building hardware -- for example, if there's going to be a backplane then we will make our DACs compatible with it, but we can't do that until the specification is written down somewhere. If you have strong opinions, feel free to put them on the Wiki and we'll be happy to accommodate them.

Turning to your more specific points:

None of the extensions are Eurocards at this moment.

You're correct, my language was imprecise. I mean that the boards should fit within the Eurocard form-factor, so that all boards are 100mm wide and no deeper than 160mm. AFAIK all currently planned boards do fit this. I'll amend the Wiki.

please undo your sprinkling of "EEM" everywhere. Inventing new and wrong acronyms is not helpful.

When I've had conversations with people about this, I found that I needed a convenient short-hand way of referring to this hardware (and to differentiate it from the uTCA sub-ecosystem). EEM emerged as an obvious convenient choice. I think it's important (and certainly not unhelpful) to have some name for them. If you have a better name then please introduce it.

Your pin count is off and not how we prioritize of even approach this. I don't understand what you are trying to achieve.

Which pin count? Do you mean the 2x15 pin header pin count? That was taken from the current VHDCI/3U_BNC designs. What mistake have I made? My goal is to have a pinout that I can design our DAC/ADC boards around.

Backplane support will be there but probably just secondary. The backplane might -- for example --- end up being just an alternative to using the first four IDC connectors. ... We see no problem with VHDCI breakout compatibility but where adequate we may choose to give that a low priority.

Can we at least try to agree on specifications (pinout etc) for the headers + backplane now? We plan to start building hardware based on the current VHDCI carrier design soon. It seems perverse to start building a Kasli carrier in a few months that won't work with DAC/ADC extensions we start building now.

I.e. you can still only connect 8 extensions. But you can use the backplane to conveniently (mechanically and electrically) wire the first four extensions.

The backplane might -- for example --- end up being just an alternative to using the first four IDC connectors. I.e. you can still only connect 8 extensions. But you can use the backplane to conveniently (mechanically and electrically) wire the first four extensions.

If you have strong opinions about this, please add them to the Wiki, and we'll design our hardware around them.

[hartytp]

Also, a small bugbear: in a few places, people are calling the 100mil male pin header on these boards "IDC". IDC (insulation-displacement contact) is a method of crimping ribbon cable onto housings. These housings mate with the pin headers on the board, but are not the connectors used on the boards themselves. Can we stick to "pin header" or similar please to avoid confusing things?

[gkasprow]

The 3U boards have such pin headers surrounded by walls which are mating parts for crimped ribbon cable IDC connectors.
So the pin headers are like this:

while IDC headers are like that:

[hartytp]

@gkasprow Okay, I'm used to seeing that referred to a "shrouded" or "boxed" pin header, rather than an "IDC" header. But, if this is standard terminology then I'm wrong and I rescind my objections. Live and learn ;)

[gkasprow]

Whatever you google it gives the same pics :) But indeed, shrouded is official name in many producers catalogues. But since IDC header means that you plug IDC jumper we can leave this for clarity.

[hartytp]

ack

[gkasprow]

To define backplane structure we first have to think of communication channel for backplane enabled devices. IMHO we should reserve backplane option for special use cases. Individual IO channels like SMA, BNC, RJ45 need dedicated LVDS links and IDCs do the job nicely. Backplane enabled modules will benefit from clock distribution so we can install there DDS, RF generators, DACs, ADCs that need external synchronization.
For backplane devices we can make dedicated VHDCI adapter that supports high speed MLVDS.
So we can define backplane as MLVDS bus with limited number of supported protocols.
In this way all modules may have identical pinout and routed i.e. 16 LVDS lines + one dedicated selection line.
The interface that fits best both simple and complex devices is SPI over LVDS with multiple incarnations like QSPI, double QSPI. It resembles MMC interface where clock, chip select and 8 data lines are used.
With such interface running across the backplane, we can insert as many devices as the rack fits and obtain scalable transfer rate. It has one more benefit - we can mix 4HP and 8HP modules without loosing too many Kasli IO lines - only single chip select is not used.
For simple DACs we simply use standard SPI and ignore remaining data lines.
For applications that require hundreds of MB/s transfer rate (high speed ADC, DAC modules) we simply use 8 or 16 bit interface. To make all deterministic we can define time slots for every device.
SPI clock speed can also vary depending on currently selected module. MLVDS supports up to 100MHz.
Some simpler modules (slow ADC, slow DAC) can be designed as dual function with both backplane connectivity and IDC connector to enable stand-alone operation without the backplane.

[jordens]

Rough Kasli design looks settled. Backplane discussion moved to #168.