Sinara FMC and VHDCI support

[jbqubit]

Recap of discussion in this Issue.

Physical resources

• PCB_Metlino has
  • 1 LPC FMC (plus 4 transceivers)
  • 2 VHDCI, 32 LVDS pairs/ea
• PCB_Sayma_AMC has 1 LPC FMC
• Creotech FMC to VHDCI adapter

I/O options offered by ARTIQ

• TTL_PHY (~8 ns resolution, 1 pin/ea)
• SERDES_TTL_PHY (~1 ns resolution, 1 pin/ea)
  • SERDES_TTL_PHY is a superset of TTL_PHY
• SPI_PHY ([SCLK, MOSI, MISO, CS], 4 pins/ea)
  • SPI Master PHY and a SERDES TTL PHY can't easily be without incurring an SPI speed penalty

default support

to be determined by this Issue...

[jbqubit]

I'm pulling together the list of features that are expected to be supported in v0.1 software and gateware for Sinara #139. This is intended to be a minimal list of what /will/ be implemented not a wish-list. One decision to be made is how to use the FMC on PCB_Melino and PCB_Sayma_AMC.

Proposal:

• PCB_Melino: FMC configured for DRTIO 4xSFP
  • example FMC card is http://www.hitechglobal.com/FMCModules/FMC_4SFP+_Module.htm
• PCB_Sayma_AMC: FMC configured for 32xTTL LVDS
  • example FMC card is http://creotech.pl/en/produkt/fmc-dio-32ch-ttl-a/

[jbqubit]

@dtcallcock says

Using the FMC-VHDCI card and VHDCI carrier would presumably allow reuse of Kasli extensions and gateware for some of the above (TTL/ADC/DAC/CameraLink etc).

I like this suggestion. FMC-VHDCI is more extensible than the Creotech board and fits better with the Kasli 3U ecosystem.

[jordens]

Using the FMC-VHDCI card and VHDCI carrier would presumably allow reuse of Kasli extensions and gateware for some of the above (TTL/ADC/DAC/CameraLink etc).

It is not so much about gateware reuse. It is about limiting the number of bitstreams: there are a bunch of problems with supporting dynamic reconfiguration of what e.g. one extension connector on the VHDCI board does. E.g. we can't really switch between a TTL and a SPI without having separate bitstreams.

[hartytp]

@jordens Good point. I'd naively assumed that I'd be able to run the DAC extension board from the VHDCI carrier if I need to. Forgive me if it's a stupid question, but what prevents us from muxing the SPI and TTL in gateware?

[hartytp]

answering my own question: given the number of different ways one could wire up multiple SPI busses, anything sufficiently flexible to be genuinely useful is too messy/complex to be practical to implement...

[jordens]

It is not impossible but there are many unknows.
Let's assume we restrict the possible functionalities of a given pin to one of "a TTL I/O, a fixed SPI signal, a fixed I2C signal, a fixed CameraLink signal, [insert more here]" then yes, we could mux it, assuming that (a) it still works timing-wise without introducing registers there and (b) we have the logic resources to have all those RTIO PHYs around. For muxing with the high resolution SERDES based TTLs it would be trickier again (quite some gateware and clocking; definitely increasing latency which can be problematic for SPI).

[jbqubit]

@gkasprow Please confirm the extent to which the FMC is VITA 57.1 compliant?
It's a High Pin Count (HPC) connector.

[dtcallcock]

How about if we fixed a pinout for a couple of combinations like 'all TTLs' and '1xSPI + TTLs' and just supported muxing in gateware between those. That would cover most use cases and only add a modest number of PHYs. Then we'd just accept Camera link, SERDES TTLs and non-standard pinouts would need a different bitstream.

[gkasprow]

It is LPC connectivity on Sayma. There is only one exception - the gigabit transceiver link is routed to IOSERDES so it works up to about 2Gbit/s depending on device speed grade.

[jordens]

@dtcallcock sure. we could consider restricting the extension usage the following way:
EXT0: 8xSERDES-TTL I/O or 1xCameraLink
EXT1: 8xSERDES-TTL I/O or 1xCameraLink
EXT2: 2x(4xTTL I/O or 1xSPI)
EXT2: 2x(4xTTL I/O or 1xSPI)
...
(I hope the syntax is intuitive)

It still doubles the number of PHYs and requires agreement about exactly how we pin-out an SPI bank.

[hartytp]

@jordens How bad is doubling the number of PHYs? Where are we on FPGA resources?

[jordens]

For what I outlined you need to have twice as many PHYs as you can actually use. Every pin can be used for just one of two PHYs.
I don't know where we are on resources. One would have to find some time and funding to explore this.

[dtcallcock]

@jordens Is 2x(1xSPI) compatible with the plan from #142 of driving two SPI DACs with 2xCLK, 2xMOSI, 1xMISO, 2xSYNC, 1xLDAC? I'm thinking the shared MISO would break this.

[hartytp]

@jordens Is 2x(1xSPI) compatible with the plan from #142 of driving two SPI DACs with 2xCLK, 2xMOSI, 1xMISO, 2xSYNC, 1xLDAC? I'm thinking the shared MISO would break this.

and, more generally, if an SPI phy uses 4 pins, how is LDAC handled?

[jordens]

@dtcallcock Yes. That would not work. It is one reason why we should not try to bend "standard SPI" too much. 4xTTL (for LDAC and a couple of additional CS) + 1xSPI would work.

[hartytp]

@jordens point taken. The extra DAC isn't worth the pain of having to maintain a custom bitstream. We'll go to one DAC then.

[hartytp]

@jordens

Let's assume we restrict the possible functionalities of a given pin to one of "a TTL I/O, a fixed SPI signal, a fixed I2C signal, a fixed CameraLink signal, [insert more here]"

EXT0: 8xSERDES-TTL I/O or 1xCameraLink
EXT1: 8xSERDES-TTL I/O or 1xCameraLink
EXT2: 2x(4xTTL I/O or 1xSPI)
EXT2: 2x(4xTTL I/O or 1xSPI)

It still doubles the number of PHYs

• To clarify: the proposal is to only support SPI, TTL & CameraLink directly from Kasli, right? I2C etc are supported by extension boards that communicate with Kasli via SPI.

• I'm happy with this proposal (2 CameraLink per Kasli is plenty IMO).

• Will we support something similar for Metlino VHDCI? (e.g. it would be nice to be able to run some SPI busses directly from the Metlino VHDCI carrier).

• How hard would it be to support multiplexed SERDES-TTL & SPI (rather than standard TTL and SPI)? For Metlino VHDCI, it would be nice to have lots of SERDES-TTL, but still have the option of running a few SPI busses.

and requires agreement about exactly how we pin-out an SPI bank.

This is something we should agree on before any SPI extension boards get made. What would your preferred pin-out be?

[jordens]

To clarify: the proposal is to only support SPI, TTL & CameraLink directly from Kasli, right? I2C etc are supported by extension boards that communicate with Kasli via SPI.

This was arbitrary. And just for four extensions. But yes, if there are good SPI-to-I2C converters (SC18IS600), then why not. Or one could look at making the SPIMaster core so generic that it can just as well talk I2C. Like the serial protocol engines in modern µCs. But it's quite some work.

I'm happy with this proposal (2 CameraLink per Kasli is plenty IMO).

One CameraLink would likely occupy two extension headers.

Will we support something similar for Metlino VHDCI? (e.g. it would be nice to be able to run some SPI busses directly from the Metlino VHDCI carrier).

Sure. Same problem there.

How hard would it be to support multiplexed SERDES-TTL & SPI (rather than standard TTL and SPI)? For Metlino VHDCI, it would be nice to have lots of SERDES-TTL, but still have the option of running a few SPI busses.

It costs a handful cycles of latency. i.e. SPI reads would be even slower than they are now. The coding required is not gigantic. It is more about tedious corner cases and attempting to get resource usage and timing closure lined up for these highly multiplexed designs.

This is something we should agree on before any SPI extension boards get made. What would your preferred pin-out be?

The exact order doesn't matter much. SCLK, MOSI, MISO, CS seems to be frequent.

[hartytp]

But yes, if there are good SPI-to-I2C converters (SC18IS600), then why not. Or one could look at making the SPIMaster core so generic that it can just as well talk I2C. Like the serial protocol engines in modern µCs. But it's quite some work.

Let's not bother with I2C for now. If it becomes a problem then we can look into upgrading SPIMaster later.

One CameraLink would likely occupy two extension headers.

Sorry my mistake. Thanks for clarifying.

It costs a handful cycles of latency. i.e. SPI reads would be even slower than they are now. The coding required is not gigantic. It is more about tedious corner cases and attempting to get resource usage and timing closure lined up for these highly multiplexed designs.

Okay, let's not bother with this for now, and stick to something along the lines of your original suggestion.

Will we support something similar for Metlino VHDCI? (e.g. it would be nice to be able to run some SPI busses directly from the Metlino VHDCI carrier).

Sure. Same problem there.

Maybe, use one VHDCI exclusively for SERDES IO, and multiplex the other one?

[jbqubit]

Issue title updated to reflect need to decide on support for both FMC and VHDCI. I started to summarize conclusions about this Issue in the leading post.

Metlino VHDCI anticipates applications that a) require very-low latency, high bandwidth link to ARTIQ Core Device (Metlino) or b) convenient re-use of 3U peripherals via VHDCI Carrier. CameraLink does not fit this profile for the following reasons.

• Maximum possible pixel clock rate for CameraLink is 85 MHz, 255 MB/s (single-link) ref
• CameraLink is pin-hungry >=16 pins
• Fanciest camera on the market (I know about) is Andor Ultra 888
  • readout rate is still slow relative to DRTIO: 697 frame-per-second for 128x128 sub-array
  • at best 10 ns timing accuracy
• Gateware-based image processing is likely required for feedback within the qubit coherence time. As this gateware is likely to be application specific it shouldn't be part of the standard Metlino or Sayma .bit files.
  Better to design a CameraLink 3U peripheral for Kasli, do the image processing on Kasli and transmit results back to ARTIQ Core Device using DRTIO.

A broader comment is that we don't have to anticipate all future uses of FMC/VHDCI this winter. Let's settle on something simple for now. Based on discussion thus far and assuming CameraLink is dropped I propose the following. Call it jbqubit_p1.

gateware configuration options

Configuration options for groups of 8 I/O.

• let c1 be 8 of SERDES_TTL_PHY
• let c2 be 4 of TTL_PHY and 1 of SPI_PHY
• let c3 be whatever is needed to enable 4 channel SFP breakout

VHDCI

PCB_Metlino VHDCI 1
- EXT0, EXT1, EXT2, EXT3 each independently configurable as c1 or c2
- I2C
PCB_Metlino VHDCI 2
- EXT0, EXT1, EXT2, EXT3 each independently configurable as c1 or c2
- I2C

FMC

• let d1 be support for FMC HPC [4 channel SFP breakout] (http://www.ohwr.org/projects/fmc-sfp-4cha/wiki)
  • c3
• let d2 be support for FMC LPC Creotech FMC to VHDCI adapter with gateware
  • EXT0, EXT1, EXT2, EXT3 options c1 and c2

PCB_Melino HPC FMC

• configurable as d1 or d2
• I2C

PCB_Sayma_AMC LPC FMC

• configured as d2
• I2C
• NOTE: d1 is not supported since SFP breakout requires HPC FMC

[jbqubit]

@hartytp Would c2 permit operation of Zotino using VHDCI Carrier? I've not followed exactly how Zotino uses the SPI LDAC line.

5xLVDS lines used for DAC SPI bus: CLK, MOSI, MISO, SYNC, LDAC

[jordens]

Let's not bother with I2C for now. If it becomes a problem then we can look into upgrading SPIMaster later.

I'll still bother with I2C. We need it.

Maybe, use one VHDCI exclusively for SERDES IO, and multiplex the other one?

One should probably collect use cases and determine priorities...

[hartytp]

@jbqubit : Would c2 permit operation of Zotino using VHDCI Carrier? I've not followed exactly how Zotino uses the SPI LDAC line.

yes

@jordens : I'll still bother with I2C. We need it.
One should probably collect use cases and determine priorities...

That sounds like a good idea...

[jbqubit]

Does somebody have an imminent use case not covered by the proposal I made earlier today? We don't have to anticipate all future uses of FMC/VHDCI, just enough to get Sinara off the ground. It's gateware so accommodating

@jmizrahi @jasonamini is what I proposed compatible with your many-channel PMT-readout scheme?

[gkasprow]

What camera link standard do you want to use? Base or Medium/Full one? There are 2 ways to do that – using IOSERDES or external deserializer (http://www.ti.com/product/DS90CR287) The deserializer gives just 28 signals + clock which would fit to 2 IDC cables (we treat them as 2x16 LVTTL). But it is just Base configuration + serial link + 2 control signals. The advantage is simple implementation in the FPGA. Medium configuration would need 4 IDC cables and second deserializer. Another way is of course IOSERDES where for Base configuration we could use just one IDC cable (4 LVDS @595Mbit + 1x LVDS clock @85MHz) This also means that Kasli banks have to be routed to the IDCs in standardized way. We can define first LVDS pair as the clock/io. 3 remaining LVDS signals will be used for serial communication and synchronization. Second IDC would add another video channel + remaining 3 LVDS control channels Third IDC would be just supplement for 64 bit video required by Full standard The CameraLink standard defines 4 synchronisation LVDS channels and 2 serial channels. How do you plan to use them? I assume you used frame grabber that took care of communication. Is Andor control protocol described? So do you want to have just Base configuration that consumes 2 IDCs or implement Full configuration that would need one more IDC? Of course we can make IO board wired for Full configuration and give an option to use 1, 2 or 3 IDC channels (Limited Base, Medium or Full)

[jbqubit]

Let's move CameraLink discussion to #156.

[jbqubit]

@jordens said

I'll still bother with I2C. We need it.

How is I2C used?

[jbqubit]

How is I2C used?

The wiki answers my question. "I2C from each VHDCI is routed to TCA9548 8-ch multiplexer on VHDCI Carrier."

[jbqubit]

@jordens @sbourdeauducq Is my proposed specification from 10 days ago acceptable? If so please close Issue.

[jordens]

@jbqubit It is unwise and unsound IMO but certainly acceptable. Closing.
Your description and reasoning about CameraLink are flawed though. If you don't want to use it or pay for it that's fine. But apart from technical errors, I don't see how you can seriously explain to us about how we would want to use CameraLink and argue against us doing so.

[jbqubit]

It might help to reiterate the intended scope of this Issue.

I'm pulling together the list of features that are expected to be supported in v0.1 software and gateware for Sinara #139. This is intended to be a minimal list of what /will/ be implemented not a wish-list.

I don't want to discourage others from developing other interfaces including CameraLink (#156). The present discussion is intended to come up with a simple baseline specification for the FMC and VHDCI IO.

It is unwise and unsound IMO but certainly acceptable.

What aspects of what I proposed (jbqubit_p1) cause you concern?

[jordens]

@jbqubit Who is your target audience here? Is this a specification that you plan to have implemented or would you like someone else to fund this?

Please consolidate and clean this up w.r.t. the proposals and data elsewhere (e.g. in #129, #149).

From the top of my head:

• As explained elsewhere, we can't easily mux the SPI Master PHY and a SERDES TTL PHY without incurring a SPI speed penalty.
• SERDES TTL PHY is a superset of TTL PHY. One would be wasting user, FPGA, development, funding resources.
• FMC HPC is a superset of FMC LPC. Distinguishing them means wasting resources.
• Runtime support for different extensions and FMC cards requires the same IO standard and type (differential, single ended). Someone needs to verify compatibility.

[jbqubit]

@jbqubit Who is your target audience here?

My target audience is M-Labs who is implementing v0.1 support for the Sinara hardware.

As explained elsewhere, we can't easily mux the SPI Master PHY and a SERDES TTL PHY without incurring a SPI speed penalty.

I don't recall where this conversation is at the moment. Can you link to it? Is there an estimate as to the speed penalty?

FMC HPC is a superset of FMC LPC. Distinguishing them means wasting resources.

PCB_Metlino and PCB_Sinara_AMC are distinct targets and will already have their own .bit files. I see no harm in supporting FMC HPC for PCB_Metlino and FMC LPC for PCB_Sinara_AMC.

SERDES TTL PHY is a superset of TTL PHY. One would be wasting user, FPGA, development, funding resources.

OK. VHDCI on PCB_Metlino is directly connected to the Metlino FPGA. This is the primary low-latency (10's ns) IO in the Sinara system. It is desirable to use SERDES_TTL_PHY for the VHDCI IO. The (VHDCI Carrier)[https://github.com/m-labs/sinara/wiki/VHDCI%20carrier] is intended to provide IO breakout for use with (EEM)[https://github.com/m-labs/sinara/wiki/EEM] peripherals.

Runtime support for different extensions and FMC cards requires the same IO standard and type (differential, single ended). Someone needs to verify compatibility.

Good point. I provided to concrete examples of FMC cards. One for PCB_Metlino and one for PCB_Sayma_AMC. Please plan support for each in the respective bit files. Accommodation of other FMC can be addressed down the line.

In light of your comments I propose a variation as follows labeled: jbqubit_p2. Define configuration options for groups of 8 I/O.

• let c1 be 8 of SERDES_TTL_PHY
• let c2 be 4 of TTL_PHY and 1 of SPI_PHY

PCB_Metlino

VHDCI 1

• supported hardware: VHDCI Carrier
• EXT0, EXT1, EXT2, EXT3 configured to toggle between c1 and c2
• I2C

VHDCI 2

• supported hardware: VHDCI Carrier
• EXT0, EXT1, EXT2, EXT3 configured to toggle between c1 and c2
• I2C

FMC

• supported hardware: 4 channel SFP breakout FMC board
• allocate remaining IO as EXT0, ..., EXTN configured to toggle between c1 and c2

PCB_Sayma_AMC

FMC

• supported hardware: FMC LPC Creotech FMC to VHDCI adapter with gateware
• supported hardware: VHDCI link to VHDCI Carrier
• EXT0, EXT1, EXT2, EXT3 configured to toggle between c1 and c2
• I2C

[gkasprow]

WE use FMC LPC for both Sayma and Metlino. At least from FPGA IO pins point of view, it is LPC so the IP cores would be the same. The only difference is on number of GTP links assigned. That’s why HPC connector is used and we call it HPC.

[jordens]

Again. SERDES_TTL_PHY is a superset of TTL_PHY. Toggling between them is idiotic.
Toggling between SPI and SERDES TTL will at least give you a handful of cycles latency for input and output. That's two handful of cycles slowdown per SPI bit when reading. And again, it is unclear whether we have the FPGA resources.

[jbqubit]

Toggling between SPI and SERDES TTL will at least give you a handful of cycles latency for input and output.

The intent of supporting "togging" between c1 and c2 is a) to permit attachment of either fast TTL hardware or TTL+SPI hardware b) to compile only a single .bit to cover a range of use cases. The duty cycle for switching any given EXTn from c1 to c2 is weeks or months. So if I understand your statement about latency, it's OK if toggling takes many micro-seconds.

@jordens This is what you proposed several weeks ago.

@dtcallcock sure. we could consider restricting the extension usage the following way:
EXT0: 8xSERDES-TTL I/O or 1xCameraLink
EXT1: 8xSERDES-TTL I/O or 1xCameraLink
EXT2: 2x(4xTTL I/O or 1xSPI)
EXT2: 2x(4xTTL I/O or 1xSPI)

And again, it is unclear whether we have the FPGA resources.

@jordens Do you mean the Sayma or Metlino FPGA? Do you have

@sbourdeauducq What are the resource requirements of SERDES_TTL_PHY and SPI_PHY with a FIFO depth of, say, 128?

Since the intent is for the Kasli Carrier and VHDCI Carrier to be interchangeable we need to square this away. I expect Kasli implementation will also support multiple IO configurations in a single .bit for each of the 8 EMM ports. Configurations based on planned peripherals includes 8 of SERDES_TTL_PHY (eg BNC DIO breakout) and xx TTL + SPI_PHY (eg Zotino, Novogorny).

[jordens]

No Joe. Being able to toggle is a speed penalty for SPI (always).
Either FPGA.

[jbqubit]

--Updated at 7:40pm GMT--
Based on discussion and conversation with Robert.

0. Supporting the ability to toggle between SERDES_TTL_PHY and TTL_PHY/SPI_PHY incurs a latency penalty. Toggling between TTL_PHY/SPI_PHY does not incur a latency penalty.

1. SERDES_TTL_PHY is somewhat more FPGA-resource intensive. Metlino has more available FPGA resources than Sayma.

2. The ARTIQ core device is Metlino and at present all experiment branching occurs on the core device, low-latency Sinara IO is available only on Metlino.

3. Thirty-two channels of low-latency ~1 ns-resolution IO is sufficient for near-term use cases.

4. Implementation of Kasli Carrier IO interface requires few FPGA resources since Kasli FPGA has few resources by design.

5. Based on existing EEM peripheral designs, the Kasli Carrier IO interface takes two forms. <-- @jordens create a new Issue for this -->

• 8 TTL_PHY plus 1 I2C_PHY
  • BNC DIO breakout
  • SMA DIO breakout
  • RJ45 LVDS breakout
• 4 TTL_PHY, 1 SPI_PHY plus 1 I2C_PHY
  • Zotino, Novogorny, Mirny, Urukul, Sary, Berdenish

6. PCB_Sayma_AMC FMC may be populated by a "dumb" FMC-VHDCI adapter. A Sinara system has at least one PCB_Sayma_AMC. This is an approach which permits Sinara to drive the VHDCI Carrier.

Call the following variation jbqubit_p3.

PCB_Metlino

VHDCI 1 and VHDCI 2

• EXT0, EXT1, EXT2, EXT3 configured as 8 of SERDES_TTL_PHY
• I2C
• supported hardware via VHDCI Carrier:
  • BNC DIO breakout
  • SMA DIO breakout
  • RJ45 LVDS breakout

FMC

• transceivers and TTL_PHY to support 4 channel SFP breakout FMC board as SFP DRTIO Slaves
• @jordens please suggest simpler FMC-SFP adapter

PCB_Sayma_AMC

FMC

• EXT0, EXT1, EXT2, EXT3 configured to permit toggling of each EXTn between the following configurations
  • 8 TTL_PHY
  • 4 TTL_PHY, 1 SPI_PHY
• I2C
• supported hardware
  • Creotech FMC to VHDCI adapter
  • VHDCI Carrier interface to Zotino, Novogorny, Mirny, Urukul, Sary, Berdenish