Sayma RTM - version 1 errata & tests

[gkasprow]

Mechanical:

• SMA connector clearance with PCB edge - move it 100um outside
• Align J6, J7
• adjust 4-th upper panel SMA hole
• move T7 away from the board edge
• J16, J18, J55, J66 i J64 not covered vias under connectors
• fix footprint of interdigitated capacitors

Silkscreen:

• Label on both sides of PCB for name of J1, J2, J73, J74.
• Dot marking pin1 J1, J2, J5, J7, J8, J9, J73, J74
• AFE slot names are not visible when Allaki are installed
• AFE slot names on both sides of PCB
• When the Allaki is mounted, the IN/OUT silkscreen faces the Sayma RTM PCB and makes it useless. Put it on the other layer that is visible when the card is mounted.
• add description to LD12, LD5, LD10, LD11, LD3

BOM:

• R20, R21 = DNP
• R269=1k

I2C:

• replace MAX6642ATT90+T with MAX6642ATT98+T on RTM - change I2C address from 0x48 to 0x4C
• connect some I2C pullups to P3V3 instead of P3V3MP

Power supply:

• add another LDO for 1.2V (SVDD12)
• connect 2V and 4V power module FB path to GND
• fix power sequencing of negative SEPIC converter - either by addin 1uF cap to SS or starting negative converter after positive one.

Annotations:

• IC57 address is 0x4B - change comment on schematics (PK: 0x49 is correct)
• Zero-index all channels (DACs, ADCs, SFPs, SMAs, mezzanines, etc.)

JTAG:

• consider adding config FLASH
• wire M[2:0] as 0b111 for slave serial configuration (#249)
• swap FPGA config lines - DIN with MOSI (probably),
• make sure config mode is set correctly.

Clocks:

• HMC7043 has active high reset while SI5324 is active low. Add negator to one of them or tie HMC RST to GND give HMC7043 its own dedicated reset line to avoid noise issues during initialization.
• short RSV* pins of HMC7043 to GND
• add 10k pulldown on PLL chip CLK and SEN lines to make sure that logic state is well defined and FPGA can setup correct SPI protocol.
• add CLK and SYSREF termination
• SD pins of the LTC6957 clock buffer mus be grounded
• HMC830 loop filter layout forms quite a large loop. Please try to make this smaller for EMI reasons.
• HMC830 "NC" pins are currently floating in our design. Copy the eval board for these connections.
• add HMC7043 output bias FET that disables clocks before configuration to get rid of RF noise
• pullup/pulldown on HMC830 CS/clk line to prevent it from getting stuck in the wrong SPI mode
• So, to be sure, I think we need separate pins for: Si5324 RST; HMC7043 RST; HMC7043 output enable FETs.
• We also need to make sure that all these lines (as well as the HMC830 CS) have the right pull ups/pull downs to put things in the correct default state on power up.
• PLL locked LED for the HMC830, maybe leds for the clock muxes to indicate the clock source.

DAC:

• missing 49R9 0201 resistors at the DAC outputs
• connect VTT to 1.2V

Test points:

• add GND and i2C testpoints
• More test points, e.g. on all SPI busses, power rails, JTAG, PLL v tune, etc! Also some ground points to attach a scope to (maybe use pcb mounted test loops for some of these?)
• Lots of test points (all SPI/I2C lines)

Other:

• RTM: short VREFN and VREFP to GND, DNP C546 to enable internal ADC reference
• consider adding 0R resistors between ICs and control lines (SPI, enable, etc) to allow easier debug/rework
• add LVDS terminators (SSTL)

[gkasprow]

I observed interesting lockup of analog frontend amplifier.
After power on, the 5V LDO gives -0.9V, negative -5V, VCM = 0V and outputs are at -2.5V.
Below is power sequence

yellow = VCM, green = -6V, violet = +6V.
We clearly see that negative power goes too fast.
So I increased sot=ft start capacitance from 3.3nF to 100nF.
It got slightly better but still locks up

blue = +5V after LDO
So I increased capacitance to 1uF and it works:

But when one disconnects and quickly connects 12V supply, the SS capacitor doesn't discharge completely and -6V starts too quickly:

So we will fix it in next revision - negative DC/DC will be switched by PG output of positive one.

[hartytp]

@gkasprow good catch. Are you referring to IC6 and IC7 on the AFE test board?

Out of curiosity, would it be easier/more robust to fix this by adding Schottky diodes on the LDO outputs to prevent reverse-bias-induced lockup? This is the standard trick I've used on this kind of dual supply board.

[gkasprow]

@hartytp Yes, IC6 and IC7. I'll add diodes and let you know it this solves that issue.

[gkasprow]

The BAS281 diode helped. Thanks @hartytp !
I saw such diodes in a few app notes so it seems it is common problem of LDOs.

[hartytp]

@gkasprow Great! Glad to hear that fixed it.

I've seen quite a few subtle power sequencing problems in dual supply designs in the past -- particularly when the supplies are loaded asymmetrically. But, IME adding diodes always fixes it, so these days I always add them (diodes are cheaper than thinking/debugging time ;) ).

We should be aware of this as a potential issue for other Sinara boards and add diodes where appropriate.

T

[gkasprow]

I'm doing power load tests.
And it seem to work. I loaded all supplies with 50% of rated power and for 12V voltage drop on AFE is
150mV for +6V and +3.3V and below 50mV for remaining rails. With single 12V fan temperature does not exceed 71 degrees in hottest spots

Then I loaded +12V and -12V rail with 100% and measured on last AFE:
-11.95V, +11.99
Then I loaded +6V with 100% and measured +5.77V on last AFE. Power consumption was 5.01A@12V
Temperature does not exceed 74deg in hottest spot

Then I loaded +3.3V with 100% and measured +3.07V on last AFE. Unloaded voltage was 3.30V, 50% loaded was 3.18V

cooling configuration:

[jbqubit]

@gkasprow Regarding discussion of "lockup of analog frontend amplifier" related to the LDO IC6 (ADP7102) on AFE test board. Good catch! Related question... It looks like the LDO has a EN/UVLO pin for delayed startup and user-defined hysteresis intended for sequencing multiple power supplies. Is there a reason this feature wasn't used?

[hartytp]

@gkasprow

And it seem to work.

Great!

I loaded all supplies with 50% of rated power and for 12V voltage drop on AFE is 150mV for +6V and +3.3V and below 50mV for remaining rails.

Then I loaded +12V and -12V rail with 100% and measured on last AFE: -11.95V, +11.99

Then I loaded +6V with 100% and measured +5.77V on last AFE. Power consumption was 5.01A@12V

I'm not sure I follow you here. Do you mean that when you load all AFE test mezzanines at 50% on the +12V supply (100mA per AFE) the +12V rail is at 11.85V, and when you load all AFE mezzanines with 100% on the +12V supply rail (200mA per AFE) the +12V rail is at +11.99V?

[hartytp]

@jbqubit I don't think that the ADP7102 EN/UVLO will help here. The issue is that the negative supply switches on faster than the positive supply and ends up reverse biasing the positive supply rail (through biasing circuitry/ESD diodes/etc in the ADA4927-1 OpAmp). This causes the regulator to go into lockup.

Using the EN/UVLO to delay the positive regulator's startup further will make the problem worse.

This is a pretty common issue in this kind of design, and adding diodes to prevent transient reverse bias of the regulators is the standard fix.

[hartytp]

@gkasprow One other question while I'm thinking about RTM power supplies: at the moment, the +-12V supplies for the AFEs come from a switching converter on Sayma RTM, with some LC filtering. The +12V supply is then used on Allaki to bias the pre-amp (and, less importantly, to supply the InAmp). Is that correct? If so, do you think that the noise on this rail will be good enough for the pre-amp if we're aiming for "ultra-low" phase/amplitude noise?

We probably talked about this before, so apologies if I've forgotten what was agreed!

[gkasprow]

@hartytp I've written that it is below 50mV, which also means close to zero :)
I measured it again with clock mezzanine plugged in:
12V rail
no load: 12.06V
50% load: 12.01V
100% load: 11.97V

6V rail:
no load: 5.95V
50% load: 5.85V
100% load: 5.71V. Had to connect another bench supply because current exceeded 5A

-6V rail:
no load: -6.04
50% load: -6.01
100% load: -5.99

-12V rail:
100% load -11.92. Due to bug on a AFE board the load is constantly connected

3.3V rail:
no load : 3.29
50% load: 3.15V
100% load: 3.03V

[hartytp]

@gkasprow Nice!

Thanks for clarifying. Glad to hear that's all working!

Out of curiosity, what happens when all rails are at 100% load?

[gkasprow]

@hartytp
Yes, both 12V rails come from DC/DC converter.
It's hard to measure this noise in my noisy enviorement - the scope shows 1-2mV RMS with probe grounded and 1-2mV RMS with probe connected to 12V rail.
So the noise is certainly below the noise floor of my scope.
I don't see any correlation between load and noise value.
Some figures are below. yellow probe is AC coupled to the rail, blue is DC coupled to the same rail.
The scope noise floor. Probe and clip connected to SMA ground on AFE board.

-6V rail under full load

6V rail with no load

3.3V rail

-12V rail with full load

[gkasprow]

@hartytp when I load all rails with 100% , I exceed twice the power foreseen for AMC+RTM boards.
It would consume over 120W of power.
I don't want to check it now - have only 2 boards :)

[hartytp]

@gkasprow Thanks for the info.

Fluctuations in this voltage rail directly modulate the amp's bias current, so my suspicion is that using the DC/DC output directly will degrade the noise performance somewhat. If we're worried about this/if it turns out to be a problem, we can always add an extra LDO onto Allaki. Maybe @dhslichter has thoughts?

In any case, it looks like the RTM works perfectly -- this is something to think about when we do our design review for Allaki AFE.

when I load all rails with 100% , I exceed twice the power foreseen for AMC+RTM boards. It would consume over 120W of power. I don't want to check it now - have only 2 boards :)

Fair enough, good point.

[jordens]

It's not terribly sensitive. At the low frequency (for bias current/power supply fluctuations) end, you get about 0.005 dB more gain for 1% more bias current.

[hartytp]

@jordens True. Depends on what noise floor you're trying to reach though. Anyway, this isn't something I'm particularly worried about as it's easy to test and easy to fix if we do decide it's problematic.

[jordens]

That's assuming it's linear.... the delta is 0.12 dB for a 20% bias step.

[jbqubit]

@hartytp said

adding diodes to prevent transient reverse bias of the regulators is the standard fix.

Thanks for the explanation. Inclusion of diodes seems like a general solution. I've updated the wiki PCBReviewChecklist to reflect this practice.

[dhslichter]

I thought we had an LDO on Allaki for the output amplifier supply -- if not, it would be good to add for the next iteration. The gain of the output amp is basically linear in supply current, so fractional voltage noise is turned directly into fractional gain noise. There is a choke and a capacitor (or at least, there should be) on the supply line which provides some filtering at higher frequencies, but I think an LDO is definitely a good idea here.

[jbqubit]

@dhslichter I added the LDO to Allaki schematic review check list. #110

[gkasprow]

SPI works, I had a bug in my Vivado code. For some strange reason it let me assign 2 conflicting values to same IO pin. And I didn't notice warning. But it works now - I read chip ID. I also loaded example configuration that generates 1.4GHz output.
Here is FORTH code used to init the chip
: config_hmc_830 0 0 $20 spi_hmc830 . 0 1 $2 spi_hmc830 . 0 2 $2 spi_hmc830 . 0 5 $1628 spi_hmc830 . 0 5 $60a0 spi_hmc830 . 0 5 $e110 spi_hmc830 . 0 5 $2818 spi_hmc830 . 0 5 $0 spi_hmc830 . 0 6 $303ca spi_hmc830 . 0 7 $14d spi_hmc830 . 0 8 $c1beff spi_hmc830 . 0 9 $153fff spi_hmc830 . 0 a $2046 spi_hmc830 . 0 b $7c061 spi_hmc830 . 0 f $81 spi_hmc830 . ;

And I get RF at the output:

Yellow plot is 100MHz at the HMC830 input, orange is RF output from spectrum analyser.
It is probably not in lock, I don't want to waste my time optimising PLL operation, just want to make sure it works from HW point of view.

[hartytp]

@gkasprow Great!

[gkasprow]

I'm fighting with HMC7043 SPI.
Don't get any response, the chip does not react to ID readout nor setting of GPIO.
I even generated example configuration using ADI tool - still no response. The chip keeps the data line Hi-Z all the time. I even configured GPIO as SPI data output but still no response.
I observe all signals at the chip leads, power supply, reset are OK.
Maybe it needs RF signal for SPI operation - this I will check tomorrow.
Any ideas?

[hartytp]

Hmm, not sure. Some ideas:

• Is RESET LOW?
• Have you tried grounding pins 14 and 24? The data sheet says these "must be tied to ground". On the EVAL board, it's labelled TEST_MODE_SEL and TEST_MODE

[gkasprow]

@hartytp good catch!
I will connect these pins to GND and check tomorrow.
RESET is low.
Thanks!

[gkasprow]

@hartytp
It works!

[hartytp]

The joys of undocumented diagnostic modes...

[gkasprow]

Another bizzare thing
When you configure HMC830 first, it works. But when you start config from HMC7043, the HMC830 does not. And it is correct behaviour. HMC830 has 2 modes of operation and as datasheet says:

On power up both types of modes are active and listening.
A decision to select the desired SPI protocol is made on the first occurrence of SEN or SCLK following a
hard reset, after which the protocol is fixed and only changeable by cycling the power OFF and ON.
a. If a rising edge on SEN is detected first HMC Mode is selected.
b. If a rising edge on SCLK is detected first Open mode is selected.

So one must make sure that HMC830 SEN goes high first and in our case where clock line is shared between both HMC chips it is critical.

[gkasprow]

@enjoy-digital you delivered me 2 bitstreams: one that is supposed to set logic low on SEL output but in fact it sets it randomly. On some boards including yours it is L, but in 3 of mine it is H, while on rest of them it is L. Second bit stream toggles this pin correctly (I tested on problematic board), and I see it on the scope. So it means the hardware is OK, but for some reason your FPGA bitstream (sayma_rtm.bit) disables the clock for HMC chip and I cannot test boards that want to send to UMD.
So maybe there is something wrong with your compiler.
You can also map this pin to one of registers and let me control it manually from Python script.
I simply stucked with further test and don't want to send @jbqubit HW with issues.

[enjoy-digital]

@gkasprow: ok, I'll connect that to a register then and see if it's better when controlled by the software.

[enjoy-digital]

@gkasprow: i added that:
https://github.com/enjoy-digital/sayma_test/blob/master/test/sayma_both/test_clocking.py#L54

Here is the bitstream with updated csr.csv (untested):
http://enjoy-digital.fr/open_hardware/sayma_rtm_dynamic_clk_sel.7z

[gkasprow]

@enjoy-digital for some strange reason it works properly with your recent fix. I don't even need to access this register from Python. I replaced the csv file but is says name 'wb' is not defined.
I simply loaded the new bitstream for RTM and three RTM boards that were not working due to lack of clock are working perfectly now.

[enjoy-digital]

@gkasprow: maybe another vivado bug... (the difference is that now it has to drive the pin from a register and was maybe doing buggy simplification before??)

[gkasprow]

@enjoy-digital I had such issue in the past that compiler synthesized away pins routed to GND/VCC. But in this case the pin was shorten to VCC internally and on some boards to GND. So this may be a bug. Can you check in RTL map what Vivado did ?

[jbqubit]

• Label on both sides of PCB for name of J1, J2, J73, J74.

• Dot marking pin1 J1, J2, J5, J7, J8, J9, J73, J74

• AFE slot names are not visible when Allaki are installed

• AFE slot names on both sides of PCB

[gkasprow]

@jbqubit please add these issues to the list on top. Whey will get lost in the thread :)

[jbqubit]

@gkasprow top post now updated :)

[hartytp]

@gkasprow It would be good to get a price breakdown for Sayma AMC and RTM at some point. What drives the cost of these boards? What can we do to make them a bit cheaper? IME the cost per channel is one of the biggest factors in user adoption!

[gkasprow]

@hartytp at the moment we have batch of 10 boards produced. So substantial cost is tooling, FR408 material, connectors. All is expensive because for many components we don't break the first price step.
At the moment PCB tooling, stencils, P&P program, AOI is roughly 40% of the cost.
Components, panel and assembly is remaining 60%

[hartytp]

@hartytp at the moment we have batch of 10 boards produced. So substantial cost is tooling, FR408 material, connectors. All is expensive because for many components we don't break the first price step.
At the moment PCB tooling, stencils, P&P program, AOI is roughly 40% of the cost.
Components, panel and assembly is remaining 60%

Thanks for the info.

• Is the tooling, stencils etc a fixed 1-off cost (e.g. once we've produced 1 batch with a given design version we will not have to pay it again?)
• Do you see any low-hanging fruit in terms of things we can do to save costs? e.g. I assume there are no PCB layers we could easily do without? If there are any particularly expensive components (things like the SMPs) it would be good to highlight them so we can try to think of alternatives.

[hartytp]

@jordens If I recall correctly, you felt that Sayma was much more expensive than it should be. Other than getting the cost of the mezzanines right down, do you have any concrete proposals for ways we can save money?

There are a few minor things we can do like shaving a mux or two out of the clock tree, but those are barely a rounding error on the overall cost AFAICT...

[jordens]

I am no expert. The advice to reduce cost should come from those calculating the total. To me the additional runs for AFE and clock mezzanines plus the connector space, cost, risk, time issues are significant. And yes: reduce options that nobody is paying for. You seem to be implying that "a mux or two" is just the cost of s single part. The cost for designing, reviewing, discussing, debugging, tweaking, supporting, coding, and assembling that stuff is orders of magnitude more. E.g. to reduce cost you may want to kill the RF backplane connector and that part of the clocking tree for those who are not using it.

[gkasprow]

@jordens these backplane connectors are expensive. We can always mark them as DNP

[dhslichter]

@gkasprow can you quantify "expensive" here for the backplane connectors?

Of the silicon cost, the ADCs are the dominant contribution IIRC. Certainly the SMP connectors are substantial as well. If people wanted output-only they could have some of these DNP but then there is additional cost associated with the assembly variant because you're not doing it in as big a run.

The Sayma boards will need a rather sophisticated functional testing suite after assembly -- what does that contribute to the cost, @gkasprow ?

[dhslichter]

@jordens @hartytp IMHO the cost of separate mezzanines is worth it due to the risk mitigation/customizability aspect. However, if there is low-hanging fruit to reduce those costs (e.g. cheaper substrate for Allaki, cheaper board-to-board connectors instead of SMP), I'm certainly in favor, as long as we have tested it to make sure it will work RF-wise for the former and mechanically for the latter.

[hartytp]

Maybe the thing to do is for @gkasprow to produce a list of the cost of the components on Sayma AMC+RTM assuming a sensible batch size like 25. Cost should mean the total cost of getting the components onto the PCB, so both component cost but also assembly etc. Then we can make more of a decision about this.

The cost for designing, reviewing, discussing, debugging, tweaking, supporting, coding, and assembling that stuff is orders of magnitude more.

True, but those are largely one-off costs. While that could have been more of a consideration at the design stage, I'm more concerned with the recurring per-board costs at the moment.

E.g. to reduce cost you may want to kill the RF backplane connector and that part of the clocking tree for those who are not using it.

We can take a survey and see how many people plan to use it. If the fraction of users wanting the backplane is small then we can DNP it.

However, as @dhslichter points out, one shouldn't underestimate the cost/effort involved in producing and supporting multiple design variations. IME it's generally better to pay a little more and have a single well-supported version (e.g. this makes is far easier for someone to keep them in stock, which is great since the lead time is pretty long).

@jordens @hartytp IMHO the cost of separate mezzanines is worth it due to the risk mitigation/customizability aspect. However, if there is low-hanging fruit to reduce those costs (e.g. cheaper substrate for Allaki, cheaper board-to-board connectors instead of SMP), I'm certainly in favor, as long as we have tested it to make sure it will work RF-wise for the former and mechanically for the latter.

That's exactly my feeling. These mezzanines are a good thing for a lot of reasons. We just have to make sure that the cost isn't excessive and that we fix the mechanical issues.

[hartytp]

Once M-Labs have Sayma up and running, our top priority should be testing Sayma v1.0, before moving onto v2.0 production. Ideally, I'd like to send v2.0 off to production by the end of January.

One thing I really want to avoid is delays due to design discussions (e.g. which connector we should use etc). So, I'd like to get that all out of the way asap, and ideally before we begin testing.

Edit: once we begin testing, the only discussions we have should be related to issues that emerge during testing, not longstanding ones.

To that end, I've lumped all Sayma-related issues in the v2.0 milestone (some can be moved to a different milestone later but it's nice to have everything related to Sayma in one place for now). Please have a look at these issues and post any comments/open new issues where necessary. I'll assume that anyone who doesn't contribute to those issues over the next couple of weeks is happy with the plans for Sayma v2.0!

@jordens @sbourdeauducq IIRC, you had some issues with: the current IPMI/MMC; the Ethernet PHY; the JTAG. I know that some of that has already been posted on the issues, but what I haven't seen yet (apologies if I've missed it) is a clear list of changes you'd like for v2.0. Is there anything we can change to make your life easier on those fronts? IMHO, since you guys are the ones dealing with these hardware components, you should have say in how they're designed.

[jordens]

@hartytp We have raised lots of issues. Those plus general simplification (i.e. reducing the design). But please file an issue for a Sayma 2.0 system design proposal/change list and assign it to us.

[hartytp]

Done.

[hartytp]

I have a theory about why my DAC looses lock...

Looks like a QC failure...

[hartytp]

Never mind, this is intended. I was looking at the wrong photo of what the (manually fixed) boards should look like. Should look like this: https://github.com/m-labs/sinara/blob/master/Drawings/Sayma_RTM_rev1.0_front.jpg so this is all as intended for this revision.

[hartytp]

@gkasprow Can you remind me what temp co we used for the DAC bias current resistors? Did we make sure that these are relatively low temp co to avoid degrading the DAC's performance?

[gkasprow]

It's 0.1% 4k02 10ppm

R0603_4K02_0.1%_0.063W_10PPM GENERIC

[hartytp]

Perfect.

[jbqubit]

• pullup on HMC830 CS line to prevent it from getting stuck in the wrong SPI mode (cf Sayam: startup issues sinara#554 (comment))

[gkasprow]

moving mechanical issues to #46 and closing